RISC-v on ZC706 Evaluation Board - Part II

With RISC-v on ZC706 Evaluation Board - Part I, I was successfully building the boot.bin. I thought I got the work done, but realized it was not when I tried the binary I made: after replaced the 'boot.bin' on the SD card with the one I built, the Init LED turns to green for about 1 second, then turns to red, no output from UART port, so something is wrong. Actually there is a warning while making the boot.bin:

[WARNING]: Partition FSBL.elf.0 range is overlapped with partition rocketchip_wrapper.bit.0 memory range
[INFO]   : Bootimage generated successfully

Would need to check the partition layout.

Beside this problem, I'm seeing another issue when playing more, with the fetched SD binaries: running "./fesvr-zynq pk hello" got "ERROR: No cores found", same error for any command with fesvr-zynq. Search the symptom returned same issue:

https://github.com/chipsalliance/rocket-chip/issues/1390

https://github.com/ucb-bar/fpga-zynq/issues/87

Likely there is an update for zedboard but not for zc706. So likely will have to build this fesvr-zynq by myself. Run 'make fesvr-zynq' lead me realized that all sub-modules of rocket-chip/riscv-tools didn't get cloned, especially 'riscv-fesvr' which is needed for making fesvr-zynq. With that repo cloned, the build did make some progress, but eventually would complain cannot find 'arm-xilinx-linux-gnueabi-g++'. Though I did find the executable under Xilinx Vitis, running environment setup script, or even adding the path of the executable won't help, as it needs the PetaLinux build environment, which seems hasn't been fully integrated to Xilinx Vitis. Have to download PetaLinux Tools from Xilinx site.

Will update this once get some progress on this.

Below is quoted from https://github.com/ucb-bar/fpga-zynq

Overview of System Stack

Our system will allow you to run a RISC-V binary on a rocket core instantiated on a supported Zynq FPGA. This section will outline the stack of all of the parts involved and by proxy, outline the rest of the documentation. Going top-down from the RISC-V binary to the development system:

Target Application (RISC-V binary) will run on top of whatever kernel the rocket chip is running. Compiled by riscv-gcc or riscv-llvm.

RISC-V Kernel (proxy kernel or RISC-V Linux) runs on top of the rocket chip. The proxy kernel is extremely lightweight and designed to be used with a single binary linked against Newlib while RISC-V Linux is appropriate for everything else.

Rocket Chip (rocket core with L1 instruction and data caches) is instantiated on the FPGA. Many of its structures will typically map to various hard blocks including BRAMs and DSP slices. It communicates to the host ARM core on the Zynq via AXI.

Front-end Server (riscv-fesvr) runs on the host ARM core and provides an interface to the rocket chip running on the FPGA (connected via AXI).

Zynq ARM Core (actually dual Cortex A9) runs Linux and simplifies interfacing with the FPGA.

FPGA Board (Zybo, Zedboard, or ZC706) contains the Zynq FPGA and several I/O devices. At power on, the contents of the SD card are used to configure the FPGA and boot Linux on the ARM core.

External Communication (TTY over serial on USB or telnet/ssh over ethernet) allows the development system to communicate with the FPGA board.

Development System (PC with SD card reader) generates the images to configure the FPGA.

RISC-v on ZC706 Evaluation Board - Part I

Continue my journey with RISC-v on FPGA. This time, I got a Xilinx Zynq-7000 SoC ZC706 Evaluation kit. The board is at least 7 years old. The FPGA has two Cortex-a9 cores. And there is https://github.com/ucb-bar/fpga-zynq which has everything needed to get RISC-v running on several Zynq FPGA boards, including ZC706. Again, this project has no update for more than four years now, so can foresee something might be out-of-date. Making this blog just for reference.

The fpga-zynq github repo has pretty good documentation. It's very easy to follow the Quick Instruction, just fetch images and load them to SD card. Setup the boot mode of the board, and all done. The only issue I had run into is related to the SD card. There are several links for FPGA boot from SD issue, like https://support.xilinx.com/s/question/0D52E00006hpPfLSAU/sd-card-boot-problems-zc706-board and https://support.xilinx.com/s/article/59476. In general, need to make sure all needed files are properly copied to SD card. For my case, the SD card reader I used first has some weird issue. After I used a newer SD card reader, everything goes very smoothly.

For the 2nd step, pushing Rocket modification to FPGA, I got several problems.

• Problem 1: the instruction mentioned user can pick own rocket-chip directory by override ROCKET_DIR. Since the freedom repo for Arty A7 contains rocket-chip repo, I was thinking to re-use that to reduce the time and disk usage. However, the 'make rocket' command will run into this error: 

        module not found: edu.berkeley.cs#chisel3_2.12;3.2-SNAPSHOT

The error is like https://github.com/ucb-bar/chiseltest/issues/72 and edu.berkeley.cs#chisel_2.11;3.1-SNAPSHOT: not found · Issue #518 · chipsalliance/chisel3 · GitHub. One post mentioned need to update the dependency from chisel2 to chisel3. I cannot figure out a quick way to get around this, so gave up on this. Instead, remove the setting of ROCKET_DIR. 'make init-submodules' will download rocket_chip repo, and the not found of edu.berkeley.cs:chisel3_2.12:3.3-SNAPSHOT is gone.

• Problem 2: 'make rocket' successfully created rocket_chip.stamp file, but reported no rule to make testchipip.stamp. This is a weird one, as the Makefile and common/Makefrag file does not have testchipip.stamp as target. Turns out due to the line-ending of generate-pkg-mk.sh file, the shell script failed to finish some work which leads to issue for EXTRA_PACKAGES=testchipip. One problem is the make command line doesn't generate meaningful message pointing to the shell script problem. I figured out that by turning on -d option for make, and run the script manually. For fixing this, do: "git config --global core.autocrlf false" then do git checkout to apply the setting.
  
• Problem 3: Seeing error

  Missing dependency 'object java.lang.Object in compiler mirror', 
  required by /.m2/repository/org/scala-lang/scala-library/2.10.4/scala-library-2.10.4.jar(scala/package.class)
  

  JDK version not compatible with scala. As replied for https://stackoverflow.com/questions/53942248/maven-error-missing-dependency-java-lang-object-required-by-scala, each version of scala only works with certain version of JDK (refer to docs.scala-lang.org/overviews/jdk-compatibility/overview.html). For this case, have to stick with JDK8:

sudo apt install openjdk-8-jdk-headless
sudo update-alternatives --install "/usr/bin/java" "java" "/usr/lib/jvm/java-8-openjdk-amd64/bin/java" 1

sudo update-alternatives --config java

• Problem 4: "make project" will need to invoke Vivado. On Windows, this can be achieved by running under Cygwin with Vivado executable added to PATH. However, on both Windows and Linux, I'm seeing same error as:
  ERROR: [Board 49-71] The board_part definition was not found for xilinx.com:zc706:part0:1.0. The project's board_part property was not set, but the project's part property was set to xc7z045ffg900-2. Valid board_part values can be retrieved with the 'get_board_parts' Tcl command. Check if board.repoPaths parameter is set and the board_part is installed from the tcl app store.
  Per this, this and this, as the instruction/repo is for Vivado 2016 but I'm using 2022 version, would need to update the tcl script for creating Vivado project, modify the line like this: 
  1. set_property board_part xilinx.com:zcu102:part0:3.0 [current_project]
  to: xilinx.com:zcu102:part0:3.1
  For my case, would need to update zc706/src/tcl/zc706_rocketchip_ZynqFPGAConfig.tcl, which has line as: set_property "board_part" "xilinx.com:zc706:part0:1.0" $obj
  Run get_board_parts within the tcl console shows xilinx.com:zc706:part0:1.4. So update the tcl script, run "rm -rf zc706_rocketchip_ZynqFPGAConfig" first, then run "make project" again.
  Then run into problem 5:
  ERROR: [BD_TCL-109] This script was generated using Vivado <2016.2> and is being run in <2022.1> of Vivado. Please run the script in Vivado <2016.2> then open the design in Vivado <2022.1>. Upgrade the design by running "Tools => Report => Report IP Status...", then run write_bd_tcl to create an updated script.
  This error is from zc706/src/tcl/zc706_bd.tcl. With 2022.1, "Report" is a separate menu item beside tools, and it only shows up after a project is opened. Really don't want to spend hours to install the 2016.2 version. So I just modified the zc706_bd.tcl to set the version as 2022.1 which matches the version I'm using. With that, there is some critical warning, like this: "CRITICAL WARNING: [BD 41-737] Cannot set the parameter PCW_M_AXI_GP0_FREQMHZ on /processing_system7_0. It is read-only.", but the project (.xpr) file did get created successfully. 
   
  Run "make fpga-images-zc706/boot.bin" will do synthesis and implementation with the project. This somehow not working for me, because of the bit file is missing. Likely this is due to my Vivado installation does not have valid license for creating bit for zynq. Opened the project from Vivado which has valid license, and generate bit file there. It got successfully generated on both Linux and Windows Vivado. However, when run "make fpga-images-zc706/boot.bin" again, the boot.bin would be successfully generated on Linux but not on Windows. Error for Windows/Cygwin run is like this:

  ln -sf ../../zc706_rocketchip_ZynqFPGAConfig/zc706_rocketchip_ZynqFPGAConfig.runs/impl_1/rocketchip_wrapper.bit fpga-images-zc706/boot_image/rocketchip_wrapper.bit
  cd fpga-images-zc706; bootgen -image boot.bif -w -o boot.bin
  ...
  [ERROR]  : Cannot read BIT file - boot_image/rocketchip_wrapper.bit
  

  The bit file under impl_1 did exist, however, 'ln -sf' may create a junction point on NTFS, which might not work for Xilinx Windows bootgen. To get around this, delete the symbolic link, copy the bit file directly to fpga-images-zc706/boot_image folder, then CD to fpga-images-zc706 folder, and run "bootgen -image boot.bif -w -o boot.bin" directly (don't try the make command, as it will create the soft symbolic link again), and this works under Cygwin.
   
  So finally I got the binary built on both Linux and Windows/Cygwin.  Lesson learned is: for FPGA development, better do everything on Linux but not Windows: too many hassle.
  
  
  At this point, I thought I had it completed, but in fact, it is not. So  <to be continue>
  
  
  
  
  
  
  
  
  
  

Program Arty A7 FPGA to be a RISC-V processor - Part IV

 So still fighting with the "ERROR: [Synth 8-439] module 'BootROM' not found" issue. More internet search did bring back a clue from https://github.com/sifive/freedom/issues/34, richardxia commented Sep 27, 2017: BootROM uses Chisel's sequential memory feature, so it does not refer to an existing Verilog module. You will need to use the vlsi_rom_gen script from rocket-chip to generate the Verilog file from the ROM parameters. See our Makefile rule for handling this: freedom/bootrom/xip/Makefile, Line 38 in 22ee433: $(rocketchip_dir)/scripts/vlsi_rom_gen $(ROMCONF) $< > $@

But, it didn't explain how this is invoked during the build. In the Makefile it has a line as:

 export BOOTROM_DIR := $(base_dir)/bootrom/xip

In common.mk, target 'romgen' would run make under $BOOTROM_DIR, and target 'bit' is depends on 'romgen', and target 'mcs' is depends on 'bit'. So it should be built while run make ... mcs. So I did a clean build with log captured, and found build error as:

java -Xmx2G -Xss8M -XX:MaxPermSize=256M -cp /mnt/ext4/freedom/rocket-chip/firrtl/utils/bin/firrtl.jar firrtl.Driver -i /mnt/ext4/freedom/builds/e300artydevkit/sifive.freedom.everywhere.e300artydevkit.E300ArtyDevKitConfig.fir -o /mnt/ext4/freedom/builds/e300artydevkit/sifive.freedom.everywhere.e300artydevkit.E300ArtyDevKitConfig.v -X verilog
OpenJDK 64-Bit Server VM warning: Ignoring option MaxPermSize; support was removed in 8.0
Total FIRRTL Compile Time: 55117.1 ms
make -C /mnt/ext4/freedom/bootrom/xip romgen
make[1]: Entering directory '/mnt/ext4/freedom/bootrom/xip'
dtc -I dts -O dtb -o /mnt/ext4/freedom/builds/e300artydevkit/sifive.freedom.everywhere.e300artydevkit.E300ArtyDevKitConfig.dtb /mnt/ext4/freedom/builds/e300artydevkit/sifive.freedom.everywhere.e300artydevkit.E300ArtyDevKitConfig.dts
/mnt/ext4/riscv/bin/riscv64-unknown-elf-gcc -march=rv32imac -mabi=ilp32 -O2 -std=gnu11 -Wall -I. -nostartfiles -fno-common -g -DXIP_TARGET_ADDR=0x20400000 -DDEVICE_TREE='"/mnt/ext4/freedom/builds/e300artydevkit/sifive.freedom.everywhere.e300artydevkit.E300ArtyDevKitConfig.dtb"' -static -nostdlib -o /mnt/ext4/freedom/builds/e300artydevkit/xip.elf xip.S
/mnt/ext4/riscv/bin/riscv64-unknown-elf-objcopy -O binary /mnt/ext4/freedom/builds/e300artydevkit/xip.elf /mnt/ext4/freedom/builds/e300artydevkit/xip.bin
od -t x4 -An -w4 -v /mnt/ext4/freedom/builds/e300artydevkit/xip.bin > /mnt/ext4/freedom/builds/e300artydevkit/xip.hex
/mnt/ext4/freedom/rocket-chip/scripts/vlsi_rom_gen /mnt/ext4/freedom/builds/e300artydevkit/sifive.freedom.everywhere.e300artydevkit.E300ArtyDevKitConfig.rom.conf /mnt/ext4/freedom/builds/e300artydevkit/xip.hex > /mnt/ext4/freedom/builds/e300artydevkit/rom.v
Traceback (most recent call last):
  File "/mnt/ext4/freedom/rocket-chip/scripts/vlsi_rom_gen", line 90, in iterate_by_n
    batch += (next(it),)
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/mnt/ext4/freedom/rocket-chip/scripts/vlsi_rom_gen", line 138, in <module>
    main()
  File "/mnt/ext4/freedom/rocket-chip/scripts/vlsi_rom_gen", line 134, in main
    **parse_line(line))
  File "/mnt/ext4/freedom/rocket-chip/scripts/vlsi_rom_gen", line 113, in parse_line
    kwargs = {key: try_cast_int(val)
  File "/mnt/ext4/freedom/rocket-chip/scripts/vlsi_rom_gen", line 113, in <dictcomp>
    kwargs = {key: try_cast_int(val)
RuntimeError: generator raised StopIteration
make[1]: *** [Makefile:38: /mnt/ext4/freedom/builds/e300artydevkit/rom.v] Error 1
make[1]: Leaving directory '/mnt/ext4/freedom/bootrom/xip'
make: *** [common.mk:68: /mnt/ext4/freedom/builds/e300artydevkit/sifive.freedom.everywhere.e300artydevkit.E300ArtyDevKitConfig.rom.v] Error 2

And if re-run the make command, the error would be hidden, might because of rom.v file was already generated, even it was error out for previous make run. To fix that, as mentioned https://github.com/chipsalliance/rocket-chip/issues/1991, need to update rocket-chip/scripts/vlsi_rom_gen line 97, replace 'raise' with 'return', then re-do a clean build. Eventually now it is moving forward. And make .. mcs will have the bit file generated. The bit file was generated under builds/e300artydevkit/obj/ folder. And Vivado project file is also available under builds/e300artydevkit, as E300ArtyDevKitFPGAChip.xpr. It can be opened with Vivado, to view all the settings.