Enzian BMC

TODO

• Establish, document and test a reproducible build process for BMC firmware (incl. bitstream).
• Update implemented power tree to match v3 prototype.
• Automate nightly build - up-to-date golden image should be always available.
• Comment and document all code and processes.
• Update IR3581 configuration.
• Design a reliable FW update mechanism.
• Shim board fixes:
  • PS UART MIO routing
  • Jumper to route JTAG SRST to SRST or POR on BMC
  • I²C pinouts
  • More flash
• Next revision:
  • All pin straps BMC GPIO-controlled

Power Management Software

Currently our power management software is written in Python 2 and lives here: Power Management Tools]. For documentation see [Enzian/BMC/PowerManagementTools.

Module Identification

We identify modules by the MAC address of their primary interface, which is assigned by u-boot from the environment variable {{`}}ethaddr{{`}}. This is persistent, saved in the SPI NOR flash. Refer to the user guide for how MAC addresses are assigned. Every module is labelled with its MAC address.

Modules in use

• Module Inventory
• For modules in use as BMCs see the enzian machine inventory.
  
  

Power Tree

We've extracted the Enzian Power Tree.svg (i.e. the network of voltage regulators and converters) from the schematics delivered by Dreamchip. The BMC prototype needs to model a representative subtree, preferably one device of each type and including tricky topologies e.g. configurable regulators whose logic supply is generated from other configurable regulators.

The Prototype Power Tree.svg models only the FPGA side of the system, and substitutes a MAX17030 for the ISL6334 that generates the DDR core voltage 1.2V. The Intersil part doesn't have an available development board, but both are configured in the same way, with a 7b parallel binary code setting the output voltage. The MAX20751 generating FPGA VCCINT@100A, the MAX15301 generating UTIL_3V3 (which is the logic supply voltage for the MAX20751 and input voltage to the MAX15053) and a MAX15053 generating SYS_1V8 from UTIL_3V3, as in the full tree.

We include the ispPOWR CPLD for voltage monitoring and sequencing.

Clock Tree

SiLabs ClockBuilder Pro

Main clock generator - CLK I2C bus, address 0x68:
Si5395-RevA-Project-main.slabtimeproj

CPU clock generator - CLK I2C bus, address 0x69:
Si5395-RevA-Project-cpu.slabtimeproj

FPGA clock generator - CLK I2C bus, address 0x6A:
Si5395-RevA-Project-fpga.slabtimeproj

BMC Module

The documentation for the module can be found in the hardware section of the wiki.

Enzian BMC specific documentation:

• BMC PE1 interface BMC PE1 board pinout.pdf

On the BMC runs OpenBMC. The repositories:

• OpenBMC https://gitlab.inf.ethz.ch/PROJECT-Enzian/enzian-bmc-openbmc
• Linux kernel https://gitlab.inf.ethz.ch/PROJECT-Enzian/enzian-bmc-linux, included by the OpenBMC build process
• U-Boot https://gitlab.inf.ethz.ch/PROJECT-Enzian/enzian-bmc-uboot, included by the OpenBMC build process
• FPGA bitstream https://gitlab.inf.ethz.ch/PROJECT-Enzian/enzian-bmc-fpga, included by the OpenBMC build process
• Power Management https://gitlab.inf.ethz.ch/PROJECT-Enzian/enzian-bmc-powermgmt, included by the OpenBMC build process

The OpenBMC repo is the main repo used to build the BMC image (U-Boot SPL, U-Boot, The Device Tree, the Linux kernel image, the root FS image, for now without the Power Management module), the whole configuration (the Device Tree, patches, Linux & U-Boot configurations, package list) is located in the directory /meta-enzianbmc

Overview

The BMC is running on a Zynq, two ARM A9 cores with an FPGA. The boot sequence:

1. The first stage boot loader, FSBL, is loaded and executed.
2. FSBL loads a bitstream from the flash and programs the FPGA.
3. FSBL loads and executes the actual bootloader, U-Boot.
4. U-Boot waits a few seconds and, if interrupted, starts the console.
5. U-Boot loads the device tree and the kernel and starts Linux.
6. During the Linux startup, the Enzian power management services are started.
7. OpenBMC login prompt, login: root, password: 0penBmc (note: the first character is a zero)

The U-Boot console

The BMC has two serial ports which can be used to access the U-Boot console

1. the PE1 serial port, accessible via USB, in Linux seen as "/dev/ttyPS0"
2. the Enzian serial port, accesible via USB, in Linux seen as "/dev/ttyPS1"

The U-Boot will automatically boot Linux if no key is pressed within 2 seconds after the power up.

The U-Boot important commands:

• dhcp - request an IP address from a server; needed for any network-based actions
• run <variable> - run a script contained in a variable
• printenv - print the current environment
• setenv <variable> <value> - set a variable with a value
• saveenv - save permanently the current environment to the flash,
• reset - reset the BMC

The U-Boot important environment variables:

• ethaddr - the MAC address of the BMC
• def_args - the Linux default arguments; if you want to use in Linux the PE1 serial port as a console, modify the "console" argument
• update_dtb - if run, download and flash the device tree from the server
• update_kernel - if run, download and flash the kernel from the server
• update_rootfs - if run, download and flash the root file system from the server

Programming the FPGA from u-boot

zynq-uboot> fpga info 0                       
Xilinx Device
Descriptor @ 0x3ffbb6f4
Family:         Zynq PL
Interface type: Device configuration interface (Zynq)
Device Size:    3510668 bytes
Cookie:         0x15 (21)
Device name:    7z015
No Device Function Table.
zynq-uboot> dhcp
BOOTP broadcast 1
DHCP client bound to address 192.168.1.13 (3 ms)
zynq-uboot> tftpboot 4000000 bmc/EnzianBMC.bit
Using ethernet@e000b000 device
TFTP from server 192.168.1.2; our IP address is 192.168.1.13
Filename 'bmc/EnzianBMC.bit'.
Load address: 0x4000000
Loading: #################################################################
         #################################################################
         #################################################################
         #############################################
         8.5 MiB/s
done
Bytes transferred = 3510773 (3591f5 hex)
zynq-uboot> fpga loadb 0 0x4000000 0x3591f5   
  design filename = "EnzianBMC;UserID=0XFFFFFFFF;Version=2020.1"
  part number = "7z015clg485"
  date = "2020/08/31"
  time = "02:10:06"
  bytes in bitstream = 3510668�zynq-uboot> �

Flash memories layout

The BMC has 2 flash memories, 64MB QSPI NOR and 256MB NAND. NOR and NAND flashes share the same I/O lines. U-Boot can access both flashes (execute zx_set_storage NAND to access NAND flash or zx_set_storage QSPI to access NOR flash). Once Linux boots, only NAND can be accessed. All the files comes from the OpenBMC deploy directory.

On the NOR flash there are 4 used regions (offset:size), this flash is only accessible from U-Boot, enzianbmc-nor.bin:

1. 0x00000000:0x00040000 - first stage bootloader (FSBL), U-Boot SPL, OpenBMC boot.bin-enzianbmc
2. 0x00040000:0x00380000 - BMC's FPGA bitstream EnzianBMC.bit
3. 0x003c0000:0x00040000 - U-Boot environment
4. 0x00400000:0x000c0000 - actual bootloader, U-Boot, OpenBMC u-boot-enzianbmc.img

On the NAND flash there are 4 partitions:

1. 0x00000000:0x00500000 - Linux kernel, OpenBMC uImage-enzianbmc.bin
2. 0x00500000:0x00100000 - Device Tree, OpenBMC enzianbmc.dtb
3. 0x00600000:0x00400000 - no longer used (was BMC's FPGA bitstream, moved to the NOR flash)
4. 0x00a00000:0x1f600000 - Root FS, OpenBMC obmc-phosphor-image-enzianbmc.ubi

The BMC has access to the SPI flash which is used as a boot device for ThunderX:

1. 0x000000:0x400000 - BDK
2. 0x400000:0x200000 - BL0 + BL1 + FIP (Firmware Image Package, consists of BL2 + BL31)
3. 0x600000:0x300000 - UEFI (BL33)
4. 0x900000:0x600000 - (unused)
5. 0xf00000:0x005000 - EFI variables

Flashing from Linux

Linux Memory Technology Devices (MTDs), current, temporary layout:

root@enzianbmc:~# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 02000000 00020000 "nand-linux"
mtd1: 00080000 00020000 "nand-devicetree"
mtd2: 00080000 00020000 "nand-bootscript"
mtd3: 1df00000 00020000 "nand-rootfs"
mtd4: 01000000 00001000 "enzian-bdk"

The "enzian-bdk" contains all boot modules: BDK, ATF and UEFI, usually comes in a form of a bootfs.bin file (assembled boot modules).

Linux Memory Technology Devices (MTDs), old and future layout:

root@enzianbmc:~# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00400000 00001000 "enzian-bdk"
mtd1: 00200000 00001000 "enzian-atf"
mtd2: 00300000 00001000 "enzian-uefi"
mtd3: 00700000 00001000 "enzian-unused"
mtd4: 00700000 00020000 "nand-linux"
mtd5: 00100000 00020000 "nand-device-tree"
mtd6: 1f600000 00020000 "nand-rootfs"

To flash the desired flash partition, the CPU must be power down, call from the BMC:

flashcp -v <binary image> /dev/mtd<MTD number>

Examples:

1. Flashing the Linux kernel: flashcp -v /tmp/uImage-enzianbmc.bin /dev/mtd1
2. Flashing the ThunderX boot image: flashcp -v /tmp/bdk-git-v3.bin /dev/mtd0

I2C Buses

There are 6 I2C buses, clocked 100kHz:

• Bus 0: SEQ - ISPPACs
• Bus 1: PSU
• Bus 2: CLK
• Bus 3: FPIO
• Bus 4: PWR_FAN
• Bus 5: internal ZX5 - RTC & user SRAM

NVMe backplane i2c addresses:
bp_i2c.png]]

Enzian FTDI external UARTs

1. BMC console
2. CPU UART0 - the default console
3. CPU UART1
4. FPGA UART

Compilation

OpenBMC is built using bitbake which will fetch most things from sources. This means you need an internet connection. On enzian-build you'll need to set the ETH proxy:

export https_proxy=http://proxy.ethz.ch:3128
export http_proxy=http://proxy.ethz.ch:3128

There is in theory a way to forbid bitbake to access the internet. After step 3 in the following guide, create an empty file conf/sanity.conf, for example touch conf/sanity.conf. However the webinterface for the BMC is built using npm which is hard to impossible to persuade to only use a local mirror.

To build the entire BMC firmware stack do the follwoing:

1. We need the Vitis 2020.1 (Vivado + SDK), it should be installed in the /opt/Xilinx/Vitis/2020.1/ directory. It's required to build the FPGA bitstream and to flash the NOR flash.
2. Check out the Enzian BMC OpenBMC repo https://gitlab.inf.ethz.ch/PROJECT-Enzian/enzian-bmc-openbmc.git
3. Setup the build environment: . setup enzianbmc

4. You can use a mirror to download the necessary packages (see the Yocto Project Manual for infos on how to create a mirror). To use the one we set up (works only inside the ETH network), edit the file conf/local.conf located in the build directory and add following lines:

   SOURCE_MIRROR_URL = "http://enzian-build.ethz.ch/enzian/bmc/openbmc-mirror/2.11.0/"
   INHERIT += "own-mirrors"
   

5. Build: bitbake obmc-phosphor-image

6. Be patient, it may take some time (around 40 minutes on the enzian-build)
7. The output files are located in the deploy directory enzian-bmc-openbmc/build/enzianbmc/tmp/deploy/images/enzianbmc

Deployment

1. Initial flashing of an empty BMC module (FSBL, U-Boot) using JTAG

We also need a u-boot-spl file which can be found in the OpenBMC build tree: enzian-bmc-openbmc/enzianbmc/build/tmp/work/enzianbmc-openbmc-linux-gnueabi/u-boot-enzianbmc/1.0+master+-r0/build/spl. This will flash the first board which is directly connected to the computer. If there are many connected boards and/or on a different machine, the target board has to be specify using combination of -target-name and/or -url arguments.

/opt/Xilinx/Vitis/2020.1/bin/program_flash -f enzianbmc_nor.bin -flash_type qspi_single -blank_check -verify -fsbl u-boot-spl

where enzianbmc_nor.bin is the image of the NOR flash (FSBL + Bitstream + U-Boot) from the deploy directory.

2. Flashing, the second stage, using the USB (console) and Ethernet

U-Boot boot messages:

U-Boot 2018.01-g584e21cc0c-dirty (Jun 12 2020 - 15:26:32 +0000)

DRAM:  ECC disabled 1 GiB
NAND:  512 MiB
MMC:   sdhci@e0100000 - probe failed: -6

SF: Detected s25fl512s_256k with page size 512 Bytes, erase size 256 KiB, total 64 MiB
In:    serial
Out:   serial
Err:   serial

On the actual Enzian you'll see only this:

SF: Detected s25fl512s_256k with page size 512 Bytes, erase size 256 KiB, total 64 MiB
Net:   ZYNQ GEM: e000b000, phyaddr 3, interface rgmii-id
eth0: ethernet@e000b000
Hit any key to stop autoboot:  2 

The U-Boot will try to autoboot, to cancel it, press a key within 2 seconds. U-Boot commands:

1. printenv - print current environment variables
2. saveenv - save the environment to the flash
3. setenv - set the environment variable
4. dhcp - acquire the IP and the address of the TFTP server
5. run - execute a script contained in a variable

U-Boot environment variables:

1. ethaddr - the MAC address of the BMC Ethernet MAC

The U-Boot expects a TFTP server at the address 192.168.1.1 or after invoking the 'dhcp' command from the IP address of the TFTP server, all the files should be located in the directory /bmc

Commands should be invoked with 'run':

1. update_fsbl - flash the first stage bootloader (/bmc/enzianbmc-stable/boot.bin-enzianbmc)
2. update_boot_image - flash the boot image containing U-Boot and the bitstream (/bmc/enzianbmc-stable/u-boot-enzianbmc.itb)
3. update_devicetree - flash the Device Tree (/bmc/enzianbmc-stable/enzianbmc.dtb)
4. update_kernel - flash the Linux kernel (/bmc/enzianbmc-stable/uImage-enzianbmc.bin)
5. update_rootfs - flash the root FS (/bmc/enzianbmc-stable/obmc-phosphor-image-enzianbmc.ubi)

3. Accessing

login: root

password: 0penBmc

The BMC can be accessed over the USB serial console or using SSH. The BMC tries to obtain the IP address from a DHCP server. There is a Web interface running (over HTTPS).

ISPPACs

ISPPACs can be programmed using a HW-USBN-2B cable over JTAG and Lattice Diamond suite (Tools/Programmer).

Files:
DesignCPU20.PAC DesignCPU20.jed DesignFPGA21.PAC DesignFPGA21.jed

On Linux, the cable has to be unbound from the FTDI driver:

1. Go to directory /sys/bus/usb/drivers/ftdi_sio
2. List the directory for the list of devices
3. Write a name of device to /sys/bus/usb/drivers/ftdi_sio/unbind (for example echo 2-3\:1.0 >unbind)

Resetting the module CPU remotely

If the BMC hangs it can be remotely reset over JTAG. The System reset pin of the USB JTAG module (SRST) is connected to the BMC pulling that low for a few milliseconds will reset the BMC.  You can pull this pin low using the Xilinx System Debugger (xsdb). It is installed on enzian-build in /opt/Xilinx/Vivado/2020.1/bin/xsdb. Once in the xsdb shell do the following:

1. connect -host enzian-gateway.ethz.ch
2. target -set -filter {jtag_cable_serial=="<JTAG ID>"} where <JTAG ID> looks like this: 210357A7CB89A
3. If this succeeds it means that something in the scan chain is useable and a rst -srst will trigger the system reset.

If the scan chain is down e.g. because the FPGA is not powered, xsdb refuses to do the work for you but you can coax it into manually generating a JTAG sequence to trigger the system reset. This script does that: jtag_srst.tcl

Run it as follows:

1. /opt/Xilinx/Vivado/2020.1/bin/xsdb jtag_srst.tcl
2. You'll be prompted for the host to connect to, enter e.g. enzian-gateway.ethz.ch
3. You'll be shown a list of JTAG targets and prompted for which one to reset. Enter the target number of the one that has the right JTAG ID for your machine

If all of the above fails, try power-cycling the machine using emg. If that fails as well, probably something with the boot configuration of the BMC is wrong. This can be fixed over JTAG as well. Documentation is sadly a bit behind, so shout out to the team.

Power Distributor

Distributor 3D big.png

To make it easier and safer to route the power connections for the BMC prototype, we've designed and build a breakout board for an ATX 24-pin (+ 2x8-pin CPU) power connector. All supply lines are fused and broken out to screw terminals. The board should be capable of 20A on every rail. All supply rails are current- & voltage-monitored using a TI INA226 (the IC used on the final Enzian board), which are accessible (via an isolator) through a PMBus header. Another PMBus header brings out the (isolated) PSU PMBus interface. Additionally, there are LED "traffic light" indicators giving a go/no go indication for all supply voltages. The form factor is a full-length PCIe expansion card, minus the edge connector, and should fit any standard chassis.

• Schematics, bill of materials and other generated documentation are in this Power Distributor.pdf.
• PCB sources
• FuseSafetyExperiments

Estimated Power Specs

Testing Results

CPU Thermal Simulator

Operation

ISPPACs

CPU

FPGA

Power-up sequence

5VSB_PSUP is on

1. ISPPACs: reset, check CPU: ADC12, ADC13, check FPGA: ADC12, ADC13
2. B_PSUP_ON set to 1 -> enable 3V3_PSUP, 5V_PSUP, 12V_PSUP
3. ISPPACs, CPU: wait for PSUP_PGOOD, check IN1, ADC0, ADC1, ADC2, FPGA: IN1, ADC0, ADC1
4. C_PLL_DCOK set to 0, C_RESET_N to 0, B_OCI2/3_LNK1 (ECI link configuration)
5. Program the main clock generator: U57, check if B_CLOCK_BLOL is True
6. Program the CPU and FPGA clock generators: U11, U16, check if B_CLOCK_CLOL is True and B_CLOCK_FLOL is True
7. Program the current/voltage monitors:
   1. U27 - INA226 - CPU VDD_DDRCPU13 - I2C 0x44
   2. U31 - INA226 - CPU VDD_DDRCPU24 - I2C 0x45
   3. U44 - INA226 - FPGA VDD_DDRFPGA13 - I2C 0x40
   4. U48 - INA226 - FPGA VDD_DDRFPGA24 - I2C 0x41
   5. IC4 - MAX15301 - CPU 1V5_VDD_OCT - I2C 0x10
   6. IC12 - MAX15301 - FPGA VCC1V8 - I2C 0x11
   7. IC10 - MAX15301 - FPGA UTIL_3V3 - I2C 0x1B
   8. IC11 - MAX15301 - FPGA VADJ_1V8 - I2C 0x12
   9. IC13 - MAX15301 - FPGA VCCINTIO_BRAM - I2C 0x15
8. Program the power controllers:
   1. U34 - IR3581 - CPU VDD_CORE, 0V9_VDD_OCT - I2C 0x??
   2. U26, U30 - ISL6334 - CPU VDD_DDR - GPIO B_CDV
   3. U37 - MAX20751 - FPGA MGTAVCC - I2C 0x72
   4. U41 - MAX20751 - FPGA MGTAVTT - I2C 0x73
   5. U51 - MAX20751 - FPGA VCCINT - I2C 0x70
   6. U43, U47 - ISL6334 - FPGA VDD DDR - GPIO B_FDV
9. Wait for B_CLOCK_BLOL
10. ISPPACs: enable the CPU and FPGA power
    1. enable CPU VDD_CORE (VDD_CORE_EN, OUT6)
    2. enable CPU 0V9_VDD_OCT (VDD_OCT_EN_L2, OUT7)
    3. enable CPU 1V5_VDD_OCT (EN_1V5_VDD_OCT, OUT8), check CPU:ADC5
    4. enable CPU 2V5_CPU_13 (EN_2V5_CPU13, OUT9), check CPU:ADC6
    5. enable CPU 2V5_CPU_24 (EN_2V5_CPU24, OUT10), check CPU:ADC7
    6. enable CPU VDD_DDRCPU13 (EN_VDD_DDRCPU13, OUT11)
    7. enable CPU VDD_DDRCPU24 (EN_VDD_DDRCPU24, OUT12)
    8. enable FPGA UTIL_3V3 (EN_UTIL_3V3, OUT6), check FPGA:ADC2
    9. enable FPGA VCCINT (EN_VCCINT_FPGA, OUT9), check FPGA:ADC5
    10. enable FPGA VCCINTIO_BRAM (EN_VCCINTIO_BRAM_FPGA, OUT13), check FPGA:ADC9
    11. enable FPGA VCC1V8 (EN_VCC1V8_FPGA, VCCAUX & VCCAUX_IO, OUT15, VCCADC), check FPGA:ADC10
    12. enable FPGA SYS_1V8 (EN_SYS_1V8, OUT16, VCCO), check FPGA:ADC11
    13. enable FPGA SYS_2V5_13 (EN_SYS_2V5_13, OUT8, VTT_DDRFPGA13), check FPGA:ADC4
    14. enable FPGA SYS_2V5_24 (EN_SYS_2V5_24, OUT7, VTT_DDRFPGA24), check FPGA:ADC3
    15. enable FPGA DDRFPGA13 (EN_VDD_DDRFPGA13, OUT19, VCCO)
    16. enable FPGA DDRFPGA24 (EN_VDD_DDRFPGA24, OUT18, VCCO)
    17. enable FPGA VADJ_1V8 (EN_VADJ_1V8_FPGA, OUT17, VCCO)
    18. enable FPGA MGTAVCC (EN_MGTAVCC_FPGA, OUT10), check FPGA:ADC6
    19. enable FPGA MGTAVTT (EN_MGTAVTT_FPGA, OUT14)
    20. enable FPGA MGTVCCAUX_L (EN_MGTVCCAUX_L, OUT11), check FPGA:ADC7
    21. enable FPGA MGTVCCAUX_R (EN_MGTVCCAUX_R, OUT12), check FPGA:ADC8
11. Wait 3 ms - probably not necessery
12. Wait for B_CLOCK_CLOL and B_CLOCK_FLOL
13. C_PLL_DCOK set to 1, C_RESET_N to 1

Power-down sequence

1. C_PLL_DCOK set to 0
2. ISPPACs: disable CPU and FPGA power
3. B_PSUP_ON set to 0
4. ISPPACs: wait for PSUP

Loop/monitoring

Programming the FPGA

1. The FPGA must be powered up (obviously)
2. The bitstream must be a ".bin" file, not a ".bit" file
3. The bitstream must be put in the /lib/firmware directory, eg. top.bin
4. To program the FPGA, write the name of the file to the /sys/class/fpga_manager/fpga0/firmware file, eg "echo top.bin >/sys/class/fpga_manager/fpga0/firmware"