General overview

The main software comprises of:

  1. BDK - Bring-up and Diagnostic Kit
    BIOS-like configuration and first-stage bootloader portion, together with diagnostic tools and the ability to boot arbitrary images
  2. ATF - ARM Trusted Firmware
    Secure world software, including an Exception Level 3 (EL3) Secure Monitor
    Base address: 0x00040000 (ATF source code enzian-atf/plat/thunder/include/thunder_def.h)
    Max size: 0x00079000
  3. UEFI - Unified Extensible Firmware Interface
    EDK2 + GRUB2
    Act as BL33 of the ATF
    Base address: 0x00500000
    Max size: 0x040000000
  4. Linux


BDK

BDK Building

repo - https://gitlab.inf.ethz.ch/PROJECT-Enzian/enzian-bdk

Follow the instruction

BDK Configuration

The main BDK screen:

Cavium SOC
Locking L2 cache
PASS: CRC32 verification
Transferring to thread scheduler
================
Cavium Boot Stub
================
Firmware Version: 2020-07-03 15:39:26
BDK Version: db14bf2, Branch: heads/master, Built: Fri Jul 3 15:39:25 UTC 2020

Board Model:    enzian_v3
Board Revision: 1.5
Board Serial:   1

Node:  0
SKU:   CN8890-2000BG2601-NT-Y-G
L2:    16384 KB
RCLK:  2000 Mhz
SCLK:  800 Mhz
Boot:  SPI24(5)
VRM:   Disabled
Trust: Disabled, Non-secure Boot
CCPI:  Disabled

Press 'B' within 10 seconds for boot menu

=================================
Boot Options
=================================
N) Boot Normally
S) Enter Setup
D) Enter Diagnostics
E) Enter Diagnostics, skipping Setup
F) Select Image from Flash
X) Upload File to FatFS using Xmodem
W) Burn boot flash using Xmodem
U) Change baud rate and flow control
R) Reboot

Choice:

The setup main menu

=================================
Setup
=================================
B) Board Manufacturing Data
C) Chip Features
D) DRAM Options
Q) QLM Options
P) Power Options
F) Restore factory defaults
S) Save Settings and Exit
X) Exit Setup, discarding changes

The basic board configuration menu

=================================
Setup - Board
=================================
B) Board Model Number (enzian_v3)
R) Board Revision (1.5c)
S) Serial Number (1)
M) Base MAC Address (0xc05331030020)
N) Number of MAC Addresses (8)
W) Save Board Manufacturing Data to Flash
Q) Return to main menu

Chip Features menu:

=================================
Setup - Chip
=================================
D) Delay for Boot Menu (10 s)
N) Support Multi-node (2)
B) Bring the link up, do not enable CCPI (0)
L) OCX Link Width (3)
K) OCX Link KR Training (0)
E) Support PCIe ECAM Enhanced Allocation (EA) (1)
O) Strict PCIe Ordering (0)
W) Watchdog Timeout (0 ms)
T) BDK Tracing Level (0x0 bitmask)
C) BDK Allowed Coremask (0x0 bitmask)
Q) Return to main menu

The QLM configuration

=================================
Setup - Chip
=================================
2) Switch Nodes (0)
1) Auto configure QLMs based on MCU (0)
A) N0.QLM0 - XLAUI_1X4, 10.312 Gbaud, External Reference
B) N0.QLM1 - XLAUI_1X4, 10.312 Gbaud, External Reference
C) N0.QLM2 - PCIE_1X4, 8.000 Gbaud, Common Clock 0
D) N0.QLM3 - SATA_4X1, 6.000 Gbaud, Common Clock 0
E) N0.QLM4 - PCIE_1X8, 8.000 Gbaud, Common Clock 0
F) N0.QLM5 - @, 0.000 Gbaud, External Reference
G) N0.QLM6 - PCIE_1X4, 8.000 Gbaud, Common Clock 0
H) N0.QLM7 - PCIE_1X4, 8.000 Gbaud, Common Clock 0
Q) Return to main menu


BDK Tracing

To enable the tracing, set the BDK Tracing Level bitmask in the Chip Features menu.

Bitmask (hex)

Description

0x000001

BGX networking block

0x000002

DRAM initialzation

0x000004

DRAM test code

0x000008

Early initialization, before main()

0x000010

ECAM initialization

0x000020

QLM related debug

0x000040

eMMC related debug

0x000080

PCIe link init

0x000100

PCIe config space reads / writes

0x000200

SATA/AHCI related debug

0x000400

Multi-node related debug

0x000800

FatFs related debug

0x001000

MPI related debug

0x002000

Environment variables related debug

0x004000

Free Pool Allocator

0x008000

Packet Input

0x010000

Packet Output

0x020000

SSO

0x040000

ECAM based device framework

0x080000

ECAM based device scanning detail

0x100000

Virtual NIC

0x200000

Device tree passed to OS

0x400000

USB XHCI block


BDK Testing

The main diagnostic menu

=================================
Main Menu
=================================
 1) Display current configuration
 2) SERDES configuration
 3) File options
 4) DDR options
 5) PCIe options
 6) TWSI options
 7) SMI/MDIO options
 8) SPI/MPI options
 9) eMMC/SD options
10) SATA options
11) GPIO options
12) SGPIO options
13) USB options
14) Interactive Lua prompt
15) PHY options
16) Traffic Generator
17) Power Burn options
18) CCPI options
19) CCPI forwarding
20) Reboot
21) Enzian Tests


40G PHY

VSC8248 Quad 10G Transceiver

40G PHY testing menu

=================================
40G PHY Menu
=================================
 1) PHY 1
 2) PHY 2
 3) Main menu

Selected PHY testing menu

=================================
PHY Menu 1
=================================
 1) Read register
 2) Write register
 3) Reset
 4) Enable loopback
 5) Enable pass through
 6) Power up
 7) Power down
 8) Channel 1 training
 9) Channel 2 training
10) Channel 3 training
11) Channel 4 training
12) Main menu
  1. Read register - read from a PHY's register
  2. Write register - write to PHY's register
  3. Reset - reset all the PHY's registers
  4. Enable loopback - enable a near-side (CPU side, TXIN -> RXOUT) loopback
  5. Enable pass through - enable normal routing, TXIN -> TXOUT, RXIN -> RXOUT
  6. Power up - power up transceivers
  7. Power down - power down transceivers
  8. Channel training - train manually a channel by adjusting slew rate (10), pre- (31), main- (31) and post-cursors (0). In brackets are values that give the best results with Firefly loopbacks.
PHY:1 Channel:1 trainining
Key 1 - slew rate -1
Key 2 - slew rate +1
Key 3 - pre-cursor -1
Key 4 - pre-cursor +1
Key 5 - main-cursor -1
Key 6 - main-cursor +1
Key 7 - post-cursor -1
Key 8 - post-cursor +1
Key q - quit
PHY:1  Channel:1  Status:11  Errors:0  Slew:10  Pre:15  Main:19  Post:0

The status is read from the register 0x1Ex1056, BISTCHK Status, hex number, the expected value is 0x11 (in sync, no errors, checker active).

Bit

Name

Description

7:5

Reserved

Reserved

4

SYNC

Synchronized to incoming pattern, 0: No sync, 1: Sync

3

FAIL

Bit error indicator, 0: No bit errors, 1: At least one bit error

2

ECF

Error count full, 0: No full count, 1: Full count

1

SYNCERR

Test finished with no sync, 0: No sync error, 1: Sync error

0

BUSY

Checker active, 0: Inactive, 1: Seeking or checking pattern

CPU reset configuration

  • AP_CVMMEMCTL0_EL1 - 0x009f0020'43e6d67c
    • ZVAL2CDIS - ZVA bypass L2C - disabled 

ATF

Required Ubuntu packages:

gcc-aarch64-linux-gnu
linux-libc-dev-arm64-cross
libc6-dev-arm64-cross

cd enzian-atf
make
make fip
make bootstrap


Building the boot image

tools/make-bootfs.py --bs bdk.bin --bl0 bootstrap.bin --bl1 bl1.bin --fip fip.bin -e THUNDER_EFI.fd -f bootfs.bin


UEFI

export LANG=C
export PATH=/opt/ThunderX-SDK/host/bin/:/opt/ThunderX-SDK/tools-gcc-4.7/bin/:$PATH
cd enzian-uefi
make

Enable debug (without it you will likely get synchronous exceptions when booting, the working hypothesis is that there are concurrency bugs and the debug build slightly changes some timings):

make DEBUG=1

The debug bitmask is defined in gEfiMdePkgTokenSpaceGuid.PcdDebugPrintErrorLevel, located in the file enzian-uefi/ArmPlatformPkg/ThunderPkg/Thunder.dsc.inc

Corrupted boot configuration

If the boot configuration gets corrupted and it is impossible to modify it i.e. changes are not saved, the UEFI command varclr  will clear all of the settings, including the boot configuration.

Linux


  • No labels