• Created by Unknown User (mgabtest), last updated by Anonymous on Jan 16, 2025 6 minute read

Bringup trace

captured: 06.09.2019
file name: ccpi_bringup_4lane_bidir_256G.bin

During the capture of this trace, the ECI link has been reset. We expect to see a full re-establishment of an ECI link. Layer after layer, we should be able to observer how an ECI link should be established.

Interlaken layer

It looks like the Interlaken layer has not been reset. The metaframes are all complete and span the entire trace.

Block layer

The following table shows the exchanged blocks. For each phase, only the first timestamp is shown. Until the next timestamp shown in the table, the blocks sent are the same.

Timestamps in ms

Node 0 => Node 1

Node 0 <= Node 1

Node 0 state

Node 1 state

Comment

0.0

INIT_REQ, ACK=1

INIT_REQ, ACK=0

IREQ

IREQ

 

7894.684827

INIT_REQ, ACK=1

INIT_REQ, ACK=1 (one single message)

IREQ

IACK

Node 1 accepts ACK from node 0 and sends one back

7894.684880

INIT_REQ, ACK=1

9 times SYNC, REQ=00

IACK

IACK

Node 1 waits until no more REQ!=00 blocks arrive and goes to RUN immediately

7894.685363

SYNC, REQ=00

CREDITS

IACK

RUN

Node 1 sends initial credits to node 0

7894.687292

SYNC, REQ=00

IDLE, ACK=0

IACK

RUN

Node 0 stays in state IACK and sends empty SYNC blocks as normal for IACK. Not receiving any more INIT_REQ blocks or blocks with ACK=1, it should go to RUN state.There seems to be a timeout, it only happens after approximately 10ms

7903.367620

CREDITS and 4 time ACK, first four ACK=1 others ACK=0

4x IDLE block with ACK=1, others are all ACK=0

RUN

RUN

Node 0 sends initial credits to node 1 and acknowledges the received Credits from earlier. Node 1 acknowledges the received CREDIT (=DATA) blocks. (1 ACK <=> 8 correct data blocks, 4x ACK are for 32 blocks with initial credits )

7903.369550

IDLE, ACK=0

IDLE, ACK=0

RUN

RUN

 

The first blocks observed, in both directions, are SYNC blocks, requesting link initiation. All blocks until second 7.894 are of this type. Node 0 has the ACK flag set in its initiation requests, node 1 does not.

Then both nodes send some blocks that are not quite covered by our user guide (see below).

Undocumented SYNC Blocks

The blocks sent after INIT_REQ and before the CREDITS are sent are not in a format described by the user guide document. But our older description here describes this particular field a bit differently and with that those can just be SYNC blocks with SM_REQ set to false. (TODO: Update user guide?)

Following are the details of said blocks.

They are marked as SYNC blocks (Header bits 63:61 = 110) but the field Req[53:52] contains 00, which is not explained by our block-layer documentation so far.

Node 0

Node 0 sends these blocks for about 10ms. All but the first blocks look exactly the same.

For node 0, the header information is the same for all those blocks. Disregarding the CRC field, everything except the SYNC block marker is set to 0, including the ACK field.

The data words (unused for normal SYNC blocks) are not completely 0, as you can see below. Also interesting, the first block has some different values in there, whereas all following blocks have the same values.

Here is how all blocks except the first look

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="280d5357-f452-462a-b835-63c1a05de87b"><ac:plain-text-body><![CDATA[

Unused

[0x0000000000000000]

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="40a4553e-a988-4a1f-9237-f12c6c1755f7"><ac:plain-text-body><![CDATA[

Unused

[0x00000000000000C0]

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="6ae8c183-fece-47d4-a091-d1cd4a244784"><ac:plain-text-body><![CDATA[

Unused

[0x0000000000000000]

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="7836c468-5042-4887-86c2-814d309866aa"><ac:plain-text-body><![CDATA[

Unused

[0x00000000000000C0]

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="be4a5422-5dac-4dbc-a70d-30e4869f8eb5"><ac:plain-text-body><![CDATA[

Unused

[0x0000000000000000]

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="7e0b975a-ae23-4fb2-a5f4-99824db48326"><ac:plain-text-body><![CDATA[

Unused

[0x00000000000000C0]

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="b9d5c71c-e7a1-4c51-b799-308e4cfd07ed"><ac:plain-text-body><![CDATA[

Unused

[0x0000000000000000]

]]></ac:plain-text-body></ac:structured-macro>

Header

<0xC00000000034329C>

Here is the first block:

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="616160ea-cd43-44b5-9ef3-3a31bf6098d8"><ac:plain-text-body><![CDATA[

Unused

[0x0000000000000000]

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="590c40fc-0022-4752-83ed-98052469c6fd"><ac:plain-text-body><![CDATA[

Unused

[0x00000000000000D0]

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="37900fe2-ad5a-4435-a8c6-d96e369ef713"><ac:plain-text-body><![CDATA[

Unused

[0x0000000000000000]

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="2a53fa3b-f8ff-4cd9-b138-b1951da0635f"><ac:plain-text-body><![CDATA[

Unused

[0x00000000000000D0]

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="11b3c618-a7d8-42a1-8639-b1bb3837bacd"><ac:plain-text-body><![CDATA[

Unused

[0x0000000000000000]

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="e6b61b9f-769b-450f-8df6-3f9bb8407c3f"><ac:plain-text-body><![CDATA[

Unused

[0x00000000000000D0]

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="72cb9497-02a7-460d-91e6-ed637d6653cd"><ac:plain-text-body><![CDATA[

Unused

[0x0000000000000000]

]]></ac:plain-text-body></ac:structured-macro>

Header

<0xC000000000246F62>

Node 1

The other node only sends precisely 9 SYNC blocks with REQ=00. In this case, the first 8 blocks look all the same and the last block is different.

The important difference here is that the the first 8 blocks have the ACK bit (bit 60) set to 1, otherwise the blocks look mostly the same in both directions.

First 8 blocks:

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="3db726c5-e971-4bb8-92df-21c35f42012b"><ac:plain-text-body><![CDATA[

Unused

[0x0000000000000000]

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="a2fe7174-9af9-4142-8677-b3e792d835f3"><ac:plain-text-body><![CDATA[

Unused

[0x00000000000000D0]

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="8283c777-ba0a-4995-9072-5820997b6426"><ac:plain-text-body><![CDATA[

Unused

[0x0000000000000000]

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="e23d2c92-198f-4c24-b47a-b476c164eaf3"><ac:plain-text-body><![CDATA[

Unused

[0x00000000000000D0]

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="454b1d04-7d05-482c-bffa-a4da2e5ed1b3"><ac:plain-text-body><![CDATA[

Unused

[0x0000000000000000]

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="11963a4b-8f7e-47a9-be6a-82b5f24046d6"><ac:plain-text-body><![CDATA[

Unused

[0x00000000000000D0]

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="59d754c9-7ebb-4662-8470-46db015f46aa"><ac:plain-text-body><![CDATA[

Unused

[0x0000000000000000]

]]></ac:plain-text-body></ac:structured-macro>

Header

<0xD0000000005253FB>

Last block:

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="3297d612-28e0-400d-993b-30c20b81c1f1"><ac:plain-text-body><![CDATA[

Unused

[0x0000000000000000]

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="9811ed95-48ec-43de-a91c-74bbe34447b4"><ac:plain-text-body><![CDATA[

Unused

[0x00000000000000D0]

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="af7d3a27-ea59-4f77-b4da-fa146e35ecea"><ac:plain-text-body><![CDATA[

Unused

[0x0000000000000000]

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="34934fd6-ea3b-456d-b330-9d46f7a25599"><ac:plain-text-body><![CDATA[

Unused

[0x00000000000000D0]

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="053d601a-c2c2-44c3-a411-ffefc8abddde"><ac:plain-text-body><![CDATA[

Unused

[0x0000000000000000]

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="9e9125d3-fede-4cef-a263-5e835fb602fa"><ac:plain-text-body><![CDATA[

Unused

[0x00000000000000D0]

]]></ac:plain-text-body></ac:structured-macro>

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="4c6bc348-1802-40c7-9de5-112dc91ffd64"><ac:plain-text-body><![CDATA[

Unused

[0x0000000000000000]

]]></ac:plain-text-body></ac:structured-macro>

Header

<0xC000000000246F62>

Credits

For initialization, credits in both direction are sent in the same pattern. Here is the full part of the trace where credit are sent from node 0 to node 1. (Only the timestamps differ in the other direction.)

    [7903.367620] DATA BLOCK (ACK=1) Credits: [0, 1, 2, 3, 4, 5, 6, 7]
    [7903.367674] DATA BLOCK (ACK=1) Credits: [8, 9, 10, 11, 12]
    [7903.367728] DATA BLOCK (ACK=1) Credits: [0, 1, 2, 3, 4, 5, 6, 7]
    [7903.367781] DATA BLOCK (ACK=1) Credits: [8, 9, 10, 11, 12]
    [7903.367835] DATA BLOCK (ACK=0) Credits: [0, 1, 2, 3, 4, 5, 6, 7]
    [7903.367888] DATA BLOCK (ACK=0) Credits: [8, 9, 10, 11, 12]
    [7903.367942] DATA BLOCK (ACK=0) Credits: [0, 1, 2, 3, 4, 5, 6, 7]
    [7903.367996] DATA BLOCK (ACK=0) Credits: [8, 9, 10, 11, 12]
    [7903.368049] DATA BLOCK (ACK=0) Credits: [0, 1, 2, 3, 4, 5]
    [7903.368103] DATA BLOCK (ACK=0) Credits: [0, 1, 2, 3, 4, 5]
    [7903.368156] DATA BLOCK (ACK=0) Credits: [0, 1, 2, 3, 4, 5]
    [7903.368210] DATA BLOCK (ACK=0) Credits: [0, 1, 2, 3, 4, 5]
    [7903.368264] DATA BLOCK (ACK=0) Credits: [0, 1, 2, 3, 4, 5]
    [7903.368317] DATA BLOCK (ACK=0) Credits: [0, 1, 2, 3, 4, 5]
    [7903.368371] DATA BLOCK (ACK=0) Credits: [0, 1, 2, 3, 4, 5]
    [7903.368424] DATA BLOCK (ACK=0) Credits: [0, 1, 2, 3, 4, 5]
    [7903.368478] DATA BLOCK (ACK=0) Credits: [0, 1, 2, 3, 4, 5]
    [7903.368532] DATA BLOCK (ACK=0) Credits: [0, 1, 2, 3, 4, 5]
    [7903.368585] DATA BLOCK (ACK=0) Credits: [0, 1, 2, 3, 4, 5]
    [7903.368639] DATA BLOCK (ACK=0) Credits: [0, 1, 2, 3, 4, 5]
    [7903.368692] DATA BLOCK (ACK=0) Credits: [0, 1, 2, 3, 4, 5]
    [7903.368746] DATA BLOCK (ACK=0) Credits: [0, 1, 2, 3, 4, 5]
    [7903.368800] DATA BLOCK (ACK=0) Credits: [0, 1, 2, 3, 4, 5]
    [7903.368853] DATA BLOCK (ACK=0) Credits: [0, 1, 2, 3, 4, 5]
    [7903.368907] DATA BLOCK (ACK=0) Credits: [0, 1, 2, 3, 4, 5]
    [7903.368960] DATA BLOCK (ACK=0) Credits: [0, 1, 2, 3, 4, 5]
    [7903.369014] DATA BLOCK (ACK=0) Credits: [0, 1, 2, 3, 4, 5]
    [7903.369068] DATA BLOCK (ACK=0) Credits: [0, 1, 2, 3, 4, 5]
    [7903.369121] DATA BLOCK (ACK=0) Credits: [0, 1, 2, 3, 4, 5]
    [7903.369175] DATA BLOCK (ACK=0) Credits: [0, 1, 2, 3, 4, 5]
    [7903.369228] DATA BLOCK (ACK=0) Credits: [0, 1, 2, 3, 4, 5]
    [7903.369282] DATA BLOCK (ACK=0) Credits: [0, 1, 2, 3, 4, 5]
    [7903.369336] DATA BLOCK (ACK=0) Credits: [0, 1, 2, 3, 4, 5]
    [7903.369389] DATA BLOCK (ACK=0) Credits: [0, 1, 2, 3, 4, 5]
    [7903.369443] DATA BLOCK (ACK=0) Credits: [0, 1, 2, 3, 4, 5]
    [7903.369496] DATA BLOCK (ACK=0) Credits: [0, 1, 2, 3, 4, 5]

VC layer (ECI messages)

In the table below are the first ECI CC messages in the trace. There is exactly one messages on each side of the trace that cannot be parsed with our current specification. The message is sent on VC 13 which is supposed to be used for multicore debug messages and CSR data.

Note that the timestamps are from different sides and may not be ordered exactly. (It is possible that the order below is not 100% correct)

Timestamps in ms

Node 0 => Node 1

Node 0 <= Node 1

Address

ppvid

DID

payload

9129.338690

 

0x80055E6807F80000 vc=0xD

 

 

 

 

9134.199566

 

ECI_IREQ_SLILD

0x2200004

0x0

0x7e

 

9134.199911

ECI_IRSP_SLIRSP

 

 

0x30

 

000000000000000c

9134.201121

 

ECI_IREQ_SLIST

0x2200004

0x0

0x7e

000000000000000c

9139.493800

 

ECI_IREQ_SLIST

0x2202005

0x0

0x7e

0000abcd00000000

9139.493960

 

ECI_IREQ_SLILD

0x2200005

0x0

0x7e

 

9139.494143

0x80055E6800000000 vc=0xD

 

 

 

 

 

9139.494304

ECI_IRSP_SLIRSP

 

 

0x30

 

0000000000000027

9139.494979

 

ECI_IREQ_SLIST

0x2200005

0x0

0x7e

0000000000000007

9143.399738

 

ECI_IREQ_SLILD

0x2200006

0x0

0x7e

 

9143.400082

ECI_IRSP_SLIRSP

 

 

0x30

 

0000000000000026

9143.400810

 

ECI_IREQ_SLIST

0x2200006

0x0

0x7e

0000000000000006

9162.754805

 

ECI_IREQ_SLILD

0x2203005

0x0

0x7e

 

9162.755150

ECI_IRSP_SLIRSP

 

 

0x30

 

8000abcd00ff0000

9168.309267

 

ECI_IREQ_SLILD

0x2203405

0x0

0x7e

 

9168.309610

ECI_IRSP_SLIRSP

 

 

0x30

 

0000000000000000

9172.562159

 

ECI_IREQ_SLILD

0x2203805

0x0

0x7e

 

9172.562502

ECI_IRSP_SLIRSP

 

 

0x30

 

0000000000000000

9176.815158

 

ECI_IREQ_SLILD

0x2200000

0x0

0x7e

 

9176.815502

ECI_IRSP_SLIRSP

 

 

0x30

 

000000000000000d

  • No labels