3. pcieflat

Usage:

pcieflat

Dumps trace of PCIE transactions.

The trace contains the following columns:

col description
dirtype TLP direction and type
timestamp Timestamp of TLP, in main clock cycles
delta Number of cycles since previous TLP
type type:pktype:format
be Byte enables
hit BAR hit
eof End of frame
sof Start of frame
address Request physical address
tag Request tag

The first column, dirtype, contains RX if received by FPGA, TX if transmitted by FPGA, qq for start of request, pp for start of response, and cc for continuation.

The second column, timestamp, displays the timestamp in terms of a 64-bit counter running at the PCIe user clock frequency (125MHz for gen1, 250MHz for gen2 and gen3).

The third column, delta, displays the number of cycles since the previous TLP. If a TLP is transmitted and received on the same cycle, then the transmitted TLP will have a delta of 0. The first TLP shown in the trace will have a delta of 0.

The fourth column shows the type of the TLP. If it is the start of a transaction, it will be of the form type:tlppkttype:tlpformat, where the types are:

  • CpuRReq: read request from CPU to FPGA
  • CpuWReq: write request from CPU to FPGA
  • CpuRRsp: read response from FPGA to CPU
  • DmaRReq: read request from FPGA to CPU DRAM
  • DmaWReq: write request from FPGA to CPU DRAM
  • DmaRRsp: read response from CPU DRAM to FPGA
  • CpuRCon: continuation data sent from FPGA to CPU (continuation of CpuRRsp or DmaWReq)
  • DmaRCon: continuation data sent from CPU DRAM to FPGA (continuation of DmaRRsp or CpuWReq)
  • Interru: interrupt message from FPGA to CPU

The TLP pkttype is one of the following:

  • MRW: Memory read/write
  • COMP: additional data (“completion”) of transaction

The TLP format is one of the following:

  • MEM_WRITE_3DW_DATA: 96-bit write request header containing 32-bit address and 32-bit data
  • MEM_WRITE_4DW_DATA: 128-bit write request header containing 64-bit address and no data
  • MEM_READ__3DW: 96-bit read request header containing 32-bit address
  • MEM_READ__4DW: 128-bit read request header containing 64-bit address

The sof and eof flags indicate the start and end TLPs of each transaction.

3.1. Diagnosing Crashes

If the machine crashed, look for transactions that were started but not ended.

These will generally fall before very large deltas, where the machine was rebooted before it had any more interactions with the FPGA.

3.2. Example Trace

Here are some excerpts from the output of pcieflat after running tests/memserver_copy:

pcieflat: devices are ['/dev/portal_0_6', '/dev/portal_0_5', '/dev/portal_0_4', '/dev/portal_0_3', '/dev/portal_0_2', '/dev/portal_0_1']
             ts     delta   response                     XXX          tlp          address  off   be       tag     clid  nosnp  laddr        data
                                pkttype format               foo (be hit eof sof)            (1st last)        req     stat  bcnt    length
RXpp  513858932          0 DmaRRsp:COMP:MEM_WRITE_3DW_DATA 09 0x4 tlp(ffff 0 0 1)                        tag:07 0300 0000 0 00 080 00  16 01179d1d
TXcc  513858932          0 CpuRCon                         10 0x8 tlp(ffff 0 0 0)                            data:259b17012c9b1701339b17013a9b1701
RXcc  513858933          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:249d17012b9d1701329d1701399d1701
TXcc  513858933          0 CpuRCon                         10 0x8 tlp(ffff 0 1 0)                            data:419b1701489b17014f9b1701569b1701
RXcc  513858934          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:409d1701479d17014e9d1701559d1701
RXcc  513858935          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:5c9d1701639d17016a9d1701719d1701
RXcc  513858936          1 DmaRCon                         08 0x4 tlp(ffff 0 1 0)                            data:789d17017f9d1701869d170164b19a21
RXpp  513858938          2 DmaRRsp:COMP:MEM_WRITE_3DW_DATA 09 0x4 tlp(ffff 0 0 1)                        tag:07 0300 0000 0 00 040 40  16 01179d8d
RXcc  513858939          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:949d17019b9d1701a29d1701a99d1701
RXcc  513858940          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:b09d1701b79d1701be9d1701c59d1701
RXcc  513858941          1 DmaRCon                         08 0x4 tlp(ffff 0 0 0)                            data:cc9d1701d39d1701da9d1701e19d1701
RXcc  513858942          1 DmaRCon                         08 0x4 tlp(ffff 0 1 0)                            data:e89d1701ef9d1701f69d1701af3ad2c2
TXqq  513858949          7 DmaWReq: MRW:MEM_WRITE_4DW_DATA 11 0x8 tlp(ffff 0 0 1) address: 000000035f4fc680 be(1st: f last:f) tag:05 reqid:0300 length:32
TXcc  513858950          1 CpuRCon                         10 0x8 tlp(ffff 0 0 0)                            data:5d9b1701649b17016b9b1701729b1701
TXcc  513858951          1 CpuRCon                         10 0x8 tlp(ffff 0 0 0)                            data:799b1701809b1701879b17018e9b1701
TXcc  513858952          1 CpuRCon                         10 0x8 tlp(ffff 0 0 0)                            data:959b17019c9b1701a39b1701aa9b1701
TXcc  513858953          1 CpuRCon                         10 0x8 tlp(ffff 0 0 0)                            data:b19b1701b89b1701bf9b1701c69b1701
TXcc  513858954          1 CpuRCon                         10 0x8 tlp(ffff 0 0 0)                            data:cd9b1701d49b1701db9b1701e29b1701
TXcc  513858955          1 CpuRCon                         10 0x8 tlp(ffff 0 0 0)                            data:e99b1701f09b1701f79b1701fe9b1701
TXcc  513858956          1 CpuRCon                         10 0x8 tlp(ffff 0 0 0)                            data:059c17010c9c1701139c17011a9c1701
TXcc  513858957          1 CpuRCon                         10 0x8 tlp(ffff 0 1 0)                            data:219c1701289c17012f9c1701369c1701

It may contain DMA Read Requests:

TXqq  513859009          1 DmaRReq: MRW:MEM_READ__4DW      11 0x8 tlp(ffff 0 1 1) address: 000000039e8fc800 be(1st: f last:f) tag:00 reqid:0300 length:32
TXqq  513859010          1 DmaRReq: MRW:MEM_READ__4DW      11 0x8 tlp(ffff 0 1 1) address: 000000039e8fc880 be(1st: f last:f) tag:01 reqid:0300 length:32

The trace may contain interrupts:

TXqq  513865009         12 Interru: MRW:MEM_WRITE_3DW_DATA 11 0x8 tlp(ffff 0 1 1)  fee00000    0 be(f 0) tag:00 0300                    1 0000406e

It will also contain reads and writes from the CPU to the FPGA:

RXqq  513874994       9985 CpuRReq: MRW:MEM_READ__3DW      09 0x4 tlp(ffff 4 1 1)  df400000    0 be(f 0) tag:00 0038                    1
TXpp  513875008         14 CpuRRsp:COMP:MEM_WRITE_3DW_DATA 11 0x8 tlp(ffff 4 1 1)                        tag:00 0038 0300 0 00 004 00   1 00000001
RXqq  513875159        151 CpuRReq: MRW:MEM_READ__3DW      09 0x4 tlp(ffff 4 1 1)  df402000  800 be(f 0) tag:00 0038                    1
TXpp  513875173         14 CpuRRsp:COMP:MEM_WRITE_3DW_DATA 11 0x8 tlp(ffff 4 1 1)                        tag:00 0038 0300 0 00 004 00   1 00000000
RXqq  513875323        150 CpuRReq: MRW:MEM_READ__3DW      09 0x4 tlp(ffff 4 1 1)  df401000  400 be(f 0) tag:00 0038                    1
TXpp  513875337         14 CpuRRsp:COMP:MEM_WRITE_3DW_DATA 11 0x8 tlp(ffff 4 1 1)                        tag:00 0038 0300 0 00 004 00   1 00000000
RXqq  513876236        899 CpuRReq: MRW:MEM_READ__3DW      09 0x4 tlp(ffff 4 1 1)  df40000c    3 be(f 0) tag:00 0038                    1
TXpp  513876250         14 CpuRRsp:COMP:MEM_WRITE_3DW_DATA 11 0x8 tlp(ffff 4 1 1)                        tag:00 0038 0300 0 00 004 0c   1 00000001
RXqq  513876449        199 CpuRReq: MRW:MEM_READ__3DW      09 0x4 tlp(ffff 4 1 1)  df400020    8 be(f 0) tag:00 0038                    1
TXpp  513876463         14 CpuRRsp:COMP:MEM_WRITE_3DW_DATA 11 0x8 tlp(ffff 4 1 1)                        tag:00 0038 0300 0 00 004 20   1 00000000
RXqq  513883818       7355 CpuRReq: MRW:MEM_READ__3DW      09 0x4 tlp(ffff 4 1 1)  df40000c    3 be(f 0) tag:00 0038                    1
TXpp  513883832         14 CpuRRsp:COMP:MEM_WRITE_3DW_DATA 11 0x8 tlp(ffff 4 1 1)                        tag:00 0038 0300 0 00 004 0c   1 00000000
RXqq  513883976        144 CpuWReq: MRW:MEM_WRITE_3DW_DATA 09 0x4 tlp(ffff 4 1 1)  df400004    1 be(f 0) tag:02 0000                    1 00000001
RXqq  513884007         31 CpuRReq: MRW:MEM_READ__3DW      09 0x4 tlp(ffff 4 1 1)  df40200c  803 be(f 0) tag:00 0038                    1
TXpp  513884021         14 CpuRRsp:COMP:MEM_WRITE_3DW_DATA 11 0x8 tlp(ffff 4 1 1)                        tag:00 0038 0300 0 00 004 0c   1 00000000
RXqq  513884165        144 CpuWReq: MRW:MEM_WRITE_3DW_DATA 09 0x4 tlp(ffff 4 1 1)  df402004  801 be(f 0) tag:03 0000                    1 00000001

The trace will end with a summary of the kinds of PCIe transactions:

{'DmaWReq': 115, 'CpuRReq': 45, 'Interru': 3, 'CpuRRsp': 45, 'CpuRCon': 922, 'CpuWReq': 15, 'DmaRReq': 112, 'DmaRRsp': 141, 'DmaRCon': 904}
2302