3. pcieflat¶
Usage:
pcieflat
Dumps trace of PCIE transactions.
The trace contains the following columns:
col |
description |
|---|---|
dirtype |
TLP direction and type |
timestamp |
Timestamp of TLP, in main clock cycles |
delta |
Number of cycles since previous TLP |
type |
type:pktype:format |
be |
Byte enables |
hit |
BAR hit |
eof |
End of frame |
sof |
Start of frame |
address |
Request physical address |
tag |
Request tag |
The first column, dirtype, contains RX if received by FPGA, TX if transmitted by FPGA, qq for start of request, pp for start of response, and cc for continuation.
The second column, timestamp, displays the timestamp in terms of a 64-bit counter running at the PCIe user clock frequency (125MHz for gen1, 250MHz for gen2 and gen3).
The third column, delta, displays the number of cycles since the previous TLP. If a TLP is transmitted and received on the same cycle, then the transmitted TLP will have a delta of 0. The first TLP shown in the trace will have a delta of 0.
The fourth column shows the type of the TLP. If it is the start of a transaction, it will be of the form type:tlppkttype:tlpformat, where the types are:
CpuRReq: read request from CPU to FPGA
CpuWReq: write request from CPU to FPGA
CpuRRsp: read response from FPGA to CPU
DmaRReq: read request from FPGA to CPU DRAM
DmaWReq: write request from FPGA to CPU DRAM
DmaRRsp: read response from CPU DRAM to FPGA
CpuRCon: continuation data sent from FPGA to CPU (continuation of CpuRRsp or DmaWReq)
DmaRCon: continuation data sent from CPU DRAM to FPGA (continuation of DmaRRsp or CpuWReq)
Interru: interrupt message from FPGA to CPU
The TLP pkttype is one of the following:
MRW: Memory read/write
COMP: additional data (“completion”) of transaction
The TLP format is one of the following:
MEM_WRITE_3DW_DATA: 96-bit write request header containing 32-bit address and 32-bit data
MEM_WRITE_4DW_DATA: 128-bit write request header containing 64-bit address and no data
MEM_READ__3DW: 96-bit read request header containing 32-bit address
MEM_READ__4DW: 128-bit read request header containing 64-bit address
The sof and eof flags indicate the start and end TLPs of each transaction.
3.1. Diagnosing Crashes¶
If the machine crashed, look for transactions that were started but not ended.
These will generally fall before very large deltas, where the machine was rebooted before it had any more interactions with the FPGA.
3.2. Example Trace¶
Here are some excerpts from the output of pcieflat after running tests/memserver_copy:
pcieflat: devices are ['/dev/portal_0_6', '/dev/portal_0_5', '/dev/portal_0_4', '/dev/portal_0_3', '/dev/portal_0_2', '/dev/portal_0_1']
ts delta response XXX tlp address off be tag clid nosnp laddr data
pkttype format foo (be hit eof sof) (1st last) req stat bcnt length
RXpp 513858932 0 DmaRRsp:COMP:MEM_WRITE_3DW_DATA 09 0x4 tlp(ffff 0 0 1) tag:07 0300 0000 0 00 080 00 16 01179d1d
TXcc 513858932 0 CpuRCon 10 0x8 tlp(ffff 0 0 0) data:259b17012c9b1701339b17013a9b1701
RXcc 513858933 1 DmaRCon 08 0x4 tlp(ffff 0 0 0) data:249d17012b9d1701329d1701399d1701
TXcc 513858933 0 CpuRCon 10 0x8 tlp(ffff 0 1 0) data:419b1701489b17014f9b1701569b1701
RXcc 513858934 1 DmaRCon 08 0x4 tlp(ffff 0 0 0) data:409d1701479d17014e9d1701559d1701
RXcc 513858935 1 DmaRCon 08 0x4 tlp(ffff 0 0 0) data:5c9d1701639d17016a9d1701719d1701
RXcc 513858936 1 DmaRCon 08 0x4 tlp(ffff 0 1 0) data:789d17017f9d1701869d170164b19a21
RXpp 513858938 2 DmaRRsp:COMP:MEM_WRITE_3DW_DATA 09 0x4 tlp(ffff 0 0 1) tag:07 0300 0000 0 00 040 40 16 01179d8d
RXcc 513858939 1 DmaRCon 08 0x4 tlp(ffff 0 0 0) data:949d17019b9d1701a29d1701a99d1701
RXcc 513858940 1 DmaRCon 08 0x4 tlp(ffff 0 0 0) data:b09d1701b79d1701be9d1701c59d1701
RXcc 513858941 1 DmaRCon 08 0x4 tlp(ffff 0 0 0) data:cc9d1701d39d1701da9d1701e19d1701
RXcc 513858942 1 DmaRCon 08 0x4 tlp(ffff 0 1 0) data:e89d1701ef9d1701f69d1701af3ad2c2
TXqq 513858949 7 DmaWReq: MRW:MEM_WRITE_4DW_DATA 11 0x8 tlp(ffff 0 0 1) address: 000000035f4fc680 be(1st: f last:f) tag:05 reqid:0300 length:32
TXcc 513858950 1 CpuRCon 10 0x8 tlp(ffff 0 0 0) data:5d9b1701649b17016b9b1701729b1701
TXcc 513858951 1 CpuRCon 10 0x8 tlp(ffff 0 0 0) data:799b1701809b1701879b17018e9b1701
TXcc 513858952 1 CpuRCon 10 0x8 tlp(ffff 0 0 0) data:959b17019c9b1701a39b1701aa9b1701
TXcc 513858953 1 CpuRCon 10 0x8 tlp(ffff 0 0 0) data:b19b1701b89b1701bf9b1701c69b1701
TXcc 513858954 1 CpuRCon 10 0x8 tlp(ffff 0 0 0) data:cd9b1701d49b1701db9b1701e29b1701
TXcc 513858955 1 CpuRCon 10 0x8 tlp(ffff 0 0 0) data:e99b1701f09b1701f79b1701fe9b1701
TXcc 513858956 1 CpuRCon 10 0x8 tlp(ffff 0 0 0) data:059c17010c9c1701139c17011a9c1701
TXcc 513858957 1 CpuRCon 10 0x8 tlp(ffff 0 1 0) data:219c1701289c17012f9c1701369c1701
It may contain DMA Read Requests:
TXqq 513859009 1 DmaRReq: MRW:MEM_READ__4DW 11 0x8 tlp(ffff 0 1 1) address: 000000039e8fc800 be(1st: f last:f) tag:00 reqid:0300 length:32
TXqq 513859010 1 DmaRReq: MRW:MEM_READ__4DW 11 0x8 tlp(ffff 0 1 1) address: 000000039e8fc880 be(1st: f last:f) tag:01 reqid:0300 length:32
The trace may contain interrupts:
TXqq 513865009 12 Interru: MRW:MEM_WRITE_3DW_DATA 11 0x8 tlp(ffff 0 1 1) fee00000 0 be(f 0) tag:00 0300 1 0000406e
It will also contain reads and writes from the CPU to the FPGA:
RXqq 513874994 9985 CpuRReq: MRW:MEM_READ__3DW 09 0x4 tlp(ffff 4 1 1) df400000 0 be(f 0) tag:00 0038 1
TXpp 513875008 14 CpuRRsp:COMP:MEM_WRITE_3DW_DATA 11 0x8 tlp(ffff 4 1 1) tag:00 0038 0300 0 00 004 00 1 00000001
RXqq 513875159 151 CpuRReq: MRW:MEM_READ__3DW 09 0x4 tlp(ffff 4 1 1) df402000 800 be(f 0) tag:00 0038 1
TXpp 513875173 14 CpuRRsp:COMP:MEM_WRITE_3DW_DATA 11 0x8 tlp(ffff 4 1 1) tag:00 0038 0300 0 00 004 00 1 00000000
RXqq 513875323 150 CpuRReq: MRW:MEM_READ__3DW 09 0x4 tlp(ffff 4 1 1) df401000 400 be(f 0) tag:00 0038 1
TXpp 513875337 14 CpuRRsp:COMP:MEM_WRITE_3DW_DATA 11 0x8 tlp(ffff 4 1 1) tag:00 0038 0300 0 00 004 00 1 00000000
RXqq 513876236 899 CpuRReq: MRW:MEM_READ__3DW 09 0x4 tlp(ffff 4 1 1) df40000c 3 be(f 0) tag:00 0038 1
TXpp 513876250 14 CpuRRsp:COMP:MEM_WRITE_3DW_DATA 11 0x8 tlp(ffff 4 1 1) tag:00 0038 0300 0 00 004 0c 1 00000001
RXqq 513876449 199 CpuRReq: MRW:MEM_READ__3DW 09 0x4 tlp(ffff 4 1 1) df400020 8 be(f 0) tag:00 0038 1
TXpp 513876463 14 CpuRRsp:COMP:MEM_WRITE_3DW_DATA 11 0x8 tlp(ffff 4 1 1) tag:00 0038 0300 0 00 004 20 1 00000000
RXqq 513883818 7355 CpuRReq: MRW:MEM_READ__3DW 09 0x4 tlp(ffff 4 1 1) df40000c 3 be(f 0) tag:00 0038 1
TXpp 513883832 14 CpuRRsp:COMP:MEM_WRITE_3DW_DATA 11 0x8 tlp(ffff 4 1 1) tag:00 0038 0300 0 00 004 0c 1 00000000
RXqq 513883976 144 CpuWReq: MRW:MEM_WRITE_3DW_DATA 09 0x4 tlp(ffff 4 1 1) df400004 1 be(f 0) tag:02 0000 1 00000001
RXqq 513884007 31 CpuRReq: MRW:MEM_READ__3DW 09 0x4 tlp(ffff 4 1 1) df40200c 803 be(f 0) tag:00 0038 1
TXpp 513884021 14 CpuRRsp:COMP:MEM_WRITE_3DW_DATA 11 0x8 tlp(ffff 4 1 1) tag:00 0038 0300 0 00 004 0c 1 00000000
RXqq 513884165 144 CpuWReq: MRW:MEM_WRITE_3DW_DATA 09 0x4 tlp(ffff 4 1 1) df402004 801 be(f 0) tag:03 0000 1 00000001
The trace will end with a summary of the kinds of PCIe transactions:
{'DmaWReq': 115, 'CpuRReq': 45, 'Interru': 3, 'CpuRRsp': 45, 'CpuRCon': 922, 'CpuWReq': 15, 'DmaRReq': 112, 'DmaRRsp': 141, 'DmaRCon': 904}
2302