bus_dma_dmar_set_buswide(9): KPI to indicate that whole dmar context should share page tables.

[kib]

Summary

De-facto it means that dma requests from any device on the bus are translated according to the entries loaded for the bus:0:0 device.

KPI requires that the slot and function of the device be 0:0, and that no tags for other devices on the bus were used.

The intended use are NTBs which pass TLPs from the downstream to the host with slot:func of the downstream originator.

[kib]

Alexander, I obviously cannot test this, I only ensured that machine with DMAR enabled still boots.

[mav]

I did quick test with PLX NTB with this patch to its driver:

Index: sys/dev/ntb/ntb_hw/ntb_hw_plx.c
===================================================================
--- sys/dev/ntb/ntb_hw/ntb_hw_plx.c     (revision 354799)
+++ sys/dev/ntb/ntb_hw/ntb_hw_plx.c     (working copy)
@@ -452,6 +452,9 @@ ntb_plx_attach(device_t dev)
                        sc->spad_count2 = PLX_NUM_SPAD_PATT / 2;
        }
 
+       /* This device uses all of devices and functions on the bus. */
+       bus_dma_dmar_set_buswide(dev);
+
        /* Apply static part of NTB configuration. */
        ntb_plx_init(dev);
 

, and it seems like working. The DMAR error messages have gone and my code reading from NTB window that previously failed is now working. I am not completely certain in what part of the driver it is cleaner to put this line. I though about the beginning, as I understand it should be just before the first bus_dma tag access or interrupt setup. But in case of attach failure I guess it should better be reverted somehow? Since there is no such method I put it when most of probe process is done.

I haven't investigated how exactly Intel NTB does the translation, they are not documenting it unlike PLX. I may try it with DMAR later when reconfigure the hardware. For AMD I have neither docs nor hardware, unfortunately.

[kib]

In D22434#490451, @mav wrote:

I am not completely certain in what part of the driver it is cleaner to put this line. I though about the beginning, as I understand it should be just before the first bus_dma tag access or interrupt setup. But in case of attach failure I guess it should better be reverted somehow? Since there is no such method I put it when most of probe process is done.

There are two requirements for the call to bus_dma_dmar_set_buswide():

1. the call must occur before the first busdma tag allocation for anything on this bus
2. the call must not occur before you are certain that the device identified has the bus-wide property.

It does not matter if attach fails later, as far as we know that the device does not allow any sibling on the bus, and that device can usurp all slot:function values, because attachment failure is a software thing, while presence of such hardware prevents other PCI devices on this bus to ever appear.

I considered adding a table with the list of PCI ids for this kind of devices, but then decided that since probes must identify the devices anyway, having yet another table is worse.

I haven't investigated how exactly Intel NTB does the translation, they are not documenting it unlike PLX. I may try it with DMAR later when reconfigure the hardware. For AMD I have neither docs nor hardware, unfortunately.

Could you please point me to the specific PLX datasheet and the chapter/section that explains the originator handling by the device.

[mav]

In D22434#490491, @kib wrote:

Could you please point me to the specific PLX datasheet and the chapter/section that explains the originator handling by the device.

The document I am using is "ExpressLane PEX 8717-BA/CA 16-Lane, 10-Port PCI Express Gen Multi-Root Switch with DMA Data Book", section "16.2 Requester ID Translation". Unfortunately it is under NDA.