Page MenuHomeFreeBSD

bus_dma_dmar_set_buswide(9): KPI to indicate that whole dmar context should share page tables.
ClosedPublic

Authored by kib on Sun, Nov 17, 5:59 PM.

Details

Summary

De-facto it means that dma requests from any device on the bus are translated according to the entries loaded for the bus:0:0 device.

KPI requires that the slot and function of the device be 0:0, and that no tags for other devices on the bus were used.

The intended use are NTBs which pass TLPs from the downstream to the host with slot:func of the downstream originator.

Diff Detail

Repository
rS FreeBSD src repository
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

kib created this revision.Sun, Nov 17, 5:59 PM
kib added a comment.Sun, Nov 17, 6:00 PM

Alexander, I obviously cannot test this, I only ensured that machine with DMAR enabled still boots.

mav accepted this revision.Mon, Nov 18, 3:01 AM

I did quick test with PLX NTB with this patch to its driver:

Index: sys/dev/ntb/ntb_hw/ntb_hw_plx.c
===================================================================
--- sys/dev/ntb/ntb_hw/ntb_hw_plx.c     (revision 354799)
+++ sys/dev/ntb/ntb_hw/ntb_hw_plx.c     (working copy)
@@ -452,6 +452,9 @@ ntb_plx_attach(device_t dev)
                        sc->spad_count2 = PLX_NUM_SPAD_PATT / 2;
        }
 
+       /* This device uses all of devices and functions on the bus. */
+       bus_dma_dmar_set_buswide(dev);
+
        /* Apply static part of NTB configuration. */
        ntb_plx_init(dev);

, and it seems like working. The DMAR error messages have gone and my code reading from NTB window that previously failed is now working. I am not completely certain in what part of the driver it is cleaner to put this line. I though about the beginning, as I understand it should be just before the first bus_dma tag access or interrupt setup. But in case of attach failure I guess it should better be reverted somehow? Since there is no such method I put it when most of probe process is done.

I haven't investigated how exactly Intel NTB does the translation, they are not documenting it unlike PLX. I may try it with DMAR later when reconfigure the hardware. For AMD I have neither docs nor hardware, unfortunately.

This revision is now accepted and ready to land.Mon, Nov 18, 3:01 AM
kib added a comment.Mon, Nov 18, 9:48 AM
In D22434#490451, @mav wrote:

I am not completely certain in what part of the driver it is cleaner to put this line. I though about the beginning, as I understand it should be just before the first bus_dma tag access or interrupt setup. But in case of attach failure I guess it should better be reverted somehow? Since there is no such method I put it when most of probe process is done.

There are two requirements for the call to bus_dma_dmar_set_buswide():

  1. the call must occur before the first busdma tag allocation for anything on this bus
  2. the call must not occur before you are certain that the device identified has the bus-wide property.

It does not matter if attach fails later, as far as we know that the device does not allow any sibling on the bus, and that device can usurp all slot:function values, because attachment failure is a software thing, while presence of such hardware prevents other PCI devices on this bus to ever appear.

I considered adding a table with the list of PCI ids for this kind of devices, but then decided that since probes must identify the devices anyway, having yet another table is worse.

I haven't investigated how exactly Intel NTB does the translation, they are not documenting it unlike PLX. I may try it with DMAR later when reconfigure the hardware. For AMD I have neither docs nor hardware, unfortunately.

Could you please point me to the specific PLX datasheet and the chapter/section that explains the originator handling by the device.

mav added a comment.Mon, Nov 18, 2:56 PM
In D22434#490491, @kib wrote:

Could you please point me to the specific PLX datasheet and the chapter/section that explains the originator handling by the device.

The document I am using is "ExpressLane PEX 8717-BA/CA 16-Lane, 10-Port PCI Express Gen Multi-Root Switch with DMA Data Book", section "16.2 Requester ID Translation". Unfortunately it is under NDA.

This revision was automatically updated to reflect the committed changes.