Enable Intel DMAR by default
ClosedPublic
Actions

Authored by emaste on Nov 1 2023, 6:30 PM.

Details

Reviewers

kib
jhb

Commits

rG3192fc30230a: x86: Enable Intel DMAR by default
rG2777a32588e1: iommu: disable dma by default

Summary

APIC ID 255 and above require x2APIC and DMAR interrupt remapping.

Diff Detail

Lint

Lint Skipped

Unit

Tests Skipped

Event Timeline

emaste requested review of this revision.Nov 1 2023, 6:30 PM

emaste created this revision.

We were asked to take a quick look at 13.2 and 14.0 failing on a system that was to be deployed in production. 13.2 fails for a few reasons, while 14.0 successfully boots with hw.dmar_enable set to 1 (and fails to find its NVMe devices with dmar_enable at the default of 0).

emaste removed a reviewer: lwhsu.Nov 1 2023, 6:39 PM

emaste added subscribers: markj, jhb, lwhsu.

This would cause a huge (~10x) hit on workloads that use busdma intensively, like network drivers.

To be practical, dmar should be enabled, busdma disabled, ir enabled.

In D42435#968292, @kib wrote:

This would cause a huge (~10x) hit on workloads that use busdma intensively, like network drivers.

To be practical, dmar should be enabled, busdma disabled, ir enabled.

It looks like it would sufficient to make hw.iommu.dma default to 0 instead of 1, is that right?

In D42435#968499, @markj wrote:

In D42435#968292, @kib wrote:

This would cause a huge (~10x) hit on workloads that use busdma intensively, like network drivers.

To be practical, dmar should be enabled, busdma disabled, ir enabled.

It looks like it would sufficient to make hw.iommu.dma default to 0 instead of 1, is that right?

Yes.

Also disable hw.iommu.dma by default

kib accepted this revision.Nov 3 2023, 1:14 PM

This revision is now accepted and ready to land.Nov 3 2023, 1:14 PM

It does occur to me (after uploading this update) that I'm enabling Intel-specific DMAR by default, but the hw.iommu.dma that I'm switching to default off is global (moved out of DMAR by @br in f2b2f31707bce25e3fdee9fdfcb75ddbd1ff3338). At a minimum I'll need to note this in the commit message.

I would probably split this up into two commits (even though they are so tiny) since disabling IOMMU for bus_dma by default is a MI setting.

Closed by commit rG2777a32588e1: iommu: disable dma by default (authored by emaste). · Explain WhyAug 13 2024, 8:03 PM

This revision was automatically updated to reflect the committed changes.

emaste added a commit: rG2777a32588e1: iommu: disable dma by default.

Herald added a subscriber: imp. · View Herald TranscriptAug 13 2024, 8:03 PM

emaste added a commit: rG3192fc30230a: x86: Enable Intel DMAR by default.Aug 13 2024, 8:03 PM

In D42435#969553, @emaste wrote:

It does occur to me (after uploading this update) that I'm enabling Intel-specific DMAR by default, but the hw.iommu.dma that I'm switching to default off is global (moved out of DMAR by @br in f2b2f31707bce25e3fdee9fdfcb75ddbd1ff3338). At a minimum I'll need to note this in the commit message.

This didn't get noted anywhere :(

This didn't get noted anywhere :(

Followed up with a reply to the commit mail

This crashes my laptop with a null pointer dereference unless I do hw.dmar.enable=0 in the boot loader

trap
dmar_match_by_path()+0x20 <---- null pointer dereference on unit, which is somehow NULL)
dmar_find()
iommu_get_dma_tag()
acpi_pci_get_dma_tag()
xhci_init()
...

Hacking dmar_match_by_path to return if unit == NULL results in a system where the interrupts fail in various ways:
CPU0: local APIC error 0x40
nvme0: System interrupt issues?
CPU0: local APIC error 0x40
<repeats 100s of times>

So this isn't fully baked.

crest_freebsd_rlwinm.de added a subscriber: crest_freebsd_rlwinm.de.Aug 20 2024, 9:47 AM

emaste added a reverting change: rG0875f3cd74b2: Revert "x86: Enable Intel DMAR by default".Aug 20 2024, 3:50 PM