Page MenuHomeFreeBSD

Intel IOMMU DMAR suspend/resume support (only without QI for now)
Needs ReviewPublic

Authored by on Dec 3 2019, 2:23 PM.


Group Reviewers
Contributor Reviews (base)

Trying to sort of vaguely do what Linux does. With queued invalidation disabled (hw.dmar.enable=1 hw.dmar.qi=0) my laptop actually does resume :)

(debug prints are left over in the patch, I'll remove them in a later update)

Diff Detail

rS FreeBSD src repository
Lint Skipped
Unit Tests Skipped

Event Timeline

kib added a comment.Dec 3 2019, 3:07 PM

DMARs must be suspended after, and resumed before any subordinate devices. Because DMARs do not participate in the newbus hierarchy, or rather, they are placed there in the arbitrary place determined by not relevant details of the driver implementation, newbus suspend/resume methods are really not appropriate to use. The methods should be nops, and actual suspend/resume done near nexus.


The structure is not needed.


For this to have even a chance to work, you must reinitialize the queue before enabling qi. Also, I am surprised that it worked for you, since if queued invalidation was used, queue must be re-enabled for TLB flushes to work after resume, as well.


Command invalidation must not be used if queued invalidation is enabled.


This should only be done if translations were enabled before suspend.


You need to clear potentially unprocessed faults before enabling fault interrupt. marked 2 inline comments as done.Dec 3 2019, 3:33 PM added inline comments.

of course, I just thought it was a nice thing to do :)


QI did not work for me, the DMAR-related calls from the IOAPIC driver locked the system up before getting to here.

Linux doesn't seem to be reinitializing the queue on resume, they just do the disable and the __enable which resets the head/tail but does not allocate memory, which is only done by the full enable at the original boot


Are dmar_qi_invalidate_ctx_glob_locked and dmar_qi_invalidate_iotlb_glob_locked the right functions for this when QI is enabled?

kib added inline comments.Dec 3 2019, 3:46 PM

I mean that you need to re-initialize hardware state for queue, of course you do not need to reallocate queue. But for this to work, you need to ensure that the queue is drained.

I am not sure what you get with IOAPIC and why do you claim that it deadlock the system, but it is probably just a manifestation of the ordering issue I talked about in the general comment.


AFAIR yes, but they require working queue. edited the summary of this revision. (Show Details)

Now calling from IOAPIC suspend/resume functions (if there's any better place "close to the nexus" I haven't found it.)

Moved setting head/tail registers into dmar_enable_qi, so now that gets called on resume.

Added waiting for the queue to empty, it's actually already empty (tail == head) both before suspend and right after resume (in the latter case, it's just not initialized, head and tail are 0).

Now hangs in dmar_qi_invalidate_ctx_glob_locked