Intel IOMMU DMAR suspend/resume support (only without QI for now)

[val_packett.cool]

Summary

Trying to sort of vaguely do what Linux does. With queued invalidation disabled (hw.dmar.enable=1 hw.dmar.qi=0) my laptop actually does resume :)

(debug prints are left over in the patch, I'll remove them in a later update)

[kib]

DMARs must be suspended after, and resumed before any subordinate devices. Because DMARs do not participate in the newbus hierarchy, or rather, they are placed there in the arbitrary place determined by not relevant details of the driver implementation, newbus suspend/resume methods are really not appropriate to use. The methods should be nops, and actual suspend/resume done near nexus.

sys/x86/iommu/intel_dmar.h

247

The structure is not needed.

sys/x86/iommu/intel_drv.c

676

For this to have even a chance to work, you must reinitialize the queue before enabling qi. Also, I am surprised that it worked for you, since if queued invalidation was used, queue must be re-enabled for TLB flushes to work after resume, as well.

687

Command invalidation must not be used if queued invalidation is enabled.

690

This should only be done if translations were enabled before suspend.

693

You need to clear potentially unprocessed faults before enabling fault interrupt.

[val_packett.cool]

sys/x86/iommu/intel_dmar.h

247

of course, I just thought it was a nice thing to do :)

sys/x86/iommu/intel_drv.c

676

QI did not work for me, the DMAR-related calls from the IOAPIC driver locked the system up before getting to here.

Linux doesn't seem to be reinitializing the queue on resume, they just do the disable and the __enable which resets the head/tail but does not allocate memory, which is only done by the full enable at the original boot

687

Are dmar_qi_invalidate_ctx_glob_locked and dmar_qi_invalidate_iotlb_glob_locked the right functions for this when QI is enabled?

[kib]

sys/x86/iommu/intel_drv.c

676

I mean that you need to re-initialize hardware state for queue, of course you do not need to reallocate queue. But for this to work, you need to ensure that the queue is drained.

I am not sure what you get with IOAPIC and why do you claim that it deadlock the system, but it is probably just a manifestation of the ordering issue I talked about in the general comment.

687

AFAIR yes, but they require working queue.

[val_packett.cool]

Now calling from IOAPIC suspend/resume functions (if there's any better place "close to the nexus" I haven't found it.)

Moved setting head/tail registers into dmar_enable_qi, so now that gets called on resume.

Added waiting for the queue to empty, it's actually already empty (tail == head) both before suspend and right after resume (in the latter case, it's just not initialized, head and tail are 0).

Now hangs in dmar_qi_invalidate_ctx_glob_locked…

[emaste]

Referenced in PR280817