This doesn't cause any problems in EC2, but it doesn't solve the "recognizing that we're ready for an nvme disk to go away" problem either. They're definitely using a write to the power status register as a cue to remove the device (per https://reviews.freebsd.org/D49146 ).

In D49222#1122599, @cperciva wrote:

This doesn't cause any problems in EC2, but it doesn't solve the "recognizing that we're ready for an nvme disk to go away" problem either. They're definitely using a write to the power status register as a cue to remove the device (per https://reviews.freebsd.org/D49146 ).

Humm, I can't imagine what the goal of this behavior is on the part of EC2. Can the engineers there explain why they think the PME is still pending and needs to be cleared? Do they assert a PME as part of the eject notification? (You can read the PME bit to see when it changes from 0 to 1 if that's helpful.)

In D49222#1122616, @jhb wrote:

In D49222#1122599, @cperciva wrote:

This doesn't cause any problems in EC2, but it doesn't solve the "recognizing that we're ready for an nvme disk to go away" problem either. They're definitely using a write to the power status register as a cue to remove the device (per https://reviews.freebsd.org/D49146 ).

Humm, I can't imagine what the goal of this behavior is on the part of EC2. Can the engineers there explain why they think the PME is still pending and needs to be cleared? Do they assert a PME as part of the eject notification? (You can read the PME bit to see when it changes from 0 to 1 if that's helpful.)

Yes, the PCIM_PSTAT_PMEENABLE is set to 0 until the eject request arrives at which point it is set to 1.

In D49222#1122856, @cperciva wrote:

Yes, the PCIM_PSTAT_PMEENABLE is set to 0 until the eject request arrives at which point it is set to 1.

Oops, I was misled by some earlier testing. In fact PME# never gets asserted. But we need to turn it off anyway to acknowledge the detach.

Existing device drivers cleared PME# first in their resume methods before messing with device registers, so do the same in pci_resume_child.

The updated series survived multiple sleep/resume cycles on my X1 Carbon including waking due to key press and waking due to opening the lid. I don't have a way to test any of the WOL driver changes however.