pmap_l0(pmap, sva) -> l0

I'm confused. l1 can't have ATTR_DESCR_VALID set, but the pmap's resident count isn't being incremented here (like it is below in the else).

Can't the l0 entry be invalid? In which case, l1p will be NULL.

See my earlier comment.

Assert that the l1 entry has the "wired" bit set, like we do below?

You're right, I'm not sure why I didn't just follow what the amd64 implementation does.

Hmm. This is wrong for managed mappings. We should be testing whether the page is supposed to be writeable, not whether we've dirtied it.

pmap_unuse_pt(pmap, sva, pmap_load(l0), &free);?

Suppose that you are replacing an existing large page mapping. Isn't there still a problem? Because you'll increment the resident count, but you shouldn't.

Right. We should be checking (tpte & ATTR_SW_DBM) != 0 I believe.

Does this bug impact correctness? If we fail here because the logically writable mapping is clean, then vm_fault_quick_hold_pages()'s fallback logic will invoke the fault handler and locate the page.

I'm not sure about stage2 translations yet.

Don't you need to do some kind of dsb() after store, at least for the new mapping ?

Yes.

I don't understand why we do this unconditionally. We don't use mp except in the then clause of the below if statement.

Ditto.

See my amd64 comment.

1G -> 2M

L1_SIZE -> pagesizes[psind]

L1_SIZE -> pagesizes[psind]