I have to prepare for my lecture. I'll think about this some more later.

Before talking about implementation, do we agree that the following describes the desired behavior for COW faults?

Suppose that the page was active. I think it's okay to deactivate the page, as we have previously done. If the page is actually mapped (read-only) and accessed by another process, the page daemon will reactivate it, because the object's ref count will be non-zero and pmap_ts_referenced() will return a non-zero value. In fact, with the exception of pmap_remove_pages(), because we set PGA_REFERENCED whenever we destroy a mapping with the accessed bit set in the PTE, the page will be reactivated even if it is not still mapped by this other process at the instant that the page daemon examines the page.

Suppose that the page was inactive. I think that requeueing the page suffices, and this requeueing can be deferred to the page daemon. As described in the active case, the page daemon may actually discover that the page is actually mapped and accessed by another process, and so it may actually reactivate the page.

In D14625#307878, @alc wrote:

Before talking about implementation, do we agree that the following describes the desired behavior for COW faults?

Yep, I agree that the behaviour you described seems reasonable.

In D14625#307922, @markj wrote:

In D14625#307878, @alc wrote:

Before talking about implementation, do we agree that the following describes the desired behavior for COW faults?

Yep, I agree that the behaviour you described seems reasonable.

Then, I'm going to retract my original support for the PGA_REFERENCED-based solution, because that will lead to page reactivation when we want the page to be requeued in the inactive queue. Do you have a plan for merging r330296 into HEAD?

In D14625#308095, @alc wrote:

Then, I'm going to retract my original support for the PGA_REFERENCED-based solution, because that will lead to page reactivation when we want the page to be requeued in the inactive queue. Do you have a plan for merging r330296 into HEAD?

Not quite yet. I still have some cleanup to do, and the change needs review and testing, assuming of course that we agree with the direction taken there. I'm not sure if you've looked much at r330296 but it's a fairly drastic change. I can describe it over the phone at some point if that would be more convenient.

In any case, it seems we want a variation of vm_page_deactivate() that requeues the page even if it's already in PQ_INACTIVE (and enqueues it there otherwise). As part of r330296 I'd like to make sure there is a clean separation between the higher-level page queue APIs (e.g., vm_page_activate(), vm_page_deactivate()) and the lower-level queue manipulation APIs (e.g., vm_page_requeue()). In particular, use of the latter should be confined to vm_page.c and vm_pageout.c. I don't like the

else if (m->queue != PQ_INACTIVE)
    vm_page_deactivate(m);
else
    vm_page_requeue(m);

in vfs_vmio_unwire(), for example.

Even before r330296, having a vm_page_deactivate_or_requeue() for use in release_page() would be an improvement since we'd only have to touch the page and page queue locks once instead of twice. With r330296, vm_page_deactivate_or_requeue() could be modified to use a lazy requeue, or another API could be added so that the caller can choose between strict LRU and second-chance. Does that sound like a reasonable interim approach? If so, any suggestions for a better name than vm_page_deactivate_or_requeue()? :)

In D14625#308114, @markj wrote:

... I don't like the

else if (m->queue != PQ_INACTIVE)
    vm_page_deactivate(m);
else
    vm_page_requeue(m);

in vfs_vmio_unwire(), for example.

Even before r330296, having a vm_page_deactivate_or_requeue() for use in release_page() would be an improvement since we'd only have to touch the page and page queue locks once instead of twice. With r330296, vm_page_deactivate_or_requeue() could be modified to use a lazy requeue, or another API could be added so that the caller can choose between strict LRU and second-chance. Does that sound like a reasonable interim approach? If so, any suggestions for a better name than vm_page_deactivate_or_requeue()? :)

Yes. No.

That said, there aren't that many calls to vm_page_deactivate() in the tree, and I suspect that none of them would be adversely affected by changing vm_page_deactivate() to requeue when the page is already inactive.

In D14625#308444, @alc wrote:

That said, there aren't that many calls to vm_page_deactivate() in the tree, and I suspect that none of them would be adversely affected by changing vm_page_deactivate() to requeue when the page is already inactive.

I went with this approach. Now we preserve LRU without using PGA_REFERENCED.