Add OBJ_PG_DTOR flag to VM object

[br]

Summary

Add OBJ_PG_DTOR flag to object allowing us to skip pages removal from object in vm_object_terminate() and to leave that for cdev_pg_dtor function.

This is required for Intel SGX support as we want to lookup pages in queue before removal.

[br]

move out pages removal code to separate function to avoid code duplication

[br]

make function static

[alc]

I have no objections to this change. I'm just interested in hearing a little more about why it is needed. In your SGX driver are you essentially maintaining a private free page list for managing the pool of physical memory that backs enclaves?

[br]

In D11688#249549, @alc wrote:

I have no objections to this change. I'm just interested in hearing a little more about why it is needed. In your SGX driver are you essentially maintaining a private free page list for managing the pool of physical memory that backs enclaves?

Exactly. We have enclave's page cache (EPC) memory pool reserved by processor and we register it as fictitious range of physical memory. That EPC pool is encrypted and backs enclave's control information, enclave's code, stack, data, etc.
https://reviews.freebsd.org/D11113

[alc]

In D11688#249553, @br wrote:

In D11688#249549, @alc wrote:

I have no objections to this change. I'm just interested in hearing a little more about why it is needed. In your SGX driver are you essentially maintaining a private free page list for managing the pool of physical memory that backs enclaves?

Exactly. We have enclave's page cache (EPC) memory pool reserved by processor and we register it as fictitious range of physical memory. That EPC pool is encrypted and backs enclave's control information, enclave's code, stack, data, etc.
https://reviews.freebsd.org/D11113

Ok, that helps. Because you are using OBJT_MGTDEVICE in the SGX driver, there is no list of fake pages, i.e., we don't maintain un_pager.devp.devp_pglist for that type of object.