It looks we better drop swp_pager_meta_ctl() changes from this, D22324, and let D22409 take over the lookup implementation while dropping SWM_POP for simpler diff.
D22409 change is to optimize swap_pager_haspage() with additional searches after successful pindex lookup.

Dropped self-test.

D22280 has been committed.

Correct swblk index and exclude swp_pager_swblk_free_empty() changes.

I don't think this code duplication in swap_pager_copy() is a good idea.
I updated D22256 to avoid the double lookup cost from the first revision.
I'm still testing the 2nd version though.

Fixed spaces/style.

Created swp_pager_meta_find_least private API to find and delete in a single
operation. swap_pager_find_least calls it for look up.

Suspend for now.

I've been using the comparison based optimization for 2 weeks on current
following daily, buildworld/buildkernel daily and haven't seen an issue
with this approach.

I see. I just added zfs/zvol swap space temporary from command line within the last a couple of weeks for other reasons. I see few of too early OOM on few of machines although it is very rare and spontaneous such that it hasn't been easy to find a trigger.

swap_pager_full != 0 was indeed too aggressive. I'm wondering if we can feed in swap space usage such that we can delay OOM electing at low usage and push for OOM at high usage.

That's another idea.
I will check the code and test if that is also an option.

Thanks for follow up, markj.

Use "== SWPG_LOW" comparison for almost full warning.
Drop swp_gauge > SWPG_LOW comparison for high watermark check.

The only reason I created a patch was to proceed removal of sys/dir.h in the base system.
Given the upstream source hasn't been updated for more than a couple of decade.
It looks the code still assume 20 year old standard.

But sysctl describes totally different:

Skipping is about https://svnweb.freebsd.org/base/head/sys/vm/swap_pager.c?revision=352407&view=markup#l2313

destLen -> destlen.

Thanks.

Remove uninteded debug print.

Also print length with PRIu32.

Include zutil.h instead of zlib.h in order to pick up OS_CODE and DEF_MEM_LEVEL.

We have a last item - sys/kern/subr_compressor.c includes sys/zutil.h.
I attempted and saw crc32() API conflict between gsb_crc32.h and zutil.h.
I haven't got a chance this week, though.

Thank you for explanations.
I found and had looked at that git repo while reviewing this change.
However, I didn't have enough guts to look into each of repo history to figure out which one came from where.
That explains why all of ppp use Z_PACKET_FLUSH while really NONE uses Z_SYNC_FLUSH.

The change looks reasonable based on PPP protocol and comparison to existing zlib code.

I will take a closer look.
I've never used nor looked into details of this API.
Please allow me a bit extra time for review.

I haven't looked into FreeBSD extension of zlib yet to see if this change is safe.

I wasn't aware of swap partition being the the first in a slice was the case. All of my swaps have been either one of whole device, whole slice or non-first partition, or NFS files. No wonder I never encountered any issues over years even with this change on my local tree.

I found and have one more minor comment.

I agree with the changes.

Expose functions from compress.c with modified MY_ZCALLC handling.

Enhancce MY_ZCALLOC to expose functions in compress.c.
This also allows to use default zfree.

Given we added -DZ_SOLO to zlib, pass zfree function explicitly.
Expose compressBound() function even with -DZ_SOLO.
compress2() doesn't work with -DZ_SOLO and not worth exposing as is.
Alternative is improve MY_ZCALLOC conditions along with Z_SOLO for compress2 to work as well.

Given we added -DZ_SOLO to zlib, pass zfree function explicitly.

I tested reads and writes on gzipped ZFS between 12.0-RELEASE and 13-CURRENT with this changes.
The test result was good.

It looks ZFS changes, dropping zlib files in zfs, are melded into this one.
I recall needing to add -DGZIP and/or -DGUNZIP option for zfs compression=gzip to properly uncompress gzipped zfs devices with older version; there was gzip header regression...

D21099 removed gzip'ed a.out support. Rebased and adjusted.

Someone mentioned https://svnweb.freebsd.org/base?view=revision&revision=231885 was trigger in one of the mailing list. Anyway, if that's true, it's been broken for 7 years.

The changes look okay.

D21072, "Remove support for kernel.tramp and kernel.tramp.gz", precedes this revision. This removes obsolete use of legacy zlib version.

Actually, this file rename doesn't fix https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=205822.
I don't think it is worth addressing this bug report here as removing the zlib in zfs also fixes this issue and we are aiming for this serious of changes.

Disable broken gzipped aout executable support and rename zlib in ZFS to avoid filename conflicts.

D20635 picked up the changes.

Adjusted sys/type.h include to FreeBSD section and also moved crypto_z* function implementations to top of file.

Drop NO_GZIP compile option changes in zlib as these aren't necessary at this stage.

I had missed zcalloc.c in sys/modules/Makefile, too.

Missed zcallc.h/c files from last update.

Drop -DMY_ZCALLOC from kmod.mk but put zcfree() back.

Drop -DMY_ZCALLOC as no use under -DZ_SOLO.

Use -DZ_SOLO to avoid standard C header files.
ZFS requires Non-Z_SOLO ZLIB and we will revisit when we work on ZFS.
Combined opencrypto conversion to avoid dtrace probe name changes.

I've been away and haven't been able to follow up or read updates on any of "swapoff" related changes for the last one week.
I won't be able to get back to this for another week; I will find the situations and figure out what to do later.
It is currently looking like Doug is finalizing changes based at quick look of incoming email titles. I appreciate your passion.

Added sys/contrib/zlib/crc32.c to zlib kernel module.

Drop underscore from symbols - changed prefix from _zlib104 to zlib104.

Removed commented code in Makefiles.

crc32 split has been comitted and code rebased.

Xin committed this change with minor adjustments and https://svnweb.freebsd.org/base?view=revision&revision=349151 was the revision.