More re-working of multipage mbufs.

[glebius]

Summary

This is a straight continuation of D24213.

Now that array of pages is embedded into mbuf, we no longer need
separate structure to pass around, so struct mbuf_ext_pgs is an
artifact of the first implementation. And struct mbuf_ext_pgs_data
is a crutch to accomodate the main idea D24213 with minimal churn.
This change embeds into mbuf and anonymizes both of them.

Also, M_EXT of type EXT_PGS became just a synonym of M_NOMAP, so
clear that.

Also, bring all multi-page mbuf stuff together into a consistent
namespace. Now, a multipage mbuf has M_EXTPG flag on it, and that
means that all struct members that have name starting with m_epg are
enabled. Functions and macros that work with M_EXTPG also renamed into
similar namespace.

While working on this removed some redundant fields and assignments.

We probably can pack one more page into mbuf, but this is out of scope
of this patch.

To be committed in several steps, that can be reviewed in git:

https://github.com/glebius/FreeBSD/commits/m_epg

P.S. A logical continuation of this would be anonymization of
struct m_ext, sorting out fields that are shared with M_EXT and M_EXTPG,
and those that are not, sprinkling better comments and assertions. TBD.

[• hselasky]

mlx5en changes are OK.

[bz]

sys/kern/kern_mbuf.c

855 ↗

(On Diff #71073)

That once was like 220 bytes on the stack? Is that a problem these days?

[gallatin]

sys/kern/kern_mbuf.c

855 ↗

(On Diff #71073)

Its using less space than before.. (256b for an entire mbuf) and there is precedent for other code in the kernel putting mbufs on the stack. The smallest default stacks seem to be 2 4K pages on 32 bit arm/ppc, so even on those platforms, 220b is less than 3% of available stack space.

[glebius]

sys/kern/kern_mbuf.c

855 ↗

(On Diff #71073)

It is okay to put an mbuf on stack. There are/were other places like that. Note that this patch even reduces the stack usage from full mbuf to MHLEN.