libuvmem: usermode port of vmem(9)

[kib]

Summary

The quantum cache is disabled, there is no uma.

Intent is to use this for resource allocation in bhyve(8), for start. Addition of -luvmem to bhyve linking was done to test changes to share/mk.

[markj]

sys/kern/subr_vmem.c

1467

Same comment as above wrt local variables.

1573

This block should probably be formally protected with ifdef _KERNEL as well.

[emaste]

sys/kern/subr_vmem.c

1329

I agree in general, and if this were some special case (e.g. INVARIANTS) instead of the usual case I'd argue strongly for markj's take.

[kib]

sys/kern/subr_vmem.c

1258

To not add __unused declaration, I think it is logically wrong.

1458

There were only two of them so I did not bothered.

[kib]

Rebase after commit of warning fixes.
Handle Mark notes.

[markj]

sys/contrib/openzfs/include/sys/zfs_context.h

53 ↗

(On Diff #79634)

Is this just a reminder to upstream the change? Or is there another problem?

sys/kern/subr_vmem.c

1258

I think (void)qcache_max; is more common for this scenario.

[kib]

Add M_*FIT bits to sys/vmem.h for userspace.

[bnovkov]

Hi, is there a concrete reason why this revision never landed or has it just slipped through the cracks?

I would really like to use libuvmem for one of my ongoing bhyve projects and would like to see this land, let me know if there a way I could help with that.

[kib]

In D27220#1212528, @bnovkov wrote:

Hi, is there a concrete reason why this revision never landed or has it just slipped through the cracks?

I would really like to use libuvmem for one of my ongoing bhyve projects and would like to see this land, let me know if there a way I could help with that.

The intent of the libuvmem port was to use vmem allocator for bhyve to handle physical address space. That stalled, and I do not see a reason to add unused library to base.
If you need it, I am more than happy to push the commit.

Meantime I rebased the branch.

[kib]

Rebase. Use FreeBSD 16.0 symbol version for the new lib.

[kib]

Remove undefined symbols from the version script. The library should be buildable, at least.

[kib]

Use standard way to reference libc src dir.
Silence compiler warning in _bt_fill() about flags.

[kib]

sys/contrib/openzfs/include/sys/zfs_context.h

53 ↗

(On Diff #79634)

AFAIR it did not work, there was some conflict because solaris shims also provided sys/vmem.h.

[markj]

sys/kern/subr_vmem.c

1312

Maybe add macros for these list locks, like we have for other locks?

[kib]

VMEM_LIST_LOCK()

[bnovkov]

In D27220#1212532, @kib wrote:

In D27220#1212528, @bnovkov wrote:

Hi, is there a concrete reason why this revision never landed or has it just slipped through the cracks?

I would really like to use libuvmem for one of my ongoing bhyve projects and would like to see this land, let me know if there a way I could help with that.

The intent of the libuvmem port was to use vmem allocator for bhyve to handle physical address space. That stalled, and I do not see a reason to add unused library to base.

Is there a revision with the stalled patches somewhere? I'd be interested in taking a look.

If you need it, I am more than happy to push the commit.

That would be amazing, thank you!
It would really help with managing PCI BAR allocations for my ongoing device hotplugging work.

[kib]

In D27220#1212853, @bnovkov wrote:

In D27220#1212532, @kib wrote:

In D27220#1212528, @bnovkov wrote:

Hi, is there a concrete reason why this revision never landed or has it just slipped through the cracks?

I would really like to use libuvmem for one of my ongoing bhyve projects and would like to see this land, let me know if there a way I could help with that.

The intent of the libuvmem port was to use vmem allocator for bhyve to handle physical address space. That stalled, and I do not see a reason to add unused library to base.

Is there a revision with the stalled patches somewhere? I'd be interested in taking a look.

I suspect none survived, since I only found this libuvmem diff itself on that branch.

If you need it, I am more than happy to push the commit.

That would be amazing, thank you!
It would really help with managing PCI BAR allocations for my ongoing device hotplugging work.

Yes this is exactly the place where libuvmem should fit perfectly.

[markj]

Looks ok to me. Perhaps this should be a private library instead?

lib/libuvmem/Makefile

1 ↗

(On Diff #164155)

Extra newline.

7 ↗

(On Diff #164155)

[kib]

In D27220#1213262, @markj wrote:

Looks ok to me. Perhaps this should be a private library instead?

IMO it is generally useful API. Also, I do provide symbol versioning, which does not make sense for a private library: the intent is to keep stable interfaces.

[kib]

Makefile formatting

[markj]

In D27220#1213306, @kib wrote:

In D27220#1213262, @markj wrote:

Looks ok to me. Perhaps this should be a private library instead?

IMO it is generally useful API. Also, I do provide symbol versioning, which does not make sense for a private library: the intent is to keep stable interfaces.

Well, now the kernel's internal interface must be kept stable as well. I don't object, but I'm worried it'll be easy for someone modifying vmem to forget about this.

If the library is public, it should get a man page IMO, even if it just points at vmem(9).

[kib]

In D27220#1213316, @markj wrote:

In D27220#1213306, @kib wrote:

In D27220#1213262, @markj wrote:

Looks ok to me. Perhaps this should be a private library instead?

IMO it is generally useful API. Also, I do provide symbol versioning, which does not make sense for a private library: the intent is to keep stable interfaces.

Well, now the kernel's internal interface must be kept stable as well. I don't object, but I'm worried it'll be easy for someone modifying vmem to forget about this.

Yes, I noted the same when parts of msdosfs were used in makefs (AFAIR). But there it is much simpler interface, and if the interface for kernel or userspace starts drifting, the code can be easily forked to the residual part.

[kib]

libuvmem(3)

[markj]

In D27220#1213582, @kib wrote:

In D27220#1213316, @markj wrote:

In D27220#1213306, @kib wrote:

In D27220#1213262, @markj wrote:

Looks ok to me. Perhaps this should be a private library instead?

IMO it is generally useful API. Also, I do provide symbol versioning, which does not make sense for a private library: the intent is to keep stable interfaces.

Well, now the kernel's internal interface must be kept stable as well. I don't object, but I'm worried it'll be easy for someone modifying vmem to forget about this.

Yes, I noted the same when parts of msdosfs were used in makefs (AFAIR). But there it is much simpler interface, and if the interface for kernel or userspace starts drifting, the code can be easily forked to the residual part.

My point is that no automation (e.g., tests) will catch this drift until after it is a problem for some hypothetical users. If makefs -t msdos is broken somehow, we will notice fairly quickly.

If Bojan's work lands in bhyve, maybe that will be enough.

lib/libuvmem/libuvmem.3

47 ↗

(On Diff #164268)

63 ↗

(On Diff #164268)

You might note that the allocator is thread-safe.

[bnovkov]

One minor thing I've noticed is that we don't export M_NOWAIT.
I think it would be very useful to have a way of controlling whether we wait or exit on failure, but I'm not sure if accomplishing that is as straightforward as defining M_NOWAIT in sys/vmem.h.

sys/kern/subr_vmem.c

1250–1251

I hit an error caused by this pretty early on.

Because the assertion above enforces that the quantum should always be 0, the resulting quantum mask will be set to -1 (all F's) which will cause vmem_roundup_size to always return 0.
This locks vmem_xalloc into either an infinite loop or causes it to block indefinitely.

[kib]

In D27220#1213773, @bnovkov wrote:

One minor thing I've noticed is that we don't export M_NOWAIT.
I think it would be very useful to have a way of controlling whether we wait or exit on failure, but I'm not sure if accomplishing that is as straightforward as defining M_NOWAIT in sys/vmem.h.

It should be, at least lets try it that way. I added M_NOWAIT into !_KERNEL block of sys/vmem.h. Also I inspected subr_vmem.c and M_WAITOK seems to be only used by the kernel blocks.

sys/kern/subr_vmem.c

1250–1251

Thanks. I handled it differently, forcing quantum = 1 in the initial ifdef block in vmem_init().

[kib]

Set quantum to 1.
Allow M_NOWAIT
List flags for vmem_xalloc() in man page.
Mention thread-safety in man page.
Remove unused VMEM_TRYLOCK()

[kib]

Any progress with testing?

[ziaee]

lib/libuvmem/libuvmem.3

9 ↗

(On Diff #164354)

You can omit this license text now if you want.

35 ↗

(On Diff #164354)

This doesn't need to be quoted.

36 ↗

(On Diff #164354)

The library section is deprecated, current advice is to put the Lb macro line immediately after the Sh SYNOPSIS line.

[kib]

lib/libuvmem/libuvmem.3

36 ↗

(On Diff #164354)

The result of this looks extremely weird. Where this advise came from?

[ziaee]

lib/libuvmem/libuvmem.3

36 ↗

(On Diff #164354)

It comes from the mdoc(7) manual [^0]. I imported this change from upstream mdoc.7 version 1.298 [^1] which says:

LIBRARY
	     The name of the library containing	the documented functions.  Us-
	     ing this section is no longer recommended.	 If any	 Lb  macro  is
	     needed, put it at the beginning of	the SYNOPSIS section instead.

It seems to have first been proposed on the groff mailing list [^2].

I tried to notify everyone with a mail to freebsd-current@ [^3] in June when when I imported the change in commit 4c07abdbacf4 [^4], but nobody said anything.

[^0]: https://man.freebsd.org/cgi/man.cgi?query=mdoc&apropos=0&sektion=0&manpath=FreeBSD+16.0-CURRENT&arch=default&format=html
[^1]: https://cvsweb.bsd.lv/mandoc/mdoc.7?rev=1.298&content-type=text/x-cvsweb-markup&sortby=date
[^2]: https://lists.gnu.org/archive/html/groff/2025-06/msg00004.html
[^3]: https://lists.freebsd.org/archives/freebsd-current/2025-June/007907.html
[^4]: https://cgit.freebsd.org/src/log/?qt=range&q=4c07abdbacf4&showmsg=1

[kib]

lib/libuvmem/libuvmem.3

36 ↗

(On Diff #164354)

Not that I like it.

[kib]

Some tweaks to the man page

[bnovkov]

In D27220#1233531, @kib wrote:

Any progress with testing?

Yes, I have reworked bhyve's PCI BAR allocator to use libuvmem and it works flawlessly.
I did not run into any vmem-related issues over the last couple of weeks while developing and testing my PCI hotplugging patches, so as far as I'm concerned this revision is good to go.