libuvmem: usermode port of vmem(9)

[kib]

Summary

The quantum cache is disabled, there is no uma.

Intent is to use this for resource allocation in bhyve(8), for start. Addition of -luvmem to bhyve linking was done to test changes to share/mk.

[grehan]

No objection from bhyve's PoV.

[markj]

Seems ok to me, just nitpicking.

sys/kern/subr_vmem.c

608

Commit separately?

812

I would delete the rehash comment and merge the two blocks. All of the functions below can arguably be called "vmem internal functions".

818

This can be a separate commit.

848–849

This can be a separate commit.

1363

Why?

1441

I really dislike conditionally defined top-level local variables, so I'd prefer to move the definition into the if block below. I don't insist on it though.

1574

You could eliminate some inline ifdefs with #define panic(...) assert(0).

1582–1583

Same comment as above wrt local variables.

1598

This can be a separate commit.

1694

This block should probably be formally protected with ifdef _KERNEL as well.

[emaste]

sys/kern/subr_vmem.c

1441

I agree in general, and if this were some special case (e.g. INVARIANTS) instead of the usual case I'd argue strongly for markj's take.

[kib]

sys/kern/subr_vmem.c

1363

To not add __unused declaration, I think it is logically wrong.

1574

There were only two of them so I did not bothered.

[kib]

Rebase after commit of warning fixes.
Handle Mark notes.

[markj]

sys/contrib/openzfs/include/sys/zfs_context.h

53

Is this just a reminder to upstream the change? Or is there another problem?

sys/kern/subr_vmem.c

1363

I think (void)qcache_max; is more common for this scenario.

[kib]

Add M_*FIT bits to sys/vmem.h for userspace.

[bnovkov]

Hi, is there a concrete reason why this revision never landed or has it just slipped through the cracks?

I would really like to use libuvmem for one of my ongoing bhyve projects and would like to see this land, let me know if there a way I could help with that.

[kib]

In D27220#1212528, @bnovkov wrote:

Hi, is there a concrete reason why this revision never landed or has it just slipped through the cracks?

I would really like to use libuvmem for one of my ongoing bhyve projects and would like to see this land, let me know if there a way I could help with that.

The intent of the libuvmem port was to use vmem allocator for bhyve to handle physical address space. That stalled, and I do not see a reason to add unused library to base.
If you need it, I am more than happy to push the commit.

Meantime I rebased the branch.

[kib]

Rebase. Use FreeBSD 16.0 symbol version for the new lib.

[kib]

Remove undefined symbols from the version script. The library should be buildable, at least.

[kib]

Use standard way to reference libc src dir.
Silence compiler warning in _bt_fill() about flags.

[kib]

sys/contrib/openzfs/include/sys/zfs_context.h

53

AFAIR it did not work, there was some conflict because solaris shims also provided sys/vmem.h.

[markj]

sys/kern/subr_vmem.c

1425

Maybe add macros for these list locks, like we have for other locks?

[kib]

VMEM_LIST_LOCK()

[bnovkov]

In D27220#1212532, @kib wrote:

In D27220#1212528, @bnovkov wrote:

Hi, is there a concrete reason why this revision never landed or has it just slipped through the cracks?

I would really like to use libuvmem for one of my ongoing bhyve projects and would like to see this land, let me know if there a way I could help with that.

The intent of the libuvmem port was to use vmem allocator for bhyve to handle physical address space. That stalled, and I do not see a reason to add unused library to base.

Is there a revision with the stalled patches somewhere? I'd be interested in taking a look.

If you need it, I am more than happy to push the commit.

That would be amazing, thank you!
It would really help with managing PCI BAR allocations for my ongoing device hotplugging work.

[kib]

In D27220#1212853, @bnovkov wrote:

In D27220#1212532, @kib wrote:

In D27220#1212528, @bnovkov wrote:

Hi, is there a concrete reason why this revision never landed or has it just slipped through the cracks?

I would really like to use libuvmem for one of my ongoing bhyve projects and would like to see this land, let me know if there a way I could help with that.

The intent of the libuvmem port was to use vmem allocator for bhyve to handle physical address space. That stalled, and I do not see a reason to add unused library to base.

Is there a revision with the stalled patches somewhere? I'd be interested in taking a look.

I suspect none survived, since I only found this libuvmem diff itself on that branch.

If you need it, I am more than happy to push the commit.

That would be amazing, thank you!
It would really help with managing PCI BAR allocations for my ongoing device hotplugging work.

Yes this is exactly the place where libuvmem should fit perfectly.

[markj]

Looks ok to me. Perhaps this should be a private library instead?

lib/libuvmem/Makefile

2

Extra newline.

8

[kib]

In D27220#1213262, @markj wrote:

Looks ok to me. Perhaps this should be a private library instead?

IMO it is generally useful API. Also, I do provide symbol versioning, which does not make sense for a private library: the intent is to keep stable interfaces.

[kib]

Makefile formatting

[markj]

In D27220#1213306, @kib wrote:

In D27220#1213262, @markj wrote:

Looks ok to me. Perhaps this should be a private library instead?

IMO it is generally useful API. Also, I do provide symbol versioning, which does not make sense for a private library: the intent is to keep stable interfaces.

Well, now the kernel's internal interface must be kept stable as well. I don't object, but I'm worried it'll be easy for someone modifying vmem to forget about this.

If the library is public, it should get a man page IMO, even if it just points at vmem(9).

[kib]

In D27220#1213316, @markj wrote:

In D27220#1213306, @kib wrote:

In D27220#1213262, @markj wrote:

Looks ok to me. Perhaps this should be a private library instead?

IMO it is generally useful API. Also, I do provide symbol versioning, which does not make sense for a private library: the intent is to keep stable interfaces.

Well, now the kernel's internal interface must be kept stable as well. I don't object, but I'm worried it'll be easy for someone modifying vmem to forget about this.

Yes, I noted the same when parts of msdosfs were used in makefs (AFAIR). But there it is much simpler interface, and if the interface for kernel or userspace starts drifting, the code can be easily forked to the residual part.

[kib]

libuvmem(3)

[markj]

In D27220#1213582, @kib wrote:

In D27220#1213316, @markj wrote:

In D27220#1213306, @kib wrote:

In D27220#1213262, @markj wrote:

Looks ok to me. Perhaps this should be a private library instead?

IMO it is generally useful API. Also, I do provide symbol versioning, which does not make sense for a private library: the intent is to keep stable interfaces.

Well, now the kernel's internal interface must be kept stable as well. I don't object, but I'm worried it'll be easy for someone modifying vmem to forget about this.

Yes, I noted the same when parts of msdosfs were used in makefs (AFAIR). But there it is much simpler interface, and if the interface for kernel or userspace starts drifting, the code can be easily forked to the residual part.

My point is that no automation (e.g., tests) will catch this drift until after it is a problem for some hypothetical users. If makefs -t msdos is broken somehow, we will notice fairly quickly.

If Bojan's work lands in bhyve, maybe that will be enough.

lib/libuvmem/libuvmem.3

48

64

You might note that the allocator is thread-safe.

[bnovkov]

One minor thing I've noticed is that we don't export M_NOWAIT.
I think it would be very useful to have a way of controlling whether we wait or exit on failure, but I'm not sure if accomplishing that is as straightforward as defining M_NOWAIT in sys/vmem.h.

sys/kern/subr_vmem.c

1354–1355

I hit an error caused by this pretty early on.

Because the assertion above enforces that the quantum should always be 0, the resulting quantum mask will be set to -1 (all F's) which will cause vmem_roundup_size to always return 0.
This locks vmem_xalloc into either an infinite loop or causes it to block indefinitely.

[kib]

In D27220#1213773, @bnovkov wrote:

One minor thing I've noticed is that we don't export M_NOWAIT.
I think it would be very useful to have a way of controlling whether we wait or exit on failure, but I'm not sure if accomplishing that is as straightforward as defining M_NOWAIT in sys/vmem.h.

It should be, at least lets try it that way. I added M_NOWAIT into !_KERNEL block of sys/vmem.h. Also I inspected subr_vmem.c and M_WAITOK seems to be only used by the kernel blocks.

sys/kern/subr_vmem.c

1354–1355

Thanks. I handled it differently, forcing quantum = 1 in the initial ifdef block in vmem_init().

[kib]

Set quantum to 1.
Allow M_NOWAIT
List flags for vmem_xalloc() in man page.
Mention thread-safety in man page.
Remove unused VMEM_TRYLOCK()