Page MenuHomeFreeBSD

add virtiofs kernel module
AcceptedPublic

Authored by emil_etsalapatis.com on Aug 14 2024, 3:55 PM.
Tags
None
Referenced Files
Unknown Object (File)
Sun, Dec 1, 11:39 PM
Unknown Object (File)
Fri, Nov 29, 6:17 AM
Unknown Object (File)
Sat, Nov 23, 10:38 AM
Unknown Object (File)
Thu, Nov 21, 8:42 AM
Unknown Object (File)
Mon, Nov 11, 1:38 AM
Unknown Object (File)
Nov 7 2024, 6:28 PM
Unknown Object (File)
Oct 17 2024, 7:14 AM
Unknown Object (File)
Oct 16 2024, 10:49 AM

Details

Summary

Add a virtiofs file system based on FUSE to interface with the virtio fs device. The file system communicates to a FUSE server on the host through this device.

Diff Detail

Repository
rG FreeBSD src repository
Lint
Lint Passed
Unit
No Test Coverage
Build Status
Buildable 59022
Build 55909: arc lint + arc unit

Event Timeline

I'm still a little shaky on the details of virtio-fs, but I can't find anything obviously wrong here.

sys/fs/fuse/virtiofs_vfsops.c
5

I think we're not supposed to use "All rights reserved" in new files.

This revision is now accepted and ready to land.Aug 21 2024, 9:01 PM

Thanks a lot for working on this!

sys/fs/fuse/virtiofs_vfsops.c
419

Can we use the "from" for the tag like p9fs does? It's a bit strange to have an unused arbitrary string as the "from", and I haven't managed to get root mount to work even though it seemingly is passing the options…

sys/fs/fuse/virtiofs_vfsops.c
393

The "strict access policy" is only breaking things for virtiofs, i.e. if you mount a home directory for a user, that user wouldn't be able to access it.

sys/fs/fuse/virtiofs_vfsops.c
419

ah, re: mountroot, virtiofs needs to be added to vfs_mountroot_wait_if_neccessary (ugh why don't we have proper registered flags for this)… And then, remount support is expected because it mounts as RO first.

sys/fs/fuse/virtiofs_vfsops.c
419

Actually that whole routine is bogus, as is waiting for the dev under the mountpoint. I've written it locally, but it was on a laptop that decided to die suddenly for reasons I've not been able to diagnose.

sys/fs/fuse/virtiofs_vfsops.c
375

You also have to *set* the size here with fiov_adjust like the local FUSE transport does when it copies data from userspace in fticket_aw_pull_uio right after fuse_body_audit.

Took me a few hours to debug why getdirentries was completely busted (: It relies on the buffer size to stop iteration; without fiov_adjust the buffer size was always 4096 so it would iterate past the actual message size, hitting a wall of zeroes, interpreting that as an entry with a 0-length name and bailing with EINVAL.

With that fixed, virtiofs is already looking a lot better than p9fs. Namely, I could install a package with pkg under a virtiofs root, while under p9fs I'm hitting rather strange errors. Reusing the FUSE protocol was a really clever idea!

Thanks for the feedback! I will update the diff accordingly.

sys/fs/fuse/virtiofs_vfsops.c
375

Great catch! I think this also fixes a bug I had been trying to triage for write back mode when heavily loading the VM. I will update and fix.

419

Sounds good, just to make sure I've gotten it right: Changing the VFS option from "tag" to "from" should bring virtiofs mount options in line w/ other file systems.

419

Re: remount, I can add it to this diff if it helps with testing. I omitted it to keep this diff set as small as possible, because it is already pretty large.

sys/fs/fuse/virtiofs_vfsops.c
124

This just causes all buffers to leak.

A couple more notes from testing:

There's some other place where FUSE layer can pick up zero-padded-to-power-of-two things, now with 1024 instead of 4096:

WARNING: FUSE protocol violation for server mounted at /: Returned an embedded NUL from FUSE_READLINK.  This warning will not be repeated.
Len 1024
0000   62 61 73 68 00 00 00 00 00 00 00 00 00 00 00 00  |bash............|
0010   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|
…

And trying to just find something in a src tree checkout, there are sporadic errors like

find: ./contrib/llvm-project/llvm/lib/DebugInfo/Symbolize: Bad file descriptor
find: ./sys/dev/qat/qat_hw: Invalid argument
find: ./sys/dev/qat/qat_api/common/stubs: Invalid argument
// etc…

that seem completely random, on one invocation there are none, on another there's a couple…

sys/fs/fuse/virtiofs_vfsops.c
419

yes, the "from" is the 'magic' one that's the source argument in the mount command that you have to pass anyway

472

"virtiofs.virtio" (the last fs gets cut off) is kinda redundant, so this isn't necessary

A couple more notes from testing:

There's some other place where FUSE layer can pick up zero-padded-to-power-of-two things, now with 1024 instead of 4096:

WARNING: FUSE protocol violation for server mounted at /: Returned an embedded NUL from FUSE_READLINK.  This warning will not be repeated.
Len 1024
0000   62 61 73 68 00 00 00 00 00 00 00 00 00 00 00 00  |bash............|
0010   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|
…

And trying to just find something in a src tree checkout, there are sporadic errors like

find: ./contrib/llvm-project/llvm/lib/DebugInfo/Symbolize: Bad file descriptor
find: ./sys/dev/qat/qat_hw: Invalid argument
find: ./sys/dev/qat/qat_api/common/stubs: Invalid argument
// etc…

that seem completely random, on one invocation there are none, on another there's a couple…

Thank you for the testing! Are you using writethrough or writeback mode for FUSE? Also, what command line arguments are you using for virtiofsd on the host side?

A couple more notes from testing:

There's some other place where FUSE layer can pick up zero-padded-to-power-of-two things, now with 1024 instead of 4096:

WARNING: FUSE protocol violation for server mounted at /: Returned an embedded NUL from FUSE_READLINK.  This warning will not be repeated.
Len 1024
0000   62 61 73 68 00 00 00 00 00 00 00 00 00 00 00 00  |bash............|
0010   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  |................|
…

And trying to just find something in a src tree checkout, there are sporadic errors like

find: ./contrib/llvm-project/llvm/lib/DebugInfo/Symbolize: Bad file descriptor
find: ./sys/dev/qat/qat_hw: Invalid argument
find: ./sys/dev/qat/qat_api/common/stubs: Invalid argument
// etc…

that seem completely random, on one invocation there are none, on another there's a couple…

Thank you for the testing! Are you using writethrough or writeback mode for FUSE? Also, what command line arguments are you using for virtiofsd on the host side?

Never adjusted any parameters on the client side. Adding --writeback to virtiofsd results in panic: FUSE: ticket still on answer delivery list.
I was using --inode-file-handles=prefer --allow-direct-io --allow-mmap mostly. But changing flags doesn't make the random errors go away.

(not surprising as write caching stuff shouldn't affect running find a bunch of times)