Gasket is Google's name for the layer their accelerator drivers share:
Google ASIC Software, Kernel Extensions, and Tools. The chips differ,
but they present the same shape to the host. A control register BAR
with a few windows user space may map, an on-chip page table the host
fills in so the device can reach host memory, and a block of MSI-X
vectors.
Details
amd64 only for now, and host mappings are refused under KMSAN.
only the m2 cards were tested.
Diff Detail
- Repository
- rG FreeBSD src repository
- Lint
Lint Passed - Unit
No Test Coverage - Build Status
Buildable 75309 Build 72192: arc lint + arc unit
Event Timeline
Interesting! A lot of this is mmap / dmabuf style stuff. I don't want to block landing this on waiting for freebsd native dmabuf, but i wonder if this would be another user of native freebsd dmabuf at some point.
| share/man/man4/gasket.4 | ||
|---|---|---|
| 53 | heh update email address | |
| sys/dev/gasket/gasket.h | ||
| 94 | do we need bus_* barriers anywhere? this may be mostly fine on x86 but eg arm/riscv it may need explicit barriers? | |
| sys/dev/gasket/gasket_core.c | ||
| 736 | so are all of these devices fine with 64 bit DMA? | |
| 871 | This means the consumer needs to do two mmaps, right? One for data and one for control/registers, so the correct memory attributes are used? | |
| 1009 | I think this function in particular deserves a doxygen style comment on what it does and entry/exit values. | |
| sys/dev/gasket/gasket_if.m | ||
| 56 | I think all of the methods here deserve documenting what they do, even if it is brief. | |
ok, i think this is fine. I think it'd be a good starting point to figure out what to do about page ownership between devices and userland (the same kind of issue we have with GPU, netmap, etc.) but I don't know if we want to tackle this here.
@markj and @kib - would you mind eyeballing the VM code bits here? See how they look and what kind of design we may want to draft up for things like this to eventually leverage?
Maybe dma_buf kind of stuff?
Accepted provided feedback is addressed
| share/man/man4/gasket.4 | ||
|---|---|---|
| 8 | Am I correct in understanding that this only works on amd64 for now? If so, please specify in the manpage. There is an optional architectural specifier here where you can do that. | |
| share/man/man4/gasket.4 | ||
|---|---|---|
| 8 | It tested in amd64 only . But the framework should work with any architecture . | |
| share/man/man4/gasket.4 | ||
|---|---|---|
| 34 | Why aren't they charged against those limits? The backing pages are effectively mlocked. | |
| sys/dev/gasket/gasket_chardev.c | ||
| 188 ↗ | (On Diff #189675) | Why do you do the validation separately from handling the actual ioctls? It is harder to read that way. |
| sys/dev/gasket/gasket_coherent.c | ||
| 29 ↗ | (On Diff #189675) | This loop runs exactly once, I don't think that's the intent. |
| sys/dev/gasket/gasket_mmap.c | ||
| 266 ↗ | (On Diff #189675) | As I said below I think this code shouldn't be messing with the vm_map at all. |
| 295 ↗ | (On Diff #189675) | This should be a list of allowed object types (or better, some characterization of the object properties that we want), not a list that needs to be updated each time a new object type is added. |
| 373 ↗ | (On Diff #189675) | Suppose a process maps some of its pages into device page tables. Then it forks, and the host mapping becomes read-only in order to implement COW semantics. Now the device has a writable mapping to read-only memory. What should happen in this case? |
| 396 ↗ | (On Diff #189675) | You already faulted the pages above, why look them up again? The last parameter to vm_fault() can be used to return the faulted page. Or use vm_fault_quick_hold_pages(), which is probably better here. That way, gasket_classify_range_locked() doesn't need to deal with vm_map entries at all. You can just look at the returned pages to see which type of VM object you're dealing with. |
| 401 ↗ | (On Diff #189675) | What does this mean? Why is it sufficient to check for only those two flags? |
| 411 ↗ | (On Diff #189675) | Isn't this handled by vm_fault()? |
| sys/conf/files.amd64 | ||
|---|---|---|
| 117 | This looks incomplete? | |
| sys/dev/gasket/gasket_chardev.c | ||
| 43 ↗ | (On Diff #190286) | Suppose a writer opens this device. Then it can pass the resulting fd to another process, e.g., via fork() or a SCM_RIGHTS message, and you end up with multiple processes that have write access, so gs_owner isn't well-defined. And the owner can exit, in which case you're left with a dangling pointer here. Is there some desire to support sharing the device among multiple processes? If not, then you should probably set D_NONPASSABLE in the cdevsw flags. If so, then we should maintain per-fd state using devfs_set_cdevpriv(). That is probably cleaner than trying to count opens, keep track of an owner, etc.. |
| sys/dev/gasket/gasket_core.c | ||
| 129 | Is there some specific reason this needs to be 0660 instead of 0600? If not I'd prefer to use the latter. | |
| sys/dev/gasket/gasket_dma.c | ||
| 168 ↗ | (On Diff #190286) | Why do you need to busy the page? vm_page_dirty() requires the VM object lock. |
| sys/dev/gasket/gasket_mmap.c | ||
| 49 ↗ | (On Diff #190286) | This comment looks stale, the cdev implements d_mmap_single, not d_mmap. |
| 60 ↗ | (On Diff #190286) | Does gd_coherent_base get set anywhere? |
| 248 ↗ | (On Diff #190286) | It took me a long time to understand what this is actually saying. I think it goes like this: vm_fault_quick_hold_pages() doesn't work as you want when applied to the coherent region: if the pages are mapped into user PTs, then pmap_extract_and_hold() will return the real vm_page structure, which belongs to kernel_object, since PHYS_TO_VM_PAGE() checks the vm_page_array first. But if we end up going into the fault handler, then we'll get back fake pages.[1] So you are working around this inconsistency by looking at the vm_map. But that's racy. Can't you instead look at the physical address of the pages returned by vm_fault_quick_hold_pages() to know whether they belong to the coherent region? [1] This is surprising and I bet it causes problems elsewhere. Possibly pmap_extract_and_hold() should be returning fake pages here... |
| 342 ↗ | (On Diff #190286) | I traced how this prot is used elsewhere. It looks like the device mappings of these pages just ignore these protection bit, so how do you ensure that the device can't write to memory that's supposed to be read-only? |