Gasket is Google's name for the layer their accelerator drivers share:
Google ASIC Software, Kernel Extensions, and Tools. The chips differ,
but they present the same shape to the host. A control register BAR
with a few windows user space may map, an on-chip page table the host
fills in so the device can reach host memory, and a block of MSI-X
vectors.
Details
Details
amd64 only for now, and host mappings are refused under KMSAN.
only the m2 cards were tested.
Diff Detail
Diff Detail
- Repository
- rG FreeBSD src repository
- Lint
Lint Passed - Unit
No Test Coverage - Build Status
Buildable 75309 Build 72192: arc lint + arc unit
Event Timeline
Comment Actions
Interesting! A lot of this is mmap / dmabuf style stuff. I don't want to block landing this on waiting for freebsd native dmabuf, but i wonder if this would be another user of native freebsd dmabuf at some point.
| share/man/man4/gasket.4 | ||
|---|---|---|
| 53 | heh update email address | |
| sys/dev/gasket/gasket.h | ||
| 94 | do we need bus_* barriers anywhere? this may be mostly fine on x86 but eg arm/riscv it may need explicit barriers? | |
| sys/dev/gasket/gasket_core.c | ||
| 736 | so are all of these devices fine with 64 bit DMA? | |
| 871 | This means the consumer needs to do two mmaps, right? One for data and one for control/registers, so the correct memory attributes are used? | |
| 1009 | I think this function in particular deserves a doxygen style comment on what it does and entry/exit values. | |
| sys/dev/gasket/gasket_if.m | ||
| 56 | I think all of the methods here deserve documenting what they do, even if it is brief. | |
Comment Actions
ok, i think this is fine. I think it'd be a good starting point to figure out what to do about page ownership between devices and userland (the same kind of issue we have with GPU, netmap, etc.) but I don't know if we want to tackle this here.
@markj and @kib - would you mind eyeballing the VM code bits here? See how they look and what kind of design we may want to draft up for things like this to eventually leverage?
Maybe dma_buf kind of stuff?
Comment Actions
Accepted provided feedback is addressed
| share/man/man4/gasket.4 | ||
|---|---|---|
| 8 | Am I correct in understanding that this only works on amd64 for now? If so, please specify in the manpage. There is an optional architectural specifier here where you can do that. | |
| share/man/man4/gasket.4 | ||
|---|---|---|
| 8 | It tested in amd64 only . But the framework should work with any architecture . | |
| share/man/man4/gasket.4 | ||
|---|---|---|
| 34 | Why aren't they charged against those limits? The backing pages are effectively mlocked. | |
| sys/dev/gasket/gasket_chardev.c | ||
| 188 ↗ | (On Diff #189675) | Why do you do the validation separately from handling the actual ioctls? It is harder to read that way. |
| sys/dev/gasket/gasket_coherent.c | ||
| 29 ↗ | (On Diff #189675) | This loop runs exactly once, I don't think that's the intent. |
| sys/dev/gasket/gasket_mmap.c | ||
| 266 ↗ | (On Diff #189675) | As I said below I think this code shouldn't be messing with the vm_map at all. |
| 295 ↗ | (On Diff #189675) | This should be a list of allowed object types (or better, some characterization of the object properties that we want), not a list that needs to be updated each time a new object type is added. |
| 373 ↗ | (On Diff #189675) | Suppose a process maps some of its pages into device page tables. Then it forks, and the host mapping becomes read-only in order to implement COW semantics. Now the device has a writable mapping to read-only memory. What should happen in this case? |
| 396 ↗ | (On Diff #189675) | You already faulted the pages above, why look them up again? The last parameter to vm_fault() can be used to return the faulted page. Or use vm_fault_quick_hold_pages(), which is probably better here. That way, gasket_classify_range_locked() doesn't need to deal with vm_map entries at all. You can just look at the returned pages to see which type of VM object you're dealing with. |
| 401 ↗ | (On Diff #189675) | What does this mean? Why is it sufficient to check for only those two flags? |
| 411 ↗ | (On Diff #189675) | Isn't this handled by vm_fault()? |