Page MenuHomeFreeBSD

Change the device shared memory allocation routines to use the busdma API. One side effect is that all memory, including the PRP segments, are now allocated in a single slab.
ClosedPublic

Authored by scottl on Nov 6 2016, 4:46 AM.
Tags
None
Referenced Files
Unknown Object (File)
Fri, Nov 29, 10:03 PM
Unknown Object (File)
Fri, Nov 29, 9:51 PM
Unknown Object (File)
Nov 20 2024, 12:01 AM
Unknown Object (File)
Nov 19 2024, 2:01 AM
Unknown Object (File)
Nov 17 2024, 3:17 PM
Unknown Object (File)
Nov 14 2024, 1:24 AM
Unknown Object (File)
Nov 9 2024, 1:46 AM
Unknown Object (File)
Nov 5 2024, 7:47 PM
Subscribers
None

Diff Detail

Repository
rS FreeBSD src repository - subversion
Lint
Lint Passed
Unit
No Test Coverage
Build Status
Buildable 5879
Build 6166: arc lint + arc unit

Event Timeline

scottl retitled this revision from to Change the device shared memory allocation routines to use the busdma API. One side effect is that all memory, including the PRP segments, are now allocated in a single slab..
scottl updated this object.
scottl edited the test plan for this revision. (Show Details)
scottl added reviewers: imp, gallatin, jimharris.

Remove a spurious printf

Updating D8453: Change the device shared memory allocation routines to use the busdma API.

One side effect is that all memory, including the PRP segments, are now
allocated in a single slab.

sys/dev/nvme/nvme_qpair.c
492

Nit - prplistsz would be a more accurate variable name here.

524โ€“529

This should be removed before committing.

605

Should you still unload the dmamap here?

Address feedback from Jim

Updating D8453: Change the device shared memory allocation routines to use the busdma API.

One side effect is that all memory, including the PRP segments, are now
allocated in a single slab.

scottl added inline comments.
sys/dev/nvme/nvme_qpair.c
492

Decided to remove the variable, it wasn't necessary.

605

Good catch, thanks.

sys/dev/nvme/nvme_qpair.c
502โ€“503

4 needs to change to PAGE_SIZE. See below.

525

Both cmd and cpl need to be page-aligned. Previously the contigmalloc call took care of this. Since these are allocated in one buffer now, you'll also need to assert that cmd is a PAGE_SIZE multiple. The default number of queue entries for admin and io pairs (128 and 256 respectively) will ensure the cmd queue is a PAGE_SIZE multiple but would be good to double-check that here.

564

A PRP list cannot span a page boundary, so you'll need to make sure you account for that here when allocating chunks of the bigger buffer for each tracker's PRP list.

scottl marked 2 inline comments as done.

Add safety belts for page alignment and page crossing.

Updating D8453: Change the device shared memory allocation routines to use the busdma API.

One side effect is that all memory, including the PRP segments, are now
allocated in a single slab.

scottl added inline comments.
sys/dev/nvme/nvme_qpair.c
525

Thanks for the insight. I've added seats belts to the next revision

sys/dev/nvme/nvme_qpair.c
550

This is probably fine, since realistically we probably won't have a PRP list size that's not even divisible into a page. If anyone ever set MAXPHYS to something like 1.5MB, this would abort out after the first tracker and we'd be stuck with just one tracker per qpair though.

Maybe instead you could do something like this above:

prpmemsz = PAGE_SIZE * (qpair->num_trackers / (PAGE_SIZE / perps));

This should allocate enough space for PRP lists that when you trunc_page check here fails, you can do a roundup2 to the next page boundary.

The other option would be to punt on this for now and just assert that the PRP list divides evenly into PAGE_SIZE.

scottl marked an inline comment as done.

Updating D8453: Change the device shared memory allocation routines to use the busdma API.

One side effect is that all memory, including the PRP segments, are now
allocated in a single slab.

jimharris edited edge metadata.

Looks good!

This revision is now accepted and ready to land.Nov 7 2016, 11:52 PM