Page MenuHomeFreeBSD

Change the device shared memory allocation routines to use the busdma API. One side effect is that all memory, including the PRP segments, are now allocated in a single slab.
ClosedPublic

Authored by scottl on Nov 6 2016, 4:46 AM.

Diff Detail

Repository
rS FreeBSD src repository - subversion
Lint
Lint OK
Unit
No Unit Test Coverage
Build Status
Buildable 5868
Build 6155: arc lint + arc unit

Event Timeline

scottl retitled this revision from to Change the device shared memory allocation routines to use the busdma API. One side effect is that all memory, including the PRP segments, are now allocated in a single slab..
scottl updated this object.
scottl edited the test plan for this revision. (Show Details)
scottl added reviewers: imp, gallatin, jimharris.

Remove a spurious printf

Updating D8453: Change the device shared memory allocation routines to use the busdma API.

One side effect is that all memory, including the PRP segments, are now
allocated in a single slab.

sys/dev/nvme/nvme_qpair.c
491

Nit - prplistsz would be a more accurate variable name here.

523–528

This should be removed before committing.

594

Should you still unload the dmamap here?

Address feedback from Jim

Updating D8453: Change the device shared memory allocation routines to use the busdma API.

One side effect is that all memory, including the PRP segments, are now
allocated in a single slab.

scottl added inline comments.
sys/dev/nvme/nvme_qpair.c
491

Decided to remove the variable, it wasn't necessary.

594

Good catch, thanks.

sys/dev/nvme/nvme_qpair.c
501–502

4 needs to change to PAGE_SIZE. See below.

523

Both cmd and cpl need to be page-aligned. Previously the contigmalloc call took care of this. Since these are allocated in one buffer now, you'll also need to assert that cmd is a PAGE_SIZE multiple. The default number of queue entries for admin and io pairs (128 and 256 respectively) will ensure the cmd queue is a PAGE_SIZE multiple but would be good to double-check that here.

555

A PRP list cannot span a page boundary, so you'll need to make sure you account for that here when allocating chunks of the bigger buffer for each tracker's PRP list.

scottl marked 2 inline comments as done.

Add safety belts for page alignment and page crossing.

Updating D8453: Change the device shared memory allocation routines to use the busdma API.

One side effect is that all memory, including the PRP segments, are now
allocated in a single slab.

scottl added inline comments.
sys/dev/nvme/nvme_qpair.c
523

Thanks for the insight. I've added seats belts to the next revision

sys/dev/nvme/nvme_qpair.c
547

This is probably fine, since realistically we probably won't have a PRP list size that's not even divisible into a page. If anyone ever set MAXPHYS to something like 1.5MB, this would abort out after the first tracker and we'd be stuck with just one tracker per qpair though.

Maybe instead you could do something like this above:

prpmemsz = PAGE_SIZE * (qpair->num_trackers / (PAGE_SIZE / perps));

This should allocate enough space for PRP lists that when you trunc_page check here fails, you can do a roundup2 to the next page boundary.

The other option would be to punt on this for now and just assert that the PRP list divides evenly into PAGE_SIZE.

scottl marked an inline comment as done.

Updating D8453: Change the device shared memory allocation routines to use the busdma API.

One side effect is that all memory, including the PRP segments, are now
allocated in a single slab.

jimharris edited edge metadata.

Looks good!

This revision is now accepted and ready to land.Nov 7 2016, 11:52 PM