Page MenuHomeFreeBSD

Change the device shared memory allocation routines to use the busdma API. One side effect is that all memory, including the PRP segments, are now allocated in a single slab.
ClosedPublic

Authored by scottl on Nov 6 2016, 4:46 AM.

Diff Detail

Repository
rS FreeBSD src repository - subversion
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

scottl retitled this revision from to Change the device shared memory allocation routines to use the busdma API. One side effect is that all memory, including the PRP segments, are now allocated in a single slab..
scottl updated this object.
scottl edited the test plan for this revision. (Show Details)
scottl added reviewers: imp, gallatin, jimharris.

Remove a spurious printf

Updating D8453: Change the device shared memory allocation routines to use the busdma API.

One side effect is that all memory, including the PRP segments, are now
allocated in a single slab.

sys/dev/nvme/nvme_qpair.c
492 ↗(On Diff #22027)

Nit - prplistsz would be a more accurate variable name here.

525 ↗(On Diff #22027)

This should be removed before committing.

571 ↗(On Diff #22027)

Should you still unload the dmamap here?

Address feedback from Jim

Updating D8453: Change the device shared memory allocation routines to use the busdma API.

One side effect is that all memory, including the PRP segments, are now
allocated in a single slab.

scottl added inline comments.
sys/dev/nvme/nvme_qpair.c
492 ↗(On Diff #22027)

Decided to remove the variable, it wasn't necessary.

571 ↗(On Diff #22027)

Good catch, thanks.

sys/dev/nvme/nvme_qpair.c
496 ↗(On Diff #22036)

4 needs to change to PAGE_SIZE. See below.

518 ↗(On Diff #22036)

Both cmd and cpl need to be page-aligned. Previously the contigmalloc call took care of this. Since these are allocated in one buffer now, you'll also need to assert that cmd is a PAGE_SIZE multiple. The default number of queue entries for admin and io pairs (128 and 256 respectively) will ensure the cmd queue is a PAGE_SIZE multiple but would be good to double-check that here.

538 ↗(On Diff #22036)

A PRP list cannot span a page boundary, so you'll need to make sure you account for that here when allocating chunks of the bigger buffer for each tracker's PRP list.

scottl marked 2 inline comments as done.

Add safety belts for page alignment and page crossing.

Updating D8453: Change the device shared memory allocation routines to use the busdma API.

One side effect is that all memory, including the PRP segments, are now
allocated in a single slab.

scottl added inline comments.
sys/dev/nvme/nvme_qpair.c
518 ↗(On Diff #22036)

Thanks for the insight. I've added seats belts to the next revision

sys/dev/nvme/nvme_qpair.c
547 ↗(On Diff #22039)

This is probably fine, since realistically we probably won't have a PRP list size that's not even divisible into a page. If anyone ever set MAXPHYS to something like 1.5MB, this would abort out after the first tracker and we'd be stuck with just one tracker per qpair though.

Maybe instead you could do something like this above:

prpmemsz = PAGE_SIZE * (qpair->num_trackers / (PAGE_SIZE / perps));

This should allocate enough space for PRP lists that when you trunc_page check here fails, you can do a roundup2 to the next page boundary.

The other option would be to punt on this for now and just assert that the PRP list divides evenly into PAGE_SIZE.

scottl marked an inline comment as done.

Updating D8453: Change the device shared memory allocation routines to use the busdma API.

One side effect is that all memory, including the PRP segments, are now
allocated in a single slab.

jimharris edited edge metadata.

Looks good!

This revision is now accepted and ready to land.Nov 7 2016, 11:52 PM