Change the device shared memory allocation routines to use the busdma API. One side effect is that all memory, including the PRP segments, are now allocated in a single slab.

[scottl]

Remove a spurious printf

Updating D8453: Change the device shared memory allocation routines to use the busdma API.

One side effect is that all memory, including the PRP segments, are now
allocated in a single slab.

[jimharris]

sys/dev/nvme/nvme_qpair.c

492 ↗

(On Diff #22027)

Nit - prplistsz would be a more accurate variable name here.

525 ↗

(On Diff #22027)

This should be removed before committing.

571 ↗

(On Diff #22027)

Should you still unload the dmamap here?

[scottl]

Address feedback from Jim

Updating D8453: Change the device shared memory allocation routines to use the busdma API.

One side effect is that all memory, including the PRP segments, are now
allocated in a single slab.

[scottl]

sys/dev/nvme/nvme_qpair.c

492 ↗

(On Diff #22027)

Decided to remove the variable, it wasn't necessary.

571 ↗

(On Diff #22027)

Good catch, thanks.

[jimharris]

sys/dev/nvme/nvme_qpair.c

496 ↗

(On Diff #22036)

4 needs to change to PAGE_SIZE. See below.

518 ↗

(On Diff #22036)

Both cmd and cpl need to be page-aligned. Previously the contigmalloc call took care of this. Since these are allocated in one buffer now, you'll also need to assert that cmd is a PAGE_SIZE multiple. The default number of queue entries for admin and io pairs (128 and 256 respectively) will ensure the cmd queue is a PAGE_SIZE multiple but would be good to double-check that here.

538 ↗

(On Diff #22036)

A PRP list cannot span a page boundary, so you'll need to make sure you account for that here when allocating chunks of the bigger buffer for each tracker's PRP list.

[scottl]

Add safety belts for page alignment and page crossing.

Updating D8453: Change the device shared memory allocation routines to use the busdma API.

One side effect is that all memory, including the PRP segments, are now
allocated in a single slab.

[scottl]

sys/dev/nvme/nvme_qpair.c

518 ↗

(On Diff #22036)

Thanks for the insight. I've added seats belts to the next revision

[jimharris]

sys/dev/nvme/nvme_qpair.c

547 ↗

(On Diff #22039)

This is probably fine, since realistically we probably won't have a PRP list size that's not even divisible into a page. If anyone ever set MAXPHYS to something like 1.5MB, this would abort out after the first tracker and we'd be stuck with just one tracker per qpair though.

Maybe instead you could do something like this above:

prpmemsz = PAGE_SIZE * (qpair->num_trackers / (PAGE_SIZE / perps));

This should allocate enough space for PRP lists that when you trunc_page check here fails, you can do a roundup2 to the next page boundary.

The other option would be to punt on this for now and just assert that the PRP list divides evenly into PAGE_SIZE.

[scottl]

Updating D8453: Change the device shared memory allocation routines to use the busdma API.

One side effect is that all memory, including the PRP segments, are now
allocated in a single slab.

[jimharris]

Looks good!