busdma/bounce: force bouncing of full buffer

[royger]

Summary

Due to r292255 when doing partial bouncing of a buffer (ie: only bouncing
certain segments) the offsets might not be contiguous anymore, breaking
certain devices that expect them to be.

In order to make sure the segments have contiguous offsets always fully
bounce the whole buffer.

PR: 208365
Sponsored by: Citrix Systems R&D

Test Plan

TBH, I'm not sure whether to simply revert r292255 or to apply this fix. The
following needs to be taken into account:

• r292255 does compaction, so that we will always use the minimum amount of bounce pages.
• Since r292255 the bounce code now tries to always fill the bounce pages, the offset of the segments is changed and we need to bounce the whole buffer.
• Is it worth bouncing the full buffer (so more memcpy) in exchange for maybe using less bounce pages and having the bounce code slightly simplified?

[• hselasky]

If the length is <= PAGE_SIZE, compaction is safe. Else don't compact. The benifit might be less optimal when length > PAGE_SIZE? Will waste a page anyway.

[royger]

In D5917#126469, @hselasky wrote:

If the length is <= PAGE_SIZE, compaction is safe. Else don't compact. The benifit might be less optimal when length > PAGE_SIZE? Will waste a page anyway.

Not really, take the following example; a buffer of size 8192 (2*PAGE_SIZE) that starts at offset 2048. With the previous code this would require 3 bounce pages, with the current code it only requires two bounce pages, because we change the offset of the first segment to be 0 and then it just fits into 2 pages.

[• hselasky]

Sure, but if you look at how much memory you waste it is less than or equal to 50%:

Memory waste is 1.0 - (8192 / (3*4096)) = 33% ???

which I think is acceptable.

The biggest gain will be for loading less than PAGE_SIZE bytes, which cross a PAGE boundary ??

[royger]

In D5917#126484, @hselasky wrote:

Sure, but if you look at how much memory you waste it is less than or equal to 50%:

Memory waste is 1.0 - (8192 / (3*4096)) = 33% ???

which I think is acceptable.

The biggest gain will be for loading less than PAGE_SIZE bytes, which cross a PAGE boundary ??

Yes, if we have a buffer that has a size of PAGE_SIZE (or less) and it crosses a page boundary we can save 50%. But having different behaviors depending on the size/alignment of the buffer seems like opening a can of worms, TBH, I don't think we want to make the code any more complex that what it already is.

[• hselasky]

About complexity - this fix makes busdma more complicated. Should we then revert the PAGE squashing feature. We should make a decision before next week, because 11.0 is going to be branched soon!

[jah]

I think it's better to revert r292255 than to add this. Like Hans mentioned it adds complexity, and the fact that it forces entire buffers to be bounced seems wrong to me. The added memcpy probably isn't that big of a deal, since performance goes out the window if you're bouncing anyway. But the memory those bounce pages come from is likely to be somewhat precious. This could make large transfers use a lot more of that, which makes me nervous.

If you want to save on bounce buffer usage for smaller segments, it's probably better to implement bounce buffer coalescing (similar to the segment coalescing we already do), and then probably only for !BUS_DMA_KEEP_PG_OFFSET. I think ian has also been working on some changes to make arm/mips use sub-page allocations for small bounce segments, so that might be useful here too.

[royger]

In D5917#126795, @jah wrote:

I think it's better to revert r292255 than to add this. Like Hans mentioned it adds complexity, and the fact that it forces entire buffers to be bounced seems wrong to me. The added memcpy probably isn't that big of a deal, since performance goes out the window if you're bouncing anyway. But the memory those bounce pages come from is likely to be somewhat precious. This could make large transfers use a lot more of that, which makes me nervous.

If you want to save on bounce buffer usage for smaller segments, it's probably better to implement bounce buffer coalescing (similar to the segment coalescing we already do), and then probably only for !BUS_DMA_KEEP_PG_OFFSET. I think ian has also been working on some changes to make arm/mips use sub-page allocations for small bounce segments, so that might be useful here too.

I agree, and I've already reverted r292255. Thanks for the input.

Roger.