Various optimizations to software AES-CCM and AES-GCM.

[jhb]

Summary

• Make use of cursors to avoid data copies for AES-CCM and AES-GCM.
  
  Pass pointers into the request's input and/or output buffers directly to the Update, encrypt, and decrypt hooks rather than always copying all data into a temporary block buffer on the stack.

• Use encrypt/decrypt_last for partial blocks which avoids a memset to clear the rest of the block on the stack.

• Shrink the on-stack buffers to assume AES block sizes and CCM/GCM tag lengths.

• For AAD data, pass larger chunks to axf->Update. CCM can take each AAD segment in a single call. GMAC can take multiple blocks at a time.

Test Plan

• cryptocheck and KTLS TX

[jhb]

With this change, a simple KTLS TX benchmark using 100% of a 4x2 Haswell box went from ~3.04 Gbps to ~3.20 Gbps. Still paltry, but at least some improvement.

sys/opencrypto/cryptosoft.c

448 ↗

(On Diff #72432)

One other thing I didn't mention in the message (should perhaps add it), is that in general I moved the special cases for a final partial block out of the main loop for both AAD and cipher/plain text to avoid conditional zeroing, etc. It also permitted using encrypt_last/decrypt_last and simplified the main loop bodies as they could assume a length of a block.

862 ↗

(On Diff #72432)

Not sure if CCM uses "tag" like GCM or of it uses "digest" as the normal name for this. I think I found some other references (perhaps in cbc_mac.c) that made me use "digest" here.

[jhb]

• Rebase
• Use 'tag' instead of 'digest' for CCM.

[jhb]

• Rebase
• Fixes after rebasing.