Page MenuHomeFreeBSD
Differential D35750

Make blake3 work
ClosedPublic
Actions

Authored by imp on Jul 7 2022, 11:58 PM.
Tags
None
Referenced Files
Unknown Object (File)
Sat, Apr 5, 2:09 PM
Unknown Object (File)
Sun, Mar 16, 8:08 AM
Unknown Object (File)
Thu, Mar 13, 5:15 AM
Unknown Object (File)
Mar 8 2025, 6:24 AM
Unknown Object (File)
Feb 28 2025, 7:58 AM
Unknown Object (File)
Feb 11 2025, 5:13 AM
Unknown Object (File)
Feb 7 2025, 2:26 PM
Unknown Object (File)
Feb 2 2025, 2:59 PM
View All Files
Subscribers
dab
delphij
kevans

Details

Reviewers
manu
tsoome
mm
kevans
Commits
rGa61eb269dec4: stand: Fix inclusion of sys/blake3.h
rGf05568d2cae6: stand: For ZFS build, add new directory
rG75ad24775b68: stand: Add blake3 support to boot loader
rGdf76778ad5e0: stand: delete zstd_shim.c
rG119c786f7b5a: stand: Compile zfs_zstd.c directly now
rGb0a337d50192: stand: Fix inclusion of sys/blake3.h
rGfaf25f48d601: stand: Work around upstream issues in the standalone environment
rG87d3aa53973f: stand: For ZFS build, add new directory
Summary

do not push this, for review only.

Fixups needed for blake3 compiling in the standalone envorinment

As an alternative to D35592

Sponsored by: Netflix

Diff Detail

Repository
rS FreeBSD src repository - subversion
Lint
Lint Passed
Unit
No Test Coverage
Build Status
Buildable 46331
Build 43220: arc lint + arc unit

Event Timeline

imp created this revision.Jul 7 2022, 11:58 PM
Herald added a reviewer: manu. · View Herald TranscriptJul 7 2022, 11:58 PM
Herald added subscribers: dab, delphij. · View Herald Transcript
imp requested review of this revision.Jul 7 2022, 11:58 PM
Harbormaster completed remote builds in B46315: Diff 107899.Jul 7 2022, 11:58 PM
imp edited the summary of this revision. (Show Details)Jul 8 2022, 12:00 AM
imp added a reviewer: tsoome.
imp mentioned this in D35592: loader: add blake3 checksum support.
imp added a comment.Jul 8 2022, 12:29 AM

This should likely be marked co-authored by tsoome since I started with his patches and didn't change much except not having the blake3* files copied.

imp added inline comments.Jul 8 2022, 12:52 AM
sys/contrib/openzfs/include/os/freebsd/spl/sys/types.h
80–81

I think I reviewed this change for upstream, but I'm not 100% sure. It seemed awful familiar as I was making it...

sys/contrib/openzfs/include/sys/blake3.h
31

This might be better as || defined(_STANDALONE) and leaving the #else clause unchanged.

sys/contrib/openzfs/include/sys/zfs_context.h
46

this one is weird and I don't completely understand it, but it's needed because machine/atomic.h is now included in sys/sysmacros.h. Maybe we should include sys/atomic.h there instead, but I'm not sure that wouldn't break something.

imp added inline comments.Jul 8 2022, 12:54 AM
stand/libsa/zfs/zstd_shim.c
0

This likely can be removed since I added it to sys/zfs_context.h

38

This was due to a bug that the types.h changes fix, and likely would hit anybody that's landing changes.

imp updated this revision to Diff 107900.Jul 8 2022, 12:58 AM

add forgotten files

Harbormaster completed remote builds in B46316: Diff 107900.Jul 8 2022, 12:58 AM
imp updated this revision to Diff 107901.Jul 8 2022, 1:25 AM

Simplify a bit still... And include the zfs_zstd.c changes.

Harbormaster completed remote builds in B46317: Diff 107901.Jul 8 2022, 1:25 AM
imp updated this revision to Diff 107902.Jul 8 2022, 1:30 AM

one last set of cleanups

Harbormaster completed remote builds in B46318: Diff 107902.Jul 8 2022, 1:30 AM
imp added a reviewer: mm.Jul 8 2022, 4:45 AM
imp updated this revision to Diff 107933.Jul 8 2022, 5:48 PM

Rebase on changes I'm pushing to main...

Harbormaster completed remote builds in B46330: Diff 107933.Jul 8 2022, 5:48 PM
imp updated this revision to Diff 107935.Jul 8 2022, 6:43 PM

Add blake3_zfs.c from tsoome's review...

Harbormaster completed remote builds in B46331: Diff 107935.Jul 8 2022, 6:44 PM
tsoome added a comment.Jul 8 2022, 8:00 PM

It does look good, thanks! (waiting for update as noted in chat).

imp updated this revision to Diff 107938.Jul 8 2022, 10:03 PM

be extra sleezy.... and add a layer where the boot loader can interpose its .h
files to work around issues with upstream so we can bring blake3 in with
upstream files unchanged while we work to fix the warts in them...

Harbormaster completed remote builds in B46333: Diff 107938.Jul 8 2022, 10:03 PM
imp added inline comments.Jul 8 2022, 10:12 PM
stand/libsa/zfs/Makefile.inc
12

The zstd_shim.c -> zfs_zstd.c thing is arguably separate.

35

Adding this and this define is separate. This is what adds the layer we need.

imp added a comment.Jul 8 2022, 11:02 PM

I have this sliced up into 6 commits, but will reply here and add 'reviewed by' if people approve here.

kevans accepted this revision.Jul 8 2022, 11:33 PM
kevans added a subscriber: kevans.

The header shims are a bit ugly, but given the context I agree that this is a reasonable stopgap...

This revision is now accepted and ready to land.Jul 8 2022, 11:33 PM
Closed by commit rG87d3aa53973f: stand: For ZFS build, add new directory (authored by imp). · Explain WhyJul 9 2022, 5:03 AM
This revision was automatically updated to reflect the committed changes.
imp added a commit: rG87d3aa53973f: stand: For ZFS build, add new directory.
imp added a commit: rGfaf25f48d601: stand: Work around upstream issues in the standalone environment.
imp added a commit: rGb0a337d50192: stand: Fix inclusion of sys/blake3.h.
imp added a commit: rG119c786f7b5a: stand: Compile zfs_zstd.c directly now.
imp added a commit: rGdf76778ad5e0: stand: delete zstd_shim.c.
imp added a commit: rG75ad24775b68: stand: Add blake3 support to boot loader.
imp added a commit: rGf05568d2cae6: stand: For ZFS build, add new directory.Jan 24 2023, 10:14 PM
imp added a commit: rGa61eb269dec4: stand: Fix inclusion of sys/blake3.h.

Revision Contents
Changeset List

PathSize
stand/
libsa/
zfs/
14 lines
1 line
sys/
cddl/
boot/
zfs/
97 lines
1 line
7 lines
contrib/
openzfs/
include/
os/
freebsd/
spl/
sys/
7 lines
25 lines
sys/
2 lines
2 lines

Diff 107935

View Options

stand/libsa/zfs/Makefile.inc

Show First 20 Lines • Show All 56 Lines • ▼ Show 20 Lines
*/ */
#include "assym.inc" #include "assym.inc"
#include "opt_sched.h" #include "opt_sched.h"
#include <sys/syscall.h> #include <sys/syscall.h>
#include <machine/trap.h> #include <machine/trap.h>
#include <machine/spr.h>
#include <machine/param.h> #include <machine/param.h>
#include <machine/asm.h> #include <machine/asm.h>
#ifdef _CALL_ELF #ifdef _CALL_ELF
.abiversion _CALL_ELF .abiversion _CALL_ELF
#endif #endif
TOC_ENTRY(blocked_lock) TOC_ENTRY(blocked_lock)
▲ Show 20 Lines • Show All 46 Lines • ▼ Show 20 LinesENTRY(cpu_switch)
mr %r14,%r3 /* Copy the old thread ptr... */ mr %r14,%r3 /* Copy the old thread ptr... */
mr %r13,%r4 /* and the new thread ptr in curthread*/ mr %r13,%r4 /* and the new thread ptr in curthread*/
mr %r16,%r5 /* and the new lock */ mr %r16,%r5 /* and the new lock */
mr %r17,%r6 /* and the PCB */ mr %r17,%r6 /* and the PCB */
stdu %r1,-48(%r1) stdu %r1,-48(%r1)
lwz %r7,PCB_FLAGS(%r17) lwz %r7, PCB_FLAGS(%r17)
andi. %r7, %r7, PCB_CDSCR
beq .L0
/* Custom DSCR was set. Reseting it to enter kernel */
li %r7, 0x0
mtspr SPR_DSCR, %r7
jhibbitsUnsubmitted
Not Done
Inline Actions

Use SPR_DSCR here.

jhibbits: Use SPR_DSCR here.
.L0:
lwz %r7,PCB_FLAGS(%r17)
/* Save FPU context if needed */ /* Save FPU context if needed */
andi. %r7, %r7, PCB_FPU andi. %r7, %r7, PCB_FPU
beq .L1 beq .L1
bl save_fpu bl save_fpu
nop nop
.L1: .L1:
mr %r3,%r14 /* restore old thread ptr */ mr %r3,%r14 /* restore old thread ptr */
▲ Show 20 Lines • Show All 47 Lines • ▼ Show 20 Lines1: mfsprg %r7,0 /* Get the pcpu pointer */
mr %r3,%r13 /* Pass curthread to enable_fpu */ mr %r3,%r13 /* Pass curthread to enable_fpu */
bl enable_fpu bl enable_fpu
nop nop
.L3: .L3:
lwz %r6, PCB_FLAGS(%r17) lwz %r6, PCB_FLAGS(%r17)
/* Restore Altivec context if needed */ /* Restore Altivec context if needed */
andi. %r6, %r6, PCB_VEC andi. %r6, %r6, PCB_VEC
beq .L4 beq .L31
mr %r3,%r13 /* Pass curthread to enable_vec */ mr %r3,%r13 /* Pass curthread to enable_vec */
bl enable_vec bl enable_vec
nop nop
.L31:
lwz %r6, PCB_FLAGS(%r17)
/* Restore Custom DSCR if needed */
andi. %r6, %r6, PCB_CDSCR
beq .L4
ld %r6, PCB_DSCR(%r17) /* Load the DSCR register*/
mtspr SPR_DSCR, %r6
jhibbitsUnsubmitted
Not Done
Inline Actions

And here

jhibbits: And here
/* thread to restore is in r3 */ /* thread to restore is in r3 */
.L4: .L4:
addi %r1,%r1,48 addi %r1,%r1,48
mr %r3,%r17 /* Recover PCB ptr */ mr %r3,%r17 /* Recover PCB ptr */
ld %r12,PCB_CONTEXT(%r3) /* Load the non-volatile GP regs. */ ld %r12,PCB_CONTEXT(%r3) /* Load the non-volatile GP regs. */
ld %r14,PCB_CONTEXT+2*8(%r3) ld %r14,PCB_CONTEXT+2*8(%r3)
ld %r15,PCB_CONTEXT+3*8(%r3) ld %r15,PCB_CONTEXT+3*8(%r3)
▲ Show 20 Lines • Show All 83 Lines • Show Last 20 Lines
View Options

stand/libsa/zfs/zfsimpl.c

Show First 20 Lines • Show All 56 Lines • ▼ Show 20 Lines
*/ */
#include "assym.inc" #include "assym.inc"
#include "opt_sched.h" #include "opt_sched.h"
#include <sys/syscall.h> #include <sys/syscall.h>
#include <machine/trap.h> #include <machine/trap.h>
#include <machine/spr.h>
#include <machine/param.h> #include <machine/param.h>
#include <machine/asm.h> #include <machine/asm.h>
#ifdef _CALL_ELF #ifdef _CALL_ELF
.abiversion _CALL_ELF .abiversion _CALL_ELF
#endif #endif
TOC_ENTRY(blocked_lock) TOC_ENTRY(blocked_lock)
▲ Show 20 Lines • Show All 46 Lines • ▼ Show 20 LinesENTRY(cpu_switch)
mr %r14,%r3 /* Copy the old thread ptr... */ mr %r14,%r3 /* Copy the old thread ptr... */
mr %r13,%r4 /* and the new thread ptr in curthread*/ mr %r13,%r4 /* and the new thread ptr in curthread*/
mr %r16,%r5 /* and the new lock */ mr %r16,%r5 /* and the new lock */
mr %r17,%r6 /* and the PCB */ mr %r17,%r6 /* and the PCB */
stdu %r1,-48(%r1) stdu %r1,-48(%r1)
lwz %r7,PCB_FLAGS(%r17) lwz %r7, PCB_FLAGS(%r17)
andi. %r7, %r7, PCB_CDSCR
beq .L0
/* Custom DSCR was set. Reseting it to enter kernel */
li %r7, 0x0
mtspr SPR_DSCR, %r7
jhibbitsUnsubmitted
Not Done
Inline Actions

Use SPR_DSCR here.

jhibbits: Use SPR_DSCR here.
.L0:
lwz %r7,PCB_FLAGS(%r17)
/* Save FPU context if needed */ /* Save FPU context if needed */
andi. %r7, %r7, PCB_FPU andi. %r7, %r7, PCB_FPU
beq .L1 beq .L1
bl save_fpu bl save_fpu
nop nop
.L1: .L1:
mr %r3,%r14 /* restore old thread ptr */ mr %r3,%r14 /* restore old thread ptr */
▲ Show 20 Lines • Show All 47 Lines • ▼ Show 20 Lines1: mfsprg %r7,0 /* Get the pcpu pointer */
mr %r3,%r13 /* Pass curthread to enable_fpu */ mr %r3,%r13 /* Pass curthread to enable_fpu */
bl enable_fpu bl enable_fpu
nop nop
.L3: .L3:
lwz %r6, PCB_FLAGS(%r17) lwz %r6, PCB_FLAGS(%r17)
/* Restore Altivec context if needed */ /* Restore Altivec context if needed */
andi. %r6, %r6, PCB_VEC andi. %r6, %r6, PCB_VEC
beq .L4 beq .L31
mr %r3,%r13 /* Pass curthread to enable_vec */ mr %r3,%r13 /* Pass curthread to enable_vec */
bl enable_vec bl enable_vec
nop nop
.L31:
lwz %r6, PCB_FLAGS(%r17)
/* Restore Custom DSCR if needed */
andi. %r6, %r6, PCB_CDSCR
beq .L4
ld %r6, PCB_DSCR(%r17) /* Load the DSCR register*/
mtspr SPR_DSCR, %r6
jhibbitsUnsubmitted
Not Done
Inline Actions

And here

jhibbits: And here
/* thread to restore is in r3 */ /* thread to restore is in r3 */
.L4: .L4:
addi %r1,%r1,48 addi %r1,%r1,48
mr %r3,%r17 /* Recover PCB ptr */ mr %r3,%r17 /* Recover PCB ptr */
ld %r12,PCB_CONTEXT(%r3) /* Load the non-volatile GP regs. */ ld %r12,PCB_CONTEXT(%r3) /* Load the non-volatile GP regs. */
ld %r14,PCB_CONTEXT+2*8(%r3) ld %r14,PCB_CONTEXT+2*8(%r3)
ld %r15,PCB_CONTEXT+3*8(%r3) ld %r15,PCB_CONTEXT+3*8(%r3)
▲ Show 20 Lines • Show All 83 Lines • Show Last 20 Lines
View Options

stand/libsa/zfs/zstd_shim.c

Show First 20 Lines • Show All 56 Lines • ▼ Show 20 Lines
*/ */
#include "assym.inc" #include "assym.inc"
#include "opt_sched.h" #include "opt_sched.h"
#include <sys/syscall.h> #include <sys/syscall.h>
#include <machine/trap.h> #include <machine/trap.h>
#include <machine/spr.h>
#include <machine/param.h> #include <machine/param.h>
#include <machine/asm.h> #include <machine/asm.h>
#ifdef _CALL_ELF #ifdef _CALL_ELF
.abiversion _CALL_ELF .abiversion _CALL_ELF
#endif #endif
TOC_ENTRY(blocked_lock) TOC_ENTRY(blocked_lock)
▲ Show 20 Lines • Show All 46 Lines • ▼ Show 20 LinesENTRY(cpu_switch)
mr %r14,%r3 /* Copy the old thread ptr... */ mr %r14,%r3 /* Copy the old thread ptr... */
mr %r13,%r4 /* and the new thread ptr in curthread*/ mr %r13,%r4 /* and the new thread ptr in curthread*/
mr %r16,%r5 /* and the new lock */ mr %r16,%r5 /* and the new lock */
mr %r17,%r6 /* and the PCB */ mr %r17,%r6 /* and the PCB */
stdu %r1,-48(%r1) stdu %r1,-48(%r1)
lwz %r7,PCB_FLAGS(%r17) lwz %r7, PCB_FLAGS(%r17)
andi. %r7, %r7, PCB_CDSCR
beq .L0
/* Custom DSCR was set. Reseting it to enter kernel */
li %r7, 0x0
mtspr SPR_DSCR, %r7
jhibbitsUnsubmitted
Not Done
Inline Actions

Use SPR_DSCR here.

jhibbits: Use SPR_DSCR here.
.L0:
lwz %r7,PCB_FLAGS(%r17)
/* Save FPU context if needed */ /* Save FPU context if needed */
andi. %r7, %r7, PCB_FPU andi. %r7, %r7, PCB_FPU
beq .L1 beq .L1
bl save_fpu bl save_fpu
nop nop
.L1: .L1:
mr %r3,%r14 /* restore old thread ptr */ mr %r3,%r14 /* restore old thread ptr */
▲ Show 20 Lines • Show All 47 Lines • ▼ Show 20 Lines1: mfsprg %r7,0 /* Get the pcpu pointer */
mr %r3,%r13 /* Pass curthread to enable_fpu */ mr %r3,%r13 /* Pass curthread to enable_fpu */
bl enable_fpu bl enable_fpu
nop nop
.L3: .L3:
lwz %r6, PCB_FLAGS(%r17) lwz %r6, PCB_FLAGS(%r17)
/* Restore Altivec context if needed */ /* Restore Altivec context if needed */
andi. %r6, %r6, PCB_VEC andi. %r6, %r6, PCB_VEC
beq .L4 beq .L31
mr %r3,%r13 /* Pass curthread to enable_vec */ mr %r3,%r13 /* Pass curthread to enable_vec */
bl enable_vec bl enable_vec
nop nop
.L31:
lwz %r6, PCB_FLAGS(%r17)
/* Restore Custom DSCR if needed */
andi. %r6, %r6, PCB_CDSCR
beq .L4
ld %r6, PCB_DSCR(%r17) /* Load the DSCR register*/
mtspr SPR_DSCR, %r6
jhibbitsUnsubmitted
Not Done
Inline Actions

And here

jhibbits: And here
/* thread to restore is in r3 */ /* thread to restore is in r3 */
.L4: .L4:
addi %r1,%r1,48 addi %r1,%r1,48
mr %r3,%r17 /* Recover PCB ptr */ mr %r3,%r17 /* Recover PCB ptr */
ld %r12,PCB_CONTEXT(%r3) /* Load the non-volatile GP regs. */ ld %r12,PCB_CONTEXT(%r3) /* Load the non-volatile GP regs. */
ld %r14,PCB_CONTEXT+2*8(%r3) ld %r14,PCB_CONTEXT+2*8(%r3)
ld %r15,PCB_CONTEXT+3*8(%r3) ld %r15,PCB_CONTEXT+3*8(%r3)
▲ Show 20 Lines • Show All 83 Lines • Show Last 20 Lines
View Options

sys/cddl/boot/zfs/blake3_zfs.c

Show First 20 Lines • Show All 56 Lines • ▼ Show 20 Lines
*/ */
#include "assym.inc" #include "assym.inc"
#include "opt_sched.h" #include "opt_sched.h"
#include <sys/syscall.h> #include <sys/syscall.h>
#include <machine/trap.h> #include <machine/trap.h>
#include <machine/spr.h>
#include <machine/param.h> #include <machine/param.h>
#include <machine/asm.h> #include <machine/asm.h>
#ifdef _CALL_ELF #ifdef _CALL_ELF
.abiversion _CALL_ELF .abiversion _CALL_ELF
#endif #endif
TOC_ENTRY(blocked_lock) TOC_ENTRY(blocked_lock)
▲ Show 20 Lines • Show All 46 Lines • ▼ Show 20 LinesENTRY(cpu_switch)
mr %r14,%r3 /* Copy the old thread ptr... */ mr %r14,%r3 /* Copy the old thread ptr... */
mr %r13,%r4 /* and the new thread ptr in curthread*/ mr %r13,%r4 /* and the new thread ptr in curthread*/
mr %r16,%r5 /* and the new lock */ mr %r16,%r5 /* and the new lock */
mr %r17,%r6 /* and the PCB */ mr %r17,%r6 /* and the PCB */
stdu %r1,-48(%r1) stdu %r1,-48(%r1)
lwz %r7,PCB_FLAGS(%r17) lwz %r7, PCB_FLAGS(%r17)
andi. %r7, %r7, PCB_CDSCR
beq .L0
/* Custom DSCR was set. Reseting it to enter kernel */
li %r7, 0x0
mtspr SPR_DSCR, %r7
jhibbitsUnsubmitted
Not Done
Inline Actions

Use SPR_DSCR here.

jhibbits: Use SPR_DSCR here.
.L0:
lwz %r7,PCB_FLAGS(%r17)
/* Save FPU context if needed */ /* Save FPU context if needed */
andi. %r7, %r7, PCB_FPU andi. %r7, %r7, PCB_FPU
beq .L1 beq .L1
bl save_fpu bl save_fpu
nop nop
.L1: .L1:
mr %r3,%r14 /* restore old thread ptr */ mr %r3,%r14 /* restore old thread ptr */
▲ Show 20 Lines • Show All 47 Lines • ▼ Show 20 Lines1: mfsprg %r7,0 /* Get the pcpu pointer */
mr %r3,%r13 /* Pass curthread to enable_fpu */ mr %r3,%r13 /* Pass curthread to enable_fpu */
bl enable_fpu bl enable_fpu
nop nop
.L3: .L3:
lwz %r6, PCB_FLAGS(%r17) lwz %r6, PCB_FLAGS(%r17)
/* Restore Altivec context if needed */ /* Restore Altivec context if needed */
andi. %r6, %r6, PCB_VEC andi. %r6, %r6, PCB_VEC
beq .L4 beq .L31
mr %r3,%r13 /* Pass curthread to enable_vec */ mr %r3,%r13 /* Pass curthread to enable_vec */
bl enable_vec bl enable_vec
nop nop
.L31:
lwz %r6, PCB_FLAGS(%r17)
/* Restore Custom DSCR if needed */
andi. %r6, %r6, PCB_CDSCR
beq .L4
ld %r6, PCB_DSCR(%r17) /* Load the DSCR register*/
mtspr SPR_DSCR, %r6
jhibbitsUnsubmitted
Not Done
Inline Actions

And here

jhibbits: And here
/* thread to restore is in r3 */ /* thread to restore is in r3 */
.L4: .L4:
addi %r1,%r1,48 addi %r1,%r1,48
mr %r3,%r17 /* Recover PCB ptr */ mr %r3,%r17 /* Recover PCB ptr */
ld %r12,PCB_CONTEXT(%r3) /* Load the non-volatile GP regs. */ ld %r12,PCB_CONTEXT(%r3) /* Load the non-volatile GP regs. */
ld %r14,PCB_CONTEXT+2*8(%r3) ld %r14,PCB_CONTEXT+2*8(%r3)
ld %r15,PCB_CONTEXT+3*8(%r3) ld %r15,PCB_CONTEXT+3*8(%r3)
▲ Show 20 Lines • Show All 83 Lines • Show Last 20 Lines
View Options

sys/cddl/boot/zfs/zfsimpl.h

Loading...
View Options

sys/cddl/boot/zfs/zfssubr.c

Loading...
View Options

sys/contrib/openzfs/include/os/freebsd/spl/sys/types.h

Loading...
View Options

sys/contrib/openzfs/include/os/freebsd/spl/sys/zfs_context.h

Loading...
View Options

sys/contrib/openzfs/include/sys/blake3.h

Loading...
View Options

sys/contrib/openzfs/include/sys/zfs_context.h

Loading...