Page MenuHomeFreeBSD

witness: harden tunables for large settings
ClosedPublic

Authored by rlibby on Wed, Jun 24, 1:10 AM.
Tags
None
Referenced Files
F161955027: D57793.id180636.diff
Wed, Jul 8, 8:11 AM
Unknown Object (File)
Sun, Jul 5, 6:10 PM
Unknown Object (File)
Sun, Jul 5, 9:33 AM
Unknown Object (File)
Sat, Jul 4, 3:00 PM
Unknown Object (File)
Sat, Jul 4, 3:00 PM
Unknown Object (File)
Sat, Jul 4, 10:23 AM
Unknown Object (File)
Sat, Jul 4, 10:20 AM
Unknown Object (File)
Sat, Jul 4, 9:42 AM
Subscribers

Details

Summary

Harden witness against reasonable tunable settings causing boot panics
or scribblers. For now, don't bother to harden this completely. Panics
are still achievable by setting the tunables to negative or very large
values.

The recent change to respect the setting of debug.witness.witness_count
(08180f1b613b) exposed that there are no guard rails on that setting,
setting it to large but plausible values can cause integer overflow or
cause attempted use of memory beyond what is actually available. Add
some guard rails, fall back to default configuration if the tuned values
are too large, and fall back to a minimum memory allocation if even the
defaults are too large. Also don't allow witness to use over half of
the memory segment, as there are other early allocators after witness
too.

Reported by: asomers
Fixes: 08180f1b613b ("witness: actually set read-only tunables in time for witness_startup")
Sponsored by: Dell Inc.

Test Plan

Boot with the following in the loader environment:

set debug.witness.witness_count=0
set debug.witness.witness_count=49152
set debug.witness.witness_count=131072

Resulting calculated witness allocations are minimal, > 2.25 GiB, and > 16 GiB (the latter of which is larger than my largest phys segment on my dev vm).

Note that setting it to -1 or some very large value will still cause an overflow related panic. I am avoiding addressing that now to avoid adding even more complexity to validate a debug setting.

Diff Detail

Repository
rG FreeBSD src repository
Lint
Lint Skipped
Unit
Tests Skipped
Build Status
Buildable 74205
Build 71088: arc lint + arc unit

Event Timeline

rlibby added reviewers: markj, kib.
rlibby added a subscriber: asomers.
sys/kern/subr_witness.c
792–793

Why noinline?

840

How does this disable witness?

sys/vm/vm_page.c
614

You might add some comment explaining why we don't take the whole segment.

rlibby added inline comments.
sys/kern/subr_witness.c
792–793

It was an attempt to optimize for size instead of speed in run-once startup code.

However, now looking at the actual generated amd64 code, the size savings is only 16 bytes. Both clang and gcc eliminate the third inline call entirely, but both generate the third call for the __noinline case.

I'll remove the __noinline to simplify logic.

It might be nice if generally we had a convention to tag startup/init code. Just for fun I also tried tagging witness_startup_count with __attribute__(__cold__). Results below for size of witness_startup_count + witness_startup_calc.

cc \ attrplain__noinline__attribute__(__cold__)
clang448432440
gcc480448434
840

It doesn't... Setting witness_watch = 0 (or -1) would also disable checks that don't rely on these allocations, and might also be confusing as that's a separate user-controlled tunable. I was only attempting to provide a way to survive the allocation failure.

How about I replace "disabling" with "minimizing startup allocations"?

sys/vm/vm_page.c
614

Sure.

rlibby marked 3 inline comments as done.

markj feedback

markj added inline comments.
sys/kern/subr_witness.c
792–793

I think having a __sysinit annotation to wrap that would be a good idea. It'd be cool if the SYSINIT macro would automatically apply that to the corresponding C function, but I don't know if that can be done practically since the macro usually follows the function body.

IIRC Linux will put cold functions in a separate ELF section, I presume so that they don't occupy as much TLB space. Does the __cold__ attribute have that effect?

840

I mean, this is effectively disabling witness, I'm not sure it's worth being pedantic in the user-visible message. I think it's ok as it is, or maybe with a brief comment.

This revision is now accepted and ready to land.Wed, Jun 24, 3:45 PM
sys/kern/subr_witness.c
792–793

Yes, with caveat. The compiler puts them in a separate section, .text.unlikely, and then our kernel linker script collects them all in the .text section. I'm unsure if this has the effect of grouping all the .text.unlikely code in the end result.

840

I'm okay either way, will follow your preference between "disabling" and "startup allocations minimized". I agree it's functionally disabled even if certain sleep-ok checks work, hence the initial verbiage. Did you mean to add a code comment about the pedantic part but leave the user message as "disabled"?

sys/kern/subr_witness.c
792–793

I'd somewhat expect so.

840

Right, I think a code comment would make this a bit clearer, but IMO it's ok as-is too.

tweak comment and log message

This revision now requires review to proceed.Thu, Jun 25, 5:17 PM
sys/kern/subr_witness.c
797

Pedantically, all values are provided by user and all calculations could overflow.

834

In bootverbose mode, I would print the witness parameters somewhere so they appear in the dmesg. Perhaps it should be done right after the witness_startup_count().

sys/kern/subr_witness.c
797

Yes, that's true (and what I meant by the second sentence of the commit log). I saw the scope here as just addressing too-big configurations like the one asomers reported, but otherwise the current patch doesn't protect against all values that would cause overflows. I thought about providing more robust protection, but wasn't sure it was worth the complexity. I'm guessing there are a lot of tunables that have similar weaknesses.

Would you all like to see the tunables be fully robust? I don't really object to going in that direction. I might suggest we put in an intermediate step and provide some infrastructure like TUNABLE_*_FETCH_LIMIT macros to clamp or reject values beyond specified limits.

To address the robustness in a more focused way, what I thought of doing was defining

#define WITNESS_TUNE_MAX (1ul << (ULONG_WIDTH / 2 - 1)

And enforcing that on the tunables, and that limit should prevent overflow with the current math (divide bits by two because of the square).

834

Like at the top of witness_startup()? I can add that.

sys/kern/subr_witness.c
797

That was my reaction to the claim of 'prevent overflow'. Also I do not think that we need to go that far, which is why I said that my comment is pedantic. Might be, a comment somewhere that we do not care about user causing the arithmetic to fail is enough, only about user taking too much memory.

834

Perhaps yes. I am thinking of debugging cases when user get early panic and only can provide dmesg, since userspace cannot be reached.

This revision is now accepted and ready to land.Fri, Jun 26, 6:24 PM
This revision was automatically updated to reflect the committed changes.