Page MenuHomeFreeBSD

trap_cfi(9): introduce amd64 implementation
AcceptedPublic

Authored by mchoo on Sun, Sep 13, 6:19 PM.
Tags
None
Referenced Files
F172987781: D59649.diff
Tue, Sep 22, 4:19 PM
F172973179: D59649.id186619.diff
Tue, Sep 22, 1:24 PM
F172950514: D59649.id186619.diff
Tue, Sep 22, 8:37 AM
F172915520: D59649.id186696.diff
Tue, Sep 22, 1:52 AM
F172905379: D59649.id186696.diff
Tue, Sep 22, 12:02 AM
Unknown Object (File)
Mon, Sep 21, 7:16 PM
Unknown Object (File)
Mon, Sep 21, 7:11 PM
Unknown Object (File)
Mon, Sep 21, 5:04 PM
Subscribers

Details

Reviewers
jhb
kib
Summary

LLDB dev working on Darwin debugging recommends using CFI instead of trapframe sniffer callbacks. CFI directives itself doesn't any effects in actual code, but I splitted fork_trampoline() into fork_trampoline() and fork_trampoline_kthread() as CFI cannot express conditionals like PC equals to zero.

MFC after: 2 weeks
MFC to: stable/14, stable/15
Sponsored by: FreeBSD Foundation

Diff Detail

Repository
rG FreeBSD src repository
Lint
Lint Skipped
Unit
No Test Coverage
Build Status
Buildable 76886
Build 73769: arc lint + arc unit

Event Timeline

mchoo requested review of this revision.Sun, Sep 13, 6:19 PM

I do not quite understand what could be a subject for the review. It is standalone macro set, which I have no idea what supposed to do, and how are used.

sys/x86/include/frame.h
119 ↗(On Diff #186619)

This comment is perhaps misplaced. Should it go somewhere where the 'trap frame CFI information' is applied?

@kib, I was going to create revisions for actual changes once the macros are accepted, but since I tested this on amd64 with lldb, I'm posting second part of the change here.

mchoo retitled this revision from trap_cfi(9): add assembly interface for amd64 to trap_cfi(9): introduce amd64 implementation.Mon, Sep 14, 4:03 PM
mchoo edited the summary of this revision. (Show Details)
sys/amd64/amd64/exception.S
1341

ast should be handled for kthreads trampolines as well

1710

FRED event entry points need the annotations

sys/amd64/include/trap_cfi.h
16 ↗(On Diff #186696)

Shouldn't .cfi_sections directive specified once, at the beginning of the asm source? At least, this is my impression from reading the gas documentation for it.

30 ↗(On Diff #186696)

I do not see this macro as useful, it only obfuscates the reading. And the passing of '.' as 'name' is confusing.

Remove trap_cfi_end and use raw .cfi_endproc
Remove name from trap_cfi_entry
Remove .cfi_startproc from trap_cfi_entry. ENTRY() used in this patch doesn't have .cfi_startproc (the one in machine/asm.h does) but most architectures have only one ENTRY() which has the directive. For consistency between different architectures, omit it from trap_cfi_entry
.cfi_sections .eh_frame should only be emitted once (the original compiled and ran fine, but there's no need to have it under macro)
Added cfi directives to FRED code paths
kthread can return from fork_exit back to trampoline, so handle AST

How do you validate the correctness of the .cfi annotations?

sys/amd64/include/trap_cfi.h
38 ↗(On Diff #186825)

Did you checked this both with clang as and binutils as? I remember I having troubles with non-gprs, in particular segment registers names. See sys/amd64/amd64/sigtramp.S. If some #ifdef 0's can be enabled there, I think it is quite important for userspace debugging.

47 ↗(On Diff #186825)

Might be call this macro 'trap_cfi_trapframe'?

In D59649#1371990, @kib wrote:

How do you validate the correctness of the .cfi annotations?

I ran full kernel build with default buildkernel. I haven't tested gnu toolchain (although as I noted in the reply, current version of gas can correctly interpret non-gpr names except for %rflags). This eliminates having cfi directives outside .cfi_startproc and .cfi_endproc (This happened a few times while I was modifying macros).

So far, I verified identical backtrace and register info with LLDB (CFI) and KGDB (frame sniffer) for fork_trampoline_kthread() and nmi_calltrap. It's very tedious to check validity of every CFI directive because I need to obtain a dump (or dumps) that cover panics from all the relevant locations.

Because LLM is pretty good at repetitive chores, I let it check the kernel binary and see if CFIs are marked as signal frame, directives and disassmebly match, etc. I can manually go over all the symbols if you want, but I don't think llm's hallucination likely happened here.

sys/amd64/include/trap_cfi.h
38 ↗(On Diff #186825)
      .text
      .globl  cfi_segments
      .type   cfi_segments, @function
cfi_segments:
      .cfi_startproc simple
      .cfi_def_cfa    %rsp, 40
      .cfi_offset     %cs, -32
      .cfi_offset     49, -24 /* rflags */
      .cfi_offset     %ss, -8
      ret
      .cfi_endproc
      .size   cfi_segments, . - cfi_segments

      .section .note.GNU-stack,"",@progbits

Both clang 21 and as 2.44 compile this code and emit same result (through llvm-dwarfdump --eh-frame). Both compiles to fail for %rflags instead of 49.

I have not tested on older versions though.

47 ↗(On Diff #186825)

Do you want trap_cfi_entry state=full to be state=trapframe as well?

In D59649#1371990, @kib wrote:

How do you validate the correctness of the .cfi annotations?

I ran full kernel build with default buildkernel. I haven't tested gnu toolchain (although as I noted in the reply, current version of gas can correctly interpret non-gpr names except for %rflags). This eliminates having cfi directives outside .cfi_startproc and .cfi_endproc (This happened a few times while I was modifying macros).

So far, I verified identical backtrace and register info with LLDB (CFI) and KGDB (frame sniffer) for fork_trampoline_kthread() and nmi_calltrap. It's very tedious to check validity of every CFI directive because I need to obtain a dump (or dumps) that cover panics from all the relevant locations.

Because LLM is pretty good at repetitive chores, I let it check the kernel binary and see if CFIs are marked as signal frame, directives and disassmebly match, etc. I can manually go over all the symbols if you want, but I don't think llm's hallucination likely happened here.

I only wanted to understand how you do it. I do not ask about any additional actions.

sys/amd64/include/trap_cfi.h
15 ↗(On Diff #186825)

Do we need cfi info in .eh_frame? It is loaded into the memory at runtime then. AFAIR there is no consumer of that data in kernel. Since it is for benefit of the offline debugger, can it be .debug_frame instead?

And I think that the .cfi_sections directive should go into the .S file directly. It is very surprising to have include file adjust code generation.

47 ↗(On Diff #186825)

Perhaps. Also the default value for the macro arg does not need to be repeated at the use site.

sys/amd64/include/trap_cfi.h
16 ↗(On Diff #186696)

clang drops .cfi_signal_frame for .cfi_sections .dwarf_frame. I don't know if this is intended though.

On top of that, having .eh_frame in stripped kernel still gives correct backtrace when debug symbol file and full kernel are unavailable.

sys/amd64/include/trap_cfi.h
15 ↗(On Diff #186825)

clang drops .cfi_signal_frame for .cfi_sections .dwarf_frame. I don't know if this is intended though. On top of that, having .eh_frame in stripped kernel still gives correct backtrace when debug symbol file and full kernel are unavailable.

  • Move .cfi_sections to assembly files
  • Rename trap_cfi_full to trap_cfi_trapframe
  • Rename trap_cfi_entry's full state to trapframe
  • Add missing CFI directives (e.g. amd64_syscall invalidating r10, r11)
  • Add CFI directives to FRED handlers described in trap.c.
  • fork_trampoline_kthread should run for P_KPROC, not TDP_KTHREAD
  • Place nop after returning from fork_exit so return address sees terminal not trapframe
  • Guard non-gpr regs from clang lower than 17
mchoo marked an inline comment as done.
sys/amd64/amd64/apic_vector.S
52

At least apic_vector.S and atpic_vector.S are not self-contained asm compilation units. They are not compiled alone, but included into exception.S.

sys/amd64/amd64/vm_machdep.c
172

I believe the old test for TDP_KTHREAD was somewhat more correct. Most of the code uses TDP_KTHREAD to detect kernel threads. I am not even sure that the other use of P_KPROC in copy_thread() is entirely correct, we are copying the thread state there.

sys/amd64/include/trap_cfi.h
16 ↗(On Diff #186696)

Do we compile kernel with async exception tables? I remember that we do not, and then standard exception tables are not enough (because we want to unwind across interrupt frames), am I right?

sys/amd64/include/trap_cfi.h
16 ↗(On Diff #186696)

We don't for amd64, but explicit cfi directives still emit FDE in the compiled kernel. The only reason I used .cfi_signal_frame here is to use raw PC.

nop                         /* terminal CFI */
trap_cfi_trapframe
cmpb    $0, fred            /* trapframe CFI */

If exception happens at cmpb and the interrupt handler isn't marked as signal frame, it will calculate pc by subtracting 1 from return address and choose terminal frame. We want trapframe CFI here so unwinder shouldn't subtract 1 from saved PC.

  • No .cfi_sections .eh_frame for ia32_exception.S, apic_vector.S, and atpic_vector.S
  • Restore TDP_KTHREAD
  • fork_trampoline and fork_trampoline_kthread share post fork_exit code. Jump from fork_trampoline_kthread to .fork_trampoline_return after returning from fork_exit
This revision is now accepted and ready to land.Mon, Sep 21, 6:20 PM