Page MenuHomeFreeBSD
Files F173472941

D60034.id187719.diff
No OneTemporary
Actions

D60034.id187719.diff
View Options

diff --git a/documentation/content/en/books/developers-handbook/kerneldebug/_index.adoc b/documentation/content/en/books/developers-handbook/kerneldebug/_index.adoc
--- a/documentation/content/en/books/developers-handbook/kerneldebug/_index.adoc
+++ b/documentation/content/en/books/developers-handbook/kerneldebug/_index.adoc
@@ -4,10 +4,11 @@
- author: Paul Richards
- author: Jörg Wunsch
- author: Robert Watson
+ - author: Minsoo Choo
prev: books/developers-handbook/kernelbuild
next: books/developers-handbook/partiv
description: FreeBSD Kernel Debugging
-tags: ["Debugging", "Dump", "kgdb", "DDB", "GDB"]
+tags: ["Debugging", "Dump", "LLDB", "kgdb", "DDB", "GDB"]
showBookMenu: true
weight: 13
params:
@@ -145,6 +146,12 @@
Also, do not forget to specify the correct path to your swap device as it is likely different than [.filename]#/dev/ad0s1b#!
====
+[TIP]
+====
+If your system is crashing regularly and you are running out of disk space,
+deleting old [.filename]#vmcore# files in [.filename]#/var/crash# can free a considerable amount of disk space.
+====
+
=== Testing Kernel Dump Configuration
The kernel includes a man:sysctl[8] node that requests a kernel panic.
@@ -164,8 +171,147 @@
After rebooting, your system should save a dump in [.filename]#/var/crash# along with a matching summary from man:crashinfo[8].
+[[kerneldebug-crashdump]]
+== Debugging a Kernel Crash Dump
+
+FreeBSD kernel crash dumps can be debugged with man:lldb[1] or man:kgdb[1].
+Both are actively supported by FreeBSD developers.
+For LLDB issues, open an issue in the LLVM GitHub repository and notify @mchoo7 and @aokblast.
+For KGDB issues, file a report in FreeBSD Bugzilla for the package:devel/gdb[] port.
+
+.Comparison of LLDB and KGDB
+[cols="2,3,3", options="header"]
+|===
+| Feature
+| LLDB
+| KGDB
+
+| Availability
+| Included in the base system; also available through package:devel/llvm-24[].
+| Available through package:devel/gdb[].
+
+| License
+| Apache License 2.0 with LLVM Exception.
+| GPLv3.
+
+| Supported platforms
+| All platforms supported by FreeBSD except 32-bit PowerPC and big-endian 64-bit PowerPC. Trapframe unwinding is unavailable on i386.
+| All platforms supported by FreeBSD.
+
+| Supported FreeBSD versions
+| FreeBSD 14 and later. Trapframe unwinding requires FreeBSD 15.1 or later, or FreeBSD 14.6 or later on stable/14.
+| All FreeBSD versions, including some end-of-life releases.
+
+| Helper scripts
+| None provided in the FreeBSD repository.
+| Available under [.filename]#sys/tools/gdb#.
+|===
+
+The following subsections show how to obtain a backtrace and locate the relevant source code.
+Most developers will ask for at least this information if you cannot debug the problem yourself.
+If you solve the problem, submit your patch for inclusion in the source tree through a problem report or mailing list, or commit it if you have access.
+
+[[kerneldebug-lldb]]
+=== Kernel Crash Dump Debugging Using LLDB
+
+[IMPORTANT]
+====
+LLDB 24 or later is required for full FreeBSD kernel debugging support.
+Earlier versions support kernel debugging only partially.
+====
+
+To examine the dump, start LLDB:
+
+[source,bash]
+....
+# lldb /boot/kernel/kernel -c /var/crash/vmcore.0
+....
+
+You can debug the crash dump using the kernel sources as you would for any other program.
+
+This dump is from a 16-CURRENT kernel, and the crash occurred deep within the kernel.
+After obtaining a back trace and register information, the `list` command is used to print relevant source code location.
+
+[source,bash]
+....
+# lldb /boot/kernel/kernel -c /var/crash/vmcore.0
+(lldb) target create "/boot/kernel/kernel" --core "/var/crash/vmcore.0"
+
+Unread portion of the kernel message buffer:
+panic: kdb_sysctl_panic
+cpuid = 13
+time = 1790378347
+KDB: stack backtrace:
+db_trace_self_wrapper() at db_trace_self_wrapper+0x36/frame 0xfffffe027e3039d0
+vpanic() at vpanic+0x149/frame 0xfffffe027e303b00
+panic() at panic+0x43/frame 0xfffffe027e303b60
+kdb_sysctl_panic() at kdb_sysctl_panic+0x7e/frame 0xfffffe027e303ba0
+sysctl_root_handler_locked() at sysctl_root_handler_locked+0xa7/frame 0xfffffe027e303bf0
+sysctl_root() at sysctl_root+0x24b/frame 0xfffffe027e303c80
+userland_sysctl() at userland_sysctl+0x1c1/frame 0xfffffe027e303d30
+sys___sysctl() at sys___sysctl+0x86/frame 0xfffffe027e303df0
+amd64_syscall() at amd64_syscall+0x17c/frame 0xfffffe027e303f30
+fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe027e303f30
+--- syscall (202, FreeBSD ELF64, __sysctl), rip = 0x3007cba8fe1a, rsp = 0x3007c8b62d08, rbp = 0x3007c8b62d50 ---
+KDB: enter: panic
+
+Core file '/var/crash/vmcore.0' (x86_64) was loaded.
+(lldb) bt
+* thread #950, name = '(pid 5465) sysctl (crashed)', stop reason = kernel panic
+ * frame #0: 0xffffffff80c50951 kernel`__curthread at pcpu_aux.h:57:2 [inlined]
+ frame #1: 0xffffffff80c50951 kernel`doadump(textdump=0) at kern_shutdown.c:399:2
+ frame #2: 0xffffffff804cb05a kernel`db_dump(dummy=<unavailable>, dummy2=<unavailable>, dummy3=<unavailable>, dummy4=<unavailable>) at db_command.c:596:10
+ frame #3: 0xffffffff804caf0a kernel`db_command(last_cmdp=<unavailable>, cmd_table=<unavailable>, dopager=true) at db_command.c:508:3
+ frame #4: 0xffffffff804cab2d kernel`db_command_loop at db_command.c:555:3
+ frame #5: 0xffffffff804ce760 kernel`db_trap(type=<unavailable>, code=<unavailable>) at db_main.c:267:3
+ frame #6: 0xffffffff80ca717f kernel`kdb_trap(type=3, code=0, tf=0xfffffe027e303910) at subr_kdb.c:790:13
+ frame #7: 0xffffffff811ee595 kernel`trap(frame=<unavailable>) at trap.c:702:8
+ frame #8: 0xffffffff811bbea8 kernel`calltrap at exception.S:381
+ frame #9: 0xffffffff80ca6683 kernel`kdb_enter(why=<unavailable>, msg=<unavailable>) at subr_kdb.c:556:11
+ frame #10: 0xffffffff80c50b99 kernel`vpanic(fmt=<unavailable>, ap=0xfffffe027e303b40) at kern_shutdown.c:0
+ frame #11: 0xffffffff80c50a13 kernel`panic(fmt=<unavailable>) at kern_shutdown.c:887:2
+ frame #12: 0xffffffff80ca762e kernel`kdb_sysctl_panic(oidp=0xffffffff81ae2900, arg1=<unavailable>, arg2=<unavailable>, req=0xfffffe027e303ca0) at subr_kdb.c:219:2
+ frame #13: 0xffffffff80c62e37 kernel`sysctl_root_handler_locked(oid=0xffffffff81ae2900, arg1=0x0000000000000000, arg2=0, req=0xfffffe027e303ca0, tracker=0xfffffe027e303c10) at kern_sysctl.c:199:10
+ frame #14: 0xffffffff80c6203b kernel`sysctl_root(oidp=<unavailable>, arg1=0x0000000000000000, arg2=0, req=0xfffffe027e303ca0) at kern_sysctl.c:2427:10
+ frame #15: 0xffffffff80c62821 kernel`userland_sysctl(td=0xfffff8015eabd780, name=0xfffffe027e303d70, namelen=<unavailable>, old=<unavailable>, oldlenp=<unavailable>, inkernel=0, new=0x000044ccc3009000, newlen=4, retval=0xfffffe027e303d68, flags=0) at kern_sysctl.c:2586:11
+ frame #16: 0xffffffff80c62626 kernel`sys___sysctl(td=0xfffff8015eabd780, uap=0xfffff8015eabdba8) at kern_sysctl.c:2457:10
+ frame #17: 0xffffffff811ef64c kernel`syscallenter(td=0xfffff8015eabd780) at subr_syscall.c:193:11 [inlined]
+ frame #18: 0xffffffff811ef509 kernel`amd64_syscall(td=0xfffff8015eabd780, traced=0) at trap.c:1278:2
+ frame #19: 0xffffffff811bc79b kernel`fast_syscall_common at exception.S:810
+ frame #20: 0x00003007cba8fe1a
+(lldb) register read
+General Purpose Registers:
+ rbx = 0x0000000000000000
+ rbp = 0xfffffe027e303560
+ rsp = 0xfffffe027e303548
+ r12 = 0x0000000000000004
+ r13 = 0xffffffff81a18640 ddb_dbbe
+ r14 = 0xffffffff81a17ae8 db_cmds + 616
+ r15 = 0x0000000000000000
+ rip = 0xffffffff80c50951 kernel`doadump + 49 [inlined] __curthread at pcpu_aux.h:57:2
+ kernel`doadump + 49 at kern_shutdown.c:399:2
+ cs = 0x0000000000000020
+ ss = 0x0000000000000028
+14 registers were unavailable.
+
+(lldb) source list -f /usr/src/sys/kern/kern_shutdown.c -l 399
+ 399 dump_savectx();
+ 400 dumping++;
+ 401
+ 402 coredump = TRUE;
+ 403 #ifdef DDB
+ 404 if (textdump && textdump_pending) {
+ 405 coredump = FALSE;
+ 406 textdump_dumpsys(TAILQ_FIRST(&dumper_configs));
+ 407 }
+ 408 #endif
+ 409 if (coredump) {
+(lldb) quit
+Quitting LLDB will kill one or more processes. Do you really want to proceed: [Y/n] y
+....
+
[[kerneldebug-gdb]]
-== Debugging a Kernel Crash Dump with `kgdb`
+=== Kernel Crash Dump Debugging Using KGDB
[NOTE]
====
@@ -202,8 +348,6 @@
The output below has been modified to include line numbers on the left.
This first trace inspects the instruction pointer and obtains a back trace.
The address that is used on line 41 for the `list` command is the instruction pointer and can be found on line 17.
-Most developers will request having at least this information sent to them if you are unable to debug the problem yourself.
-If, however, you do solve the problem, make sure that your patch winds its way into the source tree via a problem report, mailing lists, or by being able to commit it!
[source,bash]
....
@@ -299,12 +443,6 @@
90:(kgdb) quit
....
-[TIP]
-====
-If your system is crashing regularly and you are running out of disk space,
-deleting old [.filename]#vmcore# files in [.filename]#/var/crash# could save a considerable amount of disk space!
-====
-
[[kerneldebug-online-ddb]]
== On-Line Kernel Debugging Using DDB

File Metadata

Mime Type
text/plain
Expires
Sun, Sep 27, 4:57 AM (8 h, 55 m)
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
39704589
Default Alt Text
D60034.id187719.diff (9 KB)

Event Timeline