kvm: introduce kvm_convert_to_elf()

[mchoo]

Summary

Since debugging kernel crash dumps requires libkvm, cross-debugging them is
almost impossible. This function converts crash dump to elf core format so
they can be debugged on any platforms where debuggers support elf core
format.

Signed-off-by: Minsoo Choo <[e-mail]>
Sponsored by: The FreeBSD Foundation
Co-authored-by: Bora Ozarslan <[e-mail]>

Test Plan

With D56256, run

$ sudo LD_LIBRARY_PATH=/usr/obj/usr/src/freebsd/main/arm64.aarch64/lib/libkvm ./dump2elf -k /boot/kernel/kernel -c /var/crash/vmcore.last ~/vmcore.elf
# or
$ sudo LD_LIBRARY_PATH=/usr/obj/usr/src/freebsd/main/arm64.aarch64/lib/libkvm ./dump2elf -r -k /boot/kernel/kernel -c /var/crash/vmcore.last ~/vmcore.elf.raw

The result shows:

$ ls -lh /var/crash/vmcore.3
-rw-------  1 root wheel   13G Apr 15 03:17 /var/crash/vmcore.3

$ ls -lh vmcore.elf
-rw-r--r--  1 root minsoochoo  1.1G Apr 20 06:26 vmcore.elf

[mchoo]

This is improvement for D19253, but I moved this to libkvm so utilities like savecore(8) can invoke this easily.

Should I open this to fulldump as well and thus rename it to kvm_convert_to_elf()?

The reason I'm asking this is because I'm planning to copy .gnu_build_id from kernel to elf core so debuggers can reject it if UUID doesn't match. I can implement it for fulldump while kernel is dumping, but it seems redundant to implement it again in libkvm for minidump.

lib/libkvm/kvm_minidump_to_elf.3

19 ↗

(On Diff #174900)

Is there a better name for apply_kerndisp?

137 ↗

(On Diff #174900)

what is a good macro for libkvm?

Also should "improvements" be singular or plural? (I'm not good at English grammar and not sure if improvement can be plural in formal writing)

[mchoo]

Add coauthorship

[mchoo]

update copyright notice

[mchoo]

The point of this feature is to facilitate cross-platform debugging, thus this needs to accept full dumps as well. As a result, the name has changed from kvm_minidump_to_elf() to kvm_convert_to_elf(). Live kernel (/dev/mem) is still unsupported since kvm_walk_pages() doesn't work for that.

[mchoo]

update commit message

[jhb]

Hmmm, cross debugging of cross-architectures does work on a FreeBSD host? However, this isn't really a full ELF core dump. You would need to extract register values from the PCBs and store those as register set notes if you want debuggers to really work, and that is kind of complicated anyway as kthreads only include a subset of registers in the pcb unlike userspace core dumps, so in reality NT_PRTSTATUS for core dumps just needs to look different (use a different struct reg layout in essence). This is part of why I really just want to implement this in the kernel directly instead of trying to post-process. libkvm itself is not in a position to find pcb's, for that you need the debug symbols of the kernel and you need to walk the allproc list, etc.

[mchoo]

In D56250#1295915, @jhb wrote:

Hmmm, cross debugging of cross-architectures does work on a FreeBSD host?

Oh, I meant debugging across different operating systems (e.g. FreeBSD arm64 dump on Linux amd64 LLDB). This was something we wanted since the foundation sponsored LLDB's kernel debugging project, but the approach at that time was to port libkvm to other OSes (libfbsdvmcore). The end result was portable libkvm, but the issue was that most distros didn't ship LLDB with that third-party library, hence debugging FreeBSD crash dump was impossible without building LLDB from source with that library included. Also libfbsdvmcore had to catch up with libkvm, but it has been unmaintained for years.

In D56250#1295915, @jhb wrote:

However, this isn't really a full ELF core dump. You would need to extract register values from the PCBs and store those as register set notes if you want debuggers to really work, and that is kind of complicated anyway as kthreads only include a subset of registers in the pcb unlike userspace core dumps, so in reality NT_PRTSTATUS for core dumps just needs to look different (use a different struct reg layout in essence). This is part of why I really just want to implement this in the kernel directly instead of trying to post-process. libkvm itself is not in a position to find pcb's, for that you need the debug symbols of the kernel and you need to walk the allproc list, etc.

My intent here was to have memory dump without NT_PR* notes, basically an alternative to libkvm. In this way, LLDB can reuse existing mechanism of finding offsets and iterating allproc, etc and the only modification needed is updating Process::DoLoadCore(), Process::DoMemoryRead(), and Process::DoMemoryWrite() for ELF core format.

But as you said, this is different from most ELF core format with NT_PR*. If kernel wants to save dump as ELF on crash, it would require elf(3)-ish code in kernel binary. Then it needs to save the dump somewhere, which is dumpdev, but fulldump/minidump is already stored there. Also libkvm needs be updated to accept ELF core dump so it can read it after savecore(8).

What was the reason for creating minidump format if ELF core can serve the same purpose?

Edit: I prefer having ELF core dump on crash which replaces fulldump/minidump. But again, there might be reason why developers created minidump format instead of existing ELF core format.

[jhb]

In D56250#1296293, @minsoochoo0122_proton.me wrote:

In D56250#1295915, @jhb wrote:

Hmmm, cross debugging of cross-architectures does work on a FreeBSD host?

Oh, I meant debugging across different operating systems (e.g. FreeBSD arm64 dump on Linux amd64 LLDB). This was something we wanted since the foundation sponsored LLDB's kernel debugging project, but the approach at that time was to port libkvm to other OSes (libfbsdvmcore). The end result was portable libkvm, but the issue was that most distros didn't ship LLDB with that third-party library, hence debugging FreeBSD crash dump was impossible without building LLDB from source with that library included. Also libfbsdvmcore had to catch up with libkvm, but it has been unmaintained for years.

Yes, in order for this to really work, FreeBSD would have needed to treat libkvm as a contrib piece of software really.

In D56250#1295915, @jhb wrote:

However, this isn't really a full ELF core dump. You would need to extract register values from the PCBs and store those as register set notes if you want debuggers to really work, and that is kind of complicated anyway as kthreads only include a subset of registers in the pcb unlike userspace core dumps, so in reality NT_PRTSTATUS for core dumps just needs to look different (use a different struct reg layout in essence). This is part of why I really just want to implement this in the kernel directly instead of trying to post-process. libkvm itself is not in a position to find pcb's, for that you need the debug symbols of the kernel and you need to walk the allproc list, etc.

My intent here was to have memory dump without NT_PR* notes, basically an alternative to libkvm. In this way, LLDB can reuse existing mechanism of finding offsets and iterating allproc, etc and the only modification needed is updating Process::DoLoadCore(), Process::DoMemoryRead(), and Process::DoMemoryWrite() for ELF core format.

But as you said, this is different from most ELF core format with NT_PR*. If kernel wants to save dump as ELF on crash, it would require elf(3)-ish code in kernel binary. Then it needs to save the dump somewhere, which is dumpdev, but fulldump/minidump is already stored there. Also libkvm needs be updated to accept ELF core dump so it can read it after savecore(8).

Yes dumping native ELF requires quite a few changes but ELF is a more extensible format. It is true that there is value in not having the notes yet, though mostly what I want it for is just to evaluate different possible extensions we might use to come up with a real ELF format. That is, do we want a special program header describing, e.g. the DMAP region rather than a bunch of sparse PT_LOADs. The thing I'm not sure of is how many phdrs will the kernel have to write out for different choices. For example, do you dump "physical memory" (so PT_LOADs are only paddrs, no vaddrs) and then store some metadata in an arch-independent manner (rather than today where minidumps store arch-specific PTEs) that GDB/LLDB could easily parse. Or do you try to use PT_LOADs to describe the virtual address space. The guts of this review are probably quite useful for having a test tool to prototype various layouts before settling on a final design, but I probably wouldn't want to treat any of the intermediary designs as a production thing we'd want to support as they will all need some sort of handling in the debugger eventually if they are going to be efficient (I think).

What was the reason for creating minidump format if ELF core can serve the same purpose?

Minidumps were a quick hack. Full dumps prior to minidumps were/are in fact ELF cores, albeit without any notes, just PT_LOAD's for physical (not virtual(!)) memory.

Edit: I prefer having ELF core dump on crash which replaces fulldump/minidump. But again, there might be reason why developers created minidump format instead of existing ELF core format.

No principled reason, Peter just wrote up an initial prototype and once it was working didn't revisit it. However, it is painful to extend, whereas ELF is extensible with new note types and/or phdrs, etc.

[mchoo]

Right, it makes more sense to generate ELF core directly from kernel. I'll implement ELF core on kernel side then, after reorganizing code under kern_dump.c so it can adopt a third variant. Thank you for your feedback:)