Page MenuHomeFreeBSD

Make linuxulator handle ELFOSABI_NONE ELF executables
AbandonedPublic

Authored by trasz on Jun 21 2019, 7:09 PM.

Details

Summary

Make linuxulator able to handle ELFOSABI_NONE ELF executables. Previously it was neccessary to either brand them as Linux using brandelf(1), or by adjusting kern.elf{32,64}.fallback_brand. Binaries that are correctly branded, either in the ELF header or with an ABI note, are not affected.

This can be enabled with 'sysctl compat.linux.exec_unbranded=0'. It fixes ldd(1) from Centos and Ubuntu.

Diff Detail

Repository
rS FreeBSD src repository
Lint
Lint OK
Unit
No Unit Test Coverage
Build Status
Buildable 26356
Build 24820: arc lint + arc unit

Event Timeline

trasz created this revision.Jun 21 2019, 7:09 PM
kib added a comment.Jun 22 2019, 5:39 PM

In which way does it fix ldd, and why ? [As usual, you do not bother to explain]

trasz added a comment.Jun 22 2019, 6:18 PM

Without this commit, Linux ldd is broken - it errors out due to not being able to exec rtld. No idea why rtld is like this, but, well, it is - both in Centos and Ubuntu.

It probably also fixed Android binaries; I hadn’t tested it this time.

kib added a comment.Jun 22 2019, 6:33 PM

Without this commit, Linux ldd is broken - it errors out due to not being able to exec rtld. No idea why rtld is like this, but, well, it is - both in Centos and Ubuntu.
It probably also fixed Android binaries; I hadn’t tested it this time.

So the issue is that shared objects do not specify ABI ? Does it mean that with the change, direct execution mode for our rtld (and then our ldd) is either broken or only works by chance ?

Many many many Linux binaries are ELFOSABI_NONE.

The Elf64_Brandinfo definitions directly above the added lines allow executables with ELFOSABI_NONE to run when interp_path matches. You can see one was added for musl libc, before that one was added, Alpine Linux did not work.
But that's for normal executables.

why rtld is like this

I guess because rtld can't have *itself* set as its interpreter, as that would be an infinite loop.

This is our stuff, but I guess Linux is the same:

% file /bin/ls
/bin/ls: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 13.0 (1300030), FreeBSD-style, stripped
% file /libexec/ld-elf.so.1
/libexec/ld-elf.so.1: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, stripped

I guess an alternative patch would be checking for when we're launching from interp_path when matching these Elf64_Brandinfo things..

trasz added a comment.Jun 24 2019, 8:42 AM
In D20720#448098, @kib wrote:

Without this commit, Linux ldd is broken - it errors out due to not being able to exec rtld. No idea why rtld is like this, but, well, it is - both in Centos and Ubuntu.
It probably also fixed Android binaries; I hadn’t tested it this time.

So the issue is that shared objects do not specify ABI ? Does it mean that with the change, direct execution mode for our rtld (and then our ldd) is either broken or only works by chance ?

No, our rtld and ldd work just fine, because they are marked ELFOSABI_FREEBSD, as reported by "elfdump -e". This change applies to binaries marked ELFOSABI_NONE.

trasz added a comment.Jun 24 2019, 8:54 AM

Many many many Linux binaries are ELFOSABI_NONE.
The Elf64_Brandinfo definitions directly above the added lines allow executables with ELFOSABI_NONE to run when interp_path matches. You can see one was added for musl libc, before that one was added, Alpine Linux did not work.
But that's for normal executables.

But the musl entry still says ELFOSABI_LINUX, doesn't it?

why rtld is like this

I guess because rtld can't have *itself* set as its interpreter, as that would be an infinite loop.

Okay, but why is the rtld binary marked ELFOSABI_NONE instead of ELFOSABI_LINUX,
like other binaries?

This is our stuff, but I guess Linux is the same:

% file /bin/ls
/bin/ls: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 13.0 (1300030), FreeBSD-style, stripped
% file /libexec/ld-elf.so.1
/libexec/ld-elf.so.1: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, stripped

Note that both say "FreeBSD" and not "SYSV".

I guess an alternative patch would be checking for when we're launching from interp_path when matching these Elf64_Brandinfo things..

But that wouldn't work for jails.

Many many many Linux binaries are ELFOSABI_NONE.
The Elf64_Brandinfo definitions directly above the added lines allow executables with ELFOSABI_NONE to run when interp_path matches. You can see one was added for musl libc, before that one was added, Alpine Linux did not work.
But that's for normal executables.

But the musl entry still says ELFOSABI_LINUX, doesn't it?

Yes, that doesn't mean it won't match with NONE.

https://github.com/freebsd/freebsd/blob/fc870a6df90c3876ec348720e21e74beb8b70d92/sys/kern/imgact_elf.c#L378-L395 "Lacking a known brand, search for a recognized interpreter"

why rtld is like this

I guess because rtld can't have *itself* set as its interpreter, as that would be an infinite loop.

Okay, but why is the rtld binary marked ELFOSABI_NONE instead of ELFOSABI_LINUX,
like other binaries?

Others aren't either. In some distros I've tried (including Alpine), *everything* is NONE.

Linux doesn't care about ABI markers, because Linux thinks the whole world is Linux and nothing else deserves to exist :D

trasz added a comment.Jun 24 2019, 8:59 PM

Many many many Linux binaries are ELFOSABI_NONE.
The Elf64_Brandinfo definitions directly above the added lines allow executables with ELFOSABI_NONE to run when interp_path matches. You can see one was added for musl libc, before that one was added, Alpine Linux did not work.
But that's for normal executables.

But the musl entry still says ELFOSABI_LINUX, doesn't it?

Yes, that doesn't mean it won't match with NONE.
https://github.com/freebsd/freebsd/blob/fc870a6df90c3876ec348720e21e74beb8b70d92/sys/kern/imgact_elf.c#L378-L395 "Lacking a known brand, search for a recognized interpreter"

Ah, true.

why rtld is like this

I guess because rtld can't have *itself* set as its interpreter, as that would be an infinite loop.

Okay, but why is the rtld binary marked ELFOSABI_NONE instead of ELFOSABI_LINUX,
like other binaries?

Others aren't either. In some distros I've tried (including Alpine), *everything* is NONE.

Okay, but in CentOS and Ubuntu most of things are marked correctly, the rtld seems
to be only binary there not marked as Linux. Oh well.

Linux doesn't care about ABI markers, because Linux thinks the whole world is Linux and nothing else deserves to exist :D

Still - this patch should fix it for static Alpine binaries and at least fix ldd for others.

trasz added a comment.Jul 4 2019, 6:40 PM

Ping? Does the patch look ok?

kib added a comment.Jul 5 2019, 5:51 PM

Ping? Does the patch look ok?

No, it does not look ok. It lacks analysis of impact of the change on other ABIs.

trasz added a comment.Jul 6 2019, 6:47 AM

Which other ABIs?

kib added a comment.Jul 6 2019, 7:40 PM

Which other ABIs?

Any ABI supported by FreeBSD. Your patch potentially affects selection of the the ABI for any binary without the brand.

trasz added a comment.Jul 7 2019, 11:05 AM

Well, yes, that’s precisely what it’s intended to do.

kib added a comment.Jul 7 2019, 12:17 PM

Well, yes, that’s precisely what it’s intended to do.

It would be even fine if we knew the side effects in advance. I can only repeat, such change should be accompanied by the analysis of effects on other ABIs.

trasz added a comment.Jul 7 2019, 5:07 PM
In D20720#452276, @kib wrote:

Well, yes, that’s precisely what it’s intended to do.

It would be even fine if we knew the side effects in advance. I can only repeat, such change should be accompanied by the analysis of effects on other ABIs.

Okay, but what do you mean by that? Apart from what I've done, which is looking at various binaries in FreeBSD, both base and some packages, to verify they are all properly branded?

kib added a comment.Jul 7 2019, 9:10 PM
In D20720#452276, @kib wrote:

Well, yes, that’s precisely what it’s intended to do.

It would be even fine if we knew the side effects in advance. I can only repeat, such change should be accompanied by the analysis of effects on other ABIs.

Okay, but what do you mean by that? Apart from what I've done, which is looking at various binaries in FreeBSD, both base and some packages, to verify they are all properly branded?

What you did is just a way to start examine the change and see that it does not break things big outright.

What is needed is understanding how does the change affects selection of the ABI for binary, i.e. explanation which binaries suddenly start being classified for different brand.

That said, we already have sysctl kern.elf{32,64}.fallback_brand and brandelf(8).

trasz added a comment.Jul 10 2019, 1:43 PM
In D20720#452395, @kib wrote:
In D20720#452276, @kib wrote:

Well, yes, that’s precisely what it’s intended to do.

It would be even fine if we knew the side effects in advance. I can only repeat, such change should be accompanied by the analysis of effects on other ABIs.

Okay, but what do you mean by that? Apart from what I've done, which is looking at various binaries in FreeBSD, both base and some packages, to verify they are all properly branded?

What you did is just a way to start examine the change and see that it does not break things big outright.
What is needed is understanding how does the change affects selection of the ABI for binary, i.e. explanation which binaries suddenly start being classified for different brand.
That said, we already have sysctl kern.elf{32,64}.fallback_brand and brandelf(8).

Okay. So, it will make Linuxulator execute ELF files marked as OSABI_NONE (also called 'SYSV'), unless they have an ELF note that indicates another ABI.

trasz updated this revision to Diff 59600.Jul 10 2019, 2:04 PM

Remove some cruft.

trasz edited the summary of this revision. (Show Details)Jul 10 2019, 2:07 PM
trasz edited the summary of this revision. (Show Details)
trasz updated this revision to Diff 61580.Sep 3 2019, 12:20 PM
trasz edited the summary of this revision. (Show Details)

Add sysctl to disable the new behaviour.

trasz edited the summary of this revision. (Show Details)Sep 3 2019, 12:21 PM
trasz added a reviewer: emaste.
trasz added a reviewer: brooks.Sep 3 2019, 1:02 PM
trasz edited the summary of this revision. (Show Details)Sep 3 2019, 2:33 PM
kib added a comment.Sep 4 2019, 7:50 AM

You added more people to the review, but was completely silent on both my points:

  • we already have sysctl kern.elf{32,64}.fallback_brand, which can be set by rc.d/linux or rc.d/abi. There is simply no need to disturb the brands selection process and add more knobs to it.
  • I pointed out that you in effect add a wildcard for matching, which consequences needs to be analyzed before applying, but you did not.

So I object against this fiddling, suggest adding some code to rc.d to set fallback brand.

trasz added a comment.EditedSep 4 2019, 10:51 AM

It's hard to set the sysctl from a script (there's no way to figure out what value it should be set to), but generally: doing that would be working around a bug instead of fixing it. We should ship software that works correctly by default, without the need for twiddling some options. Currently we don't: we ship broken ELF activator that can't recognize common Linux executables.

And sure I have analyzed the consequences: there's a description above. The impact is minimal: it doesn't affect systems without linuxulator loaded, and even in rare cases where someone, for some reason, had an ancient, unbranded FreeBSD executable it can be worked around by tweaking the sysctl or by using brandelf(1).

trasz updated this revision to Diff 61837.Sep 9 2019, 12:14 PM
trasz edited the summary of this revision. (Show Details)

Default to disabled, add description in the man page.

Based on feedback from Konstantin, it's too risky to make it enabled
by default. Based on discussion with Brooks, defaulting to enable _could_
have made sense for Linux jails - which are not a thing right now. And
when they do, this could be turned into a jail option.

trasz edited the summary of this revision. (Show Details)Sep 11 2019, 11:35 AM
trasz abandoned this revision.Sep 11 2019, 5:08 PM

Sigh.