USES=lib32: add support for lib32- companion ports

[dbn]

Summary

Many Linux distributions ship two flavours of a port on a amd64 systems:

1. A native 64-bit port, and
2. The 32-bit libraries (cross-compiled)

We haven't had the need for such ports, except for one major exception: Wine. Wine requires the 32-bit libraries in order to run 32-bit Windows binaries on FreeBSD/amd64. Up until now Wine ships those 32-bit libraries internally. This situation is less ideal for many reasons:

1. The libraries are provided "pre-compiled" - independent of the Ports building infrastructure
2. The libraries only get updated when wine itself is updated
3. Building locally is a non-trivial undertaking

By introducing this USES (and adapting the ports required by Wine) we will be able to build Wine (native, i386 and wow64 [a new addition]) without any issue.

[dbn]

Add missing Mk/Uses/lib32.mk file.

[theron.tarigo_gmail.com]

In D14721#358258, @dbn wrote:

Thank you all for the useful ideas. I think the first price approach is to create 'lib32-' companion packages for the 32-bit libraries we need. To this end I have created a patch that adds this functionality: https://reviews.freebsd.org/D16830. Luckily the approach is minimally invasive and often involves:

1. Adding lib32 to USES
2. Sometimes replacing uses of ${PREFIX}/lib} with ${LIBDIR}, however lib32 already handles CFLAGS, CPPFLAGS, and CONFIGURE_ARGS (this can be extended as needed)
3. Sometimes adding USE_LDCONFIG=yes

If anyone has some time to convent and test the other library dependencies of Wine that will be a great help :-).

I could try to adapt my working solution to use your conventions, since I encountered but fixed various problems which lib32.mk as-is does not seem to address.

Although use of the lib32 name follows a well-known precedent, I suggest, for use in ports, a more specific name: i386-on-amd64 or similar.

[mat]

I do not think FLAVORS is the right way to make this work. Slave ports would be a better approach.

Mk/Uses/lib32.mk

34

DEV_ERROR?

58

If you write it as:

PKGNAMEPREFIX:=  lib32-${PKGNAMEPREFIX}

You can remove PKGNAMEPREFIX from the incompatible variables above.

[theron.tarigo_gmail.com]

Yes, using FLAVORS does not seem quite right. What if one of the dependencies already uses FLAVORS? Regardless of whether this presents a real conflict, FLAVORS is intended for handling varying configurations of ports themselves, not for selecting the manner in which ports are built and installed. The only reason use of FLAVORS is helpful in the present case is that it is an already established way in which one port can generate multiple packages.

A few questions:

1. Where would the lib32- slave ports live: should these clutter the category directories of their masters, or live in a new directory?
2. Is it acceptable that the amd64 users that have no use for lib32 will see two entries instead of one for any USES=lib32 library when performing a pkg search related to the library?

[theron.tarigo_gmail.com]

Although amd64 and i386 are Tier-1 and freebsd32 compat is a part of base, adding support for lib32 in ports has similarities to porting FreeBSD Ports collection to a new architecture. What is different in this case is that the "new architecture" would be a "slave architecture" to amd64.

An idea is as follows:

• Building lib32- packages is performed in exactly the same manner as cross-compiling the ports collection for any other non-amd64 architecture.
• The lib32- prefix is applied to all package names, allowing these packages to not conflict with any native packages.
• The packages are hosted under a new amd64-lib32 ABI name on the package servers.
• A user of lib32 adds this source as an additional repository in /etc/pkg/FreeBSD.conf. Use of this repository would be an installation option provided within the amd64 installer image. Therefore, native-only amd64 users are not affected by any additional clutter.
• NOT_FOR_ARCHS+=amd64-lib32 would be on by default (set somewhere in ports/Mk/*), but overridden in ports as needed.

Would this be a very bad idea, or is it something to consider?

[dbn]

Switch to using a slave port.

[dbn]

Mk/Uses/lib32.mk

34

After your suggestion, this is no longer needed.

58

Nice suggestion, implemented :-)

[dbn]

I think taking the simplest approach of slave ports for the libraries wine needs is the easiest. Are you thinking about any use case beyond wine?

[iwtcex_gmail.com]

Ok, this looks quite neat.

1. It probably worth making lib32- ports dependent on their master ports so they wouldn't go out of sync with headers.
2. Pkgconf configs should also be handled.
3. I noticed ldconfig files contain references to lib rather than lib32 directory, that doesn't look right.

[iwtcex_gmail.com]

I've also made an attempt to modify some ports according to the initial variant of this patch (i.e., flavors). I will update my diff in a few days, but where exactly should I put proposed additions and/or changes? Not clear on the workflow.

[mat]

Mk/Uses/lib32.mk

74–80

I am unsure what this does. Having USE_LDCONFIG32 set should already be doing the right thing, no?

[theron.tarigo_gmail.com]

Adding so many slave ports seems very messy. This would introduce many architecture-specific slaves to ports which are otherwise not at all architecture-specific in most cases. Fully-functional Wine requires perhaps more libraries than you anticipate: depending on graphics vendor, all of Mesa, and therefore LLVM as well, must be compiled for i386. At least some i386 LibGL is needed for full functionality.^1

Furthermore, i386-Wine on amd64 is not the only use-case to consider. What about QEMU user emulation?

So, to avoid adding special amd64-lib32 slave ports, but to get what we want anyway, we need two things:

1. Support from ports framework mkfiles to build a port and its dependencies under a user-specified PREFIX and ARCH (see below).
2. Support from Poudriere to correctly package any ports which have been built in this manner.

Earlier, I made a similar proposal to what follows, but it was specific to amd64, whereas this would be general:
Each architecture could have a -foreign variant: the existence of i386 arch would mean that implicitly there is also an i386-foreign, which is like i386 in every way, but installs files only to /usr/local/i386/. This is already possible with only minor additions to ports framework. Any -foreign architecture package would be installable on any system, since the paths would not conflict and emulation would be the primary purpose of these packages. A user of the ports tree may request building of the foreign variants by specifying a Make variable; a user of binary packages may add the appropriate foreign-arch repository to /etc/pkg/FreeBSD.conf.

1. A workaround would be to use some kind of indirect-rendering with inter-process communication, but this is new territory.

[iwtcex_gmail.com]

In D16830#372919, @theron.tarigo_gmail.com wrote:

Adding so many slave ports

Around 50 or so (not counting optional dependencies). Slightly more than that with Mesa drivers I suppose.

What about QEMU user emulation?

Jail.

Each architecture could have a -foreign variant: the existence of i386 arch would mean that implicitly there is also an i386-foreign, which is like i386 in every way, but installs files only to /usr/local/i386/. This is already possible with only minor additions to ports framework. Any -foreign architecture package would be installable on any system, since the paths would not conflict and emulation would be the primary purpose of these packages. A user of the ports tree may request building of the foreign variants by specifying a Make variable; a user of binary packages may add the appropriate foreign-arch repository to /etc/pkg/FreeBSD.conf.

Some of us [FreeBSD users] specifically dislike Debian and their way of doing things. Just so you know. This proposal should be discussed on the appropriate mailing list (or maybe IRC channel, I'm not sure).

@dbn Can we get lib32 effort unstuck somehow?

[theron.tarigo_gmail.com]

Some of us [FreeBSD users] specifically dislike Debian and their way of doing things. Just so you know. This proposal should be discussed on the appropriate mailing list (or maybe IRC channel, I'm not sure).

It is not any sort of attempt to follow Debian. After a little more research, I see how my proposal could be seen as being similar to Debian's approach to this particular situation, which raises skepticism. I will bring it up on hackers@ once I am more prepared.

My concern about adding many slave ports, which are essentially all identical except for masters' names (All the ports work on i386 already, so no port-specific changes should be needed), still stands. Isn't there some better way? Aren't these slave ports just a hack to make sure Poudriere knows to build the packages?

[iwtcex_gmail.com]

In D16830#372943, @theron.tarigo_gmail.com wrote:

My concern about adding many slave ports, which are essentially all identical except for masters' names

I agree they are annoying, but they do not require much maintenance. They can also be easily removed once you manage to bikeshed some better solution.

Isn't there some better way?

Likely yes, though nothing quite as easy. Have you seen Ravenports?

[iwtcex_gmail.com]

Any plans on doing something with this in 2019?

[iwtcex_gmail.com]

2019?

Ok, this looks as good as dead. Was this proposal discussed elsewhere and put on hold? Anything else?

[iwtcex_gmail.com]

In D16830#358878, @iwtcex_gmail.com wrote:

made an attempt … according to the initial variant of this patch

Since I didn't receive any response in 11 months, I'm dropping additional 50 lib32- ports as well as some bugfixes to lib32.mk into a GitHub repo. This includes necessary and a few optional Wine dependencies with exception of Mesa drivers (I can't test them with Nvidia card) and FFmpeg. This is, of course, only lightly tested, although I do now build WoW64 Wine that way.

[theron.tarigo_gmail.com]

This comment concerns Mesa drivers, and libGL in general, but not much else:

In D16830#455455, @iwtcex_gmail.com wrote:

with exception of Mesa drivers (I can't test them with Nvidia card)

Using LIBGL_ALWAYS_SOFTWARE=1 after an installation to a chroot (so that you don't trash your host's Nvidia installation) should be sufficient for performing the test.
However, Wine should work for Nvidia users as well...

How are libGL dependencies handled right now for other ports? After some investigation: /usr/ports/Mk/Uses/gl.mk has hardcoded dependencies on graphics/mesa-libs - ideally this can be fixed, but as part of some other review.

Right now, do you let mesa-libs install, and then install nvidia-driver after that? I have Intel-display+Nvidia-headless hybrid graphics laptop, so the configuration is a bit different for me.
Nvidia-only users should not be forced to accept the bloat, either with Wine or with any other ports... AMD Radeon users may be next, now with Valve working on a new shader compiler.

For purposes of this review, I think it is best to not change any behavior with respect to libGL from the current situation with i386-wine binary-port, even though it is not the best situation for us Nvidia users.

[iwtcex_gmail.com]

Oh, hello Theron.

In D16830#455463, @theron.tarigo_gmail.com wrote:

How are libGL dependencies handled right now for other ports?

No idea. They likely depend on Mesa and that's it.

Right now, do you let mesa-libs install, and then install nvidia-driver after that? I have Intel-display+Nvidia-headless hybrid graphics laptop, so the configuration is a bit different for me.

You know this quite well, currently libGL.so from nvidia-driver is renamed to libGL-NVIDIA.so in order to not conflict with Mesa's libGL.so. /usr/local/etc/libmap.d/nvidia.conf bundled with nvidia-driver forces applications to use Nvidia's OpenGL implementation. Ultimately this should be handled by libglvnd.

Using LIBGL_ALWAYS_SOFTWARE=1 after an installation to a chroot (so that you don't trash your host's Nvidia installation) should be sufficient for performing the test.

env LD_LIBMAP="libGL.so.1=libGL.so.1.2.0" LIBGL_ALWAYS_SOFTWARE=1 glxinfo

actually fails for me with

libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast

I don't expect chroot to be any different.

For purposes of this review…

For purposes of this review it would nice to get any review. Doesn't look like it is coming anytime soon with @dbn being pretty much absent and @gerald leaving zero comments there.

[theron.tarigo_gmail.com]

In D16830#455465, @iwtcex_gmail.com wrote:

In D16830#455463, @theron.tarigo_gmail.com wrote:

How are libGL dependencies handled right now for other ports?

No idea. They likely depend on Mesa and that's it.

I answered my own question here:

After some investigation: /usr/ports/Mk/Uses/gl.mk has hardcoded dependencies on graphics/mesa-libs - ideally this can be fixed, but as part of some other review.

Right now, do you let mesa-libs install, and then install nvidia-driver after that? I have Intel-display+Nvidia-headless hybrid graphics laptop, so the configuration is a bit different for me.

You know this quite well, currently libGL.so from nvidia-driver is renamed to libGL-NVIDIA.so in order to not conflict with Mesa's libGL.so. /usr/local/etc/libmap.d/nvidia.conf bundled with nvidia-driver forces applications to use Nvidia's OpenGL implementation. Ultimately this should be handled by libglvnd.

Well, it's been a while since I've messed with this, thanks for the explanation.

env LD_LIBMAP="libGL.so.1=libGL.so.1.2.0" LIBGL_ALWAYS_SOFTWARE=1 glxinfo

actually fails for me with

libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast

I don't expect chroot to be any different.

Oh, I think the problem then is Nvidia's GLX extension overriding the plain Xorg one, I had forgotten about that. In that case, if it is not too much of a hassle for you and you would like to test this, you could set up an alternate Xorg configuration:

mkdir -p /usr/local/lib/xorg/modules-mesa/extensions/
cp /usr/local/lib/xorg/modules/extensions/.xorg/libglx.so /usr/local/lib/xorg/modules-mesa/extensions/libglx.so

And use

Section "Files"
    ModulePath      "/usr/local/lib/xorg/modules-mesa"
    ModulePath      "/usr/local/lib/xorg/modules"
EndSection

as /usr/local/etc/X11/xorg-mesa.conf.d/glx.conf
Now,

Xorg :1 -configdir /usr/local/etc/X11/xorg-mesa.conf.d

should start a new Xorg; Nvidia GLX won't work at all in it, but Mesa libGL software rendering should.

In general, 32-bit application using Nvidia acceleration on FreeBSD would in theory be possible by installing the userspace components of the 32-bit driver download, but this has never been part of the nvidia-driver port and Nvidia dropped this support in 400 series.
Usability of GL/D3D in Wine32 under Mesa (what we have now) needs to keep working, that is why it should be tested.

For purposes of this review it would nice to get any review. Doesn't look like it is coming anytime soon with @dbn being pretty much absent and @gerald leaving zero comments there.

I might have time to look at it this weekend. I'm not in any position to help get it accepted, but I could at least try to investigate what issues might remain.

[iwtcex_gmail.com]

In D16830#455750, @theron.tarigo_gmail.com wrote:

the userspace components of the 32-bit driver download, … and Nvidia dropped this support in 400 series.

Indeed. That's not the end of it, though.

Usability of GL/D3D in Wine32 under Mesa (what we have now) needs to keep working, that is why it should be tested.

Ok, I added lib32-mesa-dri.

I might have time to look at it this weekend. I'm not in any position to help get it accepted, but I could at least try to investigate what issues might remain.

Fair enough. I've yet to do some sanity checks to verify that Wine is actually buildable outside of my desktop machine environment, but I'll get to it.

[iwtcex_gmail.com]

In D16830#455859, @iwtcex_gmail.com wrote:

I've yet to do some sanity checks to verify that Wine is actually buildable

Should be in reasonable shape now.

[gerald]

I noticed that I was (still) listed as reviewer and want to clarify that this is way outside my comfort zone, both technically, in terms of what I can verify/test, and on top of those time plus what I could approve.

Looking at the patch this looks like something for portmgr, who I'll drop a note to in addition to seeing them as reviewer already.

When (and hopefully it's when, not if) this gets in as far as Wine goes, I see three options:

1. Someone provides a patch for the Wine ports that dbn@ and/or me maintain, and if those are simple enough for me to understand I'll be happy to take them for "my" ports and hope dbn@ does the same for "his".
2. This can be done without touching emulators/wine-devel and emulators/wine, which is a subset of option 1. If dbn@ does not have the time any more, perhaps someone else wants to step up?
3. This is getting invasive as far as emulators/wine-devel and emulators/wine go, in which case I am open towards handing over those ports to someone with the skillset and time at hand.

[iwtcex_gmail.com]

In D16830#457760, @gerald wrote:

I noticed that I was (still) listed as reviewer and want to clarify that this is way outside my comfort zone, both technically, in terms of what I can verify/test, and on top of those time plus what I could approve.

For my part I just wanted to get some opinions whether this is a palatable approach at all and if it is worth any effort. I didn't really expect dbn@ to make any patches based on that question, there is more than one way lib32- ports can turn out to be completely unacceptable. Thus I assume this patch is primarily intended for discussion and not as a ready-to-merge patch. My additions linked above serve the same purpose — it's easier to see potential issues with the complete patch and not just the nice bits. Not to mention that Wine dependencies were explicitly requested. (Note that I don't expect anybody to actually review all changes in one go. Of course this should be done in reasonably sized chunks.)

[iwtcex_gmail.com]

Regarding the recent bug tracker comment, I would actually appreciate this and https://reviews.freebsd.org/D14721 being closed. Right now for people glancing through this thread it's not clear that nothing here is being actively worked on. @dbn?

In D16830#457760, @gerald wrote:

When (and hopefully it's when, not if) this gets in as far as Wine goes, I see three options:

1. Someone provides a patch for the Wine ports that dbn@ and/or me maintain, and if those are simple enough for me to understand I'll be happy to take them for "my" ports and hope dbn@ does the same for "his".
2. This can be done without touching emulators/wine-devel and emulators/wine, which is a subset of option 1. If dbn@ does not have the time any more, perhaps someone else wants to step up?
3. This is getting invasive as far as emulators/wine-devel and emulators/wine go, in which case I am open towards handing over those ports to someone with the skillset and time at hand.

Ah, I think there is one misconception here. This approach replaces i386-wine ports, with very small Wine-specific part (20 lines diff in my case, see this and this). The bulk of the changes are just adjustments to shared library locations for existing ports (lib -> lib32). It's all extremely dumb on purpose and should be perfectly maintainable in my opinion. Now, the tradeoff is that it would not be appropriate to push changes to 50-70 ports without some kind of consensus and I'd rather switch from FreeBSD completely than try to tackle this task :/ (Not that any alternative proposals would be necessarily nicer to deal with, mind you.)

[gerald]

portmgr@, can you please advise?

[iwtcex_gmail.com]

In D16830#466233, @gerald wrote:

portmgr@, can you please advise?

Let's try that again. Pretty please?

[iwtcex_gmail.com]

@mat Ok, pretty please with a direct mention on top. Specifically, since dbn@ has essentially abandoned this patch, what should we do with it? Does it make sense for me to resubmit an updated patch? (It's quite a bit larger, though.) Who is actually supposed to review, commit or decline this?

[iwtcex_gmail.com]

Well, assuming Phabricator notifications are not broken and this is indeed intentional silence, as an alternative "solution" I can only recommend making (as an option or a slave package) i386-wine-devel coinstallable with wine-devel by stripping all conflicting files from i386-wine-devel. That should enable wow64 functionality while preserving every single annoyance about the current system.

[bapt]

I still think the ability to make lib32 things working is a good idea. I am not sure this approach is the right one. I think if we have lib32 things is should be a new kind of feature quite close to the debug packages I am working on.
I don't think USES fits correctly and having slave ports is a good approach and I don't think flavors is the right feature to use as well here. Considering the number of ports that may be useful in that configuration (only wine?) not sure it is worth writing a specific feature for that.

Another approach could be also to provide just an overlay for users willing to get that. Overlays are now supported in the ports tree and poudriere-devel has support for it. That is probably an approach to consider.

Sorry my reply is not directly useful to that review.

[iwtcex_gmail.com]

In D16830#498207, @bapt wrote:

I think if we have lib32 things is should be a new kind of feature quite close to the debug packages I am working on.

I suppose that's true from the package manager point of view. Does that have any implications for the build system?

I don't think USES fits correctly and having slave ports is a good approach and I don't think flavors is the right feature to use as well here. Considering the number of ports that may be useful in that configuration (only wine?) not sure it is worth writing a specific feature for that.

Subpackages (D16457) might be useful there, but I haven't looked into it and they are still WIP.

[bapt]

In D16830#498228, @iwtcex_gmail.com wrote:

In D16830#498207, @bapt wrote:

I think if we have lib32 things is should be a new kind of feature quite close to the debug packages I am working on.

I suppose that's true from the package manager point of view. Does that have any implications for the build system?

It does have for package manager point of view, there is no differences between flavors, subpackages or slave ports.

I don't think USES fits correctly and having slave ports is a good approach and I don't think flavors is the right feature to use as well here. Considering the number of ports that may be useful in that configuration (only wine?) not sure it is worth writing a specific feature for that.

Subpackages (D16457) might be useful there, but I haven't looked into it and they are still WIP.

Yes I am thinking about using a subset of what subpackages is doing (for both debug packages et compat (lib32 being compat) packages.

[iwtcex_gmail.com]

In D16830#498205, @iwtcex_gmail.com wrote:

as an alternative "solution" I can only recommend making (as an option or a slave package) i386-wine-devel coinstallable with wine-devel by stripping all conflicting files from i386-wine-devel. That should enable wow64 functionality while preserving every single annoyance about the current system.

Submitted as PR 242625

[iwtcex_gmail.com]

In D16830#498207, @bapt wrote:

I think if we have lib32 things is should be a new kind of feature quite close to the debug packages I am working on.

Was there any progress on debug packages?

Considering the number of ports that may be useful in that configuration (only wine?) not sure it is worth writing a specific feature for that.

Wine has dependencies, so it's at least 50 to 70 ports.

[agh_riseup.net]

Hello,

A year has passed, is there any news on the debug package proposal for supporting multilibs? This would be advantageous not just for Wine, but also facilitating FreeBSD becoming a Windows cross-development platform, capable of producing and running Microsoft PE objects—provided the MinGW-w64 port is completed.

Thanks.