Update virtualbox to 6.1.18 and repocopy version 5 to -legacy ports.

[bapt]

Summary

This comes from pr 234878 [1] on bugzilla, many people contributed patches there to create this update, my work has mainly been with cleaning up the patches and integrating in present ports tree, so credit is due to the various contributors there (jkim, Mario Lobo, lwhsu contributed patches, many other testing and suggestions)

I added patches to make additions 6.1.18 work. The module gave me a bcmp undefined symbol error, the fix I used comes from a post on the mailin lists [2]

Patch also includes a tentative entry for UPDATING.

Some notes:

• At present this compiles only on amd64, and while it produces a package on 11 and 12 it fails to work there.

• The patch includes repocopies of the version 5 ports as legacy versions. This will ensure some path for users who can't run the latest hardware/software. BTW these legacy ports could be updated to 6.0 later, AFAIK 6.0 has the same requirements as 5.x.

• I regenerated most of the patches to have them fresh

[1] https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=234878

[2] https://lists.freebsd.org/pipermail/freebsd-virtualization/2020-May/008448.html

Test Plan

I'm running this on head without any problems. It actually works smoother than old version, especially for graphic output.

I've tested it in poudriere myself, and plan to also test non default options better (performed only some quick test there)

User reports are still mixed, but it should work fine on head and 13.

[madpilot]

Some personal thoughts:

Ideally this should be committed before the next quarterly for two reason, obviously to get in quarterly, and also because I'd rather commit this work I did in subversion without need to rework it through git [1].

At the same time, unless it can be made to work on 12.2 at least, it makes little sense to commit this before 13.0 is released, so the ideal window for committing this amounts to a few days at the end of March at most.

[1] this is no critique of git, in fact this activity would have been easier with git, but having done all the patching in subversion moving it to git would be extra work

[dereks_lifeofadishwasher.com]

I was able to build, install, run, create/run VMs with this patch on 12.2-RELEASE-p3 host. Should the ignore 12 lines be removed?

emulators/virtualbox-ose-kmod/Makefile

28

What's the exact reason for ignoring 12 here?

With this patch and a 12.2-RELEASE-p3 amd64 host and poudriere I'm able to build, install, create, and run a freebsd 12 amd64 and Windows 10 64 VM a 12.2-RELEASE-p3 host.

[madpilot]

In D28871#646258, @dereks_lifeofadishwasher.com wrote:

I was able to build, install, run, create/run VMs with this patch on 12.2-RELEASE-p3 host. Should the ignore 12 lines be removed?

If it works yes. Tomorrow I'll test myself, but you report looks promising.

emulators/virtualbox-ose-kmod/Makefile

28

This is here because people reported it failing on 12 and I have been unable to test it until now.

I am reinstalling an old laptop with FreeBSD 12.2 to test just this. I plan to test it tomorrow myself.

You report is good news though. I'll do my own testing anyway, just in case, I'll be quite happy to remove this line if at all possible.

[madpilot]

Further note:

6.1.18 additions fail to compile on i386 with a strange gmake "missing separator" error. I've been unable to fix it. Any help appreciated!

[dereks_lifeofadishwasher.com]

In D28871#646261, @madpilot wrote:

In D28871#646258, @dereks_lifeofadishwasher.com wrote:

I was able to build, install, run, create/run VMs with this patch on 12.2-RELEASE-p3 host. Should the ignore 12 lines be removed?

If it works yes. Tomorrow I'll test myself, but you report looks promising.

I see where there might be an issue with 12.2-RELEASE-p3 and clang.

I was building with DEBUG and it works. Rebuild with default options (DEBUG off) you get the same error reported here https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=234878#c134 while trying to start a VM.
It seems to either be an issue with the code or the way clang (10?) is building it?

So it's possible DEBUG (for the being) might be required for 12.2?

[madpilot]

In D28871#646359, @dereks_lifeofadishwasher.com wrote:

In D28871#646261, @madpilot wrote:

In D28871#646258, @dereks_lifeofadishwasher.com wrote:

I was able to build, install, run, create/run VMs with this patch on 12.2-RELEASE-p3 host. Should the ignore 12 lines be removed?

If it works yes. Tomorrow I'll test myself, but you report looks promising.

I see where there might be an issue with 12.2-RELEASE-p3 and clang.

I was building with DEBUG and it works. Rebuild with default options (DEBUG off) you get the same error reported here https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=234878#c134 while trying to start a VM.
It seems to either be an issue with the code or the way clang (10?) is building it?

Interesting finding. Ideally we should understand why this happens. Maybe forcing another compiler for 12.2 would make it work?

Also, what are you compiling with DEBUG? Only the kmod is enough or is it required for all of virtualbox-ose?

So it's possible DEBUG (for the being) might be required for 12.2?

Could be an option, but I'd like to try forcing clang 11 or some other compiler before going down that way.

[grahamperrin]

Maybe negligible for review purposes:

• the application icons that appear, temporarily, whilst a guest starts are generic X.org icons.

[dereks_lifeofadishwasher.com]

In D28871#646424, @madpilot wrote:

Interesting finding. Ideally we should understand why this happens. Maybe forcing another compiler for 12.2 would make it work?

Yeah, finding the root cause would be best however if needed forcing gcc would be next.

Also, what are you compiling with DEBUG? Only the kmod is enough or is it required for all of virtualbox-ose?

DEBUG was only enabled on emulators/virtualbox-ose

[madpilot]

In D28871#646671, @dereks_lifeofadishwasher.com wrote:

In D28871#646424, @madpilot wrote:

Interesting finding. Ideally we should understand why this happens. Maybe forcing another compiler for 12.2 would make it work?

Yeah, finding the root cause would be best however if needed forcing gcc would be next.

Also, what are you compiling with DEBUG? Only the kmod is enough or is it required for all of virtualbox-ose?

DEBUG was only enabled on emulators/virtualbox-ose

I tested now virtualbox 6 modified to force clang 11 from ports on 12.2 and it works fine. I think this fix is cleaner than forcing DEBUG.

I'm almost sure this would work for 11.x too, but it's difficult to test because I'd have to build a full set of ports using SSL from ports, due to the QT/OpenSSL issue. Maybe I'll test that too, anyway I'll remove the restriction on 11 along with the one on 12.

This removes the need to wait for 13.0 to be released before committing.

Updated patch set coming shortly.

[madpilot]

In D28871#646441, @grahamperrin_gmail.com wrote:

Maybe negligible for review purposes:

• the application icons that appear, temporarily, whilst a guest starts are generic X.org icons.

Good catch, although I suspect this is due to the way virtualbox creates its windows and not much can be done. Anyway if it's only limited to aesthetics a fix can also be added at a later time.

[madpilot]

Improved patch:

• Make virtualbox-ose-legacy depend on virtualbox-ose-kmod-legacy.
• On FreeBSD <13 force using clang 11 from ports to build. Thiss makes it work correctly there.
• Change wording in UPDATING accordingly and add note about inability to move saved states accross the update.

[dereks_lifeofadishwasher.com]

In D28871#646753, @madpilot wrote:

I tested now virtualbox 6 modified to force clang 11 from ports on 12.2 and it works fine. I think this fix is cleaner than forcing DEBUG.

I'm almost sure this would work for 11.x too, but it's difficult to test because I'd have to build a full set of ports using SSL from ports, due to the QT/OpenSSL issue. Maybe I'll test that too, anyway I'll remove the restriction on 11 along with the one on 12.

The issue I could see with using llvm11 is that building on a fresh env you need gcc10 (kbuild), llvm10 (mesalibs), and if added llvm11 to build virtualbox-ose. That's three very large ports. If you just used gcc (assuming you can reuse gcc10) it's already there.

Is it any different you force virtualbox-ose to use lang/llvm10 vs base's? If no different then using gcc10 (or whatever kbuild uses) would require less to build.

[madpilot]

In D28871#646774, @dereks_lifeofadishwasher.com wrote:

The issue I could see with using llvm11 is that building on a fresh env you need gcc10 (kbuild), llvm10 (mesalibs), and if added llvm11 to build virtualbox-ose. That's three very large ports. If you just used gcc (assuming you can reuse gcc10) it's already there.

Is it any different you force virtualbox-ose to use lang/llvm10 vs base's? If no different then using gcc10 (or whatever kbuild uses) would require less to build.

Yes, you make a compelling point. Honestly I did not think about it.

I need to spin a new round of tests before replying. Should not take long though, since the dependencies should all be present in the poudriere jail I used.

EDIT: At the same time I'm checking the "CHOSEN_COMPILER_TYPE" variable to know what to do...Having that on clang and then forcing gcc looks strange. Not sure if there is a way to make USES=compiler choose gcc himself but only depending on OSVERSION

[madpilot]

I coerced it in using gcc, but it's failing with a bunch of errors like this:

-- SUPDrvSem.o ---
gcc10: error: unrecognized command-line option '-fformat-extensions'; did you mean '-fno-ms-extensions'?
*** [SUPDrvSem.o] Error code 1

First, I cannot find any documentation for the flag -fformat-extensions, I'd rather know what it does before removing it, maybe it should be replaced by some gcc specific one. Anyone has any clue?

Anyway I have no idea how to remove it, since grepping the whole virtualbox sources only returns a match in Config.kmk, but in a comment, so, right now, I have no idea where that flag is growing from. I'll keep looking.

EDIT:

https://stackoverflow.com/questions/48927926/cc-error-unknown-argument-fformat-extensions

Interesting enough, this flag used to be an extension from our own gcc "fork", unsupported in newer gcc. But I could try forcing gcc only on main virtualbox port. Could work anyway.

[madpilot]

Forcing gcc 10 gives errors both in kmod and in main port. I'd say it's a non starter.

Main port error with gcc 10:

kBuild: Compiling iPxeBaseBin - /wrkdirs/usr/ports/emulators/virtualbox-ose-nox11/work/VirtualBox-6.1.18/src/VBox/Devices/PC/ipxe/src/arch/i386/core/basemem_packet.c
gcc10: error: unrecognized command-line option '-mstack-alignment=2'
kmk: *** [/wrkdirs/usr/ports/emulators/virtualbox-ose-nox11/work/VirtualBox-6.1.18/out/freebsd.amd64/release/obj/iPxeBaseBin/src/arch/i386/core/basemem_packet.o] Error 1
The failing command:
@gcc10 -m32 -c -O2 -fno-pie -fcommon -ffreestanding -ffunction-sections -march=i386 -fomit-frame-pointer -fstrength-reduce -falign-jumps=1 -falign-loops=1 -Os -falign-functions=1 -mstack-alignment=2 -mregparm=3 -mrtd -freg-struct-return -fshort-wchar -Ui386 -Ulinux -include compiler.h -Wall -W -Wformat-nonliteral -fno-stack-protector -fno-dwarf2-cfi-asm -Wno-address -I/wrkdirs/usr/ports/emulators/virtualbox-ose-nox11/work/VirtualBox-6.1.18/src/VBox/Devices/PC/ipxe/src -I/wrkdirs/usr/ports/emulators/virtualbox-ose-nox11/work/VirtualBox-6.1.18/src/VBox/Devices/PC/ipxe/src/include -I/wrkdirs/usr/ports/emulators/virtualbox-ose-nox11/work/VirtualBox-6.1.18/src/VBox/Devices/PC/ipxe/src/arch/x86/include -I/wrkdirs/usr/ports/emulators/virtualbox-ose-nox11/work/VirtualBox-6.1.18/src/VBox/Devices/PC/ipxe/src/arch/i386/include -I/wrkdirs/usr/ports/emulators/virtualbox-ose-nox11/work/VirtualBox-6.1.18/src/VBox/Devices/PC/ipxe/src/arch/i386/include/pcbios -I/wrkdirs/usr/ports/emulators/virtualbox-ose-nox11/work/VirtualBox-6.1.18/include -I/wrkdirs/usr/ports/emulators/virtualbox-ose-nox11/work/VirtualBox-6.1.18/out/freebsd.amd64/release -DVBOX -DVBOX_OSE -DVBOX_WITH_64_BITS_GUESTS -DRT_OS_FREEBSD -D__FREEBSD__ -DRT_ARCH_AMD64 -D__AMD64__ -DVBOX_WITH_DEBUGGER -DVBOX_WITH_HARDENING -DRTPATH_APP_PRIVATE=\"/usr/local/share/virtualbox-ose\" -DRTPATH_APP_PRIVATE_ARCH=\"/usr/local/lib/virtualbox\" -DRTPATH_SHARED_LIBS=\"/usr/local/lib/virtualbox\" -DRTPATH_APP_DOCS=\"/usr/local/share/doc/virtualbox-ose\" -DARCH=i386 -DPLATFORM=pcbios -DVERSION_MAJOR=1 -DVERSION_MINOR=0 -DVERSION_PATCH=0 -DVERSION=\"1.0.0+\" -D_build_id=0xaffeaffe -DOBJECT=basemem_packet -Wp,-MD,/wrkdirs/usr/ports/emulators/virtualbox-ose-nox11/work/VirtualBox-6.1.18/out/freebsd.amd64/release/obj/iPxeBaseBin/src/arch/i386/core/basemem_packet.o.dep -Wp,-MT,/wrkdirs/usr/ports/emulators/virtualbox-ose-nox11/work/VirtualBox-6.1.18/out/freebsd.amd64/release/obj/iPxeBaseBin/src/arch/i386/core/basemem_packet.o -Wp,-MP -o /wrkdirs/usr/ports/emulators/virtualbox-ose-nox11/work/VirtualBox-6.1.18/out/freebsd.amd64/release/obj/iPxeBaseBin/src/arch/i386/core/basemem_packet.o /wrkdirs/usr/ports/emulators/virtualbox-ose-nox11/work/VirtualBox-6.1.18/src/VBox/Devices/PC/ipxe/src/arch/i386/core/basemem_packet.c

[madpilot]

I've made a test on 12.2 using base compiler and it fails to start VMs as expected.

For me forcing clang 11 is the only solution working across the board.

I've also got a negative report forcing clang11, and one positive using base clang (also on 12.2).

At this point it's difficult for me to tell what should be done, due to the contrasting reports.

My gut feeling is that going with forcing claang11(the present patch set here) is the safest option at present.

We need to make it work and get it into the tree. Things can be improved later anyway.

[madpilot]

Updating the patch to improve some details.

First of all the kmod can be built using base clang on 12.2.

Also cleaned up the code to choose compiler in the main port a little.

I'm still forcing clang11 on 12.2 since this seems to work fine in my tests.

I am not a member of the vbox group, so I still need approval to commit this. I'm here asking for such approval. Obviously I will do my best to address any change request if it's in my ability.

[madpilot]

Small patch update to add required USES=xorg which is missing in the virtualbox-ose port.

[madpilot]

Fix one patch, looks like some path separators where missing.

[decke]

Looks like nobody is willing to review this so I grabbed my old and dusty vbox hat. Let me first say thanks a lot for taking this challenge! Upstream says VirtualBox 5.2 is EoL and support ended 2020/07.

The decision to bring back the legacy ports is definitely a good one and we used it in the past because you cannot update vbox without breaking a few features (you don't know which) and there are usually people that rely on a few of them.
The patch itself is almost impossible to review and I really hope you just do a real repocopy (svn cp) and make sure to put -legacy on the places it belongs.

I cannot say much about the vbox 6.1 patches but it looks okay to me and parts of it are in bugzilla since 6.0 with various people reporting partial success so I assume the best. It is really about time to give this broader testing!

[decke]

As a side note there is also www/phpvirtualbox (which is also maintained by vbox@) which we will need to update to latest git (develop branch) to support vbox 6.1.

[madpilot]

Since the VBoxNetDHCP issue that has been found in the related bug report on bugzilla looks quite nasty I'll hold this for a little while, till the week end at least, to try to fix that "wrinkle".

If that ends up too hard to fix for me I would anyway commit the update. As noted version 5 is EOL and we really need to move on. A fix can be cooked later.

In D28871#649523, @decke wrote:

As a side note there is also www/phpvirtualbox (which is also maintained by vbox@) which we will need to update to latest git (develop branch) to support vbox 6.1.

This needs also a -legacy port for the old version and an update. At present it can keep working with the legacy version.

[decke]

I think we never had a legacy port for phpvirtualbox because there are not that many users and upstream is almost dead. It should be enough for now to change the dependencies to vbox-legacy. We can update it when the dust has settled.

[dereks_lifeofadishwasher.com]

In D28871#649595, @decke wrote:

I think we never had a legacy port for phpvirtualbox because there are not that many users and upstream is almost dead. It should be enough for now to change the dependencies to vbox-legacy. We can update it when the dust has settled.

I don't think upstream is dead. Are you looking at the right project it moved to github? https://github.com/phpvirtualbox/phpvirtualbox the develop branch supports 6.1 (https://github.com/phpvirtualbox/phpvirtualbox/issues/255) last updated ~month ago.

Maybe over the weekend I could try to get www/phpvirtualbox updated to use this branch.

[decke]

phpvirtualbox was a one man show (imoore76) for a long time but when he stopped it was dead for a year or more. Now a new guy has taken it over and tries to maintain it but looking at the history tells you this is basically collecting patches to keep it working. There is no improvement or development they just try to keep it working and did not manage to do a release for two years. I'd call it dead but one could also say it's legacy or feature complete.

[madpilot]

phpvirtualbox port does not have any dependency on the virtualbox ports. I guess it can be installed standalone, and only needs to communicate with virtualbox via sockets or other methods. So I don't think there is anything to be changed there right now.

[madpilot]

BTW forgot to mention, yes I used "svn cp" to create the legacy ports, then adjusted the conflicts and dependencies.

[madpilot]

New patch set with some changes.

Main thing is I think I found what was the issue with VBoxNetDHCP. The daemon working as DHCP server has been rewritten singe 5.x. In the rewrote some logic remained similar but the code has clearly been rewritten.

The old implementation was calling the icotl to get new packets with a timeout of 2 seconds. The new code does not use a timeout. Adding back a timeout (patch patch-src_VBox_NetworkServices_Dhcpd_VBoxNetDhcpd.cpp) seems to fix the issue, allowing the program to be properly killed instead of hanging in the ioctl.

This patch also includes fixes allowing to properly build DEBUG binaries. The software now sets WITH_DEBUG, which avoids stripping binaries.

To make the kmod port build DEBUG binaries I had to recover an old patch (extrapatch-Config.kmk) which was removed at some point, but is still required by the kmod port. I added it back to both the new port and legacy. The patch I'm posting here adds the file as new because arc gives me errors if I repocopy the old file. In the commit I'm going to recover the old file with history using repocopy.

Please test the VBoxNetDHCP fix to verify it works properly and really solves the problem.

I'll also ask for testing in the related bug report. If I get positive feedback I'll commit the update to the tree.

[madpilot]

Added some information about the addition in the UPDATING message.

[bapt]

committed as rP567950