Page MenuHomeFreeBSD

lang/mono: take maintainership, update to 6.8.0.105
ClosedPublic

Authored by prj_rootwyrm.com on Jan 21 2020, 6:56 PM.
Tags
Referenced Files
F99628422: D23300.id.diff
Fri, Oct 11, 12:44 PM
Unknown Object (File)
Wed, Oct 9, 2:52 AM
Unknown Object (File)
Tue, Oct 8, 2:23 AM
Unknown Object (File)
Fri, Oct 4, 11:06 PM
Unknown Object (File)
Fri, Oct 4, 11:05 AM
Unknown Object (File)
Wed, Oct 2, 11:09 PM
Unknown Object (File)
Wed, Oct 2, 3:52 PM
Unknown Object (File)
Tue, Oct 1, 1:43 PM
Tokens
"Baby Tequila" token, awarded by justin_kyryli.uk.

Details

Summary

Proposal to take ownership of this port and update significantly.
mono@ has basically been dead for over a year, and has not updated the lang/mono port. I have been working closely with upstream for about a year now, and we have made tremendous strides in improving FreeBSD support as well as making upstream more friendly to the ports architecture. I have also worked with them to integrate FreeBSD into the official Mono CI, and we have gotten to a point where nearly all tests are passing or the failure is itself a failure of the test.

At this point I believe we are ready to take the next step and start bringing lang/mono officially current, and I am comfortable enough with the intricacies to take ownership of this port. There will be additional changes needed over time due to changes in the upstream organizational structure; however, these will not affect this port.

Outstanding work currently is confirming successful builds on aarch64 and powerpc64; it is known to not compile under qemu due to an MCS crash.

Test Plan
  • aarch64: need to confirm compiling on hardware or MCS bug

Completed successfully. Additional aarch64 patches added and upstreamed

  • powerpc64: need to test compiling, impossible to test with qemu (known limitation)

Completed, ppc has been added to supported architectures

  • confirm any building dependencies continue to work as normal

Diff Detail

Repository
rP FreeBSD ports repository
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

if anyone knows how I can say "aarch64 as long as it's not qemu" that would be very helpful. Users on qemu should expect this port to NOT build, while users on native arm should have no issues.

lang/ghc has

.ifdef QEMU_EMULATING
IGNORE= qemu-user-static isn't able to build lang/ghc, but it builds fine on a real hardware
.endif
  • Update to 6.8.0.105
  • Includes fix for powerpc64 building (patch has been upstreamed)
  • Includes @greg_unrelenting.technology 's patch partially; testing is required here. All of the Mono changes were already done upstream except for the boringssl portion. Some were done slightly differently (e.g. REDZONE is for all !APPLE) and revisions were necessary to boehm being deprecated/externalized.

The issue with MCS/CSC insta-crashing on qemu is a known issue due to qemu limitations and Roslyn behaviors, so if anyone knows how I can say "aarch64 as long as it's not qemu" that would be very helpful. Users on qemu should expect this port to NOT build, while users on native arm should have no issues.

This last version update has some plist issues for me as well (amd64 on STABLE12). Please check with Poudriere for QA before updating the diff. If possible. ;-).

! In D23300#518091, @driesm.michiels_gmail.com wrote:

This last version update has some plist issues for me as well (amd64 on STABLE12). Please check with Poudriere for QA before updating the diff. If possible. ;-).

Well, this is definitely "interesting" in the "that's not supposed to happen" vein. I did get reproduction of the plist issue with poudriere, using -t, but no warnings or missing files _without_ -t. Which obviously is wrong both ways.
The problem here is that ppc64 (ELFv1 and v2 both) install everything in plist _plus_ everything %%PPC64%%. However, upon reviewing a similar situation, it looks like I took the 'old bad way' approach. Which is moot when I took the diff from the wrong svn tree, so it was missing the whole %%PPC64%% handling block anyway! (Pointy-hat awarded to: me.)

This diff should resolve the plist issue fully and much more correctly.

This diff should resolve the plist issue fully and much more correctly.

Yes looks good in Poudriere again here too. Thank you.

pkubaj requested changes to this revision.Feb 11 2020, 8:50 PM

On powerpc64, I'm still getting:

configure: error: unknown target

This is because in configure.ac, there's neither powerpc64-*-freebsd, nor powerpc*-*-freebsd:

macppc-*-openbsd* | powerpc*-*-linux* | powerpc-*-openbsd* | \
powerpc-*-sysv* | powerpc-*-darwin* | powerpc-*-netbsd* | \
powerpc-*-freebsd* | powerpc*-*-aix* | powerpc*-*-os400* )
This revision now requires changes to proceed.Feb 11 2020, 8:50 PM
Makefile
31–67 ↗(On Diff #68100)

Wrong place in the Makefile. See Chapter 15. Order of Variables in Port Makefiles.

Also, the .if will never work, PORT_OPTIONS is only defined after including bsd.port.options.mk. You would be better of using a helper in this case.

78–80 ↗(On Diff #68100)

Before any include, ARCH is not defined, so this cannot possibly work.

Also, at that point, PLIST is not set, so no need to use +=.

On powerpc64, I'm still getting:

configure: error: unknown target

This is because in configure.ac, there's neither powerpc64-*-freebsd, nor powerpc*-*-freebsd:

Wups. Classic typo error on my part when re-splitting out the patch from when I upstreamed. That actually should read as it does in upstream:

powerpc*-*-freebsd* | powerpc*-*-aix* | powerpc*-*-os400* )

Note that the behavior around the AC_CHECK_COMPILE_FLAG for -mminimal-toc that seems wrong, is actually deliberate. I put the ELFv1 and ELFv2 flags in that way strictly to expose them as hinges in the event problems are uncovered that are FreeBSD specific. The other way actively broke AIX builds, and setting things more globally would break PASE (which is non-ELF.) And yes, Mono really builds on OS/400 and PASE.

Also, could you use devel/arcanist, or at least generate a diff with full context like it does, with svn diff -x -U9999 or git diff -U9999.

  • Reorder Makefile and expand to svn diff -x -U999999 as requested by @mat
  • Fixed typo in files/patch-powerpc64; this now matches powerpc*-*-freebsd* as in upstream
  • Correctly mark BROKEN for QEMU_EMULATING; Roslyn is very picky about POSIX and does not like qemu
  • Fix .if ${ARCH} PLIST handling by switching to pre/post.mk; accurate plists are still needed for ppc64 ELFv1, arm*, and arm64* This is based on https://reviews.freebsd.org/D22451
  • Remove "EXPERIMENTAL" from Ninja; this is well-tested and perfectly safe. It will remain non-default because of the number of dependencies it pulls in.
  • Remove 'TZ' from MAKE_ENV; this actually hasn't been necessary for a while
  • Correct branch comment; 6.8.0.105 is actually branch 2020-02
pkubaj requested changes to this revision.Feb 17 2020, 9:31 PM

It still fails with:

configure: error: unknown target

on powerpc64.
Since I may have worded myself wrong before, here's a complete patch for configure.ac:

--- configure.ac.orig   2020-02-17 20:58:04.818069000 +0000
+++ configure.ac        2020-02-17 21:30:00.091801000 +0000
@@ -4467,13 +4467,27 @@
                ;;
        macppc-*-openbsd* | powerpc*-*-linux* | powerpc-*-openbsd* | \
         powerpc-*-sysv* | powerpc-*-darwin* | powerpc-*-netbsd* | \
-        powerpc-*-freebsd* | powerpc*-*-aix* | powerpc*-*-os400* )
+        powerpc*-*-freebsd* | powerpc*-*-aix* | powerpc*-*-os400* )
                if test "x$ac_cv_sizeof_void_p" = "x8"; then
                        TARGET=POWERPC64;
                        CPPFLAGS="$CPPFLAGS -D__mono_ppc__ -D__mono_ppc64__"
-                       if ! (echo $CC | grep -q -- 'clang'); then
-                               CFLAGS="$CFLAGS -mminimal-toc"
-                       fi
+                       AC_MSG_NOTICE([Checking for PowerPC ISA -mminimal-toc support])
+                       AX_CHECK_COMPILE_FLAG(
+                               [-mminimal-toc],
+                               [CFLAGS="$CFLAGS -mminimal-toc"],
+                               [CFLAGS="$CFLAGS"]
+                       )
+                       case "$host" in
+                               powerpc*-*-freebsd*)
+                                       # We need to be aware if we are ELFv1 or v2 here
+                                       AC_MSG_NOTICE([Checking FreeBSD ELF version])
+                                       if ! ( echo | cc -dM -E - | awk '/_CALL_ELF/ {print $NF}'); then
+                                               AC_DEFINE([POWERPC_ELF], 1, [PowerPC ELFv1])
+                                       else
+                                               AC_DEFINE([POWERPC_ELFV2], 1, [PowerPC ELFv2])
+                                       fi
+                                       ;;
+                       esac
                else
                        TARGET=POWERPC;
                        CPPFLAGS="$CPPFLAGS -D__mono_ppc__"
This revision now requires changes to proceed.Feb 17 2020, 9:31 PM
prj_rootwyrm.com retitled this revision from lang/mono: take maintainership, update to 6.8.0.96 to lang/mono: take maintainership, update to 6.8.0.105.

Apologies for the apparent radio silence; I've actually been working on some high priority issues in upstream and trying to sort out inotify. The patch issue with ppc64 was actually just an SVN hiccup. Upstream patch is correct and will be in next release. This should be ready to go for ppc64 and aarch64.
@greg_unrelenting.technology - your patch should ONLY need to touch external/boringssl/crypto/cpu-aarch64-linux.c at this point, but I can't test here. If that's the case, can you please submit a PR with it to https://github.com/mono/boringssl ?

The inotify issue is now fixed, however, testing with real-world identified that it is extremely necessary, and some minor linking issues were found related to NuGet handling. So some significant changes are in this commit. It should be ready to go as of now though.

  • Removed INOTIFY option; devel/libinotify is now a hard dependency due to file permission reading issues
  • Added dependency on archivers/zip; the absence could now prevent some NuGet consumers from working
  • Removed SQLITE option; databases/sqlite3 is now a hard dependency (virtually guaranteed present anyway)
  • Renamed DEVELOP option to MONODEV, clarified that it is for Mono development
  • Switched to use of autogen.sh to bring patches in line with upstream (due to how tarballs are generated)
  • Removed virtually all ${REINPLACE_CMD} as these are no longer necessary

Most of the SHEBANG_FIX files will also no longer need SHEBANG_FIX (meaning it will only need BINARY_ALIAS for python3) in the future, but 6.8.0.105 isn't there yet, so I'm putting that review off for the next release.

pkubaj requested changes to this revision.Mar 17 2020, 7:52 PM

It doesn't apply after r526962:

pkubaj@talos:$/usr/ports/lang/mono$ doas svn patch D23300.diff
C         Makefile
>         rejected hunk @@ -2,94 +2,118 @@
U         distinfo
A         files/patch-btls-aarch64
D         files/patch-configure.ac
D         files/patch-mcs_class_Mono.Security_Mono.Security.Cryptography_KeyPairPersistence.cs
D         files/patch-mcs_class_Mono.Security_Mono.Security.X509_X509StoreManager.cs
D         files/patch-mcs_tools_mono-configuration-crypto_lib_Mono.Configuration.Crypto_KeyContainerCollection.cs
D         files/patch-mcs_tools_xbuild_data_12.0_Microsoft.CSharp.targets
D         files/patch-mcs_tools_xbuild_data_14.0_Microsoft.CSharp.targets
A         files/patch-mono_configure.ac
D         files/patch-mono_eglib_gfile-posix.c
A         files/patch-mono_metadata_Makefile.am
G         files/patch-mono_mini_Makefile.am.in
>         hunk @@ -1,11 +0,0 @@ already applied
D         files/patch-mono_mini_tramp-amd64.c
D         files/patch-mono_utils_mono-context.h
D         files/patch-mono_utils_mono-threads.c
D         files/patch-scripts_mono-heapviz
U         pkg-message
U         pkg-plist
A         pkg-plist.powerpc
Summary of conflicts:
  Text conflicts: 1

Can you rebase Makefile patch?

This revision now requires changes to proceed.Mar 17 2020, 7:52 PM

@greg_unrelenting.technology - your patch should ONLY need to touch external/boringssl/crypto/cpu-aarch64-linux.c at this point, but I can't test here. If that's the case, can you please submit a PR with it to https://github.com/mono/boringssl ?

Here: https://github.com/mono/boringssl/pull/23

It doesn't apply after r526962:

Oof.. and r526962 is unnecessary with 6.8.0.105 (because the port was written to require a minimum python 3.6) so I end up bulldozing it anyways. However, testing the rebase exposed a problem I missed - misbehavior with ccache that breaks builds. Only reason I caught it is three-way testing. So if a PORTREV bump is needed and the builder uses ccacche, it WILL fail to build correctly. Added NO_CCACHE= yes to prevent this in the event PORTREV is bumped.

I also redid the patches slightly, combining the inotify fix into a single patch file.

Builds fine, but I still have pkg-plist errors, probably because of missing pkg-plist.powerpc. And since this file is missing, all the files that are installed are orphaned.

I'm attaching pkg-plist.powerpc that I myself generated.

Other than this issue, it seems ok.

This revision is now accepted and ready to land.Mar 21 2020, 1:07 AM
mikael removed reviewers: linimon, pkubaj.

It builds fine on aarch64, nice work!

It builds fine on aarch64, nice work!

hm, was that not on -current? I see the ifndef ID_AA64ISAR0_AES_VAL stuff from my github PR didn't make it here yet

In D23300#530928, @greg_unrelenting.technology wrote:

It builds fine on aarch64, nice work!

hm, was that not on -current? I see the ifndef ID_AA64ISAR0_AES_VAL stuff from my github PR didn't make it here yet

Oups, yeah, I manually patched it.

How close are we to getting this committed? Are there still some open points or is it fine as is ATM?

Time has come that someone commits this accepted review.

I am a mentored committer so I can do it once my mentors give me approval.
@gerald, @tcberner: Can I proceed?

Time has come that someone commits this accepted review.

I am a mentored committer so I can do it once my mentors give me approval.
@gerald, @tcberner: Can I proceed?

Don't forget about pkg-plist.powerpc that I attached.

Time has come that someone commits this accepted review.

I am a mentored committer so I can do it once my mentors give me approval.
@gerald, @tcberner: Can I proceed?

Don't forget about pkg-plist.powerpc that I attached.

Thank you very much for pointing at it :)

tobik added inline comments.
Makefile
70 ↗(On Diff #69657)

This does not look ready to me. All the options are broken now because they are defined after bsd.port.options.mk.

@tobik : you are right. I won't commit it blindly even it has already been approved and I will review it carefully. Thanks!

Makefile
27 ↗(On Diff #69657)

^ iconv is still present here, although you added USES=iconv.

In D23300#537392, @greg_unrelenting.technology wrote:

Thanks!

@prj_rootwyrm.com:
Can you please recreate the patch including all the files and changes requested? This include:

Thanks.

This revision now requires changes to proceed.Apr 15 2020, 3:05 PM
pkubaj requested changes to this revision.May 24 2020, 9:55 PM

Looking at the diff between pkg-plist and pkg-plist.powerpc, I noticed I missed this at the end of pkg-plist.powerpc:

@postexec %%PREFIX%%/bin/cert-sync %%LOCALBASE%%/share/certs/ca-root-nss.crt
@postunexec rm -rf %%LOCALBASE%%/share/.mono

Please add this.

prj_rootwyrm.com edited the test plan for this revision. (Show Details)

Bump to 6.8.0.123 (still the 2019-10 branch), clean things up a bit more.

  • Switched @greg_unrelenting.technology 's patch to a git diff that pulls all of the external/boringssl (aka mono/boringssl) sub up to master (which includes the approved and tested PR.)
  • Identified an intermittent regression on 6.8.0.105 that does not reproduce on 6.8.0.123, where stack traces would indicate a certificate validation failure on certain lengthy responses when these entered managed exception paths. I suspect upstream #18838 fixed this issue.
  • Removed ccache prohibition after a few dozen runs to validate the edge case is cleared up with restructured Makefile
  • Added instructions in pkg-message regarding mandatory cert-sync when security/ca_root_nss is updated separate from Mono and when Mono is upgraded. (Unrelated to the fix above, just something I forgot.)
  • Added information on the SPECTRE mitigation option in pkg-descr since pkg-help documentation remains a black hole
  • Internal Roslyn bumped to 3.5.0-beta4-20121-02
  • %%DATADIR%%-2.0/mono/eglib/eglib-config.h now available to consumers
  • MonoWebRequestHandler is now a public ctor - upstream #18835.

There is a performance regression in mono-sgen that was not observed in CI and is not wholly consistent, relating to broken system thread library behavior (not FreeBSD-specific.) The problem has always been there, just varying degrees of severity. Since it's inconsistent and only impacts building Mono itself and some NuGet tasks, it's not worth trying to chase it down out of context.

Upstreamed patches are not in 6.8.0.123 because this is strictly a bug fix release. Work is also being impacted by several show-stopper issues upstream (including a complete inability to compile) that are being handled at a lower priority. This is also significantly impacted by the Azure VMs still being stuck on 12.0-RELEASE.

Hey @salvadore, @pkubaj,

Is this OK for you two? @salvadore, if you commit this, please make sure you close https://bugs.freebsd.org/238089?

If you don't have time at the moment, I'm happy to get this in.

@crees: Feel free to get this. I took it only because I read on IRC that this review needed attention, but I don't know much about mono and you will surely do a better work than me.
However you should probably take a look at https://reviews.freebsd.org/D24988 , which tries to improve this patch.

You might want to take this bug as well: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=239614 . I took it for the same reason I took this review and then you would probably deal with it better than me.

fwiw I have been working closely with the submitter over the past few weeks. I have tested both the previous version of this patch as well as the current one.

The previous version (mono-6.8.0.105) did indeed build on amd64, aarch64, powerpc64 (as of 20200526).

The current version (mono-6.8.0.123) does not (as of 20200607). I have been working with the submitter to resolve the issue.

What I have _not_ had the cycles to do yet is to carefully evaluate the differences between this and PR 238089.

Thanks @linimon.

The submitter of ports/238089 has asked for py-pillow to be removed from dependencies unless mono-heapviz is used, as it's a heavy dependency:

"I want to specify there that py-pillow is not RUN_DEPENDS for mono in general case. Only for one debug utility mono-heapviz. But it involves a large number of dependencies."

Can it be an option?

Any chance this is going to get merged? I was running a version of this for a long time and it was working flawlessly for my Mono services, but since rolling back I'm hitting so many annoying bugs in the old version of Mono.

I just read about the pillow PR.

This review has gone on for so long: why not commit what we have and then follow up with pillow later?

Some good progress is better than no progress at all.

When last talking to Philip 3-4 weeks ago I had shown that I had been able to compile his previous patch on all the various archs. The latest version does not. We have not yet agreed on whether we should simply commit the previous patch yet. I will talk to him.

The options block is badly ordered, See https://www.freebsd.org/doc/en/books/porters-handbook/porting-order-options.html for the correct ordering of the variables.

Makefile
59–62 ↗(On Diff #72481)

no. Options must not change themselves depending on what is installed.

71 ↗(On Diff #72481)

You cannot define any options helpers or otherwise after including bsd.port.options.mk. This line should be removed. (Or all the options afterwards can be removed as they do not do anything.)

98–100 ↗(On Diff #72481)
103–105 ↗(On Diff #72481)
116–120 ↗(On Diff #72481)

The last section in a Makefile is the targets, put this before the targets.

I don't mean to spam/bump, so I apologize if this is interpreted that way.
I am just asking if anyone has plans to accelerate the merging of this review (and the necessary changes), since it has been around for quite some time (the bugs.freebsd.org port has been since 05-24-2019 and this since 01-21-2020 while the version still active on FreeBSD is incredibly dated (~5.10.1.57) and actually won't work for a few of my services. I would manually install the patched port files for my systems, but it's not practical to maintain for > 1 jail (let alone 8+ jails). 😛
If there are no active plans (which is okay, of course), I will start making plans to move my systems away from using FreeBSD as the host to a more containerized system (i.e. Docker/VM instead of my current Jails), and just want to know if that is going to be necessary! 🙂

I have had to actively avoid using FreeBSD at times because of the lack of Mono or .Net Core support. PLEASE someone push SOMETHING. I would be happy to jump back into supporting Mono and help get the .Net Core to build natively on FreeBSD.

If there is something we can do to accelerate the merging, please let us know. We are in the process of migrating all our workstations to FreeBSD, but the lack of a recent mono version is blocking and so critical that we may have to move to Linux instead, this would be a shame. A recent mono is indeed now required to have a development environment for csharp on Unix (omnisharp-roslyn requires mono version 6.4.0 at least. For context: omnisharp-roslyn is the backend providing IDE-like features for csharp to all text editors).

We have been planning this migration for years, but the impact this old mono version is having on our development team is catching us by surprise midway into the migration. Very old versions of stuff can have unexpected consequences difficult to anticipate.

@missoline_protonmail.com:

I realise this doesn't fix your longtime problem, but for now (while the maintainer situation is taken care of) there is working fix at https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=238089 :

pkg install -y ca_root_nss gcc

fetch 'https://bz-attachments.freebsd.org/attachment.cgi?id=219414' -o /tmp/mono-patch-6.8.0.105

patch -p1 -d /usr/ports/lang/mono -E < /tmp/mono-patch-6.8.0.105

cd /usr/ports/lang/mono && make -DBATCH install clean

I would love to be of more assistance, but this is not really my area of expertise :(

Just going to post this workaround here since this is one of the top Google results / a lot of links point here when you're looking for mono upgrade solutions and manually compiling on each of your jails or anything along those lines can be quite tedious.
This thread has the secondary packages needed in order to install mono 6.8, as well as someone posted the mono-6.8.0.105 patch pre-built to GitHub for ease of download.
The excerpt steps are below:

pkg install libiconv
wget https://github.com/jailmanager/jailmanager.github.io/releases/download/v0.0.1/mono-6.8.0.105.txz
pkg install -y mono-6.8.0.105.txz

via https://www.truenas.com/community/threads/how-to-manually-upgrade-mono-from-5-10-to-5-20-in-a-freenas-jail.78871/post-597815

Is anyone working on this?

I intend to fix this over Christmas break.

Is anyone working on this?

I have been told that future/current release has some nasty stuff in it which doesn't work on FreeBSD. They have chosen to use Ubuntu and while they have previously worked to have this compatible with FreeBSD, that has ended. I am basing this on 2nd hand conversations with people I trust.

Just an FYI the DotNet Core guys are ramping up as well:

https://github.com/dotnet/runtime/issues/14537

They are cross building the MI portions on Linux using dotnet core 3. It's beyond me why they don't use the Linux binaries in the ports tree, but there you go.

In D23300#614738, @dvl wrote:

Is anyone working on this?

I have been told that future/current release has some nasty stuff in it which doesn't work on FreeBSD. They have chosen to use Ubuntu and while they have previously worked to have this compatible with FreeBSD, that has ended. I am basing this on 2nd hand conversations with people I trust.

But do you blame them? My God, it's been torture watching the mono situation on FreeBSD. I bailed out to prop up Lua because I could see that there was no pushing this rock any further up the hill. I thank anyone that can do it but everything just goes stale. I think we would be better off putting some effort with FreeBSD/DotNet Core team. It's not *actually* supported by MS, it's a community initiative.

What's even more telling than MS not supporting Mono is this: The DotNet Core team could have side stepped half their problems if they helped the Mono community on FreeBSD. They can't build the DotNet Standard Library without some DotNet engine so they have to cross build. They *actively* discarded my Mono idea when I proposed bringing Mono up to date. This was some years ago, just as Core 2 was coming out.

I don't know who wants to sabotage the .Net ecosystem on FreeBSD more, FreeBSD or Microsoft. It's just bonkers. Anyways, good luck!!!!

This revision was not accepted when it landed; it landed in state Needs Review.Dec 13 2020, 2:53 AM
This revision was automatically updated to reflect the committed changes.
In D23300#614738, @dvl wrote:

Is anyone working on this?

I have been told that future/current release has some nasty stuff in it which doesn't work on FreeBSD. They have chosen to use Ubuntu and while they have previously worked to have this compatible with FreeBSD, that has ended. I am basing this on 2nd hand conversations with people I trust.

I was completely mistaken.

I was confusing mono with another product, which we use internally at $WORK and for which there is no public FreeBSD port.

My apologies.