Page MenuHomeFreeBSD

[new port] sysutils/apache-mesos: Apache Mesos cluster management software
ClosedPublic

Authored by james.wright_digital-chaos.com on May 12 2019, 11:31 PM.

Details

Summary

New port for the Apache Mesos cluster management software.

This is an initial version to gain some feedback on how the port looks thus far, and currently does not include options to enable Java or Python bindings for simplicity.

Bugzilla
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237859

Diff Detail

Repository
rP FreeBSD ports repository
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

This needs to be hooked into sysutils/Makefile

linimon retitled this revision from sysutils/apache-mesos: New port to [new port] sysutils/apache-mesos: Apache Mesos cluster management software.May 13 2019, 4:37 PM

Added SUBDIR += apache-mesos to sysutils/Makefile, as requested.

tobik requested changes to this revision.May 14 2019, 7:41 AM
tobik added a subscriber: tobik.
tobik added inline comments.
sysutils/apache-mesos/Makefile
6 ↗(On Diff #57372)

Use curly braces {} here.

10 ↗(On Diff #57372)

"Apache Foundation open-source project" seems like filler and does not say much about Mesos.

30 ↗(On Diff #57372)
  • Since you are installing shared libraries you must have USE_LDCONFIG=yes.
  • We do not install *.la files unless there is a good reason to have them such as needing lt_dlopen() and libltdl at runtime. AFAICT this is not the case here, so please add USES=libtool as pointed out by stage-qa
  • The port seems to install Python scripts even with --disable-python but does not declare a Python run dependency.
  • stage-qa found a couple of issues:
====> Running Q/A tests (stage-qa)
Error: '/usr/bin/env python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-scp'
Error: '/usr/bin/env python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-tail'
Error: '/usr/bin/env python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-cat'
Error: '/usr/bin/env python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-ps'
Warning: 'lib/mesos/modules/liburi_disk_profile_adaptor-1.8.0.so' is not stripped consider trying INSTALL_TARGET=install-strip or using ${STRIP_CMD}
Warning: 'lib/mesos/modules/libfixed_resource_estimator-1.8.0.so' is not stripped consider trying INSTALL_TARGET=install-strip or using ${STRIP_CMD}
Warning: 'lib/mesos/modules/liblogrotate_container_logger-1.8.0.so' is not stripped consider trying INSTALL_TARGET=install-strip or using ${STRIP_CMD}
Warning: 'lib/mesos/modules/libload_qos_controller-1.8.0.so' is not stripped consider trying INSTALL_TARGET=install-strip or using ${STRIP_CMD}
Warning: 'lib/libmesos-1.8.0.so' is not stripped consider trying INSTALL_TARGET=install-strip or using ${STRIP_CMD}
Error: .la libraries found, port needs USES=libtool
Warning: you may not need USES=ssl
*** Error code 1
33–46 ↗(On Diff #57372)

The build has -Werror which often leads to build failures on Clang updates, maybe add --disable-werror.

sysutils/apache-mesos/pkg-descr
7 ↗(On Diff #57372)

duplicate http://

sysutils/apache-mesos/pkg-plist
10–12 ↗(On Diff #57372)

How was pkg-plist generated? etc/mesos should appear as %%ETCDIR%% and share/mesos as %%DATADIR%%.

Are the *.template user-editable (do they need @sample)?

This revision now requires changes to proceed.May 14 2019, 7:41 AM
mat added inline comments.May 14 2019, 11:47 AM
sysutils/apache-mesos/Makefile
10 ↗(On Diff #57372)

and "to manager computer clusters" does not either.

tobik resigned from this revision.Jun 2 2019, 1:41 PM

Makefile changes;

  • Fixed incorrect braces for PORTVERSION
  • Added USES= libtool
  • Added USES= shebangfix and SHEBANG_FILES= (although make stage-qa still seems unhappy with these files?)
  • Added USE_LDCONFIG=yes
  • Added option for installing CLI tools (which require the python runtime)
  • Added option for installing Deploy scripts (which should now install the *.template files as samples)
  • Added --disable-werror flag as suggested

pkg-desc changes;

  • Removed duplicate http://

pkg-plist changes;

  • Re-generated using make makeplist
  • Added subs for options (%%CLI%% and %%DEPLOY%%)
james.wright_digital-chaos.com marked 7 inline comments as done.Jun 2 2019, 9:07 PM

Please could take a look over the updated diff, as I appreciate the constructive feedback! :-)

Updated to 1.8.1
Added patch to fix compilation error in src/master/allocator/mesos/hierarchical.cpp

arrowd added a subscriber: arrowd.Aug 12 2019, 9:10 AM

Looks good to me. If no one objects, I'll commit this after giving it a run in poudriere.

Makefile
15 ↗(On Diff #60563)

If these are libraries, I'd move them to LIB_DEPENDS. At least, Boost dependency can be changed in that way.

It's not a requirement, though, so feel free to leave it as it is.

Makefile
15 ↗(On Diff #60563)

picojson is a header file only, so I think that one has to be a build dependency. The others could probably be moved but I haven't tried it out.

lwhsu added a subscriber: lwhsu.Sep 14 2019, 1:57 PM

@arrowd: any update on this?
@james.wright_digital-chaos.com: do you want to also submit the patch to the upstream?

Sorry for forgetting about this. I didn't added it to my TODO list.

This doesn't build on FreeBSD 12 due to following error:

libtool: compile:  c++ -DPACKAGE_NAME=\"mesos\" -DPACKAGE_TARNAME=\"mesos\" -DPACKAGE_VERSION=\"1.8.1\" "-DPACKAGE_STRING=\"mesos 1.8.1\"" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -DPACKAGE=\"mesos\" -DVERSION=\"1.8.1\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -DHAVE_CXX11=1 -DHAVE_PTHREAD_PRIO_INHERIT=1 -DHAVE_PTHREAD=1 -DHAVE_FTS_H=1 -DHAVE_APR_POOLS_H=1 -DHAVE_LIBAPR_1=1 -DHAVE_BOOST_VERSION_HPP=1 -DHAVE_LIBCURL=1 -DHAVE_GLOG_LOGGING_H=1 -DHAVE_ARCHIVE_H=1 -DHAVE_GOOGLE_PROTOBUF_MAP_H=1 -DHAVE_EV_H=1 -DHAVE_PICOJSON_H=1 -DHAVE_RAPIDJSON_RAPIDJSON_H=1 -DHAVE_LIBSASL2=1 -DHAVE_OPENSSL_SSL_H=1 -DHAVE_SVN_VERSION_H=1 -DHAVE_LIBSVN_SUBR_1=1 -DHAVE_SVN_DELTA_H=1 -DHAVE_LIBSVN_DELTA_1=1 -DHAVE_ZLIB_H=1 -DHAVE_LIBZ=1 -DHAVE_GRPCPP_GRPCPP_H=1 -DHAVE_ZOOKEEPER_H=1 -I. -DLIBDIR=\"/usr/local/lib\" -DPKGLIBEXECDIR=\"/usr/local/libexec/mesos\" -DPKGDATADIR=\"/usr/local/share/mesos\" -DPKGMODULEDIR=\"/usr/local/lib/mesos/modules\" -I../include -I../include -I../include/mesos -D__STDC_FORMAT_MACROS -I../3rdparty/concurrentqueue-7b69a8f -I../3rdparty/elfio-3.2 -I../3rdparty/leveldb-1.19/include -I../3rdparty/libprocess/include -I../3rdparty/nvml-352.79 -I../3rdparty/stout/include -I/usr/local/include/subversion-1 -I/usr/local/include -I/usr/local/include -I/usr/local/include/apr-1 -I/usr/local/include/apr-1.0 -isystem /usr/local/include -I/usr/local/include -I/usr/local/include -I/usr/local/include -I/usr/local/include -I/usr/local/include -I/usr/local/include -I/usr/local/include -I/usr/local/include/zookeeper -D_THREAD_SAFE -pthread -Wall -Wsign-compare -Wformat-security -fstack-protector-strong -fPIC -O2 -pipe -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing -isystem /usr/local/include -Wno-unused-local-typedef -Wno-inconsistent-missing-override -std=c++11 -stdlib=libc++ -DGTEST_HAS_PTHREAD -D_WITH_GETLINE -Wno-sentinel -MT master/libmesos_no_3rdparty_la-constants.lo -MD -MP -MF master/.deps/libmesos_no_3rdparty_la-constants.Tpo -c master/constants.cpp  -fPIC -DPIC -o master/.libs/libmesos_no_3rdparty_la-constants.o
master/allocator/mesos/hierarchical.cpp:1649:5: error: use of undeclared identifier 'random_shuffle'; did you mean 'std::random_shuffle'?
    random_shuffle(resourceVector.begin(), resourceVector.end());
    ^~~~~~~~~~~~~~
    std::random_shuffle
/usr/include/c++/v1/algorithm:3168:1: note: 'std::random_shuffle' declared here
random_shuffle(_RandomAccessIterator __first, _RandomAccessIterator __last)
^

Sorry for forgetting about this. I didn't added it to my TODO list.
This doesn't build on FreeBSD 12 due to following error:

libtool: compile:  c++ -DPACKAGE_NAME=\"mesos\" -DPACKAGE_TARNAME=\"mesos\" -DPACKAGE_VERSION=\"1.8.1\" "-DPACKAGE_STRING=\"mesos 1.8.1\"" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -DPACKAGE=\"mesos\" -DVERSION=\"1.8.1\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -DHAVE_CXX11=1 -DHAVE_PTHREAD_PRIO_INHERIT=1 -DHAVE_PTHREAD=1 -DHAVE_FTS_H=1 -DHAVE_APR_POOLS_H=1 -DHAVE_LIBAPR_1=1 -DHAVE_BOOST_VERSION_HPP=1 -DHAVE_LIBCURL=1 -DHAVE_GLOG_LOGGING_H=1 -DHAVE_ARCHIVE_H=1 -DHAVE_GOOGLE_PROTOBUF_MAP_H=1 -DHAVE_EV_H=1 -DHAVE_PICOJSON_H=1 -DHAVE_RAPIDJSON_RAPIDJSON_H=1 -DHAVE_LIBSASL2=1 -DHAVE_OPENSSL_SSL_H=1 -DHAVE_SVN_VERSION_H=1 -DHAVE_LIBSVN_SUBR_1=1 -DHAVE_SVN_DELTA_H=1 -DHAVE_LIBSVN_DELTA_1=1 -DHAVE_ZLIB_H=1 -DHAVE_LIBZ=1 -DHAVE_GRPCPP_GRPCPP_H=1 -DHAVE_ZOOKEEPER_H=1 -I. -DLIBDIR=\"/usr/local/lib\" -DPKGLIBEXECDIR=\"/usr/local/libexec/mesos\" -DPKGDATADIR=\"/usr/local/share/mesos\" -DPKGMODULEDIR=\"/usr/local/lib/mesos/modules\" -I../include -I../include -I../include/mesos -D__STDC_FORMAT_MACROS -I../3rdparty/concurrentqueue-7b69a8f -I../3rdparty/elfio-3.2 -I../3rdparty/leveldb-1.19/include -I../3rdparty/libprocess/include -I../3rdparty/nvml-352.79 -I../3rdparty/stout/include -I/usr/local/include/subversion-1 -I/usr/local/include -I/usr/local/include -I/usr/local/include/apr-1 -I/usr/local/include/apr-1.0 -isystem /usr/local/include -I/usr/local/include -I/usr/local/include -I/usr/local/include -I/usr/local/include -I/usr/local/include -I/usr/local/include -I/usr/local/include -I/usr/local/include/zookeeper -D_THREAD_SAFE -pthread -Wall -Wsign-compare -Wformat-security -fstack-protector-strong -fPIC -O2 -pipe -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing -isystem /usr/local/include -Wno-unused-local-typedef -Wno-inconsistent-missing-override -std=c++11 -stdlib=libc++ -DGTEST_HAS_PTHREAD -D_WITH_GETLINE -Wno-sentinel -MT master/libmesos_no_3rdparty_la-constants.lo -MD -MP -MF master/.deps/libmesos_no_3rdparty_la-constants.Tpo -c master/constants.cpp  -fPIC -DPIC -o master/.libs/libmesos_no_3rdparty_la-constants.o
master/allocator/mesos/hierarchical.cpp:1649:5: error: use of undeclared identifier 'random_shuffle'; did you mean 'std::random_shuffle'?
    random_shuffle(resourceVector.begin(), resourceVector.end());
    ^~~~~~~~~~~~~~
    std::random_shuffle
/usr/include/c++/v1/algorithm:3168:1: note: 'std::random_shuffle' declared here
random_shuffle(_RandomAccessIterator __first, _RandomAccessIterator __last)
^

That's strange, did the patch not apply, as it was to fix this very issue?

lwhsu requested changes to this revision.Sep 14 2019, 8:24 PM

I can build this port in poudriere fine. I think previous error was because of patching issue.

There are some issues:

  1. Please check portlint -acbt output.
  2. poudriere errors at:
====> Running Q/A tests (stage-qa)
Error: '/usr/local/bin/python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-ps'
Error: '/usr/local/bin/python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-tail'
Error: '/usr/local/bin/python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-cat'
Error: '/usr/local/bin/python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-scp'
*** Error code 1

I think this is because USE=python:run is only defined when CLI option enabled.

Makefile
18 ↗(On Diff #60563)

This blank line is not necessary, for making portlint(1) happy.

This revision now requires changes to proceed.Sep 14 2019, 8:24 PM

@james.wright_digital-chaos.com: do you want to also submit the patch to the upstream?

I don't think I need to submit the patch upstream as they have changed this portion of code in their master branch anyway!

====> Running Q/A tests (stage-qa)
Error: '/usr/local/bin/python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-ps'
Error: '/usr/local/bin/python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-tail'
Error: '/usr/local/bin/python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-cat'
Error: '/usr/local/bin/python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-scp'
*** Error code 1

I think this is because USE=python:run is only defined when CLI option enabled.

Tried moving shebangfix from USES= to CLI_USES= but still get the same error when running make stage-qa.

Please could you provide a suggestion how to fix this issue as I'm stumped as to what is the actual problem here.

lwhsu added a comment.Sep 17 2019, 1:20 AM
====> Running Q/A tests (stage-qa)
Error: '/usr/local/bin/python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-ps'
Error: '/usr/local/bin/python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-tail'
Error: '/usr/local/bin/python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-cat'
Error: '/usr/local/bin/python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-scp'
*** Error code 1

I think this is because USE=python:run is only defined when CLI option enabled.

Tried moving shebangfix from USES= to CLI_USES= but still get the same error when running make stage-qa.
Please could you provide a suggestion how to fix this issue as I'm stumped as to what is the actual problem here.

Because python is used for those bin/* files, the quickest way is moving python:run to USE=, however, I tried this and poudriere gives many Error: Orphaned: include/*.
Can you check this part?

Makefile changes

  • Fixed typo; CLI_USE= should have been CLI_USES=
  • Added 'shebangfix' to CLI_USES=
  • Removed extraneous whitespace (to satisfy portlint)

sysutils/Makefile changes

  • Added SUBDIR += apache-mesos

Tried moving shebangfix from USES= to CLI_USES= but still get the same error when running make stage-qa.
Please could you provide a suggestion how to fix this issue as I'm stumped as to what is the actual problem here.

Because python is used for those bin/* files, the quickest way is moving python:run to USE=, however, I tried this and poudriere gives many Error: Orphaned: include/*.
Can you check this part?

I think the previous errors (regarding python and shebangfix) were caused by a typo in CLI_USES=, I have since updated the diff and they seem to have been resolved.

Please could you run a build through poudriere and let me know if there are any outstanding issues?

lwhsu added a comment.Sep 18 2019, 9:08 PM

Because python is used for those bin/* files, the quickest way is moving python:run to USE=, however, I tried this and poudriere gives many Error: Orphaned: include/*.
Can you check this part?

I think the previous errors (regarding python and shebangfix) were caused by a typo in CLI_USES=, I have since updated the diff and they seem to have been resolved.
Please could you run a build through poudriere and let me know if there are any outstanding issues?

Still:

====> Running Q/A tests (stage-qa)
Error: '/usr/bin/env python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-ps'
Error: '/usr/bin/env python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-tail'
Error: '/usr/bin/env python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-scp'
Error: '/usr/bin/env python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-cat'
*** Error code 1

The problem is those files under bin/ need Python, are installed even when CLI is not defined, while python:run and shebangfix are only defined when CLI is on.
So, the possible solution is either only install these files when CLI is defined, or having python:run and shebangfix defined globally.

Still:

====> Running Q/A tests (stage-qa)
Error: '/usr/bin/env python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-ps'
Error: '/usr/bin/env python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-tail'
Error: '/usr/bin/env python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-scp'
Error: '/usr/bin/env python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-cat'
*** Error code 1

The problem is those files under bin/ need Python, are installed even when CLI is not defined, while python:run and shebangfix are only defined when CLI is on.
So, the possible solution is either only install these files when CLI is defined, or having python:run and shebangfix defined globally.

I don't think those files should be installed when CLI is not defined, because pkg-plist is as follows;

%%CLI%%bin/mesos
%%CLI%%bin/mesos-cat
%%CLI%%bin/mesos-execute
%%CLI%%bin/mesos-local
%%CLI%%bin/mesos-log
%%CLI%%bin/mesos-ps
%%CLI%%bin/mesos-resolve
%%CLI%%bin/mesos-scp
%%CLI%%bin/mesos-tail

I thought this meant that the files would only be installed with CLI defined, is that not the case?

Please advise if there is an alternative method to prevent these files being installed if CLI is not defined, and which satisfies the stage-qa tests.

  • Updated port to 1.9.0
  • Added databases/leveldb dependency
  • Added devel/elfio dependency
  • Added www/http-parser dependency
  • Additional patches to fix compile errors
  • Removed CLI option (always installed)
lwhsu added a comment.Oct 22 2019, 8:43 PM

This patch build failed in poudriere: https://gist.github.com/5919089c2e8d247342cf9a8749df7ca7

I added these files in include/ to pkg-plist and it pass the poudriere test. Is that the right fix? (these files need to be installed and installed to the right place)

This patch build failed in poudriere: https://gist.github.com/5919089c2e8d247342cf9a8749df7ca7
I added these files in include/ to pkg-plist and it pass the poudriere test. Is that the right fix? (these files need to be installed and installed to the right place)

The orphaned files mentioned in that poudriere log were header files I deliberately excluded from the pkg-plist, as I think they are for internal libs and headers only required during the build process, so didn't need to be included in the final package.

However it is probably better to package and install them all, in which case your updated pkg-plist looks fine to me!

Updated pkg-plist to include missing header files.

lwhsu added a comment.Oct 23 2019, 2:46 AM

This patch build failed in poudriere: https://gist.github.com/5919089c2e8d247342cf9a8749df7ca7
I added these files in include/ to pkg-plist and it pass the poudriere test. Is that the right fix? (these files need to be installed and installed to the right place)

The orphaned files mentioned in that poudriere log were header files I deliberately excluded from the pkg-plist, as I think they are for internal libs and headers only required during the build process, so didn't need to be included in the final package.

If they do only needed in the process, we can remove them from the package by directly rm them in post-install target.

BTW, it fails to build on 11: https://gist.github.com/3d72c1ad6338657c385a86ec8bbb2526

If this is hard to fix, we could consider mark this IGNORE on 11.

BTW, it fails to build on 11: https://gist.github.com/3d72c1ad6338657c385a86ec8bbb2526
If this is hard to fix, we could consider mark this IGNORE on 11.

After some Googling i think the error shown in the build log occurs when one of the library dependencies (in this case I suspect Boost) has been built without the -fpic (position independant code) flag.

I'm not sure why this would happen on 11 and not 12, and I don't have system running 11 to test. Do you have any ideas why the compilation of the libs may be different between 11 and 12?

lwhsu added a comment.Fri, Oct 25, 7:55 PM

I've purged my poudriere's pkg repository of 11 and kicked out a new test.

lwhsu added a comment.Sat, Oct 26, 9:18 AM

Still fails: https://gist.github.com/987197a48472bfc4790495dfdebab8de
I suggest mark IGNORE on 11 first.

Still fails: https://gist.github.com/987197a48472bfc4790495dfdebab8de
I suggest mark IGNORE on 11 first.

What is best way to do this, I tried adding following block near the bottom of the Makfile but portlint complains about it's position no matter where I put it;

.include <bsd.port.pre.mk>
.if (${OSVERSION} < 1200000)
IGNORE= build fails on FreeBSD 11
.endif

Also should we use BROKEN= rather than IGNORE= for this case?

lwhsu added a comment.Sat, Oct 26, 6:08 PM

Still fails: https://gist.github.com/987197a48472bfc4790495dfdebab8de
I suggest mark IGNORE on 11 first.

What is best way to do this, I tried adding following block near the bottom of the Makfile but portlint complains about it's position no matter where I put it;

.include <bsd.port.pre.mk>
.if (${OSVERSION} < 1200000)
IGNORE= build fails on FreeBSD 11
.endif

Also should we use BROKEN= rather than IGNORE= for this case?

I would say just add IGNORE_FreeBSD_11= does not build between LICENSE* and *DEPENDS

I'm fine with both, IGNORE would tell people don't really need to try it, while BROKEN will still be built by the package builders.

Added IGNORE_FreeBSD_11 due to breaking build.

I would say just add IGNORE_FreeBSD_11= does not build between LICENSE* and *DEPENDS
I'm fine with both, IGNORE would tell people don't really need to try it, while BROKEN will still be built by the package builders.

Thanks for advice, I've added the IGNORE_FreeBSD_11= as suggested and portlint is now happy! :-)

This revision was not accepted when it landed; it landed in state Needs Review.Sun, Oct 27, 6:02 PM
This revision was automatically updated to reflect the committed changes.