Page MenuHomeFreeBSD

[new port] sysutils/apache-mesos: Apache Mesos cluster management software
Needs ReviewPublic

Authored by james.wright_digital-chaos.com on May 12 2019, 11:31 PM.

Details

Reviewers
lwhsu
tobik
Group Reviewers
Contributor Reviewers (ports)
Summary

New port for the Apache Mesos cluster management software.

This is an initial version to gain some feedback on how the port looks thus far, and currently does not include options to enable Java or Python bindings for simplicity.

Bugzilla
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=237859

Diff Detail

Repository
rP FreeBSD ports repository
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

This needs to be hooked into sysutils/Makefile

linimon retitled this revision from sysutils/apache-mesos: New port to [new port] sysutils/apache-mesos: Apache Mesos cluster management software.May 13 2019, 4:37 PM

Added SUBDIR += apache-mesos to sysutils/Makefile, as requested.

tobik requested changes to this revision.May 14 2019, 7:41 AM
tobik added a subscriber: tobik.
tobik added inline comments.
sysutils/apache-mesos/Makefile
7

Use curly braces {} here.

11

"Apache Foundation open-source project" seems like filler and does not say much about Mesos.

31
  • Since you are installing shared libraries you must have USE_LDCONFIG=yes.
  • We do not install *.la files unless there is a good reason to have them such as needing lt_dlopen() and libltdl at runtime. AFAICT this is not the case here, so please add USES=libtool as pointed out by stage-qa
  • The port seems to install Python scripts even with --disable-python but does not declare a Python run dependency.
  • stage-qa found a couple of issues:
====> Running Q/A tests (stage-qa)
Error: '/usr/bin/env python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-scp'
Error: '/usr/bin/env python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-tail'
Error: '/usr/bin/env python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-cat'
Error: '/usr/bin/env python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-ps'
Warning: 'lib/mesos/modules/liburi_disk_profile_adaptor-1.8.0.so' is not stripped consider trying INSTALL_TARGET=install-strip or using ${STRIP_CMD}
Warning: 'lib/mesos/modules/libfixed_resource_estimator-1.8.0.so' is not stripped consider trying INSTALL_TARGET=install-strip or using ${STRIP_CMD}
Warning: 'lib/mesos/modules/liblogrotate_container_logger-1.8.0.so' is not stripped consider trying INSTALL_TARGET=install-strip or using ${STRIP_CMD}
Warning: 'lib/mesos/modules/libload_qos_controller-1.8.0.so' is not stripped consider trying INSTALL_TARGET=install-strip or using ${STRIP_CMD}
Warning: 'lib/libmesos-1.8.0.so' is not stripped consider trying INSTALL_TARGET=install-strip or using ${STRIP_CMD}
Error: .la libraries found, port needs USES=libtool
Warning: you may not need USES=ssl
*** Error code 1
34–47

The build has -Werror which often leads to build failures on Clang updates, maybe add --disable-werror.

sysutils/apache-mesos/pkg-descr
8

duplicate http://

sysutils/apache-mesos/pkg-plist
11–13

How was pkg-plist generated? etc/mesos should appear as %%ETCDIR%% and share/mesos as %%DATADIR%%.

Are the *.template user-editable (do they need @sample)?

This revision now requires changes to proceed.May 14 2019, 7:41 AM
mat added inline comments.May 14 2019, 11:47 AM
sysutils/apache-mesos/Makefile
11

and "to manager computer clusters" does not either.

tobik resigned from this revision.Jun 2 2019, 1:41 PM

Makefile changes;

  • Fixed incorrect braces for PORTVERSION
  • Added USES= libtool
  • Added USES= shebangfix and SHEBANG_FILES= (although make stage-qa still seems unhappy with these files?)
  • Added USE_LDCONFIG=yes
  • Added option for installing CLI tools (which require the python runtime)
  • Added option for installing Deploy scripts (which should now install the *.template files as samples)
  • Added --disable-werror flag as suggested

pkg-desc changes;

  • Removed duplicate http://

pkg-plist changes;

  • Re-generated using make makeplist
  • Added subs for options (%%CLI%% and %%DEPLOY%%)
james.wright_digital-chaos.com marked 7 inline comments as done.Jun 2 2019, 9:07 PM

Please could take a look over the updated diff, as I appreciate the constructive feedback! :-)

Updated to 1.8.1
Added patch to fix compilation error in src/master/allocator/mesos/hierarchical.cpp

arrowd added a subscriber: arrowd.Aug 12 2019, 9:10 AM

Looks good to me. If no one objects, I'll commit this after giving it a run in poudriere.

Makefile
15 ↗(On Diff #60563)

If these are libraries, I'd move them to LIB_DEPENDS. At least, Boost dependency can be changed in that way.

It's not a requirement, though, so feel free to leave it as it is.

Makefile
15 ↗(On Diff #60563)

picojson is a header file only, so I think that one has to be a build dependency. The others could probably be moved but I haven't tried it out.

lwhsu added a subscriber: lwhsu.Sep 14 2019, 1:57 PM

@arrowd: any update on this?
@james.wright_digital-chaos.com: do you want to also submit the patch to the upstream?

Sorry for forgetting about this. I didn't added it to my TODO list.

This doesn't build on FreeBSD 12 due to following error:

libtool: compile:  c++ -DPACKAGE_NAME=\"mesos\" -DPACKAGE_TARNAME=\"mesos\" -DPACKAGE_VERSION=\"1.8.1\" "-DPACKAGE_STRING=\"mesos 1.8.1\"" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -DPACKAGE=\"mesos\" -DVERSION=\"1.8.1\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -DHAVE_CXX11=1 -DHAVE_PTHREAD_PRIO_INHERIT=1 -DHAVE_PTHREAD=1 -DHAVE_FTS_H=1 -DHAVE_APR_POOLS_H=1 -DHAVE_LIBAPR_1=1 -DHAVE_BOOST_VERSION_HPP=1 -DHAVE_LIBCURL=1 -DHAVE_GLOG_LOGGING_H=1 -DHAVE_ARCHIVE_H=1 -DHAVE_GOOGLE_PROTOBUF_MAP_H=1 -DHAVE_EV_H=1 -DHAVE_PICOJSON_H=1 -DHAVE_RAPIDJSON_RAPIDJSON_H=1 -DHAVE_LIBSASL2=1 -DHAVE_OPENSSL_SSL_H=1 -DHAVE_SVN_VERSION_H=1 -DHAVE_LIBSVN_SUBR_1=1 -DHAVE_SVN_DELTA_H=1 -DHAVE_LIBSVN_DELTA_1=1 -DHAVE_ZLIB_H=1 -DHAVE_LIBZ=1 -DHAVE_GRPCPP_GRPCPP_H=1 -DHAVE_ZOOKEEPER_H=1 -I. -DLIBDIR=\"/usr/local/lib\" -DPKGLIBEXECDIR=\"/usr/local/libexec/mesos\" -DPKGDATADIR=\"/usr/local/share/mesos\" -DPKGMODULEDIR=\"/usr/local/lib/mesos/modules\" -I../include -I../include -I../include/mesos -D__STDC_FORMAT_MACROS -I../3rdparty/concurrentqueue-7b69a8f -I../3rdparty/elfio-3.2 -I../3rdparty/leveldb-1.19/include -I../3rdparty/libprocess/include -I../3rdparty/nvml-352.79 -I../3rdparty/stout/include -I/usr/local/include/subversion-1 -I/usr/local/include -I/usr/local/include -I/usr/local/include/apr-1 -I/usr/local/include/apr-1.0 -isystem /usr/local/include -I/usr/local/include -I/usr/local/include -I/usr/local/include -I/usr/local/include -I/usr/local/include -I/usr/local/include -I/usr/local/include -I/usr/local/include/zookeeper -D_THREAD_SAFE -pthread -Wall -Wsign-compare -Wformat-security -fstack-protector-strong -fPIC -O2 -pipe -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing -isystem /usr/local/include -Wno-unused-local-typedef -Wno-inconsistent-missing-override -std=c++11 -stdlib=libc++ -DGTEST_HAS_PTHREAD -D_WITH_GETLINE -Wno-sentinel -MT master/libmesos_no_3rdparty_la-constants.lo -MD -MP -MF master/.deps/libmesos_no_3rdparty_la-constants.Tpo -c master/constants.cpp  -fPIC -DPIC -o master/.libs/libmesos_no_3rdparty_la-constants.o
master/allocator/mesos/hierarchical.cpp:1649:5: error: use of undeclared identifier 'random_shuffle'; did you mean 'std::random_shuffle'?
    random_shuffle(resourceVector.begin(), resourceVector.end());
    ^~~~~~~~~~~~~~
    std::random_shuffle
/usr/include/c++/v1/algorithm:3168:1: note: 'std::random_shuffle' declared here
random_shuffle(_RandomAccessIterator __first, _RandomAccessIterator __last)
^

Sorry for forgetting about this. I didn't added it to my TODO list.
This doesn't build on FreeBSD 12 due to following error:

libtool: compile:  c++ -DPACKAGE_NAME=\"mesos\" -DPACKAGE_TARNAME=\"mesos\" -DPACKAGE_VERSION=\"1.8.1\" "-DPACKAGE_STRING=\"mesos 1.8.1\"" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -DPACKAGE=\"mesos\" -DVERSION=\"1.8.1\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -DHAVE_CXX11=1 -DHAVE_PTHREAD_PRIO_INHERIT=1 -DHAVE_PTHREAD=1 -DHAVE_FTS_H=1 -DHAVE_APR_POOLS_H=1 -DHAVE_LIBAPR_1=1 -DHAVE_BOOST_VERSION_HPP=1 -DHAVE_LIBCURL=1 -DHAVE_GLOG_LOGGING_H=1 -DHAVE_ARCHIVE_H=1 -DHAVE_GOOGLE_PROTOBUF_MAP_H=1 -DHAVE_EV_H=1 -DHAVE_PICOJSON_H=1 -DHAVE_RAPIDJSON_RAPIDJSON_H=1 -DHAVE_LIBSASL2=1 -DHAVE_OPENSSL_SSL_H=1 -DHAVE_SVN_VERSION_H=1 -DHAVE_LIBSVN_SUBR_1=1 -DHAVE_SVN_DELTA_H=1 -DHAVE_LIBSVN_DELTA_1=1 -DHAVE_ZLIB_H=1 -DHAVE_LIBZ=1 -DHAVE_GRPCPP_GRPCPP_H=1 -DHAVE_ZOOKEEPER_H=1 -I. -DLIBDIR=\"/usr/local/lib\" -DPKGLIBEXECDIR=\"/usr/local/libexec/mesos\" -DPKGDATADIR=\"/usr/local/share/mesos\" -DPKGMODULEDIR=\"/usr/local/lib/mesos/modules\" -I../include -I../include -I../include/mesos -D__STDC_FORMAT_MACROS -I../3rdparty/concurrentqueue-7b69a8f -I../3rdparty/elfio-3.2 -I../3rdparty/leveldb-1.19/include -I../3rdparty/libprocess/include -I../3rdparty/nvml-352.79 -I../3rdparty/stout/include -I/usr/local/include/subversion-1 -I/usr/local/include -I/usr/local/include -I/usr/local/include/apr-1 -I/usr/local/include/apr-1.0 -isystem /usr/local/include -I/usr/local/include -I/usr/local/include -I/usr/local/include -I/usr/local/include -I/usr/local/include -I/usr/local/include -I/usr/local/include -I/usr/local/include/zookeeper -D_THREAD_SAFE -pthread -Wall -Wsign-compare -Wformat-security -fstack-protector-strong -fPIC -O2 -pipe -fstack-protector-strong -isystem /usr/local/include -fno-strict-aliasing -isystem /usr/local/include -Wno-unused-local-typedef -Wno-inconsistent-missing-override -std=c++11 -stdlib=libc++ -DGTEST_HAS_PTHREAD -D_WITH_GETLINE -Wno-sentinel -MT master/libmesos_no_3rdparty_la-constants.lo -MD -MP -MF master/.deps/libmesos_no_3rdparty_la-constants.Tpo -c master/constants.cpp  -fPIC -DPIC -o master/.libs/libmesos_no_3rdparty_la-constants.o
master/allocator/mesos/hierarchical.cpp:1649:5: error: use of undeclared identifier 'random_shuffle'; did you mean 'std::random_shuffle'?
    random_shuffle(resourceVector.begin(), resourceVector.end());
    ^~~~~~~~~~~~~~
    std::random_shuffle
/usr/include/c++/v1/algorithm:3168:1: note: 'std::random_shuffle' declared here
random_shuffle(_RandomAccessIterator __first, _RandomAccessIterator __last)
^

That's strange, did the patch not apply, as it was to fix this very issue?

lwhsu requested changes to this revision.Sep 14 2019, 8:24 PM

I can build this port in poudriere fine. I think previous error was because of patching issue.

There are some issues:

  1. Please check portlint -acbt output.
  2. poudriere errors at:
====> Running Q/A tests (stage-qa)
Error: '/usr/local/bin/python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-ps'
Error: '/usr/local/bin/python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-tail'
Error: '/usr/local/bin/python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-cat'
Error: '/usr/local/bin/python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-scp'
*** Error code 1

I think this is because USE=python:run is only defined when CLI option enabled.

Makefile
18 ↗(On Diff #60563)

This blank line is not necessary, for making portlint(1) happy.

This revision now requires changes to proceed.Sep 14 2019, 8:24 PM

@james.wright_digital-chaos.com: do you want to also submit the patch to the upstream?

I don't think I need to submit the patch upstream as they have changed this portion of code in their master branch anyway!

====> Running Q/A tests (stage-qa)
Error: '/usr/local/bin/python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-ps'
Error: '/usr/local/bin/python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-tail'
Error: '/usr/local/bin/python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-cat'
Error: '/usr/local/bin/python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-scp'
*** Error code 1

I think this is because USE=python:run is only defined when CLI option enabled.

Tried moving shebangfix from USES= to CLI_USES= but still get the same error when running make stage-qa.

Please could you provide a suggestion how to fix this issue as I'm stumped as to what is the actual problem here.

lwhsu added a comment.Tue, Sep 17, 1:20 AM
====> Running Q/A tests (stage-qa)
Error: '/usr/local/bin/python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-ps'
Error: '/usr/local/bin/python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-tail'
Error: '/usr/local/bin/python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-cat'
Error: '/usr/local/bin/python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-scp'
*** Error code 1

I think this is because USE=python:run is only defined when CLI option enabled.

Tried moving shebangfix from USES= to CLI_USES= but still get the same error when running make stage-qa.
Please could you provide a suggestion how to fix this issue as I'm stumped as to what is the actual problem here.

Because python is used for those bin/* files, the quickest way is moving python:run to USE=, however, I tried this and poudriere gives many Error: Orphaned: include/*.
Can you check this part?

Makefile changes

  • Fixed typo; CLI_USE= should have been CLI_USES=
  • Added 'shebangfix' to CLI_USES=
  • Removed extraneous whitespace (to satisfy portlint)

sysutils/Makefile changes

  • Added SUBDIR += apache-mesos

Tried moving shebangfix from USES= to CLI_USES= but still get the same error when running make stage-qa.
Please could you provide a suggestion how to fix this issue as I'm stumped as to what is the actual problem here.

Because python is used for those bin/* files, the quickest way is moving python:run to USE=, however, I tried this and poudriere gives many Error: Orphaned: include/*.
Can you check this part?

I think the previous errors (regarding python and shebangfix) were caused by a typo in CLI_USES=, I have since updated the diff and they seem to have been resolved.

Please could you run a build through poudriere and let me know if there are any outstanding issues?

lwhsu added a comment.Wed, Sep 18, 9:08 PM

Because python is used for those bin/* files, the quickest way is moving python:run to USE=, however, I tried this and poudriere gives many Error: Orphaned: include/*.
Can you check this part?

I think the previous errors (regarding python and shebangfix) were caused by a typo in CLI_USES=, I have since updated the diff and they seem to have been resolved.
Please could you run a build through poudriere and let me know if there are any outstanding issues?

Still:

====> Running Q/A tests (stage-qa)
Error: '/usr/bin/env python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-ps'
Error: '/usr/bin/env python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-tail'
Error: '/usr/bin/env python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-scp'
Error: '/usr/bin/env python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-cat'
*** Error code 1

The problem is those files under bin/ need Python, are installed even when CLI is not defined, while python:run and shebangfix are only defined when CLI is on.
So, the possible solution is either only install these files when CLI is defined, or having python:run and shebangfix defined globally.

Still:

====> Running Q/A tests (stage-qa)
Error: '/usr/bin/env python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-ps'
Error: '/usr/bin/env python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-tail'
Error: '/usr/bin/env python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-scp'
Error: '/usr/bin/env python' is an invalid shebang you need USES=shebangfix for 'bin/mesos-cat'
*** Error code 1

The problem is those files under bin/ need Python, are installed even when CLI is not defined, while python:run and shebangfix are only defined when CLI is on.
So, the possible solution is either only install these files when CLI is defined, or having python:run and shebangfix defined globally.

I don't think those files should be installed when CLI is not defined, because pkg-plist is as follows;

%%CLI%%bin/mesos
%%CLI%%bin/mesos-cat
%%CLI%%bin/mesos-execute
%%CLI%%bin/mesos-local
%%CLI%%bin/mesos-log
%%CLI%%bin/mesos-ps
%%CLI%%bin/mesos-resolve
%%CLI%%bin/mesos-scp
%%CLI%%bin/mesos-tail

I thought this meant that the files would only be installed with CLI defined, is that not the case?

Please advise if there is an alternative method to prevent these files being installed if CLI is not defined, and which satisfies the stage-qa tests.

  • Updated port to 1.9.0
  • Added databases/leveldb dependency
  • Added devel/elfio dependency
  • Added www/http-parser dependency
  • Additional patches to fix compile errors
  • Removed CLI option (always installed)