Page MenuHomeFreeBSD

First version of net/ceph-devel
ClosedPublic

Authored by wjw_digiware.nl on Feb 14 2017, 1:50 AM.

Details

Summary

First version of a port for Ceph to FreeBSD.
This is based on development code.

Once a new major version is released (Luminous),
there will also be a ceph-port based on the release schedule of Ceph.

Diff Detail

Repository
rP FreeBSD ports repository
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
wjw_digiware.nl added inline comments.
net/ceph-devel/files/ceph.in
26 ↗(On Diff #25275)

Yah, that is such a fixed brain pattern.

wjw_digiware.nl marked an inline comment as done.

net/ceph-devel: Fixed some more paths

Poudriere still shows a lot of orphaned files that aren't present in pkg-plist. If it's your intentional not to install these things, you can either modify the build to not build or not install them, or you can simply delete them from the STAGEDIR during the post-install target. Examples:

Error: Orphaned: bin/ceph_tpbench
Error: Orphaned: %%PYTHON_SITELIBDIR%%/ceph_detect_init/__init__.pyc
Error: Orphaned: sbin/ceph-disk-udev
Error: Orphaned: @dir etc/init.d

Poudriere still shows a lot of orphaned files that aren't present in pkg-plist. If it's your intentional not to install these things, you can either modify the build to not build or not install them, or you can simply delete them from the STAGEDIR during the post-install target. Examples:

Error: Orphaned: bin/ceph_tpbench
Error: Orphaned: %%PYTHON_SITELIBDIR%%/ceph_detect_init/__init__.pyc
Error: Orphaned: sbin/ceph-disk-udev
Error: Orphaned: @dir etc/init.d

Is this a requirement?
Because it is not a real option to modify the CMakefiles to only install what is packaged...

What options are you running poudriere with, because my runs 'just work'

The difference between my Poudriere and yours is probably the following line in /etc/make.conf:
DEVELOPER=yes
If modifying the cmake files is hard, then just delete those files during post-install. Plenty of ports do that. For example, net/corosync.

Alan,

I'll set the Developer=yes in /etc/make.conf.
I guess it needs to go in the jail?

You did not answer the first question: is it required?
Because it is quite a number of files that need to be deleted.

In corosync it is only 2 files that need to be removed. So that is easy.
And I'd then run find ${STAGEDIR} against the list in pkg-plist to remove what is not there.
But then again that is not trivial, since all the %%_%% codes need to be resolved..

Alan,

I'll set the Developer=yes in /etc/make.conf.
I guess it needs to go in the jail?

Nope. If you set it in the base file system, Poudriere will automatically copy it to the jail.

You did not answer the first question: is it required?
Because it is quite a number of files that need to be deleted.

As far as I'm concerned, it is. Plus, it will make things much easier the next time you update the port. If Ceph adds a new file, you'll notice.

In corosync it is only 2 files that need to be removed. So that is easy.
And I'd then run find ${STAGEDIR} against the list in pkg-plist to remove what is not there.
But then again that is not trivial, since all the %%_%% codes need to be resolved..

Poudriere makes it obvious when DEVELOPER=yes. I'll email you my full Poudriere log so you can see.

net/ceph-devel: fixed the staging directory for not used items

The difference between my Poudriere and yours is probably the following line in /etc/make.conf:
DEVELOPER=yes
If modifying the cmake files is hard, then just delete those files during post-install. Plenty of ports do that. For example, net/corosync.

poudriere testport sets DEVELOPER=yes automatically. (as does bulk -t)

And I'd then run find ${STAGEDIR} against the list in pkg-plist to remove what is not there.
But then again that is not trivial, since all the %%_%% codes need to be resolved..

make makeplist does the find and %% dance for you, why bother doing it manually ?

net/ceph-devel/Makefile
73 ↗(On Diff #25292)

This does nothing.

76–85 ↗(On Diff #25292)

I know those are supposed to work, but booleans should be defined as, for example, ENABLE_GIT_VERSION:BOOL=YES

**>>>! In D9584#199162, @wjw_digiware.nl wrote:

And I'd then run find ${STAGEDIR} against the list in pkg-plist to remove what is not there.
But then again that is not trivial, since all the %%_%% codes need to be resolved..

make makeplist does the find and %% dance for you, why bother doing it manually ?

I do not want all the ceph_tests_* and ceph_bench* tools in the package.
People that want to run tests or other things should actually use the stuff in github.

net/ceph-devel: added type of CMake defines

net/ceph-devel: Fixed a type on pkg-plist

Mat, Alan,

I think that this version runs poudriere testport without complaints/errors.

Almost done. But there are a few more portlint warnings. As @mat said, you don't need to worry about portlint's "NLS knob" warning, but you should fix the others.

net/ceph-devel: fixing portlint -A warnings and errors

asomers edited edge metadata.

It looks like you accidentally excluded UIDs, GIDs, and net/Makefile from the most recent revision. Remember, when you're using Phabricator to update a revision, you must always put every modified file on the command line, even if you haven't touched that file since the last update. Otherwise, Phabricator thinks that you want to remove those files from the review.

Apart from that, lgtm. You still need approval from @mat, because i'm not a ports committer.

wjw_digiware.nl edited edge metadata.

net/ceph-devel: need to include everything

UIDs/GIDs
net/Makefile

asomers edited edge metadata.
mat requested changes to this revision.Feb 21 2017, 12:06 PM
mat edited edge metadata.

Other than that, it looks good, I assume that you have tested that poudriere testport does not complain about anything ? (the "I think" you said makes me wonder)

net/ceph-devel/Makefile
34–36 ↗(On Diff #25447)

You have devel/nspr listed more than once, remove all but one.

37–43 ↗(On Diff #25447)

You have devel/boost-libs listed more than once, please remove all but one.

44 ↗(On Diff #25447)

You already have USES=python:2.7, so it is not needed.

net/ceph-devel/pkg-plist
270–271 ↗(On Diff #25447)

Does this file come with ceph, or is it generated during the build ?

If it is generated during the build, it means everyone installing the package will get the same key, and that the key will change every time the package is rebuilt.

273–274 ↗(On Diff #25447)

If those are:

  • example configuration files, they should probably go in EXAMPLESDIR
  • default configuration files, they should probably go in ETCDIR as .sample
This revision now requires changes to proceed.Feb 21 2017, 12:06 PM
In D9584#200509, @mat wrote:

Other than that, it looks good, I assume that you have tested that poudriere testport does not complain about anything ? (the "I think" you said makes me wonder)

That says a bit about me.... I'm rarely confident that everything is satisfied.
But yes, poudriere testport had no complaints/warnings/errors.

The duplicate lips are due to either portlint or poudriere complaining about missing these references.
But it indeed needs the library only one time...
And it is a nice cleanup.

wjw_digiware.nl added inline comments.
net/ceph-devel/pkg-plist
270–271 ↗(On Diff #25447)

The keys come with Ceph. So the are the same every build.

273–274 ↗(On Diff #25447)

I will put the sample.ceph.conf in /usr/local/etc/ceph as ceph.conf.sample
and remove fetch_config

wjw_digiware.nl edited edge metadata.
wjw_digiware.nl marked 2 inline comments as done.

net/ceph-devel: Fixed location of sample files

In D9584#200509, @mat wrote:

Other than that, it looks good, I assume that you have tested that poudriere testport does not complain about anything ? (the "I think" you said makes me wonder)

Used your suggestion for the DOC files
Ran everything trhu portlint and poudriere porttest, no errors there,
3 libs are installed not stripped, but that is purposely, because they are to be linked when users start building tools on librados.

So that is it, ready to go?

net/ceph-devel/pkg-plist
273 ↗(On Diff #25516)

This is not properly sorted, it should be earlier, with the other etc files. Please generate the plist with make makeplist.

wjw_digiware.nl edited edge metadata.

net/ceph-devel: sorted pkg-plist

net/ceph-devel/pkg-plist
51 ↗(On Diff #25620)

This is not needed, it is added by USE_RC_SUBR.

net/ceph-devel: Deleted auto file etc/rc.d/ceph

@mat
Is there anything else I need to do?

@mat
Sorry to nag, but is there a reason this does not move forward?

Regards,
--WjW

I'm satisfied. It @mat is too, then he can commit it. I can't commit it myself.

@mat
Sorry to nag, but is there a reason this does not move forward?

Well, this, here, is a code review tool, we review code, and make it better, and then commit it. For non committers, I still think that once everything is ok, a PR should be openned with the final patch so that it is added to the tree. I don't have the spare time to test-build it and commit it.

In D9584#206275, @mat wrote:

@mat
Sorry to nag, but is there a reason this does not move forward?

Well, this, here, is a code review tool, we review code, and make it better, and then commit it. For non committers, I still think that once everything is ok, a PR should be openned with the final patch so that it is added to the tree. I don't have the spare time to test-build it and commit it.

@mat,

Thanx, I'll update the PR with this patch and a link to this as review process,

Added my mentors for review-before-commit process. (@mat is already here, I added @feld too)
I made few small changes (regenerated distinfo, on PR-attached patch @wjw_digiware.nl confirmed it builds fine

I can't seem to apply this cleanly

$ arc patch D9584
A         net/ceph-devel
A         net/ceph-devel/files
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|Index: net/ceph-devel/pkg-plist
|===================================================================
|--- /dev/null
|+++ net/ceph-devel/pkg-plist
--------------------------
(Creating file net/ceph-devel/pkg-plist...)
Patching file net/ceph-devel/pkg-plist using Plan A...
Empty context always matches.
Hunk #1 succeeded at 1.
done
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|Index: net/ceph-devel/pkg-descr
|===================================================================
|--- /dev/null
|+++ net/ceph-devel/pkg-descr
--------------------------
(Creating file net/ceph-devel/pkg-descr...)
Patching file net/ceph-devel/pkg-descr using Plan A...
Empty context always matches.
Hunk #1 succeeded at 1.
done
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|Index: net/ceph-devel/files/patch-src_rocksdb_CMakeLists.txt
|===================================================================
|--- /dev/null
|+++ net/ceph-devel/files/patch-src_rocksdb_CMakeLists.txt
--------------------------
(Creating file net/ceph-devel/files/patch-src_rocksdb_CMakeLists.txt...)
Patching file net/ceph-devel/files/patch-src_rocksdb_CMakeLists.txt using Plan A...
Empty context always matches.
Hunk #1 succeeded at 1.
done
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|Index: net/ceph-devel/files/ceph.in
|===================================================================
|--- /dev/null
|+++ net/ceph-devel/files/ceph.in
--------------------------
(Creating file net/ceph-devel/files/ceph.in...)
Patching file net/ceph-devel/files/ceph.in using Plan A...
Empty context always matches.
Hunk #1 succeeded at 1.
done
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|Index: net/ceph-devel/distinfo
|===================================================================
|--- /dev/null
|+++ net/ceph-devel/distinfo
--------------------------
(Creating file net/ceph-devel/distinfo...)
Patching file net/ceph-devel/distinfo using Plan A...
Empty context always matches.
Hunk #1 succeeded at 1.
done
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|Index: net/ceph-devel/Makefile
|===================================================================
|--- /dev/null
|+++ net/ceph-devel/Makefile
--------------------------
(Creating file net/ceph-devel/Makefile...)
Patching file net/ceph-devel/Makefile using Plan A...
Empty context always matches.
Hunk #1 succeeded at 1.
done
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|Index: net/Makefile
|===================================================================
|--- net/Makefile
|+++ net/Makefile
--------------------------
Patching file net/Makefile using Plan A...
Reversed (or previously applied) patch detected!  Assume -R? [y]
Hunk #1 succeeded at 67.
done
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|Index: UIDs
|===================================================================
|--- UIDs
|+++ UIDs
--------------------------
Patching file UIDs using Plan A...
Hunk #1 failed at 113.
1 out of 1 hunks failed--saving rejects to UIDs.rej
done
Hmm...  Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|Index: GIDs
|===================================================================
|--- GIDs
|+++ GIDs
--------------------------
Patching file GIDs using Plan A...
Hunk #1 failed at 108.
1 out of 1 hunks failed--saving rejects to GIDs.rej
done
svn: warning: W150002: '/usr/ports/net/ceph-devel/files' is already under version control
svn: E200009: Could not add all targets because some targets are already versioned
svn: E200009: Illegal target for the requested operation
A         net/ceph-devel/pkg-plist
A         net/ceph-devel/pkg-descr
A         net/ceph-devel/files/patch-src_rocksdb_CMakeLists.txt
A         net/ceph-devel/files/ceph.in
A         net/ceph-devel/distinfo
A         net/ceph-devel/Makefile
svn: warning: W150002: '/usr/ports/net/ceph-devel' is already under version control
svn: E200009: Could not add all targets because some targets are already versioned
svn: E200009: Illegal target for the requested operation
property 'fbsd:nokeywords' set on 'net/ceph-devel/pkg-plist'
svn: E135001: Unrecognized line ending style 'native
\ No newline at end of property' for '/usr/ports/net/ceph-devel/pkg-plist'
svn: E125004: MIME type 'text/plain
\ No newline at end of property' contains invalid character '
' in media type
property 'fbsd:nokeywords' set on 'net/ceph-devel/pkg-descr'
svn: E135001: Unrecognized line ending style 'native
\ No newline at end of property' for '/usr/ports/net/ceph-devel/pkg-descr'
svn: E125004: MIME type 'text/plain
\ No newline at end of property' contains invalid character '
' in media type
property 'fbsd:nokeywords' set on 'net/ceph-devel/files/patch-src_rocksdb_CMakeLists.txt'
svn: E135001: Unrecognized line ending style 'native
\ No newline at end of property' for '/usr/ports/net/ceph-devel/files/patch-src_rocksdb_CMakeLists.txt'
svn: E125004: MIME type 'text/plain
\ No newline at end of property' contains invalid character '
' in media type
svn: E135001: Unrecognized line ending style 'native
\ No newline at end of property' for '/usr/ports/net/ceph-devel/files/ceph.in'
property 'svn:executable' set on 'net/ceph-devel/files/ceph.in'
svn: E125004: MIME type 'text/plain
\ No newline at end of property' contains invalid character '
' in media type
property 'fbsd:nokeywords' set on 'net/ceph-devel/distinfo'
svn: E135001: Unrecognized line ending style 'native
\ No newline at end of property' for '/usr/ports/net/ceph-devel/distinfo'
svn: E125004: MIME type 'text/plain
\ No newline at end of property' contains invalid character '
' in media type
svn: E135001: Unrecognized line ending style 'native
\ No newline at end of property' for '/usr/ports/net/ceph-devel/Makefile'
property 'svn:keywords' set on 'net/ceph-devel/Makefile'
svn: E125004: MIME type 'text/plain
\ No newline at end of property' contains invalid character '
' in media type

@feld
Not sure why it would fail.
Other than that @mmokhi 's changes are perhaps not in this review.
But that should not prevent you from importing the basic files.
But it seems to confilect on the ceph entries in GUIDs and UIDs.

What version are you using to patch on?

Then there is a lot of svn complaints, about which I know very little.
I'll try and find a virgin system here, and see if I can repeat

`arc patch D9584`

to see what comes of it.

That was on a clean modern ports tree, r436539

Hmm, yeah.
@feld I'd suggest to test the patch that I attached on the PR (or @wjw_digiware.nl can update diffs here :D)
I forgot to state this problem on PR (but it's fixed on attached patch) that recently things are added to GID/UID files right before our entry, which makes this patch fail.

Hmm, yeah.
@feld I'd suggest to test the patch that I attached on the PR (or @wjw_digiware.nl can update diffs here :D)
I forgot to state this problem on PR (but it's fixed on attached patch) that recently things are added to GID/UID files right before our entry, which makes this patch fail.

Oke, I see... Sorry about this.
It took me about a year to get used to the processes in the Ceph community, so I'll manage here as well.
Will work on it tonight. Doesn't seem like much work, and then both sides are again equal.
Should prevent a lot of annoyance.

Hmm, yeah.
@feld I'd suggest to test the patch that I attached on the PR (or @wjw_digiware.nl can update diffs here :D)
I forgot to state this problem on PR (but it's fixed on attached patch) that recently things are added to GID/UID files right before our entry, which makes this patch fail.

Oke, I see... Sorry about this.
It took me about a year to get used to the processes in the Ceph community, so I'll manage here as well.
Will work on it tonight. Doesn't seem like much work, and then both sides are again equal.
Should prevent a lot of annoyance.

Yeah. :-) It'd be great if you can sync phabricato patch with latest patch on Bugzilla, for tonight. (generally syncing patches in these two reduces confusings :-])

Also, you did great job on this ceph ports, personally I appreciate __/|\__

wjw_digiware.nl edited edge metadata.

net/ceph-devel: integrating mmokhi patches.

  • redo the UIDs/GIDs patches
  • fix some SHA values

net/ceph-devel: integrating mmokhi patches.

  • redo arc to include higher up files.
  • redo the UIDs/GIDs patches
  • fix some SHA values

@feld
Hi Mark,

Any other things I need to do to get this committed?
Bot phabricator and bugzilla should be in sync now.

feld requested changes to this revision.Mar 24 2017, 7:50 PM
feld added inline comments.
net/ceph-devel/Makefile
121–122 ↗(On Diff #26454)

What happened here? It appears the ports/head/Makefile has been appended to the end of the net/ceph-devel/Makefile by accident?

This revision now requires changes to proceed.Mar 24 2017, 7:50 PM
wjw_digiware.nl edited edge metadata.

net/ceph-devel: Update the submission

Clean the Makefile

Looks good now, builds cleanly.

Macro goforit:

This revision is now accepted and ready to land.Mar 28 2017, 1:36 PM
In D9584#210073, @feld wrote:

Looks good now, builds cleanly.

Macro goforit:

@feld

Thanx Mark,

  • wjw@cephtest> grep gopher /etc/services

gopher 70/tcp
gopher 70/udp

I used to run Gopher servers in the old days.... :)

This revision was automatically updated to reflect the committed changes.