Prepare the ports tree for reproducible builds
Needs ReviewPublic

Authored by emaste on Dec 4 2015, 7:34 PM.

Details

Reviewers
bapt
bdrewery
antoine
Group Reviewers
portmgr
Summary

Add a default timestamp arbritrary to when this will be committed
Add a timestamp in distinfo each time one runs make makesum
This timestamp is only updated if the content of distinfo actually changes.

Export that timestamp as SOURCE_DATE_EPOCH respecting the reproducible build:
https://reproducible-builds.org/specs/source-date-epoch/

(patched pkg is needed to have this actually working)
https://github.com/freebsd/pkg/commit/69ee4f6417f5fa5f8204e113deb82af3e73bb97f

Diff Detail

Lint
Lint Skipped
Unit
Unit Tests Skipped
bapt updated this revision to Diff 10748.Dec 4 2015, 7:34 PM
bapt retitled this revision from to Prepare the ports tree for reproducible builds.
bapt updated this object.
bapt edited the test plan for this revision. (Show Details)
bapt added a reviewer: portmgr.
bapt updated this revision to Diff 10750.Dec 4 2015, 7:39 PM
bapt edited edge metadata.

Remove CROSS_BUILD stuff that crept in

bdrewery requested changes to this revision.Dec 4 2015, 7:48 PM
bdrewery added a reviewer: bdrewery.
bdrewery added a subscriber: bdrewery.
bdrewery added inline comments.
Mk/bsd.port.mk
2053–2056

This will hurt dependency calculation for Poudriere. Please wrap in something like this to prevent running on depends targets and -V, unless looking at -V TIMESTAMP .if !make(*depends*) && empty(.MAKEFLAGS:M-V) && empty(.MAKEFLAGS:M*TIMESTAMP)

This revision now requires changes to proceed.Dec 4 2015, 7:48 PM
bdrewery added inline comments.Dec 4 2015, 7:48 PM
Mk/bsd.port.mk
2053–2056

No *, just :MTIMESTAMP

emaste added a subscriber: emaste.Dec 7 2015, 7:40 PM
swills added a subscriber: swills.Apr 6 2016, 1:32 AM
adamw added a subscriber: adamw.Apr 10 2016, 4:11 PM

What do you think about committing the makesum part by itself first, so that these start getting populated?

Mk/bsd.port.mk
2053

1449231300 is Fri 4 Dec 2015 12:15:00 UTC
Presumably will update if/when ready to commit.

3918

Maybe TIMESTAMP = ##### for consistency with other lines in the file?

3923

just rm -f rather than testing first?

emaste added inline comments.Apr 19 2016, 2:29 PM
Mk/bsd.port.mk
2053

We should also explain why there is a default value for TIMESTAMP in a comment, since the value seems strangely arbitrary. (I mean, it is arbitrary, but reasonably so.)

emaste commandeered this revision.May 12 2016, 7:39 PM
emaste updated this revision to Diff 16264.
emaste added a reviewer: bapt.
emaste edited edge metadata.
emaste removed rP FreeBSD ports repository as the repository for this revision.

Rebase after rP415078

Wasn't this committed? Can we close?

Wasn't this committed? Can we close?

This is not yet committed. I extracted and committed one part of it to start populating TIMESTAMP = in distinfo, but nothing in tree makes use of it yet.

Ping, any progress on this?

emaste updated this revision to Diff 35557.Nov 21 2017, 6:44 PM
  • Bump default timestamp to nice round 1500000000 since it s no longer a future time
  • Set SOURCE_DATE_EPOCH only for the pkg env, not the build env
  • Add a comment
antoine requested changes to this revision.EditedNov 21 2017, 10:21 PM
antoine added a subscriber: antoine.

This causes problem with at least all python ports:

  • They record the timestamp of the .py file in .pyc/.pyo files
  • Your change changes the last modification of the .py file to a later date
  • When importing the python module as root, python notices that the timestamp in .pyc/.pyo is earlier than the last modification of the .py file and regenerates them

See also: https://github.com/python/cpython/pull/296 and https://bugs.python.org/issue29708

This revision now requires changes to proceed.Nov 21 2017, 10:21 PM

What would you suggest for selectively setting PKG_ENV (e.g., for all but python ports, for now)?

mat added a subscriber: mat.Nov 24 2017, 3:59 PM

How does debian deal with the python modules ?

Debian doesn't ship the .pyc / .pyo files; they generate them at install time.

Would it be reasonable to scan the plist for any port that sets USES=python, and avoid overriding the package archive metadata if any .pyo/.pyc found?

I found 342 ports contain .pyc and .pyo files, 3 ports contain only .pyo, and 132 contain only .pyc. Of the 477 ports with either .pyc or .pyo, 425 have a Makefile with a USES=python line. 52 do not have USES=python (3211 ports have USES=python but do not contain .pyo or .pyc files.) The ones without USES=python appear to all be subordinate ports.

emaste updated this revision to Diff 35702.Nov 24 2017, 6:51 PM

For now avoid setting the reproducible build timestamp in PKG_ENV for all ports that set USES=python

emaste added inline comments.Nov 24 2017, 6:59 PM
Mk/bsd.port.mk
2053–2056

What about:

.if !make(*depends*) && empty(.MAKEFLAGS:M-V) && \
    empty(.MAKEFLAGS:MTIMESTAMP) && exists(${DISTINFO_FILE})
TIMESTAMP!= ${AWK} -F' = ' -v ts=${TIMESTAMP} '/^TIMESTAMP/ { ts=$$2 } END { print ts}' ${DISTINFO_FILE}
.endif
emaste added a comment.Mon, Apr 2, 1:50 PM

@bdrewery, any comment on my proposal above?

bdrewery added inline comments.Mon, Apr 2, 8:29 PM
Mk/bsd.port.mk
2053–2056

This is probably OK but can we just whitelist the target it is needed in instead? make package?

emaste added inline comments.Mon, Apr 2, 8:31 PM
Mk/bsd.port.mk
2053–2056

Some ports may wish to use it for the build (via SOURCE_DATE_EPOCH=` in the environment or similar approaches)

emaste updated this revision to Diff 41004.Mon, Apr 2, 8:33 PM
  • Skip distinfo timestamp fetching for dep targets (requested by @bdrewery)
mat added inline comments.Tue, Apr 3, 1:55 PM
Mk/bsd.port.mk
2073–2075

Why the guard on USES=python?

emaste added inline comments.Tue, Apr 3, 2:09 PM
Mk/bsd.port.mk
2073–2075

See @antoine's comment above,

This causes problem with at least all python ports:

We have to somehow address python ports for reproducible builds but I hope we can apply to everything else at first at least.