Page MenuHomeFreeBSD

Prepare the ports tree for reproducible builds
Needs ReviewPublic

Authored by emaste on Dec 4 2015, 7:34 PM.


Group Reviewers

Add a default timestamp arbritrary to when this will be committed
Add a timestamp in distinfo each time one runs make makesum
This timestamp is only updated if the content of distinfo actually changes.

Export that timestamp as SOURCE_DATE_EPOCH respecting the reproducible build:

(patched pkg is needed to have this actually working)

Diff Detail

Lint Skipped
Unit Tests Skipped

Event Timeline

bapt retitled this revision from to Prepare the ports tree for reproducible builds.
bapt updated this object.
bapt edited the test plan for this revision. (Show Details)
bapt added a reviewer: portmgr.
bapt edited edge metadata.

Remove CROSS_BUILD stuff that crept in

bdrewery requested changes to this revision.Dec 4 2015, 7:48 PM
bdrewery added a reviewer: bdrewery.
bdrewery added a subscriber: bdrewery.
bdrewery added inline comments.

This will hurt dependency calculation for Poudriere. Please wrap in something like this to prevent running on depends targets and -V, unless looking at -V TIMESTAMP .if !make(*depends*) && empty(.MAKEFLAGS:M-V) && empty(.MAKEFLAGS:M*TIMESTAMP)

This revision now requires changes to proceed.Dec 4 2015, 7:48 PM

No *, just :MTIMESTAMP

What do you think about committing the makesum part by itself first, so that these start getting populated?


1449231300 is Fri 4 Dec 2015 12:15:00 UTC
Presumably will update if/when ready to commit.


Maybe TIMESTAMP = ##### for consistency with other lines in the file?


just rm -f rather than testing first?


We should also explain why there is a default value for TIMESTAMP in a comment, since the value seems strangely arbitrary. (I mean, it is arbitrary, but reasonably so.)

emaste updated this revision to Diff 16264.
emaste added a reviewer: bapt.
emaste edited edge metadata.
emaste removed rP FreeBSD ports repository as the repository for this revision.

Rebase after rP415078

Wasn't this committed? Can we close?

Wasn't this committed? Can we close?

This is not yet committed. I extracted and committed one part of it to start populating TIMESTAMP = in distinfo, but nothing in tree makes use of it yet.

  • Bump default timestamp to nice round 1500000000 since it s no longer a future time
  • Set SOURCE_DATE_EPOCH only for the pkg env, not the build env
  • Add a comment
antoine requested changes to this revision.EditedNov 21 2017, 10:21 PM
antoine added a subscriber: antoine.

This causes problem with at least all python ports:

  • They record the timestamp of the .py file in .pyc/.pyo files
  • Your change changes the last modification of the .py file to a later date
  • When importing the python module as root, python notices that the timestamp in .pyc/.pyo is earlier than the last modification of the .py file and regenerates them

See also: and

This revision now requires changes to proceed.Nov 21 2017, 10:21 PM

What would you suggest for selectively setting PKG_ENV (e.g., for all but python ports, for now)?

How does debian deal with the python modules ?

Debian doesn't ship the .pyc / .pyo files; they generate them at install time.

Would it be reasonable to scan the plist for any port that sets USES=python, and avoid overriding the package archive metadata if any .pyo/.pyc found?

I found 342 ports contain .pyc and .pyo files, 3 ports contain only .pyo, and 132 contain only .pyc. Of the 477 ports with either .pyc or .pyo, 425 have a Makefile with a USES=python line. 52 do not have USES=python (3211 ports have USES=python but do not contain .pyo or .pyc files.) The ones without USES=python appear to all be subordinate ports.

For now avoid setting the reproducible build timestamp in PKG_ENV for all ports that set USES=python


What about:

.if !make(*depends*) && empty(.MAKEFLAGS:M-V) && \
TIMESTAMP!= ${AWK} -F' = ' -v ts=${TIMESTAMP} '/^TIMESTAMP/ { ts=$$2 } END { print ts}' ${DISTINFO_FILE}

@bdrewery, any comment on my proposal above?


This is probably OK but can we just whitelist the target it is needed in instead? make package?


Some ports may wish to use it for the build (via SOURCE_DATE_EPOCH=` in the environment or similar approaches)

  • Skip distinfo timestamp fetching for dep targets (requested by @bdrewery)

Why the guard on USES=python?


See @antoine's comment above,

This causes problem with at least all python ports:

We have to somehow address python ports for reproducible builds but I hope we can apply to everything else at first at least.