Prepare the ports tree for reproducible builds

[emaste]

Summary

Add a default timestamp arbritrary to when this will be committed
Add a timestamp in distinfo each time one runs make makesum
This timestamp is only updated if the content of distinfo actually changes.

Export that timestamp as SOURCE_DATE_EPOCH respecting the reproducible build:
https://reproducible-builds.org/specs/source-date-epoch/

(patched pkg is needed to have this actually working)
https://github.com/freebsd/pkg/commit/69ee4f6417f5fa5f8204e113deb82af3e73bb97f

[bapt]

Remove CROSS_BUILD stuff that crept in

[bdrewery]

Mk/bsd.port.mk

2053–2056

This will hurt dependency calculation for Poudriere. Please wrap in something like this to prevent running on depends targets and -V, unless looking at -V TIMESTAMP .if !make(*depends*) && empty(.MAKEFLAGS:M-V) && empty(.MAKEFLAGS:M*TIMESTAMP)

[bdrewery]

Mk/bsd.port.mk

2053–2056

No *, just :MTIMESTAMP

[emaste]

What do you think about committing the makesum part by itself first, so that these start getting populated?

Mk/bsd.port.mk

2053

1449231300 is Fri 4 Dec 2015 12:15:00 UTC
Presumably will update if/when ready to commit.

3918

Maybe TIMESTAMP = ##### for consistency with other lines in the file?

3923

just rm -f rather than testing first?

[emaste]

Mk/bsd.port.mk

2053

We should also explain why there is a default value for TIMESTAMP in a comment, since the value seems strangely arbitrary. (I mean, it is arbitrary, but reasonably so.)

[emaste]

Rebase after rP415078

[swills]

Wasn't this committed? Can we close?

[emaste]

Wasn't this committed? Can we close?

This is not yet committed. I extracted and committed one part of it to start populating TIMESTAMP = in distinfo, but nothing in tree makes use of it yet.

[swills]

Ping, any progress on this?

[emaste]

• Bump default timestamp to nice round 1500000000 since it s no longer a future time
• Set SOURCE_DATE_EPOCH only for the pkg env, not the build env
• Add a comment

[antoine]

This causes problem with at least all python ports:

• They record the timestamp of the .py file in .pyc/.pyo files
• Your change changes the last modification of the .py file to a later date
• When importing the python module as root, python notices that the timestamp in .pyc/.pyo is earlier than the last modification of the .py file and regenerates them

See also: https://github.com/python/cpython/pull/296 and https://bugs.python.org/issue29708

[emaste]

What would you suggest for selectively setting PKG_ENV (e.g., for all but python ports, for now)?

[mat]

How does debian deal with the python modules ?

[emaste]

Debian doesn't ship the .pyc / .pyo files; they generate them at install time.

[emaste]

Would it be reasonable to scan the plist for any port that sets USES=python, and avoid overriding the package archive metadata if any .pyo/.pyc found?

I found 342 ports contain .pyc and .pyo files, 3 ports contain only .pyo, and 132 contain only .pyc. Of the 477 ports with either .pyc or .pyo, 425 have a Makefile with a USES=python line. 52 do not have USES=python (3211 ports have USES=python but do not contain .pyo or .pyc files.) The ones without USES=python appear to all be subordinate ports.

[emaste]

For now avoid setting the reproducible build timestamp in PKG_ENV for all ports that set USES=python

[emaste]

Mk/bsd.port.mk

2053–2056

What about:

.if !make(*depends*) && empty(.MAKEFLAGS:M-V) && \
    empty(.MAKEFLAGS:MTIMESTAMP) && exists(${DISTINFO_FILE})
TIMESTAMP!= ${AWK} -F' = ' -v ts=${TIMESTAMP} '/^TIMESTAMP/ { ts=$$2 } END { print ts}' ${DISTINFO_FILE}
.endif

[emaste]

@bdrewery, any comment on my proposal above?

[bdrewery]

Mk/bsd.port.mk

2053–2056

This is probably OK but can we just whitelist the target it is needed in instead? make package?

[emaste]

Mk/bsd.port.mk

2053–2056

Some ports may wish to use it for the build (via SOURCE_DATE_EPOCH=` in the environment or similar approaches)

[emaste]

• Skip distinfo timestamp fetching for dep targets (requested by @bdrewery)

[mat]

Mk/bsd.port.mk

2073–2075

Why the guard on USES=python?

[emaste]

Mk/bsd.port.mk

2073–2075

See @antoine's comment above,

This causes problem with at least all python ports:

We have to somehow address python ports for reproducible builds but I hope we can apply to everything else at first at least.