Page MenuHomeFreeBSD
Differential D4385

Prepare the ports tree for reproducible builds
Needs ReviewPublic
Actions

Authored by emaste on Dec 4 2015, 7:34 PM.
Tags
None
Referenced Files
F102860695: D4385.id35557.diff
Mon, Nov 18, 2:57 AM
Unknown Object (File)
Tue, Nov 12, 9:58 PM
Unknown Object (File)
Thu, Nov 7, 7:44 PM
Unknown Object (File)
Oct 16 2024, 10:58 AM
Unknown Object (File)
Oct 15 2024, 12:22 PM
Unknown Object (File)
Oct 13 2024, 9:45 PM
Unknown Object (File)
Oct 12 2024, 6:05 PM
Unknown Object (File)
Oct 12 2024, 6:05 PM
View All Files
Subscribers
adamw
antoine
bdrewery
emaste
mat
swills
ygy

Details

Reviewers
bapt
bdrewery
antoine
Group Reviewers
portmgr
Summary

Add a default timestamp arbritrary to when this will be committed
Add a timestamp in distinfo each time one runs make makesum
This timestamp is only updated if the content of distinfo actually changes.

Export that timestamp as SOURCE_DATE_EPOCH respecting the reproducible build:
https://reproducible-builds.org/specs/source-date-epoch/

(patched pkg is needed to have this actually working)
https://github.com/freebsd/pkg/commit/69ee4f6417f5fa5f8204e113deb82af3e73bb97f

Diff Detail

Repository
rP FreeBSD ports repository
Lint
No Lint Coverage
Unit
No Test Coverage
Build Status
Buildable 1409
Build 1414: arc lint + arc unit

Event Timeline

bapt updated this revision to Diff 10748.Dec 4 2015, 7:34 PM
bapt retitled this revision from to Prepare the ports tree for reproducible builds.
bapt updated this object.
bapt edited the test plan for this revision. (Show Details)
bapt added a reviewer: portmgr.
Herald edited edge metadata. · View Herald TranscriptDec 4 2015, 7:34 PM
bapt updated this revision to Diff 10750.Dec 4 2015, 7:39 PM
bapt edited edge metadata.

Remove CROSS_BUILD stuff that crept in

Herald edited edge metadata. · View Herald TranscriptDec 4 2015, 7:39 PM
bdrewery requested changes to this revision.Dec 4 2015, 7:48 PM
bdrewery added a reviewer: bdrewery.
bdrewery added a subscriber: bdrewery.
bdrewery added inline comments.
Mk/bsd.port.mk
1973–1976

This will hurt dependency calculation for Poudriere. Please wrap in something like this to prevent running on depends targets and -V, unless looking at -V TIMESTAMP .if !make(*depends*) && empty(.MAKEFLAGS:M-V) && empty(.MAKEFLAGS:M*TIMESTAMP)

This revision now requires changes to proceed.Dec 4 2015, 7:48 PM
bdrewery added inline comments.Dec 4 2015, 7:48 PM
Mk/bsd.port.mk
1973–1976

No *, just :MTIMESTAMP

emaste added a subscriber: emaste.Dec 7 2015, 7:40 PM
swills added a subscriber: swills.Apr 6 2016, 1:32 AM
adamw added a subscriber: adamw.Apr 10 2016, 4:11 PM
emaste added a comment.Apr 18 2016, 8:22 PM

What do you think about committing the makesum part by itself first, so that these start getting populated?

Mk/bsd.port.mk
1973

1449231300 is Fri 4 Dec 2015 12:15:00 UTC
Presumably will update if/when ready to commit.

4225

Maybe TIMESTAMP = ##### for consistency with other lines in the file?

4230

just rm -f rather than testing first?

emaste added inline comments.Apr 19 2016, 2:29 PM
Mk/bsd.port.mk
1973

We should also explain why there is a default value for TIMESTAMP in a comment, since the value seems strangely arbitrary. (I mean, it is arbitrary, but reasonably so.)

emaste mentioned this in D6031: Record TIMESTAMP in `make makesum`.Apr 20 2016, 7:05 PM
emaste commandeered this revision.May 12 2016, 7:39 PM
emaste updated this revision to Diff 16264.
emaste added a reviewer: bapt.
emaste edited edge metadata.
emaste removed rP FreeBSD ports repository as the repository for this revision.

Rebase after rP415078

swills added a comment.May 16 2016, 2:38 AM

Wasn't this committed? Can we close?

emaste added a comment.May 16 2016, 2:45 AM

Wasn't this committed? Can we close?

This is not yet committed. I extracted and committed one part of it to start populating TIMESTAMP = in distinfo, but nothing in tree makes use of it yet.

emaste mentioned this in D2032: Set the timestamp of files in the stage dir to the timestamp of the newest distfile. This is to support reproducable builds. Note, doesn't work on 8.x, need to find a solution to that..May 31 2016, 6:47 PM
swills added a comment.Jan 27 2017, 3:21 PM

Ping, any progress on this?

emaste updated this revision to Diff 35557.Nov 21 2017, 6:44 PM
  • Bump default timestamp to nice round 1500000000 since it s no longer a future time
  • Set SOURCE_DATE_EPOCH only for the pkg env, not the build env
  • Add a comment
antoine requested changes to this revision.EditedNov 21 2017, 10:21 PM
antoine added a subscriber: antoine.

This causes problem with at least all python ports:

  • They record the timestamp of the .py file in .pyc/.pyo files
  • Your change changes the last modification of the .py file to a later date
  • When importing the python module as root, python notices that the timestamp in .pyc/.pyo is earlier than the last modification of the .py file and regenerates them

See also: https://github.com/python/cpython/pull/296 and https://bugs.python.org/issue29708

This revision now requires changes to proceed.Nov 21 2017, 10:21 PM
emaste added a comment.Nov 21 2017, 10:53 PM

What would you suggest for selectively setting PKG_ENV (e.g., for all but python ports, for now)?

mat added a subscriber: mat.Nov 24 2017, 3:59 PM

How does debian deal with the python modules ?

emaste added a comment.Nov 24 2017, 4:01 PM

Debian doesn't ship the .pyc / .pyo files; they generate them at install time.

emaste added a comment.Nov 24 2017, 6:41 PM

Would it be reasonable to scan the plist for any port that sets USES=python, and avoid overriding the package archive metadata if any .pyo/.pyc found?

I found 342 ports contain .pyc and .pyo files, 3 ports contain only .pyo, and 132 contain only .pyc. Of the 477 ports with either .pyc or .pyo, 425 have a Makefile with a USES=python line. 52 do not have USES=python (3211 ports have USES=python but do not contain .pyo or .pyc files.) The ones without USES=python appear to all be subordinate ports.

emaste updated this revision to Diff 35702.Nov 24 2017, 6:51 PM

For now avoid setting the reproducible build timestamp in PKG_ENV for all ports that set USES=python

emaste added inline comments.Nov 24 2017, 6:59 PM
Mk/bsd.port.mk
1973–1976

What about:

.if !make(*depends*) && empty(.MAKEFLAGS:M-V) && \
    empty(.MAKEFLAGS:MTIMESTAMP) && exists(${DISTINFO_FILE})
TIMESTAMP!= ${AWK} -F' = ' -v ts=${TIMESTAMP} '/^TIMESTAMP/ { ts=$$2 } END { print ts}' ${DISTINFO_FILE}
.endif
emaste added a comment.Apr 2 2018, 1:50 PM

@bdrewery, any comment on my proposal above?

bdrewery added inline comments.Apr 2 2018, 8:29 PM
Mk/bsd.port.mk
1973–1976

This is probably OK but can we just whitelist the target it is needed in instead? make package?

emaste added inline comments.Apr 2 2018, 8:31 PM
Mk/bsd.port.mk
1973–1976

Some ports may wish to use it for the build (via SOURCE_DATE_EPOCH=` in the environment or similar approaches)

emaste updated this revision to Diff 41004.Apr 2 2018, 8:33 PM
  • Skip distinfo timestamp fetching for dep targets (requested by @bdrewery)
mat added inline comments.Apr 3 2018, 1:55 PM
Mk/bsd.port.mk
1989–1991

Why the guard on USES=python?

emaste added inline comments.Apr 3 2018, 2:09 PM
Mk/bsd.port.mk
1989–1991

See @antoine's comment above,

This causes problem with at least all python ports:

We have to somehow address python ports for reproducible builds but I hope we can apply to everything else at first at least.

ygy added a subscriber: ygy.Jan 12 2021, 5:05 PM

Revision Contents
Changeset List

PathSize
Mk/
32 lines

Diff 10748

View Options

Mk/bsd.port.mk

Loading...