Page MenuHomeFreeBSD

Fix -jN META_MODE bootstrap for usr.bin/netstat
AbandonedPublic

Authored by lifanov_mail.lifanov.com on Sep 16 2015, 4:02 PM.

Details

Reviewers
sjg
gnn
bdrewery
Summary

add a local dependency to Makefile.depend, which is the smallest
change that fixes -jN build of usr.bin/netstat in META_MODE.

Test Plan

make -C usr.bin/netstat -j8 -WITH_META_MODE=y

Diff Detail

Repository
rS FreeBSD src repository - subversion
Lint
Lint ErrorsExcuse: phabricator needs to accept c++ in file names :)
SeverityLocationCodeMessage
Errorgnu/lib/libstdc++/Makefile.depend:NAME1Bad Filename
Errorgnu/lib/libsupc++/Makefile.depend:NAME1Bad Filename
Errorlib/atf/libatf-c++/Makefile.depend:NAME1Bad Filename
Unit
No Unit Test Coverage
Build Status
Buildable 567
Build 567: arc lint + arc unit

Event Timeline

lifanov_mail.lifanov.com retitled this revision from to Fix -jN META_MODE bootstrap for usr.bin/netstat.
lifanov_mail.lifanov.com updated this object.
lifanov_mail.lifanov.com edited the test plan for this revision. (Show Details)
lifanov_mail.lifanov.com added a subscriber: allanjude.

It's not enough. I was just preparing a tree-wide update.

My build generated this:

~/svn/clean # svn diff usr.bin/netstat

Index: usr.bin/netstat/Makefile.depend

  • usr.bin/netstat/Makefile.depend (revision 287844)

+++ usr.bin/netstat/Makefile.depend (working copy)
@@ -21,4 +21,14 @@

.if ${DEP_RELDIR} == ${_DEP_RELDIR}

  1. local dependencies - needed for -jN in clean tree

+main.o: nl_defs.h
+main.po: nl_defs.h
+mroute.o: nl_defs.h
+mroute.po: nl_defs.h
+netisr.o: nl_defs.h
+netisr.po: nl_defs.h
+nl_symbols.o: nl_symbols.c
+nl_symbols.po: nl_symbols.c
+route.o: nl_defs.h
+route.po: nl_defs.h
.endif

That's correct. This is what my build generates also.
The main.o <- nl_defs.h is the minimal change to get dirdeps to generate the rest of them.

Maybe I'm not sure what the goal of Makefile.depend files in the tree is.
I understood that they are there to bootstrap the auto-generated Makefile.depend files
from a build with local changes/options in it?

Maybe I'm not sure what the goal of Makefile.depend files in the tree is.
I understood that they are there to bootstrap the auto-generated Makefile.depend files
from a build with local changes/options in it?

Think of DIRDEPS in Makefile.depend like an auto-generated ports DEPENDS list. It will build all dependencies "first".

The file: dep targets are to support -j builds in a clean tree. Without them -j0 would work fine as the dependencies are typically found otherwise, but -jN would not build them correctly.

The Makefile.depend are highly option dependent. It's a flaw in the design for FreeBSD. Committers have to be careful to not commit non-default-option-dependent changes of Makefile.depend. Most people who build will have changes to these files which is fine. It arguably would be better to just do this like ports and have a manual dependency list to avoid the list being fickle depending on options.

In theory committers would keep these files up-to-date with all dependencies.

You're right that your fix is sufficient to get it working. The rest will be auto added. However the next time someone does a build and commits the file the rest that I mentioned will be added. So I could commit your main.o change, but I'll be adding the rest later today anyhow! We have many missing -j dependencies in the tree. I have a commit almost ready with about 32 Makefile.depend updated.

Also as for this specific change, I suspect that DPSRCS is not being utilized by META_MODE properly as nl_defs.h is in the DPSRCS list. I haven't checked this theory yet. (Unsure if -j0 on this actually works or not, I would expect it to since DPSRCS has information that could be used)

Thanks for the explanation! By the way, this does not work with -j0 without adding this dependency either.

Firstly - way cool to see folk playing with this ;-)

Most Makefile.depend files start with a comment that they are auto generated.
They *should* be self correcting.

That doesn't mean that (until everyone is using them) they won't get out of date occasionally.
If they get updated though and your tree is up to date they *should* be correct.

Q. are you using meta2deps.sh or meta2deps.py ?

Also as for this specific change, I suspect that DPSRCS is not being utilized by META_MODE properly as nl_defs.h is in the DPSRCS list. I haven't checked this theory yet. (Unsure if -j0 on this actually works or not, I would expect it to since DPSRCS has information that could be used)

DPSRCS is something used by bsd.dep.mk - which isn't used in meta mode since we avoid the whole notion of separate 'make depend' step.
The local dependencies that should be captured in Makefile.depend replace the need for such things.

Cool, thanks! Actually, how do you install the built product?
There is a populated stage dir but no install targets.

Thanks for the explanation! By the way, this does not work with -j0 without adding this dependency either.

Great. It would be good to investigate if we can make it work by bootstrapping the DPSRCS list

In D3683#76141, @sjg wrote:

Also as for this specific change, I suspect that DPSRCS is not being utilized by META_MODE properly as nl_defs.h is in the DPSRCS list. I haven't checked this theory yet. (Unsure if -j0 on this actually works or not, I would expect it to since DPSRCS has information that could be used)

DPSRCS is something used by bsd.dep.mk - which isn't used in meta mode since we avoid the whole notion of separate 'make depend' step.
The local dependencies that should be captured in Makefile.depend replace the need for such things.

Yes I know. I was suggesting there may be a way to utilize it to avoid clean build failures.

lifanov_mail.lifanov.com edited edge metadata.

Actually, how are you generating these?
How did you come up with r287905, for example?

I come up with quite a bigger list. My method was:

  1. empty OBJDIR
  2. run the-lot with no customizations (no src.conf, GENERIC, etc.)
  3. fixup anything that doesn't build by building it individually with -j1
  4. repeat 2-3 until the-lot succeeds
  5. repeat 1-2 until there is no difference in Makefile.depend files

Actually, how are you generating these?
How did you come up with r287905, for example?

It was a partial commit of some missing local dependencies.

I come up with quite a bigger list. My method was:

  1. empty OBJDIR
  2. run the-lot with no customizations (no src.conf, GENERIC, etc.)
  3. fixup anything that doesn't build by building it individually with -j1
  4. repeat 2-3 until the-lot succeeds
  5. repeat 1-2 until there is no difference in Makefile.depend files

Yes, it will take several builds to get everything right. I've been working through various bugs
and have not yet gotten to a full 2 builds to resolve all of the Makefile.depend missing entries
but am getting there.

The reason it takes several builds is that the committted Makefile.depend files are missing much
of the build host tools so they use your /usr/bin version during the first build. Once the-lot is finished
though and you do another build it will then find these tools (such as lex, and yacc) in the stagedir
and use those, thus updating Makefile.depend with these yacc.host and lex.host versions. We need
to get the fully correct Makefile.depend files updated to stop them changing so constantly.

I have not gotten to the compiler yet to see how it impacts the DIRDEPS as well.

Also see this section for ensuring the various host tools are built first: http://crufty.net/sjg/docs/freebsd-meta-mode.htm#getting-started

There is nothing more to do here.