Re: [RFC] building postgres with meson - v12

Hi,

On 2022-09-02 09:35:15 -0700, Andres Freund wrote:
> Part of that is due to some ugly dependencies of src/common on backend headers
> that IMO probably shouldn't exist (e.g. src/common/relpath.c includes
> catalog/pg_tablespace_d.h). Looks like it'd not be hard to get at least the
> _shlib version of src/common and libpq build without waiting for that. But for
> all the backend code I don't really see a way, so it'd be nice to make genbki
> faster at some point.

This reminded me of a question I had. Requires a bit of an intro...

Because ninja's build specification ends up as a fairly clean DAG, and because
it also collects compiler generated dependencies as a DAG, it's possible to
check whether the build specification contains sufficient dependencies.

Before ninja 1.11 there was
https://github.com/llvm/llvm-project/blob/main/llvm/utils/check_ninja_deps.py
and ninja 1.11 has "ninja -t missingdeps" built in.

Intentionally removing some of the dependencies to show the output:

$ ninja -t missingdeps
Missing dep: src/interfaces/ecpg/preproc/ecpg.p/.._ecpglib_typename.c.o uses src/include/catalog/pg_type_d.h (generated by CUSTOM_COMMAND)
...
Missing dep: src/bin/scripts/reindexdb.p/reindexdb.c.o uses src/include/catalog/pg_class_d.h (generated by CUSTOM_COMMAND)
Missing dep: contrib/oid2name/oid2name.p/oid2name.c.o uses src/include/catalog/pg_class_d.h (generated by CUSTOM_COMMAND)
Missing dep: contrib/vacuumlo/vacuumlo.p/vacuumlo.c.o uses src/include/catalog/pg_class_d.h (generated by CUSTOM_COMMAND)
Missing dep: src/test/modules/libpq_pipeline/libpq_pipeline.p/libpq_pipeline.c.o uses src/include/catalog/pg_type_d.h (generated by CUSTOM_COMMAND)
Processed 2299 nodes.
Error: There are 62 missing dependency paths.
62 targets had depfile dependencies on 25 distinct generated inputs (from 1 rules) without a non-depfile dep path to the generator.
There might be build flakiness if any of the targets listed above are built alone, or not late enough, in a clean output directory.

Obviously that can only work after building, as the compiler generated
dependencies are needed.

I find this exceedingly helpful, because it supplies a very high guarantee
that the build specification will not fail on a different machine due to
different performance characteristics.

The question:

Is it worth running ninja -t missingdeps as a test? At the time we run tests
we'll obviously have built and thus collected "real" dependencies, so we would
have the necessary information to determine whether dependencies are missing.
I think it'd be fine to do so only for ninja >= 1.11, rather than falling back
to the llvm python implementation, which is much slower (0.068s vs
3.760s). And also because it's not as obvious how to include the python script.

Alternatively, we could just document that ninja -t missingdeps is worth
running. Perhaps at the top of the toplevel build.meson file?

Greetings,

Andres Freund