- Goal:
- Provisioning: Slim down the prov scripts as much as possible.
- TODO: Remove prov of Python (@fmecca)
- TODO: Replace Intel MKL with OpenBLAS and tag @fmecca as reviewer
- Building: Reliable multi-platform shared caching of build artifacts
- Impl
- Try wrapping provision-win.sh child-scripts
in https://bazel.build/reference/be/shell. Investigate if
such Bazel-wrapping of scripts can be opt-in. I think it can.
- Use dub with recently added shared dub artifact cache
https://github.com/dlang/dub/pull/2542 via NFS (or perhaps redis).
TODO: Add SIL issue for adding such a shared cache and tag Martin and Atila.
- TODO: Ask Mathias Lang: Are there any more input states missing from the hash in dub on order to make more reliable?
- Mental Model
- Plant a modern model in developers of what a build tool should be
- Are we using all that we are provisioning?
- DAG-model
- Extend the mental model of a pure function to a build action
- Bazel SkyLark
- Expose some automatation model to the developers
- Use seccomp to limit IO
- Safe in the sense that end-user cannot accidentally break a build dependency
- Be able to specify dependencies to source and object files inside zip files and repos
- Discuss with fmecca how to enable actions to be pure via a
pure simulation layer of file-io; one solution is to make the Input and Outpu
only operate on mmapped files; read-only mapped for inputs and write-only
mapped for outputs.
- Call actions through ptrace supervision and detect violations
- Discuss and synthesize a (graph model) concepts:
- actions
- connection/link with input with output
- nodes:
- input:
- mids
- output:
- Borrow concepts and constructs by Bazel's SkyLark language semantics
- Integrate with https://code.dlang.org/packages/iopipe
- Transactional IO API: mmap copy-on-write (CoW) implement a mirroring of a directory and
writing it out in one go.
Process and function execution directed acyclic graph (DAG).
For background see Joe Armstrong's citation "A dependency is just a blob with an optional hash" at https://www.youtube.com/watch?v=lKXe3HUG2l4&ab_channel=StrangeLoopConference.
TODO/Questions:
- Goal: - Provisioning: Slim down the prov scripts as much as possible. - TODO: Remove prov of Python (@fmecca) - TODO: Replace Intel MKL with OpenBLAS and tag @fmecca as reviewer - Building: Reliable multi-platform shared caching of build artifacts
- Impl - Try wrapping provision-win.sh child-scripts in https://bazel.build/reference/be/shell. Investigate if such Bazel-wrapping of scripts can be opt-in. I think it can. - Use dub with recently added shared dub artifact cache https://github.com/dlang/dub/pull/2542 via NFS (or perhaps redis). TODO: Add SIL issue for adding such a shared cache and tag Martin and Atila. - TODO: Ask Mathias Lang: Are there any more input states missing from the hash in dub on order to make more reliable?
- Mental Model - Plant a modern model in developers of what a build tool should be - Are we using all that we are provisioning? - DAG-model - Extend the mental model of a pure function to a build action - Bazel SkyLark - Expose some automatation model to the developers - Use seccomp to limit IO - Safe in the sense that end-user cannot accidentally break a build dependency
- Tools: - A low-hanging fruit is be to be able to conveniently specify a dep as URLs+hash in D. Where URLs are a set of mirrors for the same file. - OverlayFS - Transactional: - Emulates writes over a read-only partition - Commit or loose - Cache artifacts on shared (NFS) storage - sccache - Shared Compilation Cache: https://github.com/mozilla/sccache - Read up on Bazel's and Buck's internals. See https://blog.mozilla.org/nfroyd/2019/10/28/evaluating-bazel-for-building-firefox-part-1/.
- Be able to specify dependencies to source and object files inside zip files and repos - Discuss with fmecca how to enable actions to be pure via a pure simulation layer of file-io; one solution is to make the Input and Outpu only operate on mmapped files; read-only mapped for inputs and write-only mapped for outputs. - Call actions through ptrace supervision and detect violations - Discuss and synthesize a (graph model) concepts: - actions - connection/link with input with output - nodes: - input: - mids - output: - Borrow concepts and constructs by Bazel's SkyLark language semantics - Integrate with https://code.dlang.org/packages/iopipe
- Transactional IO API: mmap copy-on-write (CoW) implement a mirroring of a directory and writing it out in one go.