1 /** Process and function execution directed acyclic graph (DAG). 2 * 3 * For background see Joe Armstrong's citation 4 * "A dependency is just a blob with an optional hash" 5 * at https://www.youtube.com/watch?v=lKXe3HUG2l4&ab_channel=StrangeLoopConference. 6 * 7 * TODO/Questions: 8 * 9 * - Goal: 10 * - Provisioning: Slim down the prov scripts as much as possible. 11 * - TODO: Remove prov of Python (@fmecca) 12 * - TODO: Replace Intel MKL with OpenBLAS and tag @fmecca as reviewer 13 * - Building: Reliable multi-platform shared caching of build artifacts 14 * 15 * - Impl 16 * - Try wrapping provision-win.sh child-scripts 17 * in https://bazel.build/reference/be/shell. Investigate if 18 * such Bazel-wrapping of scripts can be opt-in. I think it can. 19 * - Use dub with recently added shared dub artifact cache 20 * https://github.com/dlang/dub/pull/2542 via NFS (or perhaps redis). 21 * TODO: Add SIL issue for adding such a shared cache and tag Martin and Atila. 22 * - TODO: Ask Mathias Lang: Are there any more input states missing from the hash in dub on order to make more reliable? 23 * 24 * - Mental Model 25 * - Plant a modern model in developers of what a build tool should be 26 * - Are we using all that we are provisioning? 27 * - DAG-model 28 * - Extend the mental model of a pure function to a build action 29 * - Bazel SkyLark 30 * - Expose some automatation model to the developers 31 * - Use seccomp to limit IO 32 * - Safe in the sense that end-user cannot accidentally break a build dependency 33 * 34 * - Tools: 35 * - A low-hanging fruit is be to be able to conveniently specify a dep as 36 * URLs+hash in D. Where URLs are a set of mirrors for the same file. 37 * - OverlayFS 38 * - Transactional: 39 * - Emulates writes over a read-only partition 40 * - Commit or loose 41 * - Cache artifacts on shared (NFS) storage 42 * - sccache - Shared Compilation Cache: https://github.com/mozilla/sccache 43 * - Read up on Bazel's and Buck's internals. See https://blog.mozilla.org/nfroyd/2019/10/28/evaluating-bazel-for-building-firefox-part-1/. 44 * 45 * - Be able to specify dependencies to source and object files inside zip files and repos 46 * - Discuss with fmecca how to enable actions to be pure via a 47 * pure simulation layer of file-io; one solution is to make the Input and Outpu 48 * only operate on mmapped files; read-only mapped for inputs and write-only 49 * mapped for outputs. 50 * - Call actions through ptrace supervision and detect violations 51 * - Discuss and synthesize a (graph model) concepts: 52 * - actions 53 * - connection/link with input with output 54 * - nodes: 55 * - input: 56 * - mids 57 * - output: 58 * - Borrow concepts and constructs by Bazel's SkyLark language semantics 59 * - Integrate with https://code.dlang.org/packages/iopipe 60 * 61 * - Transactional IO API: mmap copy-on-write (CoW) implement a mirroring of a directory and 62 * writing it out in one go. 63 * 64 * See_Also: https://bazel.build/ 65 */ 66 module nxt.iodag; 67 68 import std.digest.sha : SHA256; 69 import nxt.fs; 70 import mir.algebraic : Algebraic; 71 72 alias Hash = SHA256; ///< Default hash. 73 74 /** Untyped source of bytes. 75 * 76 * Currently cannot be `void[]` because of limitations of mir.algebraic. 77 */ 78 alias Data = ubyte[]; 79 80 /** Input. 81 */ 82 struct In { 83 Hash hash; ///< Mandatory hash. 84 Algebraic!(Data, Path) src; ///< Source. 85 } 86 87 /// Output. 88 struct Out { 89 Algebraic!(Data, Path) dst; ///< Destination. 90 } 91 92 @safe pure unittest { 93 // TODO: auto y = sh(x); 94 }