Tool-independent incremental compilation

[mstade]

Because of the way ez-build assumes you have two target environments (i.e. dev which is implicit and prod which is defined with --production) we can also implement tooling independent incremental compilation. This is a side effect of how in dev we assume all output files are unlinked, and that there's a 1:1 mapping between source file and destination file. This should also hold true for 1:n mappings (e.g. css modules, which implies you have a destination css file, and also a json mapping of selectors, so 1:2) since we only have to keep track of the source files. However, for n:1 mappings it becomes more difficult, since if only one of the inputs (n) change it might only affect part of the output. This requires intimate knowledge of what the output is, and so quickly becomes tool dependent.

Why do we want this to be tool independent? A couple of reasons:

• Recompiling everything can take a long time – even a trivial project is in the seconds
• Not all tools implement file watching, and most of them do it differently and as an after thought making it difficult to reconcile across platforms and concerns (i.e. css v. js v. image assets)

How do we do this? By realizing the following truths:

• A project can be decomposed into several subsets
  • Perhaps the most straightforward way to represent this is by having separate files, but it can be represented in other ways as well
• Any compiler or tool that can work on a subset of the input is parallelizable, which means it can also be processed independently at different times
• Linking is often a non-parallelizable step without intimate knowledge of the input/output relationships, so not easy to do incrementally in a tool independent way; thus we don't support incremental linking and only include that step in the --production template (tools enable incremental work by representing units of work using metadata, as opposed to inferring things based on files – ASTs are often used, since trees can be broken up in independent branches)
• We invoke tools multiple times if need be, and do per-tool optimizations to keep this fast during --interactive builds