Design Doc for component system

[viega]

Design for Chalk component/module system

John + Theo

Intro

Config file bits in examples and HOWTOs need to not require people to
have to edit configuration files, and should be COMPOSABLE.

For instance, we should have one component for the https server that
people can use with pretty much ANY content-based component. If some
component REQUIRES such a server component be installed, it could
explicitly require it.

We've spent some time thinking about how this might all manifest in con4m
in a way that wouldn't take too long to implement, and I have a strawman
here, that I'm going to introduce by example.

❗ I've since updated this doc to reflect what I actually implemented, and
things relegated to potential future work are discussed at the end.

I'm going to take the existing app-inventory config
(chalkdust.io/app-inventory.c4m) and rewrite it using our strawman.
It would span multiple components; components would be individual files
that sort of act like parameterless functions in the code, you would invoke
them with a use statement, and you can invoke them multiple times from
multiple places.

We'll go into more detail about the semantics and the rational below. First,
we want to go through an example.

The how-to linked did a few things that are really a collection of small
things that some other configs we provide will want to do:

1. Set up a server config for the container
2. Turn off the default reporting (which shouldn't be required, but should
   the default for when we're adding a server config)
3. Optionally turn off the terminal reports (this was commented on, and
   the code was left commented in the source).
4. Turn on docker as the default command. LOTS of things will do this.
5. Turns on entry point wrapping.

Those bits really should be composable across examples -- all of them
should be usable across multiple configs, and people should be able to
combine bits from different configs to their hearts' content.

But, whenever a component DOES get used, it should be configured the
same way, and only should need to be configured once, for instance, if
a micro-component X is used by A, B and C, and my config uses all
three of those.

The code below is presented w/o comments first, so you can
get the sense of the scope. I'll do a deep dive after. For now, just
note that things in "parameter" blocks will be the bits that can
appear in UI elements. The rest is just the module's code that gets run.

terminal_report.c4m

parameter var disable_terminal_reports {
    default: false  
    shortdoc: "Disable terminal summary reports"
    doc: """
Controls whether to force off the default summary reports that get
print to your terminal. If you set this to 'true', any conflicting
code that attempts to ensure this is on would fail when the
configuration loads.
"""
}

# The parameter declaration in the file allows us
# to use the below variable, even if it's not explicitly defined outside
# of the parameter spec above.  Either it will be provided by
# the runtime system before this module loads, or via inline input 
# from the user.

if disable_terminal_reports {
    custom_report.terminal_chalk_time.enabled: false
    custom_report.terminal_other_op.enabled:   false
}

log_report.c4m

parameter var disable_default_report {
  shortdoc: "Disable the log report"
  doc: """
Causes Chalk to turn off the default log report when enabled.
"""
}

if disable_default_report {
  unsubscribe("report", "default_out")
}

impersonate_docker.c4m

"Forces setting of the default command to 'docker'."
default_command: "docker"

wrap_entrypoints.c4m

"Ensures entrypoint wrapping is enabled in the config"
docker.wrap_entrypoint: true  

reporting_server.c4m

# This function isn't the world's greatest validator, but whatever!
func validate_url(url) {
  result := ""

  if (not url.starts_with("http://")) and (not url.starts_with("https://")) {
    return "Only http / https URLs are supported"
  }
}

# Rough and ready default value dynamically calculated.
func get_local_url() {
  out, code := system("ifconfig -a | grep inet | grep broadcast | head -1 | " +
                     "awk '{ print $2 }'")
  if code != 0 {
    return "https://localhost:7890"
  }

  return "https://" + out.strip() + ":7890"
}

parameter sink_config.output_to_http.uri {
  shortdoc:  "URL for reporting server"
  doc: """
A config for sending reports to a custom implementation of the test
reporting server.
"""
  validator: func validate_url(string) -> string
  default: func get_local_url() -> string
}

sink_config output_to_http {
  enabled: true
  sink:    "post"

  # The URI will get filled in automatically (the field sink_config.output_to_http.uri above)
}

compliance_docker.c4m

## Compliance Reporting Example
## Note: these aren't installed at the below URL at the time of writing;
## For testing I've been using file system components instead.
use impersonate_docker  from "https://chalkdust.io/component_store/new" 
use reporting_server from "https://chalkdust.io/component_store/new" 
use entrypoint_wrapping  from "https://chalkdust.io/component_store/new" 

Component basics, and the use statement

At runtime, the use statement causes a component to run a component.
The 'from url' bit is optional, and defaults to chalkdust.io. The source of all
dependent components gets cached in the chalk mark of the chalk exe,
and currently do not get reloaded unless you remove your entire configuration
(chalk load default) .

Statically, these statements are analyzed before we execute,
and any dependent components are fetched and validated.
Caching happens if the config successfully loads.

Therefore, at runtime, the use statement really is just a call into
the named module. And if you put a use statement in a loop,
the module's code gets run each time through the loop.

Similarly, if the module is in an if conditional, it won't run unless that
branch gets taken at runtime.

However, if a module uses any parameters (see below), a value
for each parameter must be provided before any execution begins. This is
true even for parameters that can be indirectly used if a module itself uses
other modules.

❗ Components share the global configuration (attributes). But they get their
own local variable scope, unless those variables are explicitly exported,
in which case they become truly global.

The static checking forbids cycles in module importing.
Every component is uniquely identified via a hash of its source code,
which will be useful for versioning when we know what we're doing there.

Conflicts across components

When components execute, any values they explicitly set in their body
are locked, meaning that future assignments to those variables will fail if
they try to change the value.

This is how we ensure that multiple components are truly compatible, and
how we ensure that any custom config code written directly in con4m is
compatible. For example, if one component always sets
default_command = docker, and another always sets
default_command = help, then the configuration validation phase, where
the configuration gets run, would fail, because the second component will
blow up execution when it's now allowed to set the value.

In cases where there's conditional code based on the runtime environment,
if two components have conflicting code, but one doesn't run, then the
configuration is fine under those conditions.

❗ While we do test-run the config before loading it, it's possible, due to
environmental-based conditionals, to have the config pass when loading,
but blow up when Chalk is starting up; Chalk won't get past config loading
if this happens, due to the attribute lock. However, it's best practice to
minimize conditionals around components, to maximize the value of the
static checking.

The 'locking' semantics apply to any (non-builtin) con4m function called
during a module's execution. If those modules export functions that are
not run, but are expected to be called from the top-level user context,
they will not lock.

Parameter sections

Parameters must be either attributes or variables that are used inside
the code of the component.

If parameter declarations are variables, the variable name must be
proceeded by var; variable names can never be dotted the way
attributes can be.

The default field of a parameter would set the default value when
the component is loaded, and this will override whatever is the config
default from chalk.c42spec or the base configuration. If this is not
provided, then the component will not run without a value being given
by the runtime first.

If the default field is set to a function pointer, then chalk will call
it when it needs the default value, allowing dynamic defaults based
on the environment.

Note that if there were ever a reason to have a parameter that is itself
a callback of some sort, I believe the default field would only be usable
as a value. I thought about just disallowing it, and chalk would not be
able to marshal callbacks anyway. For now, we will just disallow this in any
components we publish, as it's not important enough to worry about now.

The default value / callback ONLY applies when the user has not agreed
upon a configured value. See the command-line usage section below for
details on the config process.

Once the config process completes, Chalk keeps the value of any parameters
cached in the chalk mark, next to the cached configuration. Once these values
are set, the cached values are always used unless one reconfigures.

Parameters also get an optional validation field, allowing you to specify
a callback that will examine the value and test it, for when the provided
type checking isn't enough (for instance, the URL example used in the
sample code above).

Parameters also get a doc and shortdoc fields. This documentation
is presented in the configuration process. It's optional, but highly
recommended.

Command-lind usage

In the initial implementation, one uses chalk load to install a component
into an existing configuration, and to configure any parameters used by that
component.

For instance, chalk load foo.c4m would cache foo.c4m from your
working directory, and add a statement to your embedded configuration like:

use foo.c4m

Actually, it will add the "from" part too, but since it doesn't update its cached
copy unless you reset your config, that's not particularly relevant as long
as you don't edit the from portion in your config (the caching is done by
the full location at the time installed).

To avoid people overwriting their configs every time they want to install a
new module, the default behavior of chalk load is now to assume you are
only looking to add a component. Or, if you've already added it, to reconfigure
it.

However, the --replace flag (or the load.replace_conf field in the config
if you want to change the default behavior) will cause chalk load to remove
any existing configuration before adding the specified component.

When you call chalk load, any parameters used by the module you're
loading will go through the configuration process. For now, that process
consists of interactive prompting, allowing you to just hit 'enter' to accept
the default (or the previous value).

This only triggers parameters that can be used by the component loaded,
but does include components used by any dependent components.

Notes on specific components above

The micro-component for terminal summary reports basically makes
terminal summary reports OPTIONAL by wrapping setting stuff in a
conditional controlled by a variable the user controls.

In contrast, impersonate_docker ends up REQUIRING that the resulting
config sets default_config to true.

Similarly, entrypoint_wrapping forces docker.wrap_entrypoint to be
true. Because anything set in these components will be locked,
per above.

Roadmap Items

Most of these items have no specific time frame, but constitute the
enhancements that, as of this writing, we'd definitely like to make at
some point.

1. Better execution-time errors (including stack traces). For instance,
   when an attribute is 'locked' by virtue of a component setting its
   value, if some conflict arises, we do not currently show where in the
   config that happens. In fact, that's the case w/ any runtime con4m
   error.

2. An app-store style (terminal) interface, which would be used for
   both selecting components, and for configuring / reconfiguring. In the store
   UI, we'd probably need some comment-based annotation in modules to
   determine what to show / hide.

3. A con4m import statement that works more like a traditional module import;
   The import would only happen once, and they would NOT support parameters, or
   locking.

4. Versioning and an update approach. I originally had some semantics to allow
   version annotations, but I realized the logic around when to check for updates,
   when to auto-update and when to warn all required a lot of thinking, so decided
   to just kick the can down the road on this.

5. Currently, if module A uses module B, and module B has some parameter, there's
   no explicit mechanism for B to declare the parameter moot. For instance, if we wrote
   a component that auto-downloaded and ran the server container in the above example,
   there would still be a configuration parameter that the user would get prompted for.
   For now, that's fine with me; as a work-around can just dupe the functionality needed.

Implementation Notes

Most of the implementation was done in the con4m codebase; over time, parts
of the Chalk side of this might also get generalized and pulled into that code base.

[viega]

This has been updated to reflect what is currently implemented in the following two PRs:

• Chalk: feat(config): Load and configure components without editing configs directly #47
• Con4m: Component System (version 0.2.0) con4m#116