Deliver ROBOT plugins in ODK

[gouttegd]

This PR proposes a method to manage ROBOT plugins in the ODK. The aims are:

1. allow to easily use any plugin that is already provided by the ODK itself (currently, the ODK provides only the SSSOM plugin, but more plugins may be added in the future), both in the standard ODK-generated Makefile and in a custom Makefile;
2. allow to easily add any extra plugin that is not provided by the ODK;
3. allow to easily override a plugin that is provided by the ODK (for example, if the ODK provides version 0.6.2 of the SSSOM plugin, a ODK user must be able to locally use version 0.7.0 without having to wait for the next ODK release with the latest version).

The core principle is that we define a runtime plugins directory, which is $(TMPDIR)/plugins. The $(ROBOT_PLUGINS_DIRECTORY) variable is defined to point to that directory and exported to the environment, so that ROBOT can find any plugin located in that directory.

Once that principle is posed, the rest is only a matter of ensuring that the plugins end up being placed in that directory. This is done by the all_robot_plugins target. Any rule (whether it is a standard rule in the ODK-generated Makefile or a custom rule in the custom Makefile) that requires the use of a plugin should depend on that target, to ensure that the plugins are ready to use.

There are several ways for a user to use extra plugins and/or to override the ODK-provided plugins:

A) Putting the plugin(s) they want to use in a top-level plugins directory. The default rule to copy plugins into the runtime directory will look into that top-level plugins directory in addition to looking into the ODK’s /tools/robot-plugins directory, so any plugin found there will be automatically copied as if it had been provided by the ODK. The top-level plugins directory takes precedence over the ODK’s /tools/robot-plugins directory.

B) Using custom rules in the custom Makefile. To override a ODK-provided plugin, one may override the default installation rule with a specific rule. For example, to force the use of version 0.7.0 of the SSSOM plugin (instead of whatever version is provided in the ODK), one may do the following:

$(ROBOT_PLUGINS_DIRECTORY)/sssom.jar:
        curl -L -o $@ https://github.com/gouttegd/sssom-java/releases/download/sssom-java-0.7.0/sssom-robot-plugin-0.7.0.jar

To install a supplementary plugin (instead of overriding a default one), one can override the custom_robot_plugins target, which is itself a dependency of all_robot_plugins. For example, to add the KGCL plugin:

$(ROBOT_PLUGINS_DIRECTORY)/kgcl.jar:
	curl -L -o $@ https://github.com/gouttegd/kgcl-java/releases/download/kgcl-0.2.0/kgcl-robot-plugin-0.2.0.jar

custom_robot_plugins: $(ROBOT_PLUGINS_DIRECTORY)/kgcl.jar

C) Explicitly listing the supplementary plugins in the ODK configuration. This PR adds a new possible configuration section in the ODK config file, allowing to list extra plugins to install. For example, another way of adding the KGCL plugin would be to amend the ODK configuration as follows:

robot_plugins:
  plugins:
    - name: kgcl
      mirror_from: https://github.com/gouttegd/kgcl-java/releases/download/kgcl-0.2.0/kgcl-robot-plugin-0.2.0.jar

The generated Makefile will then contain the required code to automatically download that plugin and place it into the runtime plugin directory.

If no mirror_from key is specified, then the ODK-generated Makefile will contain a dummy rule instead – it will be up to the user to override that rule in the custom Makefile.

closes #909

[matentzn]

great!! I will test this a bit later, but the design looks good.

[matentzn]

Sorry meant to say "comment". Approval after testing !

Initial review dismissed because it was submitted prematurely.

[gouttegd]

Of note, I am open to the argument that this may be too flexible: maybe the new ODK configuration section is not needed, and it would actually be enough if supplementary plugins can only be installed through custom rules in the custom Makefile, without the possibility of declaring them in the -odk.yaml config file — presumably anyone knowledgeable enough to know that they need a plugin would know how to add a rule in the custom Makefile.

[matentzn]

FYI: Why did you use a fork for this PR? Makes it a (tiny) bit more cumbersome for me to test (not much just a bit)

[gouttegd]

Why did you use a fork for this PR?

Didn’t do that on purpose, I just forgot that on my local copy the origin remote points to my fork instead of here.

[matentzn]

As this is tested by you, I am included to leave at my line by line review and not test it (I am a bit out of breath). If you feel like I should test this, please merge this into the dev branch and I will do it next week.

[gouttegd]

I am happy with the amount of testing I’ve done myself.

In any case the PR can’t break an existing standard workflow, since there are currently no standard workflows to deal with plugins.

What’s missing now is documenting the plugin delivery mechanism – such a mechanism would not be very useful if people don’t know how to use it (or even that it exists)… Since we agree on the mechanism itself (so it won’t be changed), I’ll do that shortly.

[matentzn]

I cant see any other obvious problem. I tested three pipelines and couldn't see an obvious issue.