Initial version of the Manifest plugin (experimental)

[robertoaloi]

The manifest provider allows users to extract information about a rebar3 project, so that the information can be used by external tools such as language servers to learn about the structure of a project. The format and content of the produced manifest is totally up for debate. This is just to start the discussion. Ideally, the manifest should not be tight to rebar3 as a build tool. This would enable other build tools to produce an equivalent file.

The provider is inspired by the build_info one which is part of the eqWalizer type checker for Erlang, but uses built-in functionality to fetch dependencies.

Having a built-in provider would remove friction for language server users and implementors. ELP users are currently required to install additional plugins to be able to use the language server.
Erlang LS implementors are attempting efforts such as this one, to approximate the structure of a rebar3 project. A built-in manifest provider would simplify all of this.

Format

Ideally, one could have used JSON as the standard format for the manifest (the Rust Analyzer language server uses a rust-project.json format for a similar concept. Unfortunately, there's currently no JSON library provided out-of-the-box by Erlang/OTP or rebar3, so I'm defaulting to Erlang terms. I am also adding the ability to use EETF (Erlang External Term Format) as an alternative format, since this is the format currently used by the ELP language server via the build_info plugin.

Output

By default the provider outputs the manifest to stdout. For this reason, logging is disabled by the provider (an alternative would have been to ask users to use the QUIET=1 option or to filter out extraneous log messages). A dedicated option provides the ability to dump the manifest to file. This is mostly done to simplify testing and to avoid playing with group leaders or similar approaches. Again, this is all open for discussion.

Example

Running the command:

rebar3 manifest

For the Erlang LS project would produce something like this.

[ferd]

The biggest issue here is that this PR currently disables logging for rebar3 wholesale. This needs to be changed.

The rest of my comments are about improving the robustness for some of the weird cases rebar3 encounters and supports, and minor formatting issues.

[ferd]

The init/1 function is invoked at Rebar3 start time for all providers to know which commands are available and what their dependencies are. This means that this disables logs for rebar3 universally.

This isn't a good way to go, and if you need logs from stdout, I don't think we can prevent other providers from outputting anything, particularly if you're asking for install_deps to run already.

The only path forward I see if you want stdout content is to pass in an argument that is going to be an optional delimiter, like --delimiter="~~~content~~~" which would then give you something like:

~~~content~~~
this is the parseable output
~~~content~~~

Which frankly isn't elegant, but probably workable and without interfering with other unrelated tasks.

[ferd]

for formatting rules, Rebar3 has historically used 4 spaces indents everywhere in providers. As per the nesting, it tends to look more like https://github.com/erlang/rebar3/blob/main/apps/rebar/src/rebar_prv_repos.erl -- this is not a blocker, but generally preferable.

[ferd]

Suggested change

	otp_lib_dir := string(),
	source_root := string()
	otp_lib_dir := file:filename(),
	source_root := file:filename()

[ferd]

Some applications may not compile with that compiler; There's a set of dynamic calls the compiler uses to know if rebar3's own internal compiler handles these:

rebar3/apps/rebar/src/rebar_prv_compile.erl

Lines 330 to 331 in 8207d82

	Type = rebar_app_info:project_type(AppInfo),
	Type =:= rebar3 orelse Type =:= undefined

Both rebar3 and undefined types are treated as rebar3 projects. People can configure custom compilers for their apps (such as mix) and we sort of hand this off to a plugin to handle. So if you want to be safe, you'll want to filter these out via the list comprehensions in lines 110 and 111.

[ferd]

without forcing unicode there (with ~ts) you may get broken results when people have unicode in their file paths in the Erlang format. Then I figure you might also have the problem that the eetf one has invalid unicode sequences. It's possible you need to keep the type with the data to output it safely?

[ferd]

the tests are probably going to be okay, but there's a higher chance we accidentally break compatibility down the road if we aren't stricter on assertions. I'm assuming that your projects will be the main users of this so I'll defer to your judgment on this, we can always add more tests later.

[ferd]

inconsistent spacing here (though I imagine the init_ and end_ cases may have been copy/pasted (which isn't a problem).

[robertoaloi]

Thanks @ferd for the quick and accurate review! I will try to address all the feedback soon.

• I assumed the logging hack wouldn't stay in the final version, but I wanted to see if there was a better alternative. I think for the time I will keep it simple and just remove it. There's always the file variant that can be used as an alternative, or the caller will have to skip the extraneous output. We can introduce the delimiter when/if needed
• I only added basic tests in case you had opinions about the actual format. If not, I will proceed and refine the tests
• Regarding spaces and formatting, I copied a template from somewhere (don't recall where) and applied a local formatter. Will try to match the desired way

Regarding documentation about the provider (e.g. in the README or the website), maybe we could leave it undocumented until ELP / Erlang LS integrate with it and the output format is refined.

[ferd]

Yeah I'm good leaving it undocumented. Alternatively, we have an 'experimental' namespace that lets us change the command as we see fit and announce it and get users but without any guarantees. The issue is of course that once it's stable, if you rely on it already, you need to fallback from default to experimental to unsupported.

[robertoaloi]

@ferd Sorry for the delay in the follow-up. I think I addressed all the feedback.

• Removed logging altogether
• Leaving tests as they are until we iterate on an actual integration with ELP / Erlang LS
• Have a question about Unicode (we could address it as a follow up)
• Fixed the rest

[ferd]

This seems alright! What was the Unicode question?

Also have you given thought to the experimental status of the provider, or do you feel whatever format comes out of this can be relied on to be stable down the road?

[robertoaloi]

@ferd

• I was referring to your "It's possible you need to keep the type with the data to output it safely?". I was wondering if you meant to store the fact the content was eetf or erlang as well?
• I am not sure about the experimental namespace, so your advice is welcome here. What are the implications for the user if we go for experimental? Do they need to tweak their rebar config to get access to the plugin? The main goal of the plugin is to avoid a manual installation step when using a language server, so having to tweak the rebar config partially defeats its purpose. The format of the produced build_info (and maybe the flags) are expected to change, especially while we work on the integration itself. But after a short period they should stabilize.

[ferd]

@ferd

• I was referring to your "It's possible you need to keep the type with the data to output it safely?". I was wondering if you meant to store the fact the content was eetf or erlang as well?

Oh yeah. The Erlang format is regular text and you'd want that to be unicode-aware, ideally, to keep showing paths as text rather than lists of integers or bytes. The eetf type is however byte sequences and it may output values that aren't valid unicode or text, so you wouldn't want to use the t qualifier there, only for text.

• I am not sure about the experimental namespace, so your advice is welcome here. What are the implications for the user if we go for experimental? Do they need to tweak their rebar config to get access to the plugin? The main goal of the plugin is to avoid a manual installation step when using a language server, so having to tweak the rebar config partially defeats its purpose. The format of the produced build_info (and maybe the flags) are expected to change, especially while we work on the integration itself. But after a short period they should stabilize.

The experimental namespace just means that you have to call the task as rebar3 experimental manifest (see the vendor provider for an example) and will show up under that namespace in rebar3 help. By default, anything in the experimental namespace is expected to be unstable and prone to change. This is useful when the whole interface is still not stable, as opposed to when only some options may be worth changing.

Whenever we feel the format is solid and adequate we can promote the task outside the experimental namespace and into the default one, where rebar3 manifest will let it be called with an expectation of stability. Nothing else will need to change.

[robertoaloi]

@ferd If I mark the plugin as experimental (and update the tests accordingly), I get a failure due to a dependency not found:

{thrown,{{error,{providers,{provider_not_found,experimental,install_deps}}},
         [{providers,process_dep,2,
                     [{file,"/Users/robertoaloi/git/github/robertoaloi/rebar3/vendor/providers/src/providers.erl"},
                      {line,285}]},
                      [...]

It looks like rebar3 assumes that, if a provider is experimental, all of its dependencies also belong to the same namespace. Is this intentional? Any way to circumvent this?

[ferd]

@ferd If I mark the plugin as experimental (and update the tests accordingly), I get a failure due to a dependency not found:

{thrown,{{error,{providers,{provider_not_found,experimental,install_deps}}},
         [{providers,process_dep,2,
                     [{file,"/Users/robertoaloi/git/github/robertoaloi/rebar3/vendor/providers/src/providers.erl"},
                      {line,285}]},
                      [...]

It looks like rebar3 assumes that, if a provider is experimental, all of its dependencies also belong to the same namespace. Is this intentional? Any way to circumvent this?

oh yeah that's because the plugin dependencies look in your namespace by default. Specify the dep as {default, install_deps} and it should work.

[robertoaloi]

@ferd Yes, I eventually figured that out from the code. Does it mean the spec is wrong, then?

Updated the PR with your suggestions.

[ferd]

@ferd Yes, I eventually figured that out from the code. Does it mean the spec is wrong, then?

Updated the PR with your suggestions.

Yeah I think the spec is wrong, or pre-dates us adding namespaces to providers.

[ferd]

Okay, this seems to be in a workable format for an experimental feature. When tests pass I think we'd be good to merge. If dialyzer complains I'll see if I can fast-track a change for the type specs.

[robertoaloi]

@ferd No Dialyzer failure. It looks like the spec fix can be done as a follow-up.