Yaml/JSON pre-processing with Jackson

[fbiville]

Hello,

Just to give a little context first, I'm part of
the team who maintains the Dataflow template for
Neo4j imports on GCP.

Part of the configuration surface is a Yaml/JSON
job specification ("job spec" for short) that
describes where the data sources are and how to
map them to Neo4j nodes/relationships.

The job spec is fetched from Google Cloud Storage.
As a consequence, any change to the spec must
result in a re-upload to GCS.

To alleviate this issue, the former implementation
of the job spec parser (which is not
Jackson-based) relies on a "runtime token map"
parameter, optionally supplied by the user
alongside the job spec GCS URL.

What this token map defines is a JSON dictionary
expression, where the top-level keys are token
names and the corresponding values the token values.

For instance:

{"project": "my-project", "schema": "my-schema"}

A job spec author can then write something like
this:

version: '1'
config:
  reset_db: false
sources:
  - type: bigquery
    name: a_bigquery_source
    query: |-
      SELECT some_field FROM `$project.$schema.some_table` 
      LIMIT 100
// [...]

As a result, the job spec does not need to be
re-uploaded across environments.

The token map can also define values of JSON
primitive types other that strings.

For instance, the job spec could define a mapping
target that is dynamically disabled:

// [...]
targets:
  nodes:
    - name: some_node_target
      active: $some_node_target_active
      source: a_bigquery_source
      write_mode: merge
      labels: [ANodeLabel]
      properties:
        - source_field: some_field
          target_property: some_property
// [...]

And the corresponding runtime token map could
include {"some_node_target_active": false} for
some invocations (and true for others).

Here, I'm not sure how I can pre-process the JSON
or Yaml content of the spec, to inject the values
from the token map.

Is there any Jackson component I can hook into?

Our current Jackson integration looks like this,
i.e. the use of a YAMLMapper with a couple of custom deserializers
(the rest is annotation-based).
)

Happy new year and thanks a lot for your help!

[pjfanning]

Is the TLDR here that you want to token substitute in the yaml? If so, I don't know of a pre-built Jackson solution. I would suggest using a String based substitutor.
You probably need to worry about the values that you are substituting in containing new lines because that could throw off the indents in the yaml.

[pjfanning]

One tool that I've used in the past to replace tokens like this is https://velocity.apache.org/engine/

[fbiville]

One thing I did not mention is I would like to whitelist only a few JSON paths for interpolation.
This is harder to do if I'm dealing with raw strings (but maybe there is no way to avoid that).

[cowtowncoder]

Jackson functionality for transformations is limited (by design), so at most you can parser content (JSON, YAML, XML, Avro etc etc) into generic Tree Model (JsonNode-based; and despite name not really JSON specific) and make changes on that.

There are probably libraries based on Jackson that support various transformations but I am not familiar with any.

But if this is more about templating & String substitutions (which it sounds like), perhaps basic templating libraries work even better as per @pjfanning.

[cowtowncoder]

Oh. The possibility of post-processing Strings (which would probably allow working on something here

FasterXML/jackson-core#355

(and its databind counter-part FasterXML/jackson-databind#2971)

is a sort of long-term well-known wish. Ability to do modify Strings after low-level decoding would allow hooking in templating functionality, and make that conditional/configurable (to only apply to some sections).
Unfortunately I haven't had time to tackle this, but it is something on my mental TODO list.

[fbiville]

I think FasterXML/jackson-core#355 would cover my needs :)