Configuration to support pushing logs to multiple databases/tables using a single CH exporter config.

[tmal94]

Component(s)

exporter/clickhouse

Is your feature request related to a problem? Please describe.

We are using CH exporter to route/push logs to different databases/tables based on certain conditions/filter. Below is an example config.

    clickhouse/test1:
      create_schema: false
      database: xxxx
      endpoint: http://host?dial_timeout=10s&compress=lz4&async_insert=1
      logs_table_name: logs1
      retry_on_failure:
        enabled: true
        initial_interval: 5s
        max_elapsed_time: 300s
        max_interval: 30s
      sending_queue:
        queue_size: 10
      timeout: 60s
      ttl: 168h
    clickhouse/test2:
      create_schema: false
      database: xxxx
      endpoint: http://host?dial_timeout=10s&compress=lz4&async_insert=1
      logs_table_name: logs2
      retry_on_failure:
        enabled: true
        initial_interval: 5s
        max_elapsed_time: 300s
        max_interval: 30s
      sending_queue:
        queue_size: 10
      timeout: 60s
      ttl: 168h

Describe the solution you'd like

Since every time we add a new table or database in the config we end up copying the same set of exporter configuration which makes the resultant config bulky and error prone. So as a part of solution we would want to define the CH exporter config once and define multiple tables based on labels/markers.

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/clickhouse: @hanjm @dmitryax @Frapschen @SpencerTorres

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[SpencerTorres]

Have you tried YAML anchors or similar syntax for reusing parts of the config? (I'm not sure if the parser handles this syntax)

You could also do multiple materialized views that receive logs in a single table and then forwards them into multiple tables, but this may slightly affect the overall architecture that you're going for

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/clickhouse: @hanjm @dmitryax @Frapschen @SpencerTorres

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been closed as inactive because it has been stale for 120 days with no activity.