[exporter/clickhouseexporter] Issue with fields not populating in ClickHouse Exporter (v0.104.0+)

[AllanAlmeida]

Component(s)

exporter/clickhouse

What happened?

Description

I'm using the OpenTelemetry Collector to export trace data to ClickHouse. The fields CompanyTenantId and CompanyProductName in the otel_traces table are not being populated as expected. These fields should be mapped based on ResourceAttributes['company.tenant.id'] and ResourceAttributes['company.product.name']. However, they remain empty after data insertion.

After testing different versions, I found that this mapping works correctly in v0.103.0 of the clickhouseexporter, but fails in versions v0.104.0 through v0.107.0.

Steps to Reproduce

1. Create a ClickHouse table with fields CompanyTenantId and CompanyProductName, using the schema:

CompanyTenantId LowCardinality(String) DEFAULT ResourceAttributes['company.tenant.id'] CODEC(ZSTD(1)),
CompanyProductName LowCardinality(String) DEFAULT ResourceAttributes['company.product.name'] CODEC(ZSTD(1)),

2. Use the following Go code to send trace data:

document := map[string]interface{}{
    "resourceSpans": []interface{}{
        map[string]interface{}{
            "resource": map[string]interface{}{
                "attributes": []interface{}{
                    map[string]interface{}{
                        "key": "company.tenant.id",
                        "value": map[string]interface{}{
                            "stringValue": tenantId,
                        },
                    },
                    map[string]interface{}{
                        "key": "company.product.name",
                        "value": map[string]interface{}{
                            "stringValue": product,
                        },
                    },
                },
            },
            "scopeSpans": []interface{}{
                map[string]interface{}{
                    "spans": GetSpansApiGateway(numSpans, product),
                },
            },
        },
    },
}

3. Verify the data in ClickHouse. Notice that CompanyTenantId and CompanyProductName remain empty in versions v0.104.0 to v0.107.0, but they populate correctly in v0.103.0.

Expected Result

The CompanyTenantId and CompanyProductName fields in the otel_traces table should be populated based on the values in ResourceAttributes['company.tenant.id'] and ResourceAttributes['company.product.name'].

Actual Result

These fields remain empty in ClickHouse for versions v0.104.0 and above.

Collector version

0.107.0

Environment information

Collector Build and Configuration Details

dist:
  module: bitbucket.org/company/analytics-collector-builder
  name: company-analytics-collector
  description: Company OpenTelemetry Collector
  version: 0.107.0
  output_path: ./_build
  otelcol_version: 0.107.0

extensions:
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/extension/healthcheckextension v0.107.0

exporters:
  - gomod: go.opentelemetry.io/collector/exporter/debugexporter v0.107.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/clickhouseexporter v0.103.0 # Downgraded v0.107.0 => v0.103.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter v0.107.0
  - gomod: go.opentelemetry.io/collector/exporter/otlphttpexporter v0.107.0

processors:
  - gomod: go.opentelemetry.io/collector/processor/batchprocessor v0.107.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/filterprocessor v0.107.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/groupbyattrsprocessor v0.107.0
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/processor/routingprocessor v0.107.0

receivers:
  - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.107.0

replaces:
  - github.com/openshift/api v3.9.0+incompatible => github.com/openshift/api v0.0.1 # datadogexporter depends on openshift api that points to unknown tag v3.9.0

Environment

OS: "Ubuntu 22.04.5 LTS"
Compiler: "go 1.21"

OpenTelemetry Collector configuration

extensions:
  health_check: {}

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  filter/ottl:
    error_mode: ignore
    traces:
      span:
        - 'parent_span_id != SpanID(0x0000000000000000)'

exporters:
  debug: {}
  clickhouse:
    endpoint: http://localhost:8123
    username: admin
    password: admin
    database: otel_tst
    logs_table_name: otel_logs
    traces_table_name: otel_traces
    timeout: 15s
    retry_on_failure:
      enabled: true
      initial_interval: 15s
      max_interval: 30s
      max_elapsed_time: 120s

service:
  extensions:
    - health_check
  telemetry:
    logs:
      level: "debug"
  pipelines:
    traces:
      receivers:
        - otlp
      processors:
        - filter/ottl
      exporters:
        - clickhouse

Log output

No response

Additional context

The issue appears to have been introduced starting with version v0.104.0 of the clickhouseexporter. Reverting to version v0.103.0 resolves the issue, suggesting a possible regression.

[github-actions]

Pinging code owners:

• exporter/clickhouse: @hanjm @dmitryax @Frapschen @SpencerTorres

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[SpencerTorres]

Interesting that this was working before, I'll look for changes to the portion of the code where nested structures get encoded.

I believe your usage of DEFAULT is correct, and it's a good idea to extract these columns. Note that this may change soon if we use the new JSON type for these nested structures