IsMatch condition not matching #34499

P2W2 · 2024-08-07T21:08:51Z

Component(s)

processor/tailsampling

Describe the issue you're reporting

Description

We are using the tail sampling processor and so far it is working fine. Now we have the use case to sample traces based on the spanevent name(here: "name": "service.osrr = {String} "au:US:123test123"\n",). We try to match the identifier "au:" contained in the string with a regex. For that we use the IsMatch("name", "^.au:.$") function.
We also tried 'IsMatch(attributes["message"], "^.au:.$").
Our latest approach for the processor config:

    tail_sampling/test:
      decision_wait: 120s
      num_traces: 50000
      policies:
        [
          {
            name: test-policy,
            type: ottl_condition,
            ottl_condition:
              {
                error_mode: ignore,
                spanevent:
                  [
                  'IsMatch("name", "^.*po:.*$")',
                  'IsMatch("name", "^.*au:.*$")',
                  'IsMatch("name", "^.*vw:.*$")',
                  'IsMatch("name", "^.*sk:.*$")'
                  ],
              },
          },
        ]

Unfortunately the result is not a match:
debug [email protected]/parser.go: 338 condition evaluation result

{
    "kind": "processor",
    "name": "tail_sampling/test",
    "policy": "ottl_condition",
    "condition": "IsMatch(\"name\", \"^.*au:.*$\")",
    "match": false,
    "TransformContext": {
        "resource": {
            "attributes": {
                "service.name": "gateway",
                "telemetry.sdk.language": "java",
                "telemetry.sdk.name": "opentelemetry",
                "telemetry.sdk.version": "1.35.0",
                 ...
            },
            "dropped_attribute_count": 0
        },
        "scope": {
            "attributes": {},
            "dropped_attribute_count": 0,
            "name": "com.gateway",
            "version": "11"
        },
        "span": {
            "attributes": {
             ....some atts.
            },
            "dropped_attribute_count": 0,
            "dropped_events_count": 0,
            "dropped_links_count": 0,
            "end_time_unix_nano": 1723016747917297072,
            "events": [
                {
                    "attributes": {},
                    "dropped_attribute_count": 0,
                    "name": "service.osrr = {String} \"au:US:123test123\"\n",
                    "time_unix_nano": 1723016747917273372
                }
            ],
            "kind": "Server",
            "links": [],
            "name": "api",
            "parent_span_id": "1fd22489829bf1f4",
            "span_id": "f5703065bedb7c8a",
            "start_time_unix_nano": 1723016747867968194,
            "status.code": "Ok",
            "status.message": "No Error",
            "trace_id": "3189244eec1ca6eb7fd2237f7a9ff394",
            "trace_state": ""
        },
    "spanevent": {
            "attributes": {},
            "dropped_attribute_count": 0,
            "name": "service.osrr = {String} \"au:US:123test123\"\n",
            "time_unix_nano": 1723016747917273372
        },
        "cache": {}
    }
}

Firstly, I'm a little confused about what is meant by condition in the tail sampling documentation ottl_condition section. I can't find anything about it in the Ottl grammar documentation. I think conditions refers to all expressions that always result in a boolean value (true or false), but does this include functions like IsMatch?
Secondly, I am unsure if we are accessing the spanevent.name property correctly? In the documentation, the key 'name' is often used as an example, but I'm not sure if I can apply these examples to my case here.
Thirdly, I don't understand why it says 'name' as key in the debug message. When I look at the traces and spans, the string is located under message.

"spans": [
                        {
                            "traceId": "5c38504be6e726a0ad5dc894615ea28b",
                            "spanId": "e3a02a4b75e15e92",
                            "parentSpanId": "0000000000000000",
                            "traceState": "",
                            "name": "api:bs:climatisation:v1:Dispatcher_Actions",
                            "kind": "SPAN_KIND_SERVER",
                            "startTimeUnixNano": 1722947829274117000,
                            "endTimeUnixNano": 1722947829511452000,
                            "attributes": [
                                {...}
                            ],
                            "droppedAttributesCount": 0,
                            "droppedEventsCount": 0,
                            "droppedLinksCount": 0,
                            "status": {
                                "code": 1,
                                "message": "No Error"
                            },
                            "events": [
                                {
                                    "timeUnixNano": 1722947829511445000,
                                    "attributes": [
                                        {
                                            "key": "message",
                                            "value": {
                                                "stringValue": "service.osrr = {String} \"au:US:123test123\"\n"
                                            }
                                        }
                                    ],
                                    "droppedAttributesCount": 0,
                                    "name": ""
                                }
                            ]
                        }
                    ]

Describe the solution you'd like

A possibility to match spanevent.name based on a regex expression.

The text was updated successfully, but these errors were encountered:

github-actions · 2024-08-07T21:09:07Z

Pinging code owners:

processor/tailsampling: @jpkrohling

See Adding Labels via Comments if you do not have permissions to add labels yourself.

bacherfl · 2024-08-08T10:43:27Z

Hi @P2W2 it seems like in the IsMatch functions the name is provided as a literal string - can you try to modify it as follows, e.g. change 'IsMatch("name", "^.*po:.*$") to 'IsMatch(name, "^.*po:.*$")'?

P2W2 · 2024-08-08T16:52:02Z

Hi @bacherfl,
thanks for the fast help.
The result is also false, see below.

2024-08-08T15: 51: 08.007Z	debug	[email protected]/parser.go: 338	condition evaluation result	{
    "kind": "processor",
    "name": "tail_sampling/test",
    "pipeline": "traces/test_sampling",
    "policy": "ottl_condition",
    "condition": "IsMatch(name, \"^.*vw:.*$\")",
    "match": false,
    "TransformContext": {
...some other stuff ...
        "spanevent": {
            "attributes": {},
            "dropped_attribute_count": 0,
            "name": "service.osrr = {String} \"vw:DE:1234test1234\"\n",
            "time_unix_nano": 1723132143552836997
        },
        "cache": {}
    }
}

I don't understand why it says 'name' as key in the debug message. When I look at the traces and spans, the string is located under message attribute. Therefore we also tried 'IsMatch(attributes["message"], "^.au:.$")' without success.

"spans": [
                        {
                            "traceId": "5c38504be6e726a0ad5dc894615ea28b",
                            "spanId": "e3a02a4b75e15e92",
                            "parentSpanId": "0000000000000000",
                            "traceState": "",
                            "name": "api:bs:climatisation:v1:Dispatcher_Actions",
                            "kind": "SPAN_KIND_SERVER",
                            "startTimeUnixNano": 1722947829274117000,
                            "endTimeUnixNano": 1722947829511452000,
                            "attributes": [
                                {...}
                            ],
                            "droppedAttributesCount": 0,
                            "droppedEventsCount": 0,
                            "droppedLinksCount": 0,
                            "status": {
                                "code": 1,
                                "message": "No Error"
                            },
                            "events": [
                                {
                                    "timeUnixNano": 1722947829511445000,
                                    "attributes": [
                                        {
                                            "key": "message",
                                            "value": {
                                                "stringValue": "service.osrr = {String} \"au:US:123test123\"\n"
                                            }
                                        }
                                    ],
                                    "droppedAttributesCount": 0,
                                    "name": ""
                                }
                            ]
                        }
                    ]

bacherfl · 2024-08-09T05:58:57Z

thank you for the update on this @P2W2 - I will have a closer look today and try to reproduce this behaviour on my end. I will give you an update as soon as I find something

bacherfl · 2024-08-09T07:46:46Z

@P2W2 Alright I was now able to reproduce this. It seems like the line breaks at the end of the message value caused the regex to not match. I managed to make it match by altering the regex to the following:

IsMatch(attributes["message"], "^.*au:.*[\n]+$")

Can you try to check if this works for you as well?

I'm not exactly sure why name has been used as a key in the debug message above though. When i tried it out it seems like it was using the message key as expected:

2024-08-09T09:39:38.352+0200    debug   [email protected]/parser.go:394     condition evaluation result 
{
    "kind": "processor",
    "name": "tail_sampling",
    "pipeline": "traces",
    "policy": "ottl_condition",
    "condition": "IsMatch(attributes[\"message\"], \"^.*au:.*[\\n]+$\")",
    "match": true,
    "TransformContext": {
        "resource": {
            "attributes": {
                "service": "flo-service"
            },
            "dropped_attribute_count": 0
        },
        "scope": {
            "attributes": {},
            "dropped_attribute_count": 0,
            "name": "my-scope",
            "version": "0.1"
        },
        "span": {
            "attributes": {
                "attr1": "test",
                "foo": 11
            },
            "dropped_attribute_count": 0,
            "dropped_events_count": 0,
            "dropped_links_count": 0,
            "end_time_unix_nano": 1723189161664386000,
            "events": [
                {
                    "attributes": {
                        "message": "service.osrr = {String} \"au:US:123test123\"\n"
                    },
                    "dropped_attribute_count": 0,
                    "name": "",
                    "time_unix_nano": 1723189158664386000
                }
            ],
            "kind": "Client",
            "links": [],
            "name": "flo-span",
            "parent_span_id": "0000000000000000",
            "span_id": "3064323638353764",
            "start_time_unix_nano": 1723189156664385000,
            "status.code": "Ok",
            "status.message": "successful",
            "trace_id": "33303631363434612d633538632d3430",
            "trace_state": ""
        },
        "spanevent": {
            "attributes": {
                "message": "service.osrr = {String} \"au:US:123test123\"\n"
            },
            "dropped_attribute_count": 0,
            "name": "",
            "time_unix_nano": 1723189158664386000
        },
        "cache": {}
    }
}

Is there maybe another processor in the pipeline that could have altered the name of this attribute?

github-actions · 2024-08-13T13:28:01Z

Pinging code owners for pkg/ottl: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley. See Adding Labels via Comments if you do not have permissions to add labels yourself.

jpkrohling · 2024-08-13T13:28:13Z

This is probably something related to OTTL, not tail-sampling.

meSATYA · 2024-08-23T08:53:09Z

Hi @bacherfl , can you please share the full configuration for the tail sampling with the working regex?

IsMatch(attributes["message"], "^.*au:.*[\n]+$")

bacherfl · 2024-08-26T05:22:06Z

Hi @bacherfl , can you please share the full configuration for the tail sampling with the working regex?

IsMatch(attributes["message"], "^.*au:.*[\n]+$")

Sure, this is what I tried it out with:

  tail_sampling:
    decision_wait: 5s
    policies:
      [
        {
          name: test-policy,
          type: ottl_condition,
          ottl_condition:
            {
              error_mode: ignore,
              spanevent:
                [
                  'IsMatch(attributes["message"], "^.*po:.*[\n]+$")',
                  'IsMatch(attributes["message"], "^.*au:.*[\n]+$")',
                  'IsMatch(attributes["message"], "^.*vw:.*[\n]+$")',
                  'IsMatch(attributes["message"], "^.*sk:.*[\n]+$")'
                ],
            },
        },
      ]

P2W2 · 2024-08-28T06:55:44Z

I tried the same configuration and it is still not working.

Is there maybe another processor in the pipeline that could have altered the name of this attribute?

No, we only using the tail_sampling processor.

    tail_sampling/test:
      decision_wait: 120s
      policies:
        [
          {
            name: test-policy,
            type: ottl_condition,
            ottl_condition:
              {
                error_mode: ignore,
                spanevent:
                  [
                    'IsMatch(attributes["message"], "^.*po:.*[\n]+$")',
                    'IsMatch(attributes["message"], "^.*au:.*[\n]+$")',
                    'IsMatch(attributes["message"], "^.*vw:.*[\n]+$")',
                    'IsMatch(attributes["message"], "^.*sk:.*[\n]+$")'
                  ],
              },
          },
        ]
...
traces/test_sampling:
        exporters:
          - debug
          - otlphttp/tempo-processor
        processors:
          - tail_sampling/test
        receivers:
          - otlp

I also tested it by sampling all traces and there was no alterations. The String was still placed under ...events.attributes.attributes["message"] in the resulting trace and no name was given.

                            "events": [
                                {
                                    "timeUnixNano": 1724755911666150100,
                                    "attributes": [
                                        {
                                            "key": "message",
                                            "value": {
                                                "stringValue": "service.osrr = {String} \"vw:DE:test123457\"\n"
                                            }
                                        }
                                    ],
                                    "droppedAttributesCount": 0,
                                    "name": ""
                                }
                            ]
                        }
                    ],

But in contrast to your successful example, the string is displayed under the name in the debug message:

        "spanevent": {
            "attributes": {},
            "dropped_attribute_count": 0,
            "name": "service.osrr = {String} \"vw:DE:test123457\"\n",
            "time_unix_nano": 1724754699817154578
        },

I can't figure out why?
Can you also provide me the trace you use for testing?

P2W2 · 2024-08-28T22:22:33Z

With ‘IsMatch(name, \’^.*xx:.*[\\n]+$\")’ it now also works on our site.
But it still leaves me a little perplexed why I have to specify name as the key here and not attributes[‘message’]

Anyway, a big thank you for your help @bacherfl

bacherfl · 2024-08-29T05:24:53Z

Thanks for the update @P2W2 - good to hear it is working with using name as the targeted attribute. I cannot find an explanation for why it's not working with attributes['message'] though - I agree that is rather confusing.

Not sure if that helps, but I was using a plain grpc client and the protobuf structs in Go when I was creating me trace with which it was working, i.e. like so:

        conn, err := grpc.NewClient("localhost:4317", grpc.WithTransportCredentials(insecure.NewCredentials()))
	if err != nil {
		log.Panic(err.Error())
	}

	client := coltracepb.NewTraceServiceClient(conn)

	req := &coltracepb.ExportTraceServiceRequest{ResourceSpans: make([]*v1.ResourceSpans, 1)}

	req.ResourceSpans[0] = &v1.ResourceSpans{
		Resource: &v12.Resource{
			Attributes: []*v13.KeyValue{
				{
					Key:   "service",
					Value: &v13.AnyValue{Value: &v13.AnyValue_StringValue{StringValue: "flo-service"}},
				},
			},
		},
		ScopeSpans: []*v1.ScopeSpans{
			{
				Scope: &v13.InstrumentationScope{
					Name:    "my-scope",
					Version: "0.1",
				},
				Spans: []*v1.Span{
					{
						TraceId:           []byte(uuid.New().String()[:16]),
						SpanId:            []byte(uuid.New().String()[:8]),
						Name:              "flo-span",
						Kind:              v1.Span_SPAN_KIND_CLIENT,
						StartTimeUnixNano: uint64(time.Now().UTC().Add(-time.Second * 5).UnixNano()),
						EndTimeUnixNano:   uint64(time.Now().UTC().UnixNano()),
						Events: []*v1.Span_Event{
							{
								TimeUnixNano: uint64(time.Now().UTC().Add(-time.Second * 3).UnixNano()),
								Name:         "",
								Attributes: []*v13.KeyValue{
									{
										Key:   "message",
										Value: &v13.AnyValue{Value: &v13.AnyValue_StringValue{StringValue: "service.osrr = {String} \"au:US:123test123\"\n"}},
									},
								},
								DroppedAttributesCount: 0,
							},
						},
						Status: &v1.Status{
							Message: "successful",
							Code:    v1.Status_STATUS_CODE_OK,
						},
					},
				},
			},
		},
	}

	export, err := client.Export(context.Background(), req)

	if err != nil {
		log.Fatalf("could not export trace: %v", err)
	}

github-actions · 2024-10-29T03:34:57Z

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

pkg/ottl: @TylerHelmuth @kentquirk @bogdandrutu @evan-bradley

See Adding Labels via Comments if you do not have permissions to add labels yourself.

github-actions · 2024-12-28T05:20:02Z

This issue has been closed as inactive because it has been stale for 120 days with no activity.

P2W2 added the needs triage New item requiring triage label Aug 7, 2024

github-actions bot added the processor/tailsampling Tail sampling processor label Aug 7, 2024

github-actions bot mentioned this issue Aug 13, 2024

Weekly Report: 2024-08-06 - 2024-08-13 #34626

Closed

jpkrohling added pkg/ottl and removed processor/tailsampling Tail sampling processor needs triage New item requiring triage labels Aug 13, 2024

TylerHelmuth added the discussion needed Community discussion needed label Aug 13, 2024

github-actions bot added the Stale label Oct 29, 2024

github-actions bot added the closed as inactive label Dec 28, 2024

github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Dec 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IsMatch condition not matching #34499

IsMatch condition not matching #34499

P2W2 commented Aug 7, 2024 •

edited

Loading

github-actions bot commented Aug 7, 2024

bacherfl commented Aug 8, 2024

P2W2 commented Aug 8, 2024 •

edited

Loading

bacherfl commented Aug 9, 2024

bacherfl commented Aug 9, 2024 •

edited

Loading

github-actions bot commented Aug 13, 2024

jpkrohling commented Aug 13, 2024

meSATYA commented Aug 23, 2024

bacherfl commented Aug 26, 2024 •

edited

Loading

P2W2 commented Aug 28, 2024 •

edited

Loading

P2W2 commented Aug 28, 2024

bacherfl commented Aug 29, 2024 •

edited

Loading

github-actions bot commented Oct 29, 2024

github-actions bot commented Dec 28, 2024

IsMatch condition not matching #34499

IsMatch condition not matching #34499

Comments

P2W2 commented Aug 7, 2024 • edited Loading

Component(s)

Describe the issue you're reporting

Description

Describe the solution you'd like

github-actions bot commented Aug 7, 2024

bacherfl commented Aug 8, 2024

P2W2 commented Aug 8, 2024 • edited Loading

bacherfl commented Aug 9, 2024

bacherfl commented Aug 9, 2024 • edited Loading

github-actions bot commented Aug 13, 2024

jpkrohling commented Aug 13, 2024

meSATYA commented Aug 23, 2024

bacherfl commented Aug 26, 2024 • edited Loading

P2W2 commented Aug 28, 2024 • edited Loading

P2W2 commented Aug 28, 2024

bacherfl commented Aug 29, 2024 • edited Loading

github-actions bot commented Oct 29, 2024

github-actions bot commented Dec 28, 2024

P2W2 commented Aug 7, 2024 •

edited

Loading

P2W2 commented Aug 8, 2024 •

edited

Loading

bacherfl commented Aug 9, 2024 •

edited

Loading

bacherfl commented Aug 26, 2024 •

edited

Loading

P2W2 commented Aug 28, 2024 •

edited

Loading

bacherfl commented Aug 29, 2024 •

edited

Loading