[Bug] TTL Cleaned up delay-delivered messages

[KamenRiderKuuga]

Search before asking

• I searched in the issues and found nothing similar.

Version

pulsar running on docker: apachepulsar/pulsar:3.1.1

Minimal reproduce step

set the broker.conf: ttlDurationDefaultInSeconds=7200
code for sending message:

var messageBuilder = producer.newMessage().value(payload).deliverAfter(21_600_000L, TimeUnit.MILLISECONDS);
messageBuilder.sendAsync();

What did you expect to see?

The consumer received the message after 21,600,000ms(6 hours).

What did you see instead?

The message was cleared by TTL and the consumer did not receive anything.

Anything else?

On the one hand, I want to confirm whether this is the design, because this is quite counter-intuitive. We generally think that the expiration time is calculated from the delivery time of the message.

If this is expected, then how can I coordinate my TTL configuration along with the sending options.

Are you willing to submit a PR?

• I'm willing to submit a PR!

[Technoboy-]

Now the broker side calculate the expire time based on the publishTime of the message.

[KamenRiderKuuga]

Now the broker side calculate the expire time based on the publishTime of the message.

Then some delayed messages based on days or even months will restrict the TTL configuration, making it difficult to configure a reasonable TTL duration.

[Technoboy-]

Now the broker side calculate the expire time based on the publishTime of the message.

Then some delayed messages based on days or even months will restrict the TTL configuration, making it difficult to configure a reasonable TTL duration.

we will discuss this and work out a good way to handle this case.

[visxu]

@KamenRiderKuuga Well, I think your actual behavior is reasonable, since your message delay time > TTL. Once if send lots of long-long time delayed messages but TTL don't clean up them, the storage will be under great pressure.

BTW, from my case, TTL didn't clean up the delayed message, which is your expectation.
#23739