Copy-Topic behaviour when destination is source

[HailToDodongo]

While using the new "Copy Topic" feature, i tried to copy a topic to itself.
My intention was to duplicate the data, but it ended up filling it without stopping.
I guess copying works by streaming, and it will copy the already copied data again and again.
Depending on the kafka setup this can be dangerous since the only way i could stop this was by killing the akhq container.

IMO either the UI should prevent self-copies, or fall back to a non-streamed copy.

[tchiotludo]

Good report ! thanks :)
Yes it use stream leading to infinite stream :
2 solutions :

• capture the end offset at start
• prevent the same destination topic

[AntAreS24]

I'm planning on using the same feature as @HailToDodongo mentioned, so the option "prevent the same destination topic" wouldn't work for me. I'd like to re-initialise my system in the morning or if something goes wrong.

[archetec]

This is a MAJOR BUG that caused A LOT of troubles where I work, many hours searching for the cause of message duplications and GBs of data being loaded in a topic­ coming out nowhere that created a cascade of other issues in other systems. We figured out someone used that "feature", probably by accident... and it created a feedback loop. We were "lucky" this wasn't in production.

I hope this is being prioritized to TOP LEVEL BUG with possible MAJOR IMPACT on organizations. I recommend all people reading this to block the route to that form in your ingress if you can, because to make matters worse, there's no way to revoke access to this without removing data/write completely.

I also highly suggest you implement much more granular access control on features that are added to current roles. Write access shouldn't include "copy".

[tchiotludo]

@archetec maybe your company could contribute to this easy fix, it will be simplest than report with caps lock 🙃

[archetec]

I agree and totally understand you view point. Unfortunately the company I work for doesn't allow this. Not my decision. As a subcontractor I could do something on my own, but TBH I haven't spent much time coding in Java in the last 15 years so.... yeah. There's that.

The all caps and bold was to warn others of the possibility of major problems in production:

• data duplication with possible business impacts if systems aren't designed to cope with this
• possible downtime if not caught in time, depending on the kind of Kafka deployment... and downtime = $$$$$

TBH, if this was my project, I would release a "quick fix" and deactivate the feature until it is fixed. IMHO, it is that bad. I understand open source projects, and how the software is provided "as is", but as a community, this kind of bug should be top priority.

[HailToDodongo]

@archetec I have to disagree on that, it's not an all-caps major bug.
I think the problem is more on how that tool is used / what your expectations are, and speaks more about the infrastructure of your system.

For me, putting AKHQ in an production environment and letting people use it, is the equivalent of giving someone a root-mysql shell on a production database without transactions.
Kafka is usually used in an enclosed system where no one from the outside writes data, so you should test beforehand how copying topics from A to B (or A to A 🙃 ) behaves.
And AKHQ is just like any other normal producer, but with the ability to write arbitrary data.

What i'm getting at is: AKHQ (when writing data) can be dangerous and you should have some mechanism or fail-safes to stop it when necessary, especially when you do this more often.
Also judging from your post about searching for hours, with proper monitoring it should have been easier to spot since AKHQs CPU and Network usage goes bonkers while copying GBs of data.

Sorry if i sound a bit angry, but i just don't like it when people scream at open source projects without contributing.

Btw, my workaround for now is to create a temporary dummy topic where i copy the data to, to then copy it back to the original topic.
That way it can also be done multiple times.

[tchiotludo]

Thanks @HailToDodongo for the explanation !! ❤️
I don't have more time to explain since I was on holiday but you catch my point of view.

If you are a company using open source software, made by personal people, you just couldn't expect professional support that allow you to use bold and caps lock on issue that could be solved with few hours of works. Just open a PR to solve the issue on your personal time, like the whole software was built.

AKHQ was built because I need a soft like that and I couldn't find that at this time, on my personal time and offer free to all people.

Just keep in mind open source is hard work with no support from most companies that are using it and don't pay for it (look at my sponsor page on GitHub...).

Sending caps lock comment don't help me personally to continue this project, PR on YOUR personal time help more 🤪

[archetec]

Well, I wasn't screaming, sorry if it came through like this, it wasn't my intention. I'm also very sorry for not contributing to the project.

I'm not going to debate further as I don't think it's contributing to anything. It's noted that we are on our own with this issue for the moment and will review our usage of AKHQ for production support.

I understand your position and thank you for your work on this project, and the same to all contributors. AKHQ is a great tool, we just need to adjust.

[timtebeek]

So for now copying onto the same topic is disabled; one can use an intermediate topic for duplication to prevent an endless loop.