Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unexpected response exception #2577

Closed
WeihanLi opened this issue Oct 19, 2023 · 10 comments
Closed

Unexpected response exception #2577

WeihanLi opened this issue Oct 19, 2023 · 10 comments

Comments

@WeihanLi
Copy link
Contributor

When read from redis slave, get unexpected response exception, while the redis slave is read-only and there's no error when from redis master writes, how could we handle or avoid this kind of error? thanks very much

Unexpected response to XREAD: SimpleString: none

Unexpected response to TYPE: MultiBulk: 1 items

Unexpected response to XREVRANGE: BulkString: 570 bytes
@mgravell
Copy link
Collaborator

That seems odd; how are you targeting the replica here - via the optional enum? Or...? And is there anything unusual in your usage?

Does this happen reliably / always? Do you have code that shows such? And: are you talking to an unusual server in any way? (For example, "not actually redis, but redis compatible")

The short version here is "this should just work", so: I'm trying to see how we can reproduce this, as a precursor to identifying and fixing it.

@WeihanLi
Copy link
Contributor Author

@mgravell thanks very much for the quick reply

how are you targeting the replica here - via the optional enum? Or...? And is there anything unusual in your usage?

What does it mean targeting the replica, sorry that I do not really get it.

The Redis we're using is the master-slave mode, and the slave is readonly, we use Redis directly

The API we use likes follows:

await redis.StreamReadAsync(_typedCacheConfig.Subscription, latestMsgId, count: _typedCacheConfig.PrefetchCount)

await redis.StreamRangeAsync(_typedCacheConfig.Subscription, maxId: msgId, count: 1,
                messageOrder: Order.Descending)

await redis.KeyTypeAsync(name)

This exception happened recently, no experience with this before, think it may relate to the Redis server, but no errors from other services using the same Redis server.

@mgravell
Copy link
Collaborator

When read from redis slave

replica and slave are synonyms in this context (with replica being the terminology redis uses more recently); my question is: "what does your setup/code look like, for targeting the replica?". Based on the comment above, it sounds like your config string only mentions the replica endpoint, is that correct? (the library can connect to both, and use different nodes for different requests).

This sounds like a de-sync between client and server, hence me trying to find whether it happens routinely / all the time. And: what is the server here? Is this on-prem redis vanilla, or some hosted cloud offering? Or something redis-like (I think AWS offer a redis-like server that isn't redis, for example). I'm asking because the setup and server change a few things - for example, hosted offerings often use a proxy on the connection, so I'm wondering if the proxy might have de-sync'd. Since redis doesn't have a correlation identifier, this is hard to investigate.

@WeihanLi
Copy link
Contributor Author

what is the server here

We're using a self-hosted contained-based Redis deployed in k8s

it sounds like your config string only mentions the replica endpoint, is that correct?

Yeah, we only have the replica host address in the connection config

From the Redis metrics, it seems the replica is up with the master

image

@mgravell
Copy link
Collaborator

And again: does this happen repeatedly / reliably? Trying to think how we can investigate this.

@WeihanLi
Copy link
Contributor Author

Yes, it happens repeatedly these days from 10 days ago, may need more research and testing

@mgravell
Copy link
Collaborator

ok, allow me to rephrase: is there code, that I can run, that would show it happening reliably, so that I can investigate? without a repro, I don't have much of a route in here. Oh, and : what server version and library version is this?

@WeihanLi
Copy link
Contributor Author

The errors disappeared when I restarted the service, I see OOM exception in the early error logs.

image

So it may not relate to the Redis server(6.0.5) or the library(2.2.4), but to the memory pressure, can not reproduce it now, would try more to reproduce, so sorry for your time.

@DTTerastar
Copy link

We are seeming something similar to this. It happens on one container every week or so. The heartbeat option has been tried but it takes > 20 mins to correct it.

@rlenoci-conga
Copy link

rlenoci-conga commented Apr 25, 2024

I'm seeing a similar issue. Many occurrences of StackExchange.Redis.RedisConnectionException: Unexpected response to ZADD: MultiBulk: 26 items.

Could many of these occurrences lead to OutOfMemoryExceptions? Would low memory possibly cause these kinds of RedisConnectionExceptions?

Other occurrences of StackExchange.Redis.RedisConnectionException that happen around the same time include

Unexpected response to HGETALL: Integer: 0
Unexpected response to HMSET: Integer: 1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants