Describe the bug when you subscribe on huge number of keys with KeySpace, redis will block. when i check procedure i found that in keyspace, redis will store all patterns and iterate on patterns to check client subscribed on key or not. when we subscribed on huge number of keys , the list size will be to large and iterate on list take long time.
To reproduce
enable keyspace notification and subscribed on huge number of keys and update them.
Expected behavior
redis works normaly and did not block.
Comment From: sundb
~~Some of my thoughts.~~
~~In practice, when keyspace notify is used, most clients should be using the same pattern to listen, so it would be too much cpu overhead to iterate through all users' patterns.~~
~~Maybe we should add a new pubsubtype (e.g pubSubNotifyType) to send keyspace notifications by traversing the serverPubSubChannels by pattern, instead of traversing all users.~~
Misunderstanding, please ignore me.
Comment From: sundb
@KhDanial Each of your clients is using a different pattern for keyspace? Which version of redis are you using?
Comment From: KhDanial
4.0.14
they can use same pattern, for example both of user subscribe on key "SampleKey".
imagine a situation that single client subscribe on 1e7 key then redis will be block and does not work properly,
Comment From: sundb
@KhDanial The same patern optimization for pubsub has been implemented at https://github.com/redis/redis/commit/dfb12f06283f22c157d99830de21700a7f86c139. But it is after version 6, maybe you can upgrade redis to a newer version.
Comment From: KhDanial
@sundb thanks Was I able to clarify the problem correctly?
Comment From: sundb
@KhDanial You mean you use a client that subscribes to a large number of patterns? Can you tell us about your usage scenario? (or the pattern you use)
Comment From: KhDanial
@sundb If you mean a lot of patterns, a lot of keys, yes, that's my problem. i tested this https://github.com/redis/redis/commit/dfb12f06283f22c157d99830de21700a7f86c139 and it was not ok.
In a scenario imagine just one user subscribe on a large number of keys with KeySpace. as number of keys incresed, insert new keys will be so slow, because after each insertion redis will iterate on all keys(patterns) and it is not okay. Did the problem clear up?
Comment From: sundb
If you use keyspace notifications with a large number of patterns, it should be inevitable to iterate through all of them. I'm not sure why you're using so many patterns, can you tell us what patterns you're using?
Comment From: KhDanial
@sundb psubscribe --keyspace@3--:key1 psubscribe --keyspace@3--:key2 psubscribe --keyspace@3--:key3 ..... psubscribe --keyspace@3--:key_100000
if you subscribe on these keys, redis will be to slow
Comment From: KhDanial
It's important for me to understand what happened to some of the keys. To further clarify the issue, keep in mind that two clients may make changes to a database and shared keys that both parties need to be aware of.
Comment From: sundb
@KhDanial Using pattern matching a lot is bound to consume a lot of cpu time, just as we recommend using scan instead of keys to traverse all keys.
I'm surprised that you subscribed to 100000 patterns, why not use __keyspace@3__:key*?
If you have N clients subscribed to 100000 patterns, if one of them changes the key, it will cause the key to be matched by 100000 patterns, and then send a notification to (N-1) clients.
Comment From: KhDanial
what does this do? keyspace@3:key*
Comment From: sundb
psubscribe __keyspace@3__:key*
Comment From: KhDanial
i use your pattern, but i cant write in same format as you write :))))
Comment From: sundb
i use your pattern, but i cant write in same format as you write :))))
I cant follow you, can't use it anymore?