Redis [BUG] - Nineya|java/go/python

Describe the bug

We are getting failures since version 7.2 when using SSCAN passing the cursor.

To reproduce

As reference https://github.com/docker-library/redis/issues/376.

Expected behavior

That SSCAN respect the cursor that is provided.

Additional information

Before the new release image of docker redis, it was working as expected.

Comment From: sundb

@roggervalf Through reading the failed test of bullmq, do you mean that the scan speed is quicker than expected? Since #11290, the zset starts to support listpack encoding. When scanning a Listpack encoding zset, it does not have a maximum sampling limit like hash encoding, it is scanned from the entire zset, so faster speeds are possible. btw, I have checked the first failed commit is #11290.

Comment From: roggervalf

hi @sundb, sorry I must add an example, so for example I have a set with 65 keys, but when I use count as 50, I'm getting all values:

> sscan bull:parent-queue-2b81e4e6-12f8-4218-bec4-00c63e3cbcbe:1:dependencies 0 count 50
1) "0"
2) 1) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:1"
   2) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:2"
   3) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:3"
   4) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:4"
   5) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:5"
   6) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:6"
   7) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:7"
   8) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:8"
   9) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:9"
   10) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:10"
   11) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:11"
   12) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:12"
   13) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:13"
   14) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:14"
   15) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:15"
   16) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:16"
   17) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:17"
   18) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:18"
   19) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:19"
   20) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:20"
   21) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:21"
   22) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:22"
   23) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:23"
   24) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:24"
   25) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:25"
   26) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:26"
   27) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:27"
   28) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:28"
   29) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:29"
   30) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:30"
   31) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:31"
   32) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:32"
   33) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:33"
   34) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:34"
   35) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:35"
   36) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:36"
   37) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:37"
   38) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:38"
   39) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:39"
   40) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:40"
   41) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:41"
   42) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:42"
   43) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:43"
   44) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:44"
   45) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:45"
   46) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:46"
   47) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:47"
   48) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:48"
   49) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:49"
   50) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:50"
   51) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:51"
   52) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:52"
   53) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:53"
   54) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:54"
   55) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:55"
   56) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:56"
   57) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:57"
   58) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:58"
   59) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:59"
   60) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:60"
   61) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:61"
   62) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:62"
   63) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:63"
   64) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:64"
   65) "bull:test-a2039e6a-7939-406b-9719-d35ed08ccdca:65"

even with any other number, I'm testing this behavior also using redis-cli on version 7.2

Comment From: Nibbler999

https://redis.io/docs/manual/keyspace/#why-scan-may-return-all-the-items-of-an-aggregate-data-type-in-a-single-call

Comment From: oranagra

The COUNT argument to the SCAN command family doesn't mean that you will get exactly or at most this count. SCAN and it's cursor is designed so that with the hashtable mechanics, it will never stop in the middle of a hash bucket, so it'll usually return a few items more than you requested with COUNT, and that's normal. The COUNT feature is just a general guideline, to prevent the scan from consuming too much time, or producing a result that's too large. With ziplist and listpack encoding, it is not possible to scan using a cursor, so in these cases SCAN must return all the items in one go, and that's fine considering these collections are always rather small, so it doesn't conflict with the above concern.

Comment From: manast

It seems that COUNT did work differently before 7.2 though, as the tests above work well with all versions of Redis before 7.2. It is also worth mentioning that there is no current documentation for the COUNT argument specifying its semantics, not sure if it has been like this before, but I kind of recall that it was documented.

https://redis.io/commands/sscan/ https://redis.io/commands/scan/

Comment From: manast

Ok I found it here: https://redis.io/docs/manual/keyspace/

"While SCAN does not provide guarantees about the number of elements returned at every iteration, it is possible to empirically adjust the behavior of SCAN using the COUNT option. Basically with COUNT the user specified the amount of work that should be done at every call in order to retrieve elements from the collection. This is just a hint for the implementation, however generally speaking this is what you could expect most of the times from the implementation."

Comment From: oranagra

since in 7.2 we changed small sets from ziplist encoding to listpack encoding, it could be that some edge cases that in the past were encoded with dict are now encoded as listpack, and thus in these cases in the past it used to SSCAN gradually and now it does in one go.

maybe you can validate that and provide some info by running SCARD <key> and OBJECT ENCODING <key> on the old and new version (as well as state if you're using the default config)

Comment From: sundb

since in 7.2 we changed small sets from ziplist encoding to listpack encoding, it could be that some edge cases that in the past were encoded with dict are now encoded as listpack, and thus in these cases in the past it used to SSCAN gradually and now it does in one go.

@oranagra set doesn't change from ziplist encoding to listpack, it adds listpack encoding. The test fails because the set in this test is a 65 element hash, but in 7.2 it is encoded as a listpack.

Comment From: oranagra

ohh right. i'm mixed it up with hash.