Describe the bug
We recently upgraded our redises from 6.0.8 to 6.2.3. We are now seeing very odd behavior on streams where a command of the form
XADD <stream> MAXLEN ~ 10 <payload>
is causing stream lengths to reach >20k entries.
We rolled back our redis instances to 6.0.8 with the same config file and cannot repro -- the stream stays put around 10 entries fairly well. We've been running this code/stream paradigm for 2+ years at this point without issue.
In investigating further, we ran the XINFO command on the offending stream (20k+ entries) and found the following:
length
27969
radix-tree-keys
1
radix-tree-nodes
2
To compare, when we try to repro on 6.0.8, we get:
length
10
radix-tree-keys
10
radix-tree-nodes
23
In looking at the code changes between 6.0.8 and 6.2.3 it seems a few things might have changed, not sure which is the culprit here, but perhaps:
1. Radix tree implementation changed s.t. all entries are being put in one node. This would impact the node removal logic since there's not "one node that can be removed s.t. the stream's length is greater than the max length"
2. LIMIT operation/changes impacted the trimming logic
Other notes:
- We're using large binary payloads for this XADD (JPEG compressed images/thumbnails) on the order of 1-100kB each. Not sure if it impacts the debugging here.
- We're setting the stream ID in the XADD and not using *
- We're not setting the LIMIT argument
To reproduce
Steps to reproduce the behavior and/or a minimal code sample.
- Add items to stream with approximate maxlen
- Observe stream is not capped at maxlen
- Use XINFO to observe radix tree allocation
Expected behavior
The stream should be capped around 10 entries, give or take tens of entries (per the XADD docs)
Additional information
Running the command without the approximate (~) flag does not show this issue, i.e. it caps the stream as expected.
Comment From: huangzhw
What is config of
stream-node-max-bytes
stream-node-max-entries
Comment From: dpipemazo
stream-node-max-bytes 0
stream-node-max-entries 0
The same config was used for the tests for both 6.0.8 and 6.2.3 . The only difference between the two tests was building from the redis:6.0.8 docker container vs the redis:6.2.3 docker container in our Dockerfile.
Comment From: guybe7
@dpipemazo can you please change them tp 4096 and 100 respectively and try again?
Comment From: guybe7
just for the record: the behavior in 6.2.3 is as expected: if all stream entries are in a single node, using ~ is futile
it seems like there's a bug in 6.0.8, where it ignores the stream-node-max-entries config
Comment From: dpipemazo
That seemed to work, here is the config change that I did dynamically on a system showing the issue (XADDs paused, previously had been written with MAXLEN ~ 10)
127.0.0.1:6383> CONFIG GET stream-node-max-bytes
1) "stream-node-max-bytes"
2) "0"
127.0.0.1:6383> CONFIG GET stream-node-max-entries
1) "stream-node-max-entries"
2) "0"
127.0.0.1:6383> CONFIG SET stream-node-max-bytes 4096
OK
127.0.0.1:6383> CONFIG SET stream-node-max-entries 100
OK
127.0.0.1:6383> CONFIG GET stream-node-max-entries
1) "stream-node-max-entries"
2) "100"
127.0.0.1:6383> CONFIG GET stream-node-max-bytes
1) "stream-node-max-bytes"
2) "4096"
127.0.0.1:6383> XLEN some-stream
(integer) 58859
We then resumed XADDs with MAXLEN ~ 10 and the stream is now trimming itself
127.0.0.1:6383> XLEN some-stream
(integer) 10
XINFO STREAM some-stream also returned what looks to be nominal results:
length
10
radix-tree-keys
10
radix-key-nodes
23
Did the defaults/settings change across these versions? We're using an off-the-shelf config file that I believe is just set to defaults; would this still be a bug or expected behavior?
Comment From: guybe7
as i mentioned, it seems like 6.0.8 is buggy (ignores stream-node-max-entries)
can you please try with 6.0.15?
and where did you get the config file? using 0 for any of the datatypes sizing limits is almost never a good idea...
Comment From: dpipemazo
@guybe7 @huangzhw thanks for the help. I thought that I had pulled these from the redis conf file but can see from the git blame that the values in this file are the correct defaults and have been more or less since streams were released.
This is on me then, and I appreciate the help in diagnosing/debugging. Sorry for the trouble.