Redis Version: 7.2.0 OS: Rocky Linux release 8.8
Issue: After upgrading Redis from 6.2.6 to 7.2.0, we noticed disk getting full consistently on Redis nodes. I couldn't find any files taking up disk space so checked deleted files using lsof and found many of these on all nodes:
redis-ser 1029514 2079310 thread-po redis 20w REG 259,37 1749209225 33554577 /data/redis/dev/dump.rdb (deleted) redis-ser 1029514 2079310 thread-po redis 27w REG 259,37 1744609492 33554573 /data/redis/dev/dump.rdb (deleted) redis-ser 1029514 2079310 thread-po redis 28w REG 259,37 1744435205 33554563 /data/redis/dev/dump.rdb (deleted) redis-ser 1029514 2079310 thread-po redis 29w REG 259,37 1744439045 33554566 /data/redis/dev/dump.rdb (deleted) redis-ser 1029514 2079310 thread-po redis 30w REG 259,37 1744679573 33554567 /data/redis/dev/dump.rdb (deleted)
Redis is installed in a different partition than the dump.rdb directory. Restarting Redis removes these files and clears up the disk space.
Redis.conf:
bind 0.0.0.0 port 6380 daemonize yes loadmodule /home/redis/redis_sentinel_dev/data/modules/redistimeseries.so loadmodule /home/redis/redis_sentinel_dev/data/modules/redisearch.so TIMEOUT 0 MAXSEARCHRESULTS -1 MINPREFIX 2 loadmodule /home/redis/redis_sentinel_dev/data/modules/rejson.so loadmodule /home/redis/redis_sentinel_dev/data/modules/libredisgears.so v8-plugin-path /home/redis/redis_sentinel_dev/data/modules/libredisgears_v8_plugin.so pidfile "/home/redis/redis_sentinel_dev/data/run/redis.pid" logfile "/home/redis/redis_sentinel_dev/data/log/redis.log" dir "/data/redis/dev"
Generated by CONFIG REWRITE
save 300 100 save 60 10000 latency-tracking-info-percentiles 50 99 99.9
Please suggest a permanent solution for this as its not feasible to restart redis regularly.
Comment From: yossigo
@rishinair19 Anything interesting in the log file?
Comment From: oranagra
i think i was able to reproduce it with default configuration. looks like the replica keeps the rdb file open after full sync. need to debug it later, and also trace the change that caused it.
Comment From: rishinair19
@yossigo
Nope, nothing like that in the log file
@oranagra Apologies for late reply. In any way If you can prioritise this, it would be very helpful. We didn't observe this issue on test so upgraded Redis on Prod environment as well. Currently, we're restarting Redis weekly on Prod as workaround. Let me know if there is any other.
Comment From: oranagra
i found the issue, introduced in 7.2 with this change: https://github.com/redis/redis/pull/11248/files#diff-955c0fee32b574a1d9adb4540b8ff1564b15db9695d88a8dabc66b3c19c825daL2205 @uvletter do you have any idea why we deleted that line?
Comment From: uvletter
i found the issue, introduced in 7.2 with this change: https://github.com/redis/redis/pull/11248/files#diff-955c0fee32b574a1d9adb4540b8ff1564b15db9695d88a8dabc66b3c19c825daL2205 @uvletter do you have any idea why we deleted that line?
Just investigated the commit history, I found in the first several version I replace the close with a bioCreateCloseJob to reclaim cache and close fd, in the later version I moved the reclaim function to rdbLoad as mentioned in conversation , but forgot to recover the close...
Comment From: uvletter
https://github.com/redis/redis/pull/11248/commits/b141aadbd55234be973522af479e70e6d31af8be#diff-955c0fee32b574a1d9adb4540b8ff1564b15db9695d88a8dabc66b3c19c825da is the commit that involved the bug
Comment From: rishinair19
@oranagra Is it possible to know when new version with this fix will be released?
Comment From: oranagra
i'm looking into the possibility of releasing it next week.
Comment From: rishinair19
Thanks, much appreciated!