Describe the bug I have a system that uses Redis for metadata storing and this issue has happened for 2 times since a few months ago. The hkey that i use for metadata just disappear at a random moment.
To reproduce
I have checked every thing related to redis in my code and all of what i've used are:
redis_client = redis.Redis(host=REDIS_HOST, port=REDIS_PORT)
redis_client.ping()
redis_client.hset(collection_name, str(face_id), entities)
redis_client.hexists(collection_name, str(face_id))
redis_client.hget(collection_name, str(face_id))
redis_client.hdel(collection_name, str(face_id))
where collection_name is the hkey that i lost
Expected behavior
In a beautiful unplanned day, the only key that i use disappear, others live healthily. As the missing key is the one i use, do you think the one of the commands above could delete the key any cases???
Additional information
I'm running redis in the docker from https://hub.docker.com/_/redis
with this config
redis.zip
Comment From: sundb
After hdel, if the num of fileds in hash is empty, the key will be deleted.
Comment From: TannedCung
Thank you for your answer. I've just checked my logs (not Redis logs), There was no call to hdel before the moment the key disappeared. I have another DB that save different information of the same object, whenever the client calls a delete API, it should remove from both DBs. In my case, only information in Redis went away. Btw, my logs doesn't show any client request. So i assume this should be an internal problem.
Comment From: sundb
hi @TannedCung
Normally, redis will only delete a key if eviction is enabled, or if the key is set to expire, you don't have eviction enabled, do you have an expire time?
Can you provide info all?
Comment From: TannedCung
Hi @sundb. Here is info
# Server
redis_version:6.2.3
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:dc20d908b7b619b4
redis_mode:standalone
os:Linux 4.15.0-32-generic x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:8.3.0
process_id:1
process_supervised:no
run_id:7300306f0814471d9a51d07080d73d26b4881f1d
tcp_port:6379
server_time_usec:1622774705538322
uptime_in_seconds:86182
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:12161969
executable:/data/redis-server
config_file:/redis.conf
io_threads_active:0
# Clients
connected_clients:6
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:32
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
# Memory
used_memory:2914992
used_memory_human:2.78M
used_memory_rss:10260480
used_memory_rss_human:9.79M
used_memory_peak:3304056
used_memory_peak_human:3.15M
used_memory_peak_perc:88.22%
used_memory_overhead:935968
used_memory_startup:809896
used_memory_dataset:1979024
used_memory_dataset_perc:94.01%
allocator_allocated:3014728
allocator_active:3612672
allocator_resident:6488064
total_system_memory:12591013888
total_system_memory_human:11.73G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.20
allocator_frag_bytes:597944
allocator_rss_ratio:1.80
allocator_rss_bytes:2875392
rss_overhead_ratio:1.58
rss_overhead_bytes:3772416
mem_fragmentation_ratio:3.57
mem_fragmentation_bytes:7388248
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:123048
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
# Persistence
loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1622769258
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:729088
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
# Stats
total_connections_received:11
total_commands_processed:921
instantaneous_ops_per_sec:0
total_net_input_bytes:1108407
total_net_output_bytes:18296007
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:1688
evicted_keys:0
keyspace_hits:847
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:390
total_forks:15
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:3
dump_payload_sanitizations:0
total_reads_processed:1040
total_writes_processed:926
io_threaded_reads_processed:0
io_threaded_writes_processed:0
# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:51508964902a0dfcf45fc70ea4b1a98a0284bea7
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:57.217578
used_cpu_user:75.233882
used_cpu_sys_children:0.059562
used_cpu_user_children:0.337459
used_cpu_sys_main_thread:57.194327
used_cpu_user_main_thread:75.207285
# Modules
# Errorstats
errorstat_ERR:count=3
# Cluster
cluster_enabled:0
# Keyspace
db0:keys=62,expires=0,avg_ttl=0
Comment From: sundb
Is it info all? I can't see the Commandstats information.
Comment From: TannedCung
My bad, this is Commandstats
# Commandstats
cmdstat_set:calls=4,usec=92,usec_per_call=23.00,rejected_calls=0,failed_calls=0
cmdstat_hgetall:calls=2,usec=967,usec_per_call=483.50,rejected_calls=0,failed_calls=0
cmdstat_ping:calls=5,usec=4,usec_per_call=0.80,rejected_calls=0,failed_calls=0
cmdstat_keys:calls=4,usec=145,usec_per_call=36.25,rejected_calls=0,failed_calls=0
cmdstat_info:calls=1,usec=123,usec_per_call=123.00,rejected_calls=0,failed_calls=0
cmdstat_hdel:calls=5,usec=77,usec_per_call=15.40,rejected_calls=0,failed_calls=0
cmdstat_command:calls=3,usec=2217,usec_per_call=739.00,rejected_calls=0,failed_calls=0
cmdstat_hget:calls=850,usec=18940,usec_per_call=22.28,rejected_calls=0,failed_calls=0
cmdstat_config:calls=4,usec=3144,usec_per_call=786.00,rejected_calls=0,failed_calls=3
cmdstat_hset:calls=49,usec=1578,usec_per_call=32.20,rejected_calls=0,failed_calls=0
cmdstat_hexists:calls=7,usec=57,usec_per_call=8.14,rejected_calls=0,failed_calls=0
The full info is as below
Comment From: sundb
@TannedCung Right now i have no way of knowing exactly what the problem is.
It seems that only hdel command will delete the key, the possibility of redis randomly delete the key should be very low.
Can you provide some info about might be going wrong?
Comment From: TannedCung
Right now, i have no other clue but will try to log more info just in case this problem occurs again. In the bad scenario, Can i have any workarounds to avoid data loss ??
Comment From: sundb
@TannedCung You can write the data to other database(ex mysql) as well. Read from redis first, then read from database when you can't found it. You can also turn on aof, because it can record all the commands, and you can check if it's deleted normally.
Comment From: TannedCung
I will give it a try. This issue will be close for the moment and. If the issue happens again, i will reopen and favor your help with some more information. Thank you @sundb for your help.