Crash report
538912:C 09 Mar 2024 09:44:19.056 * RDB: 14 MB of memory used by copy-on-write
152693:M 09 Mar 2024 09:44:19.201 * Background saving terminated with success
152693:M 09 Mar 2024 09:49:20.047 * 10 changes in 300 seconds. Saving...
152693:M 09 Mar 2024 09:49:20.136 * Background saving started by pid 538951
538951:C 09 Mar 2024 09:49:45.068 * DB saved on disk
538951:C 09 Mar 2024 09:49:45.139 * RDB: 30 MB of memory used by copy-on-write
152693:M 09 Mar 2024 09:49:45.329 * Background saving terminated with success
152693:M 09 Mar 2024 09:54:46.028 * 10 changes in 300 seconds. Saving...
152693:M 09 Mar 2024 09:54:46.117 * Background saving started by pid 538990
538990:C 09 Mar 2024 09:54:57.495 * DB saved on disk
538990:C 09 Mar 2024 09:54:57.563 * RDB: 11 MB of memory used by copy-on-write
152693:M 09 Mar 2024 09:54:57.653 * Background saving terminated with success
152693:M 09 Mar 2024 09:59:58.075 * 10 changes in 300 seconds. Saving...
152693:M 09 Mar 2024 09:59:58.165 * Background saving started by pid 539030
=== REDIS BUG REPORT START: Cut & paste starting from here ===
539030:C 09 Mar 2024 10:00:00.008 # ------------------------------------------------
539030:C 09 Mar 2024 10:00:00.008 # !!! Software Failure. Press left mouse button to continue
539030:C 09 Mar 2024 10:00:00.008 # Guru Meditation: Unknown object type #rdb.c:669
539030:C 09 Mar 2024 10:00:00.008 # (forcing SIGSEGV in order to print the stack trace)
539030:C 09 Mar 2024 10:00:00.008 # ------------------------------------------------
539030:C 09 Mar 2024 10:00:00.008 # Redis 6.0.16 crashed by signal: 11, si_code: 1
539030:C 09 Mar 2024 10:00:00.008 # Crashed running the instruction at: 0x5590c8cfc46c
539030:C 09 Mar 2024 10:00:00.008 # Accessing address: 0xffffffffffffffff
539030:C 09 Mar 2024 10:00:00.008 # Failed assertion: <no assertion failed> (<no file>:0)
------ STACK TRACE ------
EIP:
redis-rdb-bgsave 127.0.0.1:6379(_serverPanic+0x14c)[0x5590c8cfc46c]
Backtrace:
redis-rdb-bgsave 127.0.0.1:6379(logStackTrace+0x5c)[0x5590c8cfa83c]
redis-rdb-bgsave 127.0.0.1:6379(sigsegvHandler+0xd0)[0x5590c8cfd320]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f21f5a57520]
redis-rdb-bgsave 127.0.0.1:6379(_serverPanic+0x14c)[0x5590c8cfc46c]
redis-rdb-bgsave 127.0.0.1:6379(rdbSaveObjectType+0x1e3)[0x5590c8cc6943]
redis-rdb-bgsave 127.0.0.1:6379(rdbSaveKeyValuePair+0x60)[0x5590c8ccce90]
redis-rdb-bgsave 127.0.0.1:6379(rdbSaveRio+0x25d)[0x5590c8ccd28d]
redis-rdb-bgsave 127.0.0.1:6379(rdbSave+0x117)[0x5590c8ccd8c7]
redis-rdb-bgsave 127.0.0.1:6379(rdbSaveBackground+0xf3)[0x5590c8ccdbe3]
redis-rdb-bgsave 127.0.0.1:6379(serverCron+0x6ff)[0x5590c8c9cd2f]
redis-rdb-bgsave 127.0.0.1:6379(aeProcessEvents+0x477)[0x5590c8c93ca7]
redis-rdb-bgsave 127.0.0.1:6379(aeMain+0x1d)[0x5590c8c93edd]
redis-rdb-bgsave 127.0.0.1:6379(main+0x466)[0x5590c8c8f5f6]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f21f5a3ed90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f21f5a3ee40]
redis-rdb-bgsave 127.0.0.1:6379(_start+0x25)[0x5590c8c8f9d5]
------ INFO OUTPUT ------
# Server
redis_version:6.0.16
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:a3fdef44459b3ad6
redis_mode:standalone
os:Linux 5.15.0-100-generic x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:11.2.0
process_id:539030
run_id:f3c11c529316b49dacfae5dbf2a6f692ce6f06b9
tcp_port:6379
uptime_in_seconds:902776
uptime_in_days:10
hz:10
configured_hz:10
lru_clock:15468782
executable:/usr/bin/redis-server
config_file:/etc/redis/redis.conf
io_threads_active:0
# Clients
connected_clients:0
client_recent_max_input_buffer:40960
client_recent_max_output_buffer:20504
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
# Memory
used_memory:2147145520
used_memory_human:2.00G
used_memory_rss:2349441024
used_memory_rss_human:2.19G
used_memory_peak:2213792816
used_memory_peak_human:2.06G
used_memory_peak_perc:96.99%
used_memory_overhead:33957936
used_memory_startup:809688
used_memory_dataset:2113187584
used_memory_dataset_perc:98.46%
allocator_allocated:2149015288
allocator_active:2310848512
allocator_resident:2349686784
total_system_memory:32519483392
total_system_memory_human:30.29G
used_memory_lua:1177600
used_memory_lua_human:1.12M
used_memory_scripts:232
used_memory_scripts_human:232B
number_of_cached_scripts:1
maxmemory:2147483648
maxmemory_human:2.00G
maxmemory_policy:allkeys-lru
allocator_frag_ratio:1.08
allocator_frag_bytes:161833224
allocator_rss_ratio:1.02
allocator_rss_bytes:38838272
rss_overhead_ratio:1.00
rss_overhead_bytes:-245760
mem_fragmentation_ratio:1.09
mem_fragmentation_bytes:202297056
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:0
mem_aof_buffer:0
mem_allocator:jemalloc-5.2.1
active_defrag_running:0
lazyfree_pending_objects:0
# Persistence
loading:0
rdb_changes_since_last_save:7425
rdb_bgsave_in_progress:0
rdb_last_save_time:1709967297
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:11
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:12197888
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
# Stats
total_connections_received:7642451
total_commands_processed:512781344
instantaneous_ops_per_sec:98
total_net_input_bytes:529670311736
total_net_output_bytes:2727120476161
instantaneous_input_kbps:121.13
instantaneous_output_kbps:660.95
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:599830
expired_stale_perc:0.14
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:131722
evicted_keys:27458547
keyspace_hits:791007429
keyspace_misses:115909222
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:89648
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_reads_processed:439100208
total_writes_processed:422699329
io_threaded_reads_processed:0
io_threaded_writes_processed:0
# Replication
role:master
connected_slaves:0
master_replid:06998fd1f209f0a8d0fe1696f933e16cd2b55619
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:0.112500
used_cpu_user:1.418312
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
# Modules
# Commandstats
cmdstat_ping:calls=1718956,usec=5370834,usec_per_call=3.12
cmdstat_mget:calls=75026149,usec=1594634959,usec_per_call=21.25
cmdstat_zremrangebyscore:calls=251,usec=2820620,usec_per_call=11237.53
cmdstat_get:calls=266235121,usec=1987156090,usec_per_call=7.46
cmdstat_del:calls=750050,usec=1166131572,usec_per_call=1554.74
cmdstat_expire:calls=1298874,usec=16763276,usec_per_call=12.91
cmdstat_info:calls=1718956,usec=130415427,usec_per_call=75.87
cmdstat_setex:calls=1496905,usec=15882371,usec_per_call=10.61
cmdstat_select:calls=4624500,usec=17582859,usec_per_call=3.80
cmdstat_zrangebyscore:calls=510,usec=3340846,usec_per_call=6550.68
cmdstat_zadd:calls=1718935,usec=41542308,usec_per_call=24.17
cmdstat_keys:calls=1,usec=112823,usec_per_call=112823.00
cmdstat_set:calls=158192135,usec=828164722,usec_per_call=5.24
cmdstat_eval:calls=1,usec=174770,usec_per_call=174770.00
# Cluster
cluster_enabled:0
# Keyspace
db0:keys=603425,expires=15011,avg_ttl=8309983
------ CLIENT LIST OUTPUT ------
------ REGISTERS ------
539030:C 09 Mar 2024 10:00:00.012 #
RAX:0000000000000000 RBX:000000000000029d
RCX:0000000000000001 RDX:ffffffffffffe7b8
RDI:00007f21f61dd040 RSI:00007f21f660b938
RBP:00005590c8d819c0 RSP:00007ffce3bdd900
R8 :00007f21f52008c0 R9 :0000000000000200
R10:0000000000000008 R11:00007f21f660d000
R12:00005590c8d796da R13:00007ffce3bdd920
R14:00007f21f50b1000 R15:000000000086967f
RIP:00005590c8cfc46c EFL:0000000000010206
CSGSFS:002b000000000033
539030:C 09 Mar 2024 10:00:00.012 # (00007ffce3bdd90f) -> 00000000000002c0
539030:C 09 Mar 2024 10:00:00.012 # (00007ffce3bdd90e) -> 00007f21ee8d7200
539030:C 09 Mar 2024 10:00:00.012 # (00007ffce3bdd90d) -> 00007f21f5aa0665
539030:C 09 Mar 2024 10:00:00.012 # (00007ffce3bdd90c) -> 0000000000000002
539030:C 09 Mar 2024 10:00:00.012 # (00007ffce3bdd90b) -> 00007f21f5aa19e1
539030:C 09 Mar 2024 10:00:00.012 # (00007ffce3bdd90a) -> 00007f21e7865ba6
539030:C 09 Mar 2024 10:00:00.012 # (00007ffce3bdd909) -> 00007f21f5aa2e6c
539030:C 09 Mar 2024 10:00:00.012 # (00007ffce3bdd908) -> 0000000000001000
539030:C 09 Mar 2024 10:00:00.012 # (00007ffce3bdd907) -> 00007f21ee8d7200
539030:C 09 Mar 2024 10:00:00.012 # (00007ffce3bdd906) -> 0000000000657079
539030:C 09 Mar 2024 10:00:00.012 # (00007ffce3bdd905) -> 74207463656a626f
539030:C 09 Mar 2024 10:00:00.012 # (00007ffce3bdd904) -> 206e776f6e6b6e55
539030:C 09 Mar 2024 10:00:00.012 # (00007ffce3bdd903) -> 0000000000001000
539030:C 09 Mar 2024 10:00:00.012 # (00007ffce3bdd902) -> 00007ffce3bdda30
539030:C 09 Mar 2024 10:00:00.012 # (00007ffce3bdd901) -> 00007ffce3bddb10
539030:C 09 Mar 2024 10:00:00.012 # (00007ffce3bdd900) -> 0000003000000018
------ MODULES INFO OUTPUT ------
------ FAST MEMORY TEST ------
539030:C 09 Mar 2024 10:00:00.012 # Bio thread for job type #0 terminated
152693:M 09 Mar 2024 10:01:29.649 # Background saving terminated by signal 11
152693:M 09 Mar 2024 10:01:29.749 * 10 changes in 300 seconds. Saving...
152693:M 09 Mar 2024 10:01:29.836 * Background saving started by pid 539081
=== REDIS BUG REPORT START: Cut & paste starting from here ===
539081:C 09 Mar 2024 10:01:32.174 # ------------------------------------------------
539081:C 09 Mar 2024 10:01:32.174 # !!! Software Failure. Press left mouse button to continue
539081:C 09 Mar 2024 10:01:32.174 # Guru Meditation: Unknown object type #rdb.c:669
539081:C 09 Mar 2024 10:01:32.174 # (forcing SIGSEGV in order to print the stack trace)
539081:C 09 Mar 2024 10:01:32.174 # ------------------------------------------------
539081:C 09 Mar 2024 10:01:32.174 # Redis 6.0.16 crashed by signal: 11, si_code: 1
539081:C 09 Mar 2024 10:01:32.174 # Crashed running the instruction at: 0x5590c8cfc46c
539081:C 09 Mar 2024 10:01:32.174 # Accessing address: 0xffffffffffffffff
539081:C 09 Mar 2024 10:01:32.174 # Failed assertion: <no assertion failed> (<no file>:0)
------ STACK TRACE ------
EIP:
redis-rdb-bgsave 127.0.0.1:6379(_serverPanic+0x14c)[0x5590c8cfc46c]
Backtrace:
redis-rdb-bgsave 127.0.0.1:6379(logStackTrace+0x5c)[0x5590c8cfa83c]
redis-rdb-bgsave 127.0.0.1:6379(sigsegvHandler+0xd0)[0x5590c8cfd320]
/lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f21f5a57520]
redis-rdb-bgsave 127.0.0.1:6379(_serverPanic+0x14c)[0x5590c8cfc46c]
redis-rdb-bgsave 127.0.0.1:6379(rdbSaveObjectType+0x1e3)[0x5590c8cc6943]
redis-rdb-bgsave 127.0.0.1:6379(rdbSaveKeyValuePair+0x60)[0x5590c8ccce90]
redis-rdb-bgsave 127.0.0.1:6379(rdbSaveRio+0x25d)[0x5590c8ccd28d]
redis-rdb-bgsave 127.0.0.1:6379(rdbSave+0x117)[0x5590c8ccd8c7]
redis-rdb-bgsave 127.0.0.1:6379(rdbSaveBackground+0xf3)[0x5590c8ccdbe3]
redis-rdb-bgsave 127.0.0.1:6379(serverCron+0x6ff)[0x5590c8c9cd2f]
redis-rdb-bgsave 127.0.0.1:6379(aeProcessEvents+0x477)[0x5590c8c93ca7]
redis-rdb-bgsave 127.0.0.1:6379(aeMain+0x1d)[0x5590c8c93edd]
redis-rdb-bgsave 127.0.0.1:6379(main+0x466)[0x5590c8c8f5f6]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f21f5a3ed90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f21f5a3ee40]
redis-rdb-bgsave 127.0.0.1:6379(_start+0x25)[0x5590c8c8f9d5]
------ INFO OUTPUT ------
# Server
redis_version:6.0.16
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:a3fdef44459b3ad6
redis_mode:standalone
os:Linux 5.15.0-100-generic x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:11.2.0
process_id:539081
run_id:f3c11c529316b49dacfae5dbf2a6f692ce6f06b9
tcp_port:6379
uptime_in_seconds:902867
uptime_in_days:10
hz:10
configured_hz:10
lru_clock:15468873
executable:/usr/bin/redis-server
config_file:/etc/redis/redis.conf
io_threads_active:0
# Clients
connected_clients:2
client_recent_max_input_buffer:917504
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
# Memory
used_memory:2144686192
used_memory_human:2.00G
used_memory_rss:2349576192
used_memory_rss_human:2.19G
used_memory_peak:2213792816
used_memory_peak_human:2.06G
used_memory_peak_perc:96.88%
used_memory_overhead:33961320
used_memory_startup:809688
used_memory_dataset:2110724872
used_memory_dataset_perc:98.45%
allocator_allocated:2146554184
allocator_active:2311208960
allocator_resident:2349600768
total_system_memory:32519483392
total_system_memory_human:30.29G
used_memory_lua:1177600
used_memory_lua_human:1.12M
used_memory_scripts:232
used_memory_scripts_human:232B
number_of_cached_scripts:1
maxmemory:2147483648
maxmemory_human:2.00G
maxmemory_policy:allkeys-lru
allocator_frag_ratio:1.08
allocator_frag_bytes:164654776
allocator_rss_ratio:1.02
allocator_rss_bytes:38391808
rss_overhead_ratio:1.00
rss_overhead_bytes:-24576
mem_fragmentation_ratio:1.10
mem_fragmentation_bytes:204850600
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:40992
mem_aof_buffer:0
mem_allocator:jemalloc-5.2.1
active_defrag_running:0
lazyfree_pending_objects:0
# Persistence
loading:0
rdb_changes_since_last_save:9279
rdb_bgsave_in_progress:0
rdb_last_save_time:1709967297
rdb_last_bgsave_status:err
rdb_last_bgsave_time_sec:91
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:12197888
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
# Stats
total_connections_received:7643019
total_commands_processed:512816665
instantaneous_ops_per_sec:582
total_net_input_bytes:529706875997
total_net_output_bytes:2727312640497
instantaneous_input_kbps:512.04
instantaneous_output_kbps:1899.72
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:599891
expired_stale_perc:0.77
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:131736
evicted_keys:27461006
keyspace_hits:791061210
keyspace_misses:115916979
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:89332
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_reads_processed:439130686
total_writes_processed:422728467
io_threaded_reads_processed:0
io_threaded_writes_processed:0
# Replication
role:master
connected_slaves:0
master_replid:06998fd1f209f0a8d0fe1696f933e16cd2b55619
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:0.088005
used_cpu_user:1.388093
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
# Modules
# Commandstats
cmdstat_ping:calls=1719081,usec=5371217,usec_per_call=3.12
cmdstat_mget:calls=75031521,usec=1594741803,usec_per_call=21.25
cmdstat_zremrangebyscore:calls=251,usec=2820620,usec_per_call=11237.53
cmdstat_get:calls=266253659,usec=1987299375,usec_per_call=7.46
cmdstat_del:calls=750053,usec=1166131591,usec_per_call=1554.73
cmdstat_expire:calls=1298958,usec=16764642,usec_per_call=12.91
cmdstat_info:calls=1719081,usec=130424804,usec_per_call=75.87
cmdstat_setex:calls=1496974,usec=15883160,usec_per_call=10.61
cmdstat_select:calls=4624859,usec=17584307,usec_per_call=3.80
cmdstat_zrangebyscore:calls=510,usec=3340846,usec_per_call=6550.68
cmdstat_zadd:calls=1719059,usec=41546349,usec_per_call=24.17
cmdstat_keys:calls=1,usec=112823,usec_per_call=112823.00
cmdstat_set:calls=158202657,usec=828225896,usec_per_call=5.24
cmdstat_eval:calls=1,usec=174770,usec_per_call=174770.00
# Cluster
cluster_enabled:0
# Keyspace
db0:keys=602480,expires=15019,avg_ttl=8353182
------ CLIENT LIST OUTPUT ------
id=7643021 addr=127.0.0.1:46604 fd=10 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default
id=7643022 addr=127.0.0.1:46608 fd=8 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20488 events=r cmd=NULL user=default
------ REGISTERS ------
539081:C 09 Mar 2024 10:01:32.175 #
RAX:0000000000000000 RBX:000000000000029d
RCX:0000000000000001 RDX:ffffffffffffe7b8
RDI:00007f21f61dd040 RSI:00007f21f660b938
RBP:00005590c8d819c0 RSP:00007ffce3bdd900
R8 :00007f21f52008c0 R9 :0000000000000200
R10:0000000000000008 R11:00007f21f660d000
R12:00005590c8d796da R13:00007ffce3bdd920
R14:00007f21f50b1000 R15:00000000008696da
RIP:00005590c8cfc46c EFL:0000000000010206
CSGSFS:002b000000000033
539081:C 09 Mar 2024 10:01:32.175 # (00007ffce3bdd90f) -> 00000000000000f9
539081:C 09 Mar 2024 10:01:32.175 # (00007ffce3bdd90e) -> 00007f21ed2b4000
539081:C 09 Mar 2024 10:01:32.175 # (00007ffce3bdd90d) -> 00007f21f5aa0665
539081:C 09 Mar 2024 10:01:32.175 # (00007ffce3bdd90c) -> 00007ffce3bdd9ae
539081:C 09 Mar 2024 10:01:32.175 # (00007ffce3bdd90b) -> 00007f21f5aa19e1
539081:C 09 Mar 2024 10:01:32.175 # (00007ffce3bdd90a) -> 00007f21d739f436
539081:C 09 Mar 2024 10:01:32.175 # (00007ffce3bdd909) -> 00007f21f5aa2e6c
539081:C 09 Mar 2024 10:01:32.175 # (00007ffce3bdd908) -> 0000000000001000
539081:C 09 Mar 2024 10:01:32.175 # (00007ffce3bdd907) -> 00007f21ed2b4000
539081:C 09 Mar 2024 10:01:32.175 # (00007ffce3bdd906) -> 0000000000657079
539081:C 09 Mar 2024 10:01:32.175 # (00007ffce3bdd905) -> 74207463656a626f
539081:C 09 Mar 2024 10:01:32.175 # (00007ffce3bdd904) -> 206e776f6e6b6e55
539081:C 09 Mar 2024 10:01:32.175 # (00007ffce3bdd903) -> 0000000000001000
539081:C 09 Mar 2024 10:01:32.175 # (00007ffce3bdd902) -> 00007ffce3bdda30
539081:C 09 Mar 2024 10:01:32.175 # (00007ffce3bdd901) -> 00007ffce3bddb10
539081:C 09 Mar 2024 10:01:32.175 # (00007ffce3bdd900) -> 0000003000000018
------ MODULES INFO OUTPUT ------
------ FAST MEMORY TEST ------
539081:C 09 Mar 2024 10:01:32.175 # Bio thread for job type #0 terminated
152693:M 09 Mar 2024 10:01:32.446 # Background saving terminated by signal 11
152693:M 09 Mar 2024 10:01:35.061 * 10 changes in 300 seconds. Saving...
152693:M 09 Mar 2024 10:01:35.150 * Background saving started by pid 539083
=== REDIS BUG REPORT START: Cut & paste starting from here ===
539083:C 09 Mar 2024 10:01:36.804 # ------------------------------------------------
Additional information
-
OS distribution and version Ubuntu 22.04.4 LTS
-
Steps to reproduce (if any)
The setup is the following:
Wordpress site with plugins redis-object-cache (for caching DB queries), nginx-helper (for clearing page cache on page edit)
nginx with sr-cache module, MariaDB, redis
Comment From: DeeKey
Also noticed the following errors:
TP/1.1", upstream: "fastcgi://unix:/run/php/php8.1-fpm.sock:", host: "www.hidden" 2024/03/09 11:07:43 [error] 530938#530938: *8978459 FastCGI sent in stderr: "PHP message: Predis\Response\ServerException: MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk. Commands that may modify the data set are disabled, because this instance is configured to report errors during writes if RDB snapshotting fails (stop-writes-on-bgsave-error option). Please check the Redis logs for details about the RDB error. in /hidden/wp-content/plugins/redis-cache/dependencies/predis/predis/src/Client.php:384
syslog
Mar 9 10:00:00 server kernel: [1247337.804079] redis-server[539030]: segfault at 7f21f4115910 ip 00007f21f5aa7bdd sp 00007ffce3bdd140 error 4 in libc.so.6[7f21f5a3d000+195000] Mar 9 10:00:00 server kernel: [1247337.804374] Code: 08 5b 5d c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 55 48 81 ec a0 00 00 00 64 48 8b 04 25 28 00 00 00 48 89 84 24 98 00 00 00 <8b> 87 d0 02 00 00 85 c0 75 29 48 8b 84 24 98 00 00 00 64 48 2b 04
Comment From: sundb
@DeeKey seems that the rdb is corrupted, can you use redis-check-rdb to check it?
Comment From: DeeKey
There is definitely something strange happening in the system.
When doing backup with rsync had a bunch of errors like:
rsync: [sender] readlink_stat("/home/768x768.jpg") failed: Bad message (74)
Upon examination turns out that it was not possible to read or delete these files. After reboot these files were accessible without any problems. Also few times had problems with dadlock processes. The process was hanging marked with D state. It was not possible to kill the process. Full RAM and disk tests showed no anomaly.
Currently there are two files:
/var/lib/redis# ls -la
total 810632
drwxr-x--- 2 redis redis 4096 Mar 12 13:35 .
drwxr-xr-x 54 root root 4096 May 31 2023 ..
-rw-rw---- 1 redis redis 726848361 Mar 12 13:35 dump.rdb
-rw-rw---- 1 redis redis 103223282 Mar 9 11:09 temp-152693.rdb
# redis-check-rdb temp-152693.rdb
[offset 0] Checking RDB file temp-152693.rdb
[offset 27] AUX FIELD redis-ver = '6.0.16'
[offset 41] AUX FIELD redis-bits = '64'
[offset 53] AUX FIELD ctime = '1709971786'
[offset 68] AUX FIELD used-mem = '1884522816'
[offset 84] AUX FIELD aof-preamble = '0'
[offset 86] Selecting DB ID 0
--- RDB ERROR DETECTED ---
[offset 103223244] Unexpected EOF reading RDB file
[additional info] While doing: read-object-value
[additional info] Reading key 'asi:terms:11348'
[additional info] Reading type 0 (string)
[info] 98014 keys read
[info] 1897 expires
[info] 1897 already expired
# redis-check-rdb dump.rdb
[offset 0] Checking RDB file dump.rdb
[offset 27] AUX FIELD redis-ver = '6.0.16'
[offset 41] AUX FIELD redis-bits = '64'
[offset 53] AUX FIELD ctime = '1710240014'
[offset 68] AUX FIELD used-mem = '2146421192'
[offset 84] AUX FIELD aof-preamble = '0'
[offset 86] Selecting DB ID 0
[offset 728165086] Checksum OK
[offset 728165086] \o/ RDB looks OK! \o/
[info] 640519 keys read
[info] 11447 expires
[info] 140 already expired
Comment From: sundb
I'm wondering why temp-152693.rdb and crash time don't match.
did you restart the server after crash and them everything works?
could you check if asi:terms:11348 exist and its time is stirng?
Comment From: DeeKey
did you restart the server after crash and them everything works? Yes - I restarted redis-server right after the crush. It was successful. After I made apt upgrade and rebooted the server (as new Linux kernel requested this).
could you check if asi:terms:11348 exist and its time is stirng?
How can I do this?
Comment From: sundb
object encoding asi:terms:11348, maybe other key like asi:terms:xxxx.
Comment From: DeeKey
redis-cli --scan --pattern 'asi:terms:11348'
asi:terms:11348
Comment From: sundb
@DeeKey can you give the output of object encoding asi:terms:11348?
Comment From: DeeKey
Could you please provide exact command how to do this?
Comment From: sundb
./src/redis-cli object encoding asi:terms:11348.
Comment From: DeeKey
# redis-cli object encoding asi:terms:11348
"raw"
Comment From: DeeKey
Just in case it might be useful for this case. As I mentioned above I use two Wodpress plugins with Redis. Both are opensourced:
https://wordpress.org/plugins/redis-cache/ https://github.com/rtCamp/nginx-helper
Could they cause some sort of collisions?
Comment From: sundb
Just in case it might be useful for this case. As I mentioned above I use two Wodpress plugins with Redis. Both are opensourced:
wordpress.org/plugins/redis-cache rtCamp/nginx-helper
Could they cause some sort of collisions?
I don't think it has anything to do with these two plugins. I'm not sure if it's related to the content of this key. If possible, could you provide the content of this key? If there are privacy issues please send me email `debing.sun@redis.com', thanks. If you could provide the corrupted rdb file even better.
Comment From: DeeKey
I have sent you corrupted file.
Comment From: sundb
@DeeKey thanks, the value of asi:terms:11348 doesn't appear to be corrupt, it just wasn't written in its entirety due to the crash, and I still can't find any valid clues from this corrupted rdb, so perhaps we can wait to see if the next crash will provide more information.