=== REDIS BUG REPORT START: Cut & paste starting from here === 1:S 18 Aug 2022 08:06:55.099 # ------------------------------------------------ 1:S 18 Aug 2022 08:06:55.099 # !!! Software Failure. Press left mouse button to continue 1:S 18 Aug 2022 08:06:55.099 # Guru Meditation: Duplicated key found in RDB file #rdb.c:2366 1:S 18 Aug 2022 08:06:55.099 # (forcing SIGSEGV in order to print the stack trace) 1:S 18 Aug 2022 08:06:55.099 # ------------------------------------------------ 1:S 18 Aug 2022 08:06:55.099 # Redis 6.0.16 crashed by signal: 11, si_code: 1 1:S 18 Aug 2022 08:06:55.099 # Crashed running the instruction at: 0x471492 1:S 18 Aug 2022 08:06:55.099 # Accessing address: 0xffffffffffffffff 1:S 18 Aug 2022 08:06:55.099 # Failed assertion: (:0)

------ STACK TRACE ------ EIP: redis-server *:6379(_serverPanic+0x112)[0x471492]

Backtrace: redis-server :6379(logStackTrace+0x2e)[0x47370e] redis-server :6379(sigsegvHandler+0xa8)[0x473db8] /lib64/libpthread.so.0(+0x12c20)[0x7f8dca255c20] redis-server :6379(_serverPanic+0x112)[0x471492] redis-server :6379(rdbLoadRio+0xba4)[0x457154] redis-server :6379(rdbLoad+0x4c)[0x4571ac] redis-server :6379(loadDataFromDisk+0x8c)[0x434d3c] redis-server :6379(main+0x3bf)[0x42685f] /lib64/libc.so.6(__libc_start_main+0xf3)[0x7f8dc9ea1493] redis-server :6379(_start+0x2e)[0x426c0e]

------ INFO OUTPUT ------

Server

redis_version:6.0.16 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:621e196d44a062ac redis_mode:standalone os:Linux 4.18.0-193.60.2.el8_2.x86_64 x86_64 arch_bits:64 multiplexing_api:epoll atomicvar_api:atomic-builtin gcc_version:8.5.0 process_id:1 run_id:8b2cc7de73df7a0f1205812a5fe1f2dd0a294eba tcp_port:6379 uptime_in_seconds:0 uptime_in_days:0 hz:10 configured_hz:10 lru_clock:16642847 executable:/data/redis-server config_file:/data/conf/redis.conf io_threads_active:0

Clients

connected_clients:0 client_recent_max_input_buffer:0 client_recent_max_output_buffer:0 blocked_clients:0 tracking_clients:0 clients_in_timeout_table:0

Memory

used_memory:13131016 used_memory_human:12.52M used_memory_rss:0 used_memory_rss_human:0B used_memory_peak:14584024 used_memory_peak_human:13.91M used_memory_peak_perc:90.04% used_memory_overhead:2903120 used_memory_startup:876360 used_memory_dataset:10227896 used_memory_dataset_perc:83.46% allocator_allocated:0 allocator_active:0 allocator_resident:0 total_system_memory:135165513728 total_system_memory_human:125.88G used_memory_lua:37888 used_memory_lua_human:37.00K used_memory_scripts:0 used_memory_scripts_human:0B number_of_cached_scripts:0 maxmemory:0 maxmemory_human:0B maxmemory_policy:volatile-lru allocator_frag_ratio:-nan allocator_frag_bytes:0 allocator_rss_ratio:-nan allocator_rss_bytes:0 rss_overhead_ratio:-nan rss_overhead_bytes:0 mem_fragmentation_ratio:-nan mem_fragmentation_bytes:0 mem_not_counted_for_evict:0 mem_replication_backlog:0 mem_clients_slaves:0 mem_clients_normal:0 mem_aof_buffer:0 mem_allocator:libc active_defrag_running:0 lazyfree_pending_objects:0

Persistence

loading:1 rdb_changes_since_last_save:0 rdb_bgsave_in_progress:0 rdb_last_save_time:1660810015 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:-1 rdb_current_bgsave_time_sec:-1 rdb_last_cow_size:0 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok aof_last_cow_size:0 module_fork_in_progress:0 module_fork_last_cow_size:0 loading_start_time:1660810015 loading_total_bytes:9560408 loading_loaded_bytes:7149229 loading_loaded_perc:74.78 loading_eta_seconds:1

Stats

total_connections_received:0 total_commands_processed:0 instantaneous_ops_per_sec:0 total_net_input_bytes:0 total_net_output_bytes:0 instantaneous_input_kbps:0.00 instantaneous_output_kbps:0.00 rejected_connections:0 sync_full:0 sync_partial_ok:0 sync_partial_err:0 expired_keys:0 expired_stale_perc:0.00 expired_time_cap_reached_count:0 expire_cycle_cpu_milliseconds:0 evicted_keys:0 keyspace_hits:0 keyspace_misses:0 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:0 migrate_cached_sockets:0 slave_expires_tracked_keys:0 active_defrag_hits:0 active_defrag_misses:0 active_defrag_key_hits:0 active_defrag_key_misses:0 tracking_total_keys:0 tracking_total_items:0 tracking_total_prefixes:0 unexpected_error_replies:0 total_reads_processed:0 total_writes_processed:0 io_threaded_reads_processed:0 io_threaded_writes_processed:0

Replication

role:slave master_host:172.30.124.120 master_port:6380 master_link_status:down master_last_io_seconds_ago:-1 master_sync_in_progress:0 slave_repl_offset:1 master_link_down_since_seconds:1660810015 slave_priority:100 slave_read_only:1 connected_slaves:0 min_slaves_good_slaves:0 master_replid:34594e93f25c248e35ae8c663129156eafaf89f0 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:0 second_repl_offset:-1 repl_backlog_active:0 repl_backlog_size:1048576 repl_backlog_first_byte_offset:0 repl_backlog_histlen:0

CPU

used_cpu_sys:0.043832 used_cpu_user:0.066480 used_cpu_sys_children:0.000000 used_cpu_user_children:0.000000

Modules

Commandstats

Cluster

cluster_enabled:0

Keyspace

db0:keys=23478,expires=23473,avg_ttl=0

------ CLIENT LIST OUTPUT ------

------ REGISTERS ------ 1:S 18 Aug 2022 08:06:55.100 # RAX:0000000000000000 RBX:00000000004eee93 RCX:00007f8dc9f6b67f RDX:0000000000000000 RDI:0000000000000000 RSI:00007f8dca23f860 RBP:000000000000093e RSP:00007fffa0fb8a90 R8 :0000000000000002 R9 :00007f8dcb592b80 R10:0000000000000000 R11:0000000000000246 R12:00007fffa0fb9150 R13:00000000028d07a3 R14:00007fffa0fb8ce0 R15:00000000028d07c0 RIP:0000000000471492 EFL:0000000000010202 CSGSFS:002b000000000033 1:S 18 Aug 2022 08:06:55.100 # (00007fffa0fb8a9f) -> 0000000000000020 1:S 18 Aug 2022 08:06:55.100 # (00007fffa0fb8a9e) -> 0000000001fcf010 1:S 18 Aug 2022 08:06:55.100 # (00007fffa0fb8a9d) -> 0000000000000007 1:S 18 Aug 2022 08:06:55.100 # (00007fffa0fb8a9c) -> 0000000000000000 1:S 18 Aug 2022 08:06:55.100 # (00007fffa0fb8a9b) -> 0000000000000007 1:S 18 Aug 2022 08:06:55.100 # (00007fffa0fb8a9a) -> 0000000000000000 1:S 18 Aug 2022 08:06:55.100 # (00007fffa0fb8a99) -> 0000000000000007 1:S 18 Aug 2022 08:06:55.100 # (00007fffa0fb8a98) -> 0000000000000000 1:S 18 Aug 2022 08:06:55.100 # (00007fffa0fb8a97) -> 656c696620424452 1:S 18 Aug 2022 08:06:55.100 # (00007fffa0fb8a96) -> 206e6920646e756f 1:S 18 Aug 2022 08:06:55.100 # (00007fffa0fb8a95) -> 662079656b206465 1:S 18 Aug 2022 08:06:55.100 # (00007fffa0fb8a94) -> 746163696c707544 1:S 18 Aug 2022 08:06:55.100 # (00007fffa0fb8a93) -> 00007fffa0fb8bb0 1:S 18 Aug 2022 08:06:55.100 # (00007fffa0fb8a92) -> 00007fffa0fb8c80 1:S 18 Aug 2022 08:06:55.100 # (00007fffa0fb8a91) -> 0000003000000018 1:S 18 Aug 2022 08:06:55.100 # (00007fffa0fb8a90) -> 0000000000000000

------ MODULES INFO OUTPUT ------

------ FAST MEMORY TEST ------ 1:S 18 Aug 2022 08:06:55.100 # Bio thread for job type #0 terminated 1:S 18 Aug 2022 08:06:55.100 # Bio thread for job type #1 terminated 1:S 18 Aug 2022 08:06:55.100 # Bio thread for job type #2 terminated *** Preparing to test memory region 740000 (122880 bytes) *** Preparing to test memory region 1fcf000 (11030528 bytes) *** Preparing to test memory region 7f8dc7f55000 (3842048 bytes) *** Preparing to test memory region 7f8dc8464000 (8388608 bytes) *** Preparing to test memory region 7f8dc8c65000 (8388608 bytes) *** Preparing to test memory region 7f8dc9466000 (8388608 bytes) *** Preparing to test memory region 7f8dc9e7d000 (4096 bytes) *** Preparing to test memory region 7f8dca23f000 (16384 bytes) *** Preparing to test memory region 7f8dca45f000 (16384 bytes) *** Preparing to test memory region 7f8dca947000 (20480 bytes) *** Preparing to test memory region 7f8dcabdf000 (4096 bytes) *** Preparing to test memory region 7f8dcb445000 (364544 bytes) *** Preparing to test memory region 7f8dcb4c0000 (884736 bytes) .O.O.O.O.O.O.O.O.O.O.O.O.O Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------ Symbol: _serverPanic (base: 0x471380) Module: redis-server *:6379 (base 0x400000) $ xxd -r -p /tmp/dump.hex /tmp/dump.bin $ objdump --adjust-vma=0x471380 -D -b binary -m i386:x86-64 /tmp/dump.bin

1:S 18 Aug 2022 08:06:55.208 # dump of function (hexdump of 402 bytes): 5589f5534889fb4881ecd801000048898c24380100004c898424400100004c898c244801000084c074400f298424500100000f298c24600100000f299424700100000f299c24800100000f29a424900100000f29ac24a00100000f29b424b00100000f29bc24c0010000488d4c2408be00010000488d8424f0010000c7442408180000004889442410488d7c2420488d842420010000c744240c300000004889442418e82845fbff8b055e5c2d0085c07476be983a4f00bf0300000031c0e85dc3fbffbed03a4f00bf0300000031c0e84cc3fbff4189e84889d9488d542420bef1524f00bf0300000031c0e830c3fbffbe103b4f00bf0300000031c0e81fc3fbffbe983a4f00bf0300000031c0e80ec3fbffc60425ffffffff784881c4d80100005b5dc30f1f4000e843f8ffffeb83904157415641554989cd41544989f4554889fd534889d3ba040000004889df4881ece80200000fb601488d74241483e00f0fc889442414e825f9ffff4c89e64889efe8ba81fdff4989c4410fb6450089c283e20f0f848f00000080fa010f84ce020000 Function at 0x42d7a0 is serverLog Function at 0x470e10 is mixDigest Function at 0x4496b0 is getExpire

=== REDIS BUG REPORT END. Make sure to include from START to END. ===

Comment From: ybjwang

redis-ha-server PVs have full, but after we extend it , still same behavior. Any suggestions, please?

Comment From: ybjwang

@sundb could you please share any suggestions please? Thanks!

Comment From: sundb

@ybjwang I'm not sure why the rdb file is corrupt when pvs is full, I may need to spend some time digging into this code.

Comment From: ybjwang

Thank you so much @sundb Is there a way to recover the PV from redis-ha-server-0? Since we have redis-ha-server-0 is 2/2 running, redis-ha-server-1 is 1/2 running.

Comment From: sundb

Maybe you can use redis-check-rdb to verify that the rdb file is working, and if it is, maybe can use some hacks to fix it.

Comment From: ybjwang

We recovered redis-ha-server pods. No help needed here. Thanks! @sundb