Crash report

Paste the complete crash log between the quotes below. Please include a few lines from the log preceding the crash report to provide some context.

3842:S 18 Feb 2021 22:14:05.199 # Server initialized
3842:S 18 Feb 2021 22:14:05.199 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
3842:S 18 Feb 2021 22:14:05.200 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
3842:S 18 Feb 2021 22:14:05.200 * Ready to accept connections
3842:S 18 Feb 2021 22:14:05.200 * Connecting to MASTER 192.168.178.109:6379
3842:S 18 Feb 2021 22:14:05.200 * MASTER <-> REPLICA sync started
3842:S 18 Feb 2021 22:14:05.201 * Non blocking connect for SYNC fired the event.
3842:S 18 Feb 2021 22:14:05.201 * Master replied to PING, replication can continue...
3842:S 18 Feb 2021 22:14:05.202 * Partial resynchronization not possible (no cached master)
3842:S 18 Feb 2021 22:14:05.209 * Full resync from master: 37e97b05f99ba4da875fda3599339936b8e42491:9869676363
3842:S 18 Feb 2021 22:14:21.253 * MASTER <-> REPLICA sync: receiving 226130426 bytes from master to disk
3842:S 18 Feb 2021 22:14:26.599 * MASTER <-> REPLICA sync: Flushing old data
3842:S 18 Feb 2021 22:14:27.209 * MASTER <-> REPLICA sync: Loading DB in memory
3842:S 18 Feb 2021 22:14:27.210 * Loading RDB produced by version 6.0.6
3842:S 18 Feb 2021 22:14:27.210 * RDB age 22 seconds
3842:S 18 Feb 2021 22:14:27.210 * RDB memory usage when created 491.54 Mb
3842:S 18 Feb 2021 22:14:27.831 # RDB has duplicated key '' in DB 0


=== REDIS BUG REPORT START: Cut & paste starting from here ===
3842:S 18 Feb 2021 22:14:27.831 # ------------------------------------------------
3842:S 18 Feb 2021 22:14:27.831 # !!! Software Failure. Press left mouse button to continue
3842:S 18 Feb 2021 22:14:27.831 # Guru Meditation: Duplicated key found in RDB file #rdb.c:2290
3842:S 18 Feb 2021 22:14:27.831 # (forcing SIGSEGV in order to print the stack trace)
3842:S 18 Feb 2021 22:14:27.831 # ------------------------------------------------
3842:S 18 Feb 2021 22:14:27.831 # Redis 6.0.6 crashed by signal: 11
3842:S 18 Feb 2021 22:14:27.831 # Crashed running the instruction at: 0x562651e3d338
3842:S 18 Feb 2021 22:14:27.831 # Accessing address: 0xffffffffffffffff
3842:S 18 Feb 2021 22:14:27.831 # Failed assertion: <no assertion failed> (<no file>:0)

------ STACK TRACE ------
EIP:
/usr/bin/redis-server 0.0.0.0:6379(_serverPanic+0x148)[0x562651e3d338]

Backtrace:
/usr/bin/redis-server 0.0.0.0:6379(logStackTrace+0x4f)[0x562651e3fbcf]
/usr/bin/redis-server 0.0.0.0:6379(sigsegvHandler+0xb5)[0x562651e403d5]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7fe77e61b3c0]
/usr/bin/redis-server 0.0.0.0:6379(_serverPanic+0x148)[0x562651e3d338]
/usr/bin/redis-server 0.0.0.0:6379(rdbLoadRio+0xc31)[0x562651e21211]
/usr/bin/redis-server 0.0.0.0:6379(rdbLoad+0x6d)[0x562651e212bd]
/usr/bin/redis-server 0.0.0.0:6379(readSyncBulkPayload+0x263)[0x562651e177e3]
/usr/bin/redis-server 0.0.0.0:6379(+0xe10ac)[0x562651e900ac]
/usr/bin/redis-server 0.0.0.0:6379(aeProcessEvents+0x303)[0x562651df0b83]
/usr/bin/redis-server 0.0.0.0:6379(aeMain+0x1d)[0x562651df0ebd]
/usr/bin/redis-server 0.0.0.0:6379(main+0x4e5)[0x562651ded3d5]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7fe77e43b0b3]
/usr/bin/redis-server 0.0.0.0:6379(_start+0x2e)[0x562651ded6ae]

------ INFO OUTPUT ------
# Server
redis_version:6.0.6
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:d463d609620685c0
redis_mode:standalone
os:Linux 5.4.78-2-pve x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:9.3.0
process_id:3842
run_id:6e1d50cc0be296357ed09f773984405b86aaaeaa
tcp_port:6379
uptime_in_seconds:22
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:3070130
executable:/usr/bin/redis-server
config_file:/etc/redis/redis-objects.conf

# Clients
connected_clients:14
client_recent_max_input_buffer:2
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0

# Memory
used_memory:271984416
used_memory_human:259.38M
used_memory_rss:5328896
used_memory_rss_human:5.08M
used_memory_peak:275485952
used_memory_peak_human:262.72M
used_memory_peak_perc:98.73%
used_memory_overhead:1946332
used_memory_startup:524648
used_memory_dataset:270038084
used_memory_dataset_perc:99.48%
allocator_allocated:1306296
allocator_active:1720320
allocator_resident:4538368
total_system_memory:16653746176
total_system_memory_human:15.51G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.32
allocator_frag_bytes:414024
allocator_rss_ratio:2.64
allocator_rss_bytes:2818048
rss_overhead_ratio:1.17
rss_overhead_bytes:790528
mem_fragmentation_ratio:5.20
mem_fragmentation_bytes:4304328
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:237804
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0

# Persistence
loading:1
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1613682845
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
aof_current_size:0
aof_base_size:0
aof_pending_rewrite:0
aof_buffer_length:0
aof_rewrite_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:0
loading_start_time:1613682867
loading_total_bytes:226130426
loading_loaded_bytes:116687810
loading_loaded_perc:51.60
loading_eta_seconds:1

# Stats
total_connections_received:14
total_commands_processed:453
instantaneous_ops_per_sec:18
total_net_input_bytes:226153939
total_net_output_bytes:825141
instantaneous_input_kbps:40822.89
instantaneous_output_kbps:45.79
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:1
pubsub_patterns:0
latest_fork_usec:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0

# Replication
role:slave
master_host:192.168.178.109
master_port:6379
master_link_status:down
master_last_io_seconds_ago:-1
master_sync_in_progress:1
slave_repl_offset:1
master_sync_left_bytes:0
master_sync_last_io_seconds_ago:1
master_link_down_since_seconds:1613682867
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:035d87a2c71af868b4b70df3feb3ed095a74ec54
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

# CPU
used_cpu_sys:0.538881
used_cpu_user:0.567898
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000

# Modules

# Commandstats
cmdstat_publish:calls=116,usec=529,usec_per_call=4.56
cmdstat_client:calls=14,usec=14,usec_per_call=1.00
cmdstat_subscribe:calls=7,usec=8,usec_per_call=1.14
cmdstat_ping:calls=140,usec=34,usec_per_call=0.24
cmdstat_info:calls=176,usec=7706,usec_per_call=43.78

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=16489,expires=0,avg_ttl=0

------ CLIENT LIST OUTPUT ------
id=13 addr=192.168.178.110:35977 fd=20 name=sentinel-9339e577-pubsub age=21 idle=0 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe user=default
id=14 addr=192.168.178.109:33203 fd=21 name=sentinel-9339e577-cmd age=21 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping user=default
id=15 addr=192.168.178.109:40435 fd=22 name=sentinel-9339e577-pubsub age=21 idle=0 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe user=default
id=4 addr=192.168.178.114:57271 fd=11 name=sentinel-9339e577-cmd age=21 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=info user=default
id=5 addr=192.168.178.114:34625 fd=12 name=sentinel-9339e577-pubsub age=21 idle=0 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe user=default
id=6 addr=192.168.178.111:39355 fd=13 name=sentinel-9339e577-cmd age=21 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=publish user=default
id=7 addr=192.168.178.111:34249 fd=14 name=sentinel-9339e577-pubsub age=21 idle=0 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe user=default
id=8 addr=192.168.178.108:60405 fd=15 name=sentinel-9339e577-cmd age=21 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping user=default
id=9 addr=192.168.178.108:48153 fd=16 name=sentinel-9339e577-pubsub age=21 idle=0 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe user=default
id=10 addr=192.168.178.113:46779 fd=17 name=sentinel-9339e577-cmd age=21 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping user=default
id=16 addr=192.168.178.112:46245 fd=23 name=sentinel-9339e577-cmd age=21 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping user=default
id=17 addr=192.168.178.112:54895 fd=24 name=sentinel-9339e577-pubsub age=21 idle=0 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe user=default
id=11 addr=192.168.178.113:56223 fd=18 name=sentinel-9339e577-pubsub age=21 idle=0 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe user=default
id=12 addr=192.168.178.110:48399 fd=19 name=sentinel-9339e577-cmd age=21 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping user=default

------ REGISTERS ------
3842:S 18 Feb 2021 22:14:27.844 # 
RAX:0000000000000000 RBX:00000000000008f2
RCX:00007fe77e5ffbb8 RDX:0000000000000000
RDI:0000562652d6c01a RSI:0000000000000001
RBP:ffffffffffffffff RSP:00007ffdab55f6c0
R8 :0000000000000007 R9 :0000000000000000
R10:0000000000004000 R11:0000000000000000
R12:0000562651f2f44b R13:00007ffdab55f6e0
R14:00007fe76a488cd0 R15:00007ffdab55fdb0
RIP:0000562651e3d338 EFL:0000000000010202
CSGSFS:002b000000000033
3842:S 18 Feb 2021 22:14:27.844 # (00007ffdab55f6cf) -> 00007fe77e40e900
3842:S 18 Feb 2021 22:14:27.844 # (00007ffdab55f6ce) -> 00007fe77e40ff80
3842:S 18 Feb 2021 22:14:27.844 # (00007ffdab55f6cd) -> 0000000000000003
3842:S 18 Feb 2021 22:14:27.844 # (00007ffdab55f6cc) -> 00007fe77e2049d8
3842:S 18 Feb 2021 22:14:27.844 # (00007ffdab55f6cb) -> 0000000000000000
3842:S 18 Feb 2021 22:14:27.844 # (00007ffdab55f6ca) -> 0000562651f481c0
3842:S 18 Feb 2021 22:14:27.844 # (00007ffdab55f6c9) -> 00007fe77e204998
3842:S 18 Feb 2021 22:14:27.844 # (00007ffdab55f6c8) -> 0000000000000000
3842:S 18 Feb 2021 22:14:27.844 # (00007ffdab55f6c7) -> 656c696620424452
3842:S 18 Feb 2021 22:14:27.844 # (00007ffdab55f6c6) -> 206e6920646e756f
3842:S 18 Feb 2021 22:14:27.844 # (00007ffdab55f6c5) -> 662079656b206465
3842:S 18 Feb 2021 22:14:27.844 # (00007ffdab55f6c4) -> 746163696c707544
3842:S 18 Feb 2021 22:14:27.844 # (00007ffdab55f6c3) -> 00007ffdab55f7f0
3842:S 18 Feb 2021 22:14:27.844 # (00007ffdab55f6c2) -> 00007ffdab55f8c0
3842:S 18 Feb 2021 22:14:27.844 # (00007ffdab55f6c1) -> 0000003000000018
3842:S 18 Feb 2021 22:14:27.844 # (00007ffdab55f6c0) -> 00007fe77e40e8f0

------ MODULES INFO OUTPUT ------

------ FAST MEMORY TEST ------
3842:S 18 Feb 2021 22:14:27.844 # Bio thread for job type #0 terminated
3842:S 18 Feb 2021 22:14:27.845 # Bio thread for job type #1 terminated
3842:S 18 Feb 2021 22:14:27.845 # Bio thread for job type #2 terminated
*** Preparing to test memory region 562651f93000 (2277376 bytes)
*** Preparing to test memory region 562652d6c000 (270336 bytes)
*** Preparing to test memory region 7fe765600000 (245501952 bytes)
*** Preparing to test memory region 7fe7786ff000 (30408704 bytes)
*** Preparing to test memory region 7fe77a400000 (8388608 bytes)
*** Preparing to test memory region 7fe77ac00000 (2097152 bytes)
*** Preparing to test memory region 7fe77aefc000 (13631488 bytes)
*** Preparing to test memory region 7fe77bbfd000 (8388608 bytes)
*** Preparing to test memory region 7fe77c3fe000 (8388608 bytes)
*** Preparing to test memory region 7fe77cbff000 (8388608 bytes)
*** Preparing to test memory region 7fe77d400000 (8388608 bytes)
*** Preparing to test memory region 7fe77dc00000 (8388608 bytes)
*** Preparing to test memory region 7fe77e40e000 (24576 bytes)
*** Preparing to test memory region 7fe77e602000 (16384 bytes)
*** Preparing to test memory region 7fe77e625000 (16384 bytes)
*** Preparing to test memory region 7fe77e8fb000 (16384 bytes)
*** Preparing to test memory region 7fe77eae7000 (8192 bytes)
*** Preparing to test memory region 7fe77eb1e000 (4096 bytes)
.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------
Symbol: _serverPanic (base: 0x562651e3d1f0)
Module: /usr/bin/redis-server 0.0.0.0:6379 (base 0x562651daf000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x562651e3d1f0 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
3842:S 18 Feb 2021 22:14:28.568 # dump of function (hexdump of 456 bytes):
f30f1efa415541544989fc5389f34881ece001000048898c24480100004c898424500100004c898c245801000084c074400f298424600100000f298c24700100000f299424800100000f299c24900100000f29a424a00100000f29ac24b00100000f29b424c00100000f29bc24d001000064488b042528000000488984242801000031c04c8d6c24204989d04c8d4c2408488d842400020000b900010000ba010000004c89ef4889442410be00010000488d842430010000c744240818000000c744240c300000004889442418e80ee5faff8b05acc5150085c00f849000000031c0488d35ef600f00bf03000000e8bd79fbff31c0bf03000000488d350f610f00e8aa79fbff4189d84c89e14c89ea31c0488d35c7790f00bf03000000e88e79fbff31c0bf03000000488d3520610f00e87b79fbff31c0bf03000000488d3595600f00e86879fbffc60425ffffffff78488b84242801000064483304252800000075174881c4e00100005b415c415dc3e8abf3ffffe966ffffffe8d1e6faff90f30f1efa415741564989f641554989fd41544989cc554889d5ba04000000534889ef4881ecc803000064488b04252800000048898424b803000031c00fb601488d74242483e00f0f

=== REDIS BUG REPORT END. Make sure to include from START to END. ===

Aditional information

Linux redis4 5.4.78-2-pve #1 SMP PVE 5.4.78-2 (Thu, 03 Dec 2020 14:26:17 +0100) x86_64 x86_64 x86_64 GNU/Linux
Ubuntu 20.04 as lxc container
I enabled appendonly persistence

Comment From: madolson

Looks like it gave us a nice concise error message, 3842:S 18 Feb 2021 22:14:27.831 # RDB has duplicated key '' in DB 0 so there is some corruption in your rdb file where you have duplicate empty keys. Do you have any idea how that could have happened?

Comment From: Apollon77

The interesting thing is that after that error he started anew automatically and it went through! I hd the same issue on 3 of 7 nodes of the same redis cluster as I turned off rbd save and enabled appendonly. I did not had the issue before and the also on the other 4 hosts it was not an issue with the same dataset

Comment From: Swiftyhu

Had almost the same error... I upgraded from PVE6 to PVE7 (maybe this is important).

=== REDIS BUG REPORT START: Cut & paste starting from here === 1393735:M 15 Jul 2021 16:39:49.763 # ------------------------------------------------ 1393735:M 15 Jul 2021 16:39:49.763 # !!! Software Failure. Press left mouse button to continue 1393735:M 15 Jul 2021 16:39:49.763 # Guru Meditation: Duplicated key found in RDB file #rdb.c:2572

------ STACK TRACE ------

Backtrace: /usr/bin/redis-server 127.0.0.1:6379(rdbLoadRio+0xc9c)[0x555cbcad570c] /usr/bin/redis-server 127.0.0.1:6379(rdbLoad+0x98)[0x555cbcad5918] /usr/bin/redis-server 127.0.0.1:6379(loadDataFromDisk+0x9c)[0x555cbcaa85bc] /usr/bin/redis-server 127.0.0.1:6379(main+0x2c1)[0x555cbca950b1] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xea)[0x7f81234cdd0a] /usr/bin/redis-server 127.0.0.1:6379(_start+0x2a)[0x555cbca9564a]

------ INFO OUTPUT ------

Server

redis_version:6.2.4 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:5b3acc0f6872c2d4 redis_mode:standalone os:Linux 5.11.22-2-pve x86_64 arch_bits:64 multiplexing_api:epoll atomicvar_api:atomic-builtin gcc_version:10.2.1 process_id:1393735 process_supervised:systemd run_id:81a7aa48ecd92588b9ebaa1c35971a7bc702af92 tcp_port:6379 server_time_usec:1626359989752310 uptime_in_seconds:1 uptime_in_days:0 hz:10 configured_hz:10 lru_clock:15747252 executable:/usr/bin/redis-server config_file:/etc/redis/redis.conf io_threads_active:0

Clients

connected_clients:8 cluster_connections:0 maxclients:10000 client_recent_max_input_buffer:0 client_recent_max_output_buffer:0 blocked_clients:0 tracking_clients:0 clients_in_timeout_table:0

Memory

used_memory:583077680 used_memory_human:556.07M used_memory_rss:598777856 used_memory_rss_human:571.04M used_memory_peak:583077680 used_memory_peak_human:556.07M used_memory_peak_perc:100.30% used_memory_overhead:451393824 used_memory_startup:816944 used_memory_dataset:131683856 used_memory_dataset_perc:22.62% allocator_allocated:581661416 allocator_active:582041600 allocator_resident:591417344 total_system_memory:33616592896 total_system_memory_human:31.31G used_memory_lua:41984 used_memory_lua_human:41.00K used_memory_scripts:0 used_memory_scripts_human:0B number_of_cached_scripts:0 maxmemory:0 maxmemory_human:0B maxmemory_policy:noeviction allocator_frag_ratio:1.00 allocator_frag_bytes:380184 allocator_rss_ratio:1.02 allocator_rss_bytes:9375744 rss_overhead_ratio:1.01 rss_overhead_bytes:7360512 mem_fragmentation_ratio:1.03 mem_fragmentation_bytes:17442080 mem_not_counted_for_evict:0 mem_replication_backlog:0 mem_clients_slaves:0 mem_clients_normal:0 mem_aof_buffer:0 mem_allocator:jemalloc-5.2.1 active_defrag_running:0 lazyfree_pending_objects:0 lazyfreed_objects:0

Persistence

loading:1 current_cow_size:0 current_cow_size_age:0 current_fork_perc:0.00 current_save_keys_processed:0 current_save_keys_total:0 rdb_changes_since_last_save:0 rdb_bgsave_in_progress:0 rdb_last_save_time:1626359988 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:-1 rdb_current_bgsave_time_sec:-1 rdb_last_cow_size:0 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok aof_last_cow_size:0 module_fork_in_progress:0 module_fork_last_cow_size:0 loading_start_time:1626359988 loading_total_bytes:3187283487 loading_rdb_used_mem:7938245344 loading_loaded_bytes:85983226 loading_loaded_perc:2.70 loading_eta_seconds:36

Stats

total_connections_received:8 total_commands_processed:0 instantaneous_ops_per_sec:0 total_net_input_bytes:6672 total_net_output_bytes:392 instantaneous_input_kbps:0.00 instantaneous_output_kbps:0.00 rejected_connections:0 sync_full:0 sync_partial_ok:0 sync_partial_err:0 expired_keys:0 expired_stale_perc:0.00 expired_time_cap_reached_count:0 expire_cycle_cpu_milliseconds:0 evicted_keys:0 keyspace_hits:0 keyspace_misses:0 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:0 total_forks:0 migrate_cached_sockets:0 slave_expires_tracked_keys:0 active_defrag_hits:0 active_defrag_misses:0 active_defrag_key_hits:0 active_defrag_key_misses:0 tracking_total_keys:0 tracking_total_items:0 tracking_total_prefixes:0 unexpected_error_replies:0 total_error_replies:8 dump_payload_sanitizations:0 total_reads_processed:8 total_writes_processed:8 io_threaded_reads_processed:0 io_threaded_writes_processed:0

Replication

role:master connected_slaves:0 master_failover_state:no-failover master_replid:8c136df8954acfb75ea745e13d2f2de89771a239 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:0 second_repl_offset:-1 repl_backlog_active:0 repl_backlog_size:1048576 repl_backlog_first_byte_offset:0 repl_backlog_histlen:0

CPU

used_cpu_sys:0.284331 used_cpu_user:1.289461 used_cpu_sys_children:0.000000 used_cpu_user_children:0.000000 used_cpu_sys_main_thread:0.280896 used_cpu_user_main_thread:1.292125

Modules

Commandstats

cmdstat_script:calls=0,usec=0,usec_per_call=0.00,rejected_calls=8,failed_calls=0

Errorstats

errorstat_LOADING:count=8

Cluster

cluster_enabled:0

Keyspace

db0:keys=784738,expires=688924,avg_ttl=0

------ CLIENT LIST OUTPUT ------ id=3 addr=127.0.0.1:14680 laddr=127.0.0.1:6379 fd=9 name= age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61472 events=r cmd=script user=default redir=-1 id=4 addr=127.0.0.1:14682 laddr=127.0.0.1:6379 fd=10 name= age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61472 events=r cmd=script user=default redir=-1 id=5 addr=127.0.0.1:14684 laddr=127.0.0.1:6379 fd=11 name= age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61472 events=r cmd=script user=default redir=-1 id=6 addr=127.0.0.1:14686 laddr=127.0.0.1:6379 fd=12 name= age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61472 events=r cmd=script user=default redir=-1 id=7 addr=127.0.0.1:14688 laddr=127.0.0.1:6379 fd=13 name= age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61472 events=r cmd=script user=default redir=-1 id=8 addr=127.0.0.1:14690 laddr=127.0.0.1:6379 fd=14 name= age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61472 events=r cmd=script user=default redir=-1 id=9 addr=127.0.0.1:14692 laddr=127.0.0.1:6379 fd=15 name= age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61472 events=r cmd=script user=default redir=-1 id=10 addr=127.0.0.1:14694 laddr=127.0.0.1:6379 fd=16 name= age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61472 events=r cmd=script user=default redir=-1

------ MODULES INFO OUTPUT ------

------ FAST MEMORY TEST ------ 1393735:M 15 Jul 2021 16:39:49.764 # Bio thread for job type #0 terminated 1393735:M 15 Jul 2021 16:39:49.764 # Bio thread for job type #1 terminated 1393735:M 15 Jul 2021 16:39:49.765 # Bio thread for job type #2 terminated *** Preparing to test memory region 555cbcbc1000 (122880 bytes) *** Preparing to test memory region 7f80e7300000 (951058432 bytes) *** Preparing to test memory region 7f811ff4f000 (8388608 bytes) *** Preparing to test memory region 7f8120750000 (8388608 bytes) *** Preparing to test memory region 7f8120f51000 (8388608 bytes) *** Preparing to test memory region 7f8121752000 (8388608 bytes) *** Preparing to test memory region 7f8122600000 (8388608 bytes) *** Preparing to test memory region 7f8122ff6000 (40960 bytes) *** Preparing to test memory region 7f8123157000 (4096 bytes) *** Preparing to test memory region 7f8123256000 (8192 bytes) *** Preparing to test memory region 7f81234a4000 (12288 bytes) *** Preparing to test memory region 7f8123668000 (16384 bytes) *** Preparing to test memory region 7f812368a000 (24576 bytes) *** Preparing to test memory region 7f8123980000 (16384 bytes) *** Preparing to test memory region 7f8123adc000 (4096 bytes) *** Preparing to test memory region 7f8123cdc000 (2256896 bytes) *** Preparing to test memory region 7f812434d000 (4096 bytes) *** Preparing to test memory region 7f8124354000 (8192 bytes) *** Preparing to test memory region 7f8124392000 (4096 bytes) .O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

=== REDIS BUG REPORT END. Make sure to include from START to END. ===

Comment From: madolson

@Swiftyhu Same thing, there is a duplicate key inside your RDB. There is the CLI tool redis-check-rdb that you can perhaps use to understand what is going on? (Also, what is PVE6 and PVE7?

Comment From: Swiftyhu

@madolson :

There is the CLI tool redis-check-rdb that you can perhaps use to understand what is going on?

Thanks, I will try it.

(Also, what is PVE6 and PVE7?

Proxmox, a virtualization solution. @Apollon77 uses this kernel: os:Linux 5.4.78-2-pve x86_64 and I use this os:Linux 5.11.22-2-pve x86_64 (Note the "pve" string :) )

Comment From: RLIndia

Facing a similar issue. Running Redis as standalone server. Keeps crashing over and over again.

3707237:M 04 Oct 2021 10:47:18.559 # Server initialized 3707237:M 04 Oct 2021 10:47:18.559 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect. 3707237:M 04 Oct 2021 10:47:18.559 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled. 3707237:M 04 Oct 2021 10:47:18.560 * Loading RDB produced by version 6.0.6 3707237:M 04 Oct 2021 10:47:18.560 * RDB age 11 seconds 3707237:M 04 Oct 2021 10:47:18.560 * RDB memory usage when created 0.85 Mb 3707237:M 04 Oct 2021 10:47:18.560 * DB loaded from disk: 0.000 seconds 3707237:M 04 Oct 2021 10:47:18.560 * Ready to accept connections 3707237:M 04 Oct 2021 10:47:19.245 * DB saved on disk 3707237:M 04 Oct 2021 10:47:19.503 * DB saved on disk 3707237:M 04 Oct 2021 10:47:22.001 # Failed opening the RDB file zzh (in server root dir /etc/cron.d) for saving: Read-only file system

=== REDIS BUG REPORT START: Cut & paste starting from here === 3707237:M 04 Oct 2021 10:47:22.001 # Redis 6.0.6 crashed by signal: 11 3707237:M 04 Oct 2021 10:47:22.001 # Crashed running the instruction at: 0x55908fee7099 3707237:M 04 Oct 2021 10:47:22.001 # Accessing address: 0x87ec0 3707237:M 04 Oct 2021 10:47:22.001 # Failed assertion: (:0)

------ STACK TRACE ------ EIP: /usr/bin/redis-server 172.31.4.153:6379(je_malloc_usable_size+0x89)[0x55908fee7099]

Backtrace: /usr/bin/redis-server 172.31.4.153:6379(logStackTrace+0x4f)[0x55908fe64bcf] /usr/bin/redis-server 172.31.4.153:6379(sigsegvHandler+0xb5)[0x55908fe653d5] /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0)[0x7f465474c3c0] /usr/bin/redis-server 172.31.4.153:6379(je_malloc_usable_size+0x89)[0x55908fee7099] /usr/bin/redis-server 172.31.4.153:6379(+0x50b79)[0x55908fe24b79] /usr/bin/redis-server 172.31.4.153:6379(rdbSave+0x2ba)[0x55908fe4545a] /usr/bin/redis-server 172.31.4.153:6379(saveCommand+0x67)[0x55908fe46ab7] /usr/bin/redis-server 172.31.4.153:6379(call+0xb1)[0x55908fe1c8b1] /usr/bin/redis-server 172.31.4.153:6379(processCommand+0x4a6)[0x55908fe1d446] /usr/bin/redis-server 172.31.4.153:6379(processCommandAndResetClient+0x14)[0x55908fe2b9e4] /usr/bin/redis-server 172.31.4.153:6379(processInputBuffer+0x18f)[0x55908fe3039f] /usr/bin/redis-server 172.31.4.153:6379(+0xe10ac)[0x55908feb50ac] /usr/bin/redis-server 172.31.4.153:6379(aeProcessEvents+0x303)[0x55908fe15b83] /usr/bin/redis-server 172.31.4.153:6379(aeMain+0x1d)[0x55908fe15ebd] /usr/bin/redis-server 172.31.4.153:6379(main+0x4e5)[0x55908fe123d5] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f465456c0b3] /usr/bin/redis-server 172.31.4.153:6379(_start+0x2e)[0x55908fe126ae]

------ INFO OUTPUT ------

Server

redis_version:6.0.6 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:d463d609620685c0 redis_mode:standalone os:Linux 5.4.0-1045-aws x86_64 arch_bits:64 multiplexing_api:epoll atomicvar_api:atomic-builtin gcc_version:9.3.0 process_id:3707237 run_id:755b810bef1a7302e9836a6cab2b276f1948002e tcp_port:6379 uptime_in_seconds:4 uptime_in_days:0 hz:10 configured_hz:10 lru_clock:5954489 executable:/usr/bin/redis-server config_file:/etc/redis/redis.conf

Clients

connected_clients:5 client_recent_max_input_buffer:2 client_recent_max_output_buffer:0 blocked_clients:0 tracking_clients:0 clients_in_timeout_table:0

Memory

used_memory:951032 used_memory_human:928.74K used_memory_rss:5697536 used_memory_rss_human:5.43M used_memory_peak:1052952 used_memory_peak_human:1.00M used_memory_peak_perc:90.32% used_memory_overhead:888186 used_memory_startup:803064 used_memory_dataset:62846 used_memory_dataset_perc:42.47% allocator_allocated:1388024 allocator_active:1712128 allocator_resident:4124672 total_system_memory:8038789120 total_system_memory_human:7.49G used_memory_lua:37888 used_memory_lua_human:37.00K used_memory_scripts:0 used_memory_scripts_human:0B number_of_cached_scripts:0 maxmemory:0 maxmemory_human:0B maxmemory_policy:noeviction allocator_frag_ratio:1.23 allocator_frag_bytes:324104 allocator_rss_ratio:2.41 allocator_rss_bytes:2412544 rss_overhead_ratio:1.38 rss_overhead_bytes:1572864 mem_fragmentation_ratio:6.27 mem_fragmentation_bytes:4788992 mem_not_counted_for_evict:0 mem_replication_backlog:0 mem_clients_slaves:0 mem_clients_normal:84930 mem_aof_buffer:0 mem_allocator:jemalloc-5.1.0 active_defrag_running:0 lazyfree_pending_objects:0

Persistence

loading:0 rdb_changes_since_last_save:6 rdb_bgsave_in_progress:0 rdb_last_save_time:1633344439 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:-1 rdb_current_bgsave_time_sec:-1 rdb_last_cow_size:0 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok aof_last_cow_size:0 module_fork_in_progress:0 module_fork_last_cow_size:0

Stats

total_connections_received:5 total_commands_processed:13 instantaneous_ops_per_sec:3 total_net_input_bytes:749 total_net_output_bytes:14277 instantaneous_input_kbps:0.35 instantaneous_output_kbps:0.02 rejected_connections:0 sync_full:0 sync_partial_ok:0 sync_partial_err:0 expired_keys:0 expired_stale_perc:0.00 expired_time_cap_reached_count:0 expire_cycle_cpu_milliseconds:0 evicted_keys:0 keyspace_hits:0 keyspace_misses:0 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:0 migrate_cached_sockets:0 slave_expires_tracked_keys:0 active_defrag_hits:0 active_defrag_misses:0 active_defrag_key_hits:0 active_defrag_key_misses:0 tracking_total_keys:0 tracking_total_items:0 tracking_total_prefixes:0 unexpected_error_replies:0

Replication

role:master connected_slaves:0 master_replid:ad477128868dd58f04bd2d423f6f92faf2512090 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:0 second_repl_offset:-1 repl_backlog_active:0 repl_backlog_size:1048576 repl_backlog_first_byte_offset:0 repl_backlog_histlen:0

CPU

used_cpu_sys:0.000000 used_cpu_user:0.008285 used_cpu_sys_children:0.000000 used_cpu_user_children:0.000000

Modules

Commandstats

cmdstat_save:calls=1,usec=2687,usec_per_call=2687.00 cmdstat_set:calls=4,usec=23,usec_per_call=5.75 cmdstat_info:calls=4,usec=243,usec_per_call=60.75 cmdstat_config:calls=3,usec=33,usec_per_call=11.00 cmdstat_flushall:calls=1,usec=2177,usec_per_call=2177.00

Cluster

cluster_enabled:0

Keyspace

db0:keys=4,expires=0,avg_ttl=0

------ CLIENT LIST OUTPUT ------ id=4 addr=172.19.0.3:50282 fd=8 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info user=default id=5 addr=172.19.0.3:50284 fd=9 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info user=default id=6 addr=172.19.0.6:48704 fd=10 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info user=default id=7 addr=172.19.0.6:48706 fd=11 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info user=default id=8 addr=118.25.242.123:39210 fd=12 name= age=4 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=14 qbuf-free=32754 obl=0 oll=0 omem=0 events=r cmd=save user=default

------ CURRENT CLIENT INFO ------ id=8 addr=118.25.242.123:39210 fd=12 name= age=4 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=14 qbuf-free=32754 obl=0 oll=0 omem=0 events=r cmd=save user=default argv[0]: 'save'

Redis [CRASH] Redis 6.0.6 crashed by signal: 11

Server

Clients

Memory

Persistence

Stats

Replication

CPU

Modules

Commandstats

Errorstats

Cluster

Keyspace

Facing a similar issue. Running Redis as standalone server. Keeps crashing over and over again.

Server

Clients

Memory

Persistence

Stats

Replication

CPU

Modules

Commandstats

Cluster

Keyspace