Hello, We are using redis with instances version as follows, +---------------+----------+ | instance_version | count | +---------------+----------+ | 2.8.19 | 6 | | 4.0.2 | 12 | | 4.0.8 | 446 | | 4.0.9 | 6 | +---------------+----------+ and 5 redis-sentinel instances (4.0.9),monitoring 229 master-slave pairs. one sentinel instance crashed and errors are reported as follows:
=== REDIS BUG REPORT START: Cut & paste starting from here ===
21518:X 29 Jan 11:08:15.522 # Redis 4.0.9 crashed by signal: 11
21518:X 29 Jan 11:08:15.522 # Crashed running the instruction at: 0x7fb636a010c2
21518:X 29 Jan 11:08:15.522 # Accessing address: 0x7fb62e400000
21518:X 29 Jan 11:08:15.522 # Failed assertion: <no assertion failed> (<no file>:0)
------ STACK TRACE ------
EIP:
/lib64/libc.so.6(+0x15e0c2)[0x7fb636a010c2]
Backtrace:
/xx/redis-sentinel *:yyyy [sentinel](logStackTrace+0x29)[0x468779]
/xx/redis-sentinel *:yyyy [sentinel](sigsegvHandler+0xac)[0x468e1c]
/lib64/libpthread.so.0(+0xf100)[0x7fb636c73100]
/lib64/libc.so.6(+0x15e0c2)[0x7fb636a010c2]
/xx/redis-sentinel *:yyyy [sentinel](sentinelRefreshInstanceInfo+0x248)[0x483aa8]
/xx/redis-sentinel *:yyyy [sentinel](redisProcessCallbacks+0x83)[0x49b173]
/xx/redis-sentinel *:yyyy [sentinel](aeProcessEvents+0x280)[0x4267f0]
/xx/redis-sentinel *:yyyy [sentinel](aeMain+0x2b)[0x426adb]
/xx/redis-sentinel *:yyyy [sentinel](main+0x49f)[0x4238ef]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7fb6368c4b15]
/xx/redis-sentinel *:yyyy [sentinel][0x423be5]
------ INFO OUTPUT ------
# Server
redis_version:4.0.9
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:8f166d4be9fa6f15
redis_mode:sentinel
os:Linux 3.10.x-xx x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:4.8.3
process_id:21518
run_id:8a5089715297a61bfd6b71b10039bcf336170456
tcp_port:yyyy
uptime_in_seconds:7161888
uptime_in_days:82
hz:12
lru_clock:5227423
executable:/xx/redis-sentinel
config_file:/xx/sentinel.conf
# Clients
connected_clients:32
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
# Memory
used_memory:9180088
used_memory_human:8.75M
used_memory_rss:13770752
used_memory_rss_human:13.13M
used_memory_peak:9302096
used_memory_peak_human:8.87M
used_memory_peak_perc:98.69%
used_memory_overhead:1350052
used_memory_startup:777824
used_memory_dataset:7830036
used_memory_dataset_perc:93.19%
total_system_memory:25113010176
total_system_memory_human:23.39G
used_memory_lua:37888
used_memory_lua_human:37.00K
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
mem_fragmentation_ratio:1.50
mem_allocator:jemalloc-4.0.3
active_defrag_running:0
lazyfree_pending_objects:0
# Persistence
loading:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1541569407
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
# Stats
total_connections_received:3431177
total_commands_processed:2695136145
instantaneous_ops_per_sec:259
total_net_input_bytes:422766034664
total_net_output_bytes:16029521750
instantaneous_input_kbps:40.12
instantaneous_output_kbps:1.03
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:5
pubsub_patterns:0
latest_fork_usec:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
# Replication
role:master
connected_slaves:0
master_replid:25f5e94146b3decf58540402f00f245c5fce8155
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:86167.11
used_cpu_user:156396.41
used_cpu_sys_children:0.00
used_cpu_user_children:0.00
# Commandstats
cmdstat_publish:calls=2659651506,usec=8700654151,usec_per_call=3.27
cmdstat_client:calls=13,usec=21,usec_per_call=1.62
cmdstat_subscribe:calls=62,usec=519,usec_per_call=8.37
cmdstat_info:calls=11757,usec=2887540,usec_per_call=245.60
cmdstat_sentinel:calls=9621239,usec=179457100,usec_per_call=18.65
cmdstat_ping:calls=25851568,usec=23685100,usec_per_call=0.92
# Cluster
cluster_enabled:0
# Keyspace
------ CLIENT LIST OUTPUT ------
id=3431171 addr=x.x.x.x:34222 fd=956 name= age=287 idle=287 flags=N db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=3431178 addr=x.x.x.x:37162 fd=962 name= age=17 idle=17 flags=N db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=455 addr=x.x.x.x:41320 fd=22 name=sentinel-213a6d4d-cmd age=6727465 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=3431173 addr=x.x.x.x:46424 fd=957 name= age=58 idle=58 flags=N db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=3431166 addr=x.x.x.x:33210 fd=951 name= age=328 idle=328 flags=N db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=452 addr=x.x.x.x:57314 fd=19 name=sentinel-efd3f557-cmd age=6727465 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=publish
id=453 addr=x.x.x.x:32828 fd=20 name=sentinel-65f29aaa-cmd age=6727465 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=ping
id=3420948 addr=x.x.x.x:55312 fd=941 name= age=61726 idle=61726 flags=N db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=3420946 addr=x.x.x.x:49372 fd=939 name= age=61735 idle=61735 flags=N db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=3431174 addr=x.x.x.x:55012 fd=958 name= age=55 idle=55 flags=N db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=3431175 addr=x.x.x.x:59898 fd=959 name= age=55 idle=55 flags=N db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=3420947 addr=x.x.x.x:55922 fd=940 name= age=61735 idle=61735 flags=N db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=3431167 addr=x.x.x.x:60648 fd=952 name= age=325 idle=325 flags=N db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=3431169 addr=x.x.x.x:36922 fd=954 name= age=316 idle=316 flags=N db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=3431168 addr=x.x.x.x:35694 fd=953 name= age=325 idle=325 flags=N db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=454 addr=x.x.x.x:56416 fd=21 name=sentinel-ec9ac407-cmd age=6727465 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=3431179 addr=x.x.x.x:58960 fd=963 name= age=2 idle=2 flags=N db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=3431177 addr=x.x.x.x:57806 fd=961 name= age=32 idle=32 flags=N db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=3421179 addr=x.x.x.x:41838 fd=943 name= age=61697 idle=61697 flags=N db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=3340970 addr=x.x.x.x:40264 fd=912 name= age=513558 idle=513558 flags=N db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=3420949 addr=x.x.x.x:59770 fd=942 name= age=61712 idle=61712 flags=N db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=3431176 addr=x.x.x.x:48606 fd=960 name= age=46 idle=46 flags=N db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=3340743 addr=x.x.x.x:43904 fd=913 name= age=513626 idle=513626 flags=N db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=3340742 addr=x.x.x.x:44742 fd=910 name= age=513693 idle=513693 flags=N db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=3421180 addr=x.x.x.x:60038 fd=944 name= age=61682 idle=61682 flags=N db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=3431165 addr=x.x.x.x:48294 fd=950 name= age=332 idle=332 flags=N db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=3421181 addr=x.x.x.x:46988 fd=945 name= age=61678 idle=61678 flags=N db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=3341197 addr=x.x.x.x:35136 fd=914 name= age=513468 idle=513468 flags=N db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=3337568 addr=x.x.x.x:60450 fd=908 name= age=516626 idle=516626 flags=N db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=3340741 addr=x.x.x.x:46964 fd=909 name= age=513783 idle=513783 flags=N db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=3340739 addr=x.x.x.x:58662 fd=911 name= age=513870 idle=513870 flags=N db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=3431170 addr=x.x.x.x:33948 fd=955 name= age=302 idle=302 flags=N db=0 sub=5 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
------ REGISTERS ------
21518:X 29 Jan 11:08:15.523 #
RAX:0000000000000000 RBX:00007fb62e3ffffb
RCX:00007fb636a010c2 RDX:000000000000000b
RDI:00007fb62e400006 RSI:000000000050764a
RBP:000000000000007f RSP:00007ffe09d6a258
R8 :00007fb636c5e060 R9 :00007fb62e07bb3b
R10:0000000000000064 R11:00007fb636a23550
R12:00007fb62dad23f8 R13:00007fb62dc1bf00
R14:0000000000000001 R15:0000000000000001
RIP:00007fb636a010c2 EFL:0000000000010203
CSGSFS:0000000000000033
21518:X 29 Jan 11:08:15.523 # (00007ffe09d6a267) -> 00007ffe09d6a518
21518:X 29 Jan 11:08:15.523 # (00007ffe09d6a266) -> 000000000049b173
21518:X 29 Jan 11:08:15.523 # (00007ffe09d6a265) -> 00000000000001c9
21518:X 29 Jan 11:08:15.523 # (00007ffe09d6a264) -> 00007fb62fe480a0
21518:X 29 Jan 11:08:15.523 # (00007ffe09d6a263) -> 0000000000000001
21518:X 29 Jan 11:08:15.523 # (00007ffe09d6a262) -> 00007ffe09d6a2e0
21518:X 29 Jan 11:08:15.523 # (00007ffe09d6a261) -> 0000000001e3fff0
21518:X 29 Jan 11:08:15.523 # (00007ffe09d6a260) -> 0000000001d42950
21518:X 29 Jan 11:08:15.523 # (00007ffe09d6a25f) -> 0000000001e3fff0
21518:X 29 Jan 11:08:15.523 # (00007ffe09d6a25e) -> 000000802e3ff960
21518:X 29 Jan 11:08:15.523 # (00007ffe09d6a25d) -> 0000000000000000
21518:X 29 Jan 11:08:15.523 # (00007ffe09d6a25c) -> 00007fb62dad2000
21518:X 29 Jan 11:08:15.523 # (00007ffe09d6a25b) -> 00007fb62e2de671
21518:X 29 Jan 11:08:15.523 # (00007ffe09d6a25a) -> 00007fb600000002
21518:X 29 Jan 11:08:15.523 # (00007ffe09d6a259) -> 0000000000000000
21518:X 29 Jan 11:08:15.523 # (00007ffe09d6a258) -> 0000000000483aa8
------ FAST MEMORY TEST ------
21518:X 29 Jan 11:08:15.523 # Bio thread for job type #0 terminated
21518:X 29 Jan 11:08:15.523 # Bio thread for job type #1 terminated
21518:X 29 Jan 11:08:15.523 # Bio thread for job type #2 terminated
*** Preparing to test memory region 744000 (98304 bytes)
*** Preparing to test memory region 1d26000 (135168 bytes)
*** Preparing to test memory region 1d47000 (1110016 bytes)
*** Preparing to test memory region 7fb62da00000 (10485760 bytes)
*** Preparing to test memory region 7fb62e5fe000 (8388608 bytes)
*** Preparing to test memory region 7fb62edff000 (8388608 bytes)
*** Preparing to test memory region 7fb62f600000 (8388608 bytes)
*** Preparing to test memory region 7fb62fe00000 (2097152 bytes)
*** Preparing to test memory region 7fb636600000 (2097152 bytes)
*** Preparing to test memory region 7fb636c5f000 (20480 bytes)
*** Preparing to test memory region 7fb636e7c000 (16384 bytes)
*** Preparing to test memory region 7fb63759c000 (16384 bytes)
*** Preparing to test memory region 7fb6375a5000 (4096 bytes)
*** Preparing to test memory region 7fb6375a6000 (4096 bytes)
*** Preparing to test memory region 7fb6375a9000 (4096 bytes)
.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
------ DUMPING CODE AROUND EIP ------
Symbol: (null) (base: (nil))
Module: /lib64/libc.so.6 (base 0x7fb6368a3000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=(nil) -D -b binary -m i386:x86-64 /tmp/dump.bin
------
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Comment From: yossigo
Hello @cumthaozi, this might have been fixed by #7443. Please upgrade to 6.0.9 and if the problem persists file a new issue with an updated crash log. Thanks!