I have been investigating redis sentinel and its all going well except then sentinel sometimes crashes. It seems to occur when I instigate a failover (using Ctrl-C) but never deterministically.

I have three sentinels and two redis instances. It is always the third sentinel that crashes. The others are fine.

If I provide any further information then please let me know.

=== REDIS BUG REPORT START: Cut & paste starting from here === 84259:X 22 May 16:44:36.921 # Redis 3.1.999 crashed by signal: 11 84259:X 22 May 16:44:36.921 # Failed assertion: (:0) 84259:X 22 May 16:44:36.921 # --- STACK TRACE 0 redis-sentinel 0x000000010e268a87 logStackTrace + 103 1 redis-sentinel 0x000000010e27bb09 sentinelEvent + 137 2 libsystem_platform.dylib 0x00007fff8ce64f1a _sigtramp + 26 3 ??? 0x0000000000000000 0x0 + 0 4 redis-sentinel 0x000000010e27fea0 sentinelProcessHelloMessage + 592 5 redis-sentinel 0x000000010e27b319 sentinelPublishCommand + 73 6 redis-sentinel 0x000000010e22da3a call + 202 7 redis-sentinel 0x000000010e22e22b processCommand + 1259 8 redis-sentinel 0x000000010e23bd06 processInputBuffer + 198 9 redis-sentinel 0x000000010e226ad4 aeProcessEvents + 548 10 redis-sentinel 0x000000010e226dcb aeMain + 43 11 redis-sentinel 0x000000010e2313ef main + 1279 12 libdyld.dylib 0x00007fff9c1b55c9 start + 1 13 ??? 0x0000000000000002 0x0 + 2 84259:X 22 May 16:44:36.921 # --- INFO OUTPUT 84259:X 22 May 16:44:36.921 # # Server redis_version:3.1.999 redis_git_sha1:c54de703 redis_git_dirty:0 redis_build_id:29ece0649d874d55 redis_mode:sentinel os:Darwin 14.0.0 x86_64 arch_bits:64 multiplexing_api:kqueue gcc_version:4.2.1 process_id:84259 run_id:c9444f8cb971ae545111caa6015330134cdbee69 tcp_port:8002 uptime_in_seconds:1258 uptime_in_days:0 hz:14 lru_clock:6246116 config_file:/Users/richardscreene/redis/7002/sentinel.conf

Clients

connected_clients:2 client_longest_output_list:0 client_biggest_input_buf:0 blocked_clients:0

Memory

used_memory:1035424 used_memory_human:1011.16K used_memory_rss:2158592 used_memory_rss_human:2.06M used_memory_peak:1085264 used_memory_peak_human:1.03M total_system_memory:17179869184 total_system_memory_human:16.00G used_memory_lua:35840 used_memory_lua_human:35.00K maxmemory:0 maxmemory_human:0B maxmemory_policy:volatile-lru mem_fragmentation_ratio:2.08 mem_allocator:libc

Persistence

loading:0 rdb_changes_since_last_save:0 rdb_bgsave_in_progress:0 rdb_last_save_time:1432308218 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:-1 rdb_current_bgsave_time_sec:-1 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok

Stats

total_connections_received:256 total_commands_processed:4397 instantaneous_ops_per_sec:4 total_net_input_bytes:257237 total_net_output_bytes:39187 instantaneous_input_kbps:0.33 instantaneous_output_kbps:0.09 rejected_connections:0 sync_full:0 sync_partial_ok:0 sync_partial_err:0 expired_keys:0 evicted_keys:0 keyspace_hits:0 keyspace_misses:0 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:0 migrate_cached_sockets:0

Replication

role:master connected_slaves:0 master_repl_offset:0 repl_backlog_active:0 repl_backlog_size:1048576 repl_backlog_first_byte_offset:0 repl_backlog_histlen:0

CPU

used_cpu_sys:1.38 used_cpu_user:1.06 used_cpu_sys_children:0.00 used_cpu_user_children:0.00

Commandstats

Cluster

cluster_enabled:0

Keyspace

hash_init_value: 1431908985

84259:X 22 May 16:44:36.922 # --- CLIENT LIST OUTPUT 84259:X 22 May 16:44:36.922 # id=3 addr=127.0.0.1:62186 fd=8 name=sentinel-7702f227-cmd age=1257 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=publish id=257 addr=127.0.0.1:50845 fd=9 name=sentinel-76e9f137-cmd age=454 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sentinel

84259:X 22 May 16:44:36.922 # --- CURRENT CLIENT INFO 84259:X 22 May 16:44:36.922 # client: id=3 addr=127.0.0.1:62186 fd=8 name=sentinel-7702f227-cmd age=1257 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=publish 84259:X 22 May 16:44:36.922 # argv[0]: 'PUBLISH' 84259:X 22 May 16:44:36.922 # argv[1]: 'sentinel:hello' 84259:X 22 May 16:44:36.922 # argv[2]: '127.0.0.1,8000,7702f22737ef45e0b85282d39c288ba877369444,27,mymaster,127.0.0.1,7002,27' 84259:X 22 May 16:44:36.922 # --- REGISTERS 84259:X 22 May 16:44:36.922 # RAX:0000000000000025 RBX:000000010e2bda04 RCX:0000000000000040 RDX:0000000000000000 RDI:0000000000000003 RSI:000000010e2be109 RBP:00007fff519dd6f0 RSP:00007fff519dd1c0 R8 :0000000000000002 R9 :00007fc87b700000 R10:0000000000000010 R11:00007fc87b700000 R12:0000000000000003 R13:000000010e2be109 R14:00007fc87b5003d0 R15:0000000000000000 RIP:000000010e27bb09 EFL:0000000000010246 CS :000000000000002b FS:0000000000000000 GS:0000000000000000 84259:X 22 May 16:44:36.922 # (00007fff519dd1cf) -> 000000020e323000 84259:X 22 May 16:44:36.922 # (00007fff519dd1ce) -> 00007fff519dd559 84259:X 22 May 16:44:36.922 # (00007fff519dd1cd) -> 00007fff90c06d82 84259:X 22 May 16:44:36.922 # (00007fff519dd1cc) -> 00007fff519dd250 84259:X 22 May 16:44:36.922 # (00007fff519dd1cb) -> 00007fc87b700000 84259:X 22 May 16:44:36.922 # (00007fff519dd1ca) -> 0000000000000002 84259:X 22 May 16:44:36.922 # (00007fff519dd1c9) -> 00007fff7d3fa6b8 84259:X 22 May 16:44:36.922 # (00007fff519dd1c8) -> 00007fff519dd558 84259:X 22 May 16:44:36.922 # (00007fff519dd1c7) -> ffffffffffffffff 84259:X 22 May 16:44:36.922 # (00007fff519dd1c6) -> 0000000000000000 84259:X 22 May 16:44:36.922 # (00007fff519dd1c5) -> 0000000000000004 84259:X 22 May 16:44:36.922 # (00007fff519dd1c4) -> 0e0096b1623a9716 84259:X 22 May 16:44:36.922 # (00007fff519dd1c3) -> 000000010e323000 84259:X 22 May 16:44:36.922 # (00007fff519dd1c2) -> 0000000000000001 84259:X 22 May 16:44:36.922 # (00007fff519dd1c1) -> 000000010e323000 84259:X 22 May 16:44:36.922 # (00007fff519dd1c0) -> 000000000e326a00 84259:X 22 May 16:44:36.922 # === REDIS BUG REPORT END. Make sure to include from START to END. ===

Comment From: antirez

Hello, you are using the unstable branch which is currently under active development and is not stable. Lots of changes going into Sentinel right now, so the code will take time to go into testing first, and stable later. 2.8 / 3.0 has no such issues. Please could you tell me something about how did you triggered this crash? Thanks.

Comment From: antirez

Ah sorry I saw you already provided some info. I'll need to replicate anyway even if this is not happening in the stable release since I've to fix incrementally bugs in the new version...

Comment From: rscreene

Sorry. My bad. I should have used the stable release. I'm out of the office this week, but I'll try with stable on Monday. On 22 May 2015 19:44, "Salvatore Sanfilippo" notifications@github.com wrote:

Ah sorry I saw you already provided some info. I'll need to replicate anyway even if this is not happening in the stable release since I've to fix incrementally bugs in the new version...

— Reply to this email directly or view it on GitHub https://github.com/antirez/redis/issues/2590#issuecomment-104724302.

Comment From: rscreene

I've installed the stable version (Redis server v=3.0.1 sha=9c0a6886:0 malloc=libc bits=64 build=f43cf14f2285b0ee) and I have been unable to reproduce. So, please close this issue if you see fit.

Let me know if you need any help diagnosing the problem on the unstable branch.

Comment From: antirez

Thanks, I'm taking this open in order to fix the issue into the unstable branch.

Comment From: filipecosta90

I've installed the stable version (Redis server v=3.0.1 sha=9c0a6886:0 malloc=libc bits=64 build=f43cf14f2285b0ee) and I have been unable to reproduce. So, please close this issue if you see fit.

Closing this issue given it was solved prior release.