=== REDIS BUG REPORT START: Cut & paste starting from here ===
31416:M 10 Apr 2020 12:57:25.209 # Redis 5.0.8 crashed by signal: 11
31416:M 10 Apr 2020 12:57:25.209 # Crashed running the instruction at: 0x55767bdb88cd
31416:M 10 Apr 2020 12:57:25.209 # Accessing address: (nil)
31416:M 10 Apr 2020 12:57:25.209 # Failed assertion: <no assertion failed> (<no file>:0)
------ STACK TRACE ------
EIP:
/usr/bin/redis-server 127.0.0.1:6379(zslDeleteNode+0x6d)[0x55767bdb88cd]
Backtrace:
/usr/bin/redis-server 127.0.0.1:6379(logStackTrace+0x41)[0x55767bdd3c81]
/usr/bin/redis-server 127.0.0.1:6379(sigsegvHandler+0xc9)[0x55767bdd4409]
/lib64/libpthread.so.0(+0xf5f0)[0x7f7dc3e945f0]
/usr/bin/redis-server 127.0.0.1:6379(zslDeleteNode+0x6d)[0x55767bdb88cd]
/usr/bin/redis-server 127.0.0.1:6379(zslDeleteRangeByRank+0xd6)[0x55767bdb90a6]
/usr/bin/redis-server 127.0.0.1:6379(zremrangeGenericCommand+0x35c)[0x55767bdbbd6c]
/usr/bin/redis-server 127.0.0.1:6379(call+0x9d)[0x55767bd8de3d]
/usr/bin/redis-server 127.0.0.1:6379(execCommand+0xd2)[0x55767bdd1082]
/usr/bin/redis-server 127.0.0.1:6379(call+0x9d)[0x55767bd8de3d]
/usr/bin/redis-server 127.0.0.1:6379(processCommand+0x39f)[0x55767bd8f1af]
/usr/bin/redis-server 127.0.0.1:6379(processInputBuffer+0x19d)[0x55767bd9ee7d]
/usr/bin/redis-server 127.0.0.1:6379(aeProcessEvents+0x2d8)[0x55767bd879b8]
/usr/bin/redis-server 127.0.0.1:6379(aeMain+0x2b)[0x55767bd87c9b]
/usr/bin/redis-server 127.0.0.1:6379(main+0x4e3)[0x55767bd848b3]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f7dc3ad9505]
/usr/bin/redis-server 127.0.0.1:6379(+0x33b31)[0x55767bd84b31]
------ INFO OUTPUT ------
# Server
redis_version:5.0.8
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:18ca9d7ea1c3e9cc
redis_mode:standalone
os:Linux 5.3.11-x86_64-linode131 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:4.8.5
process_id:31416
run_id:dda7308c2de2756ecaf58c007843ac30445352b4
tcp_port:6379
uptime_in_seconds:1959552
uptime_in_days:22
hz:10
configured_hz:10
lru_clock:9454341
executable:/usr/bin/redis-server
config_file:/etc/redis.conf
# Clients
connected_clients:114
client_recent_max_input_buffer:152179
client_recent_max_output_buffer:0
blocked_clients:0
# Memory
used_memory:27444284088
used_memory_human:25.56G
used_memory_rss:29248839680
used_memory_rss_human:27.24G
used_memory_peak:27733438728
used_memory_peak_human:25.83G
used_memory_peak_perc:98.96%
used_memory_overhead:471725718
used_memory_startup:791408
used_memory_dataset:26972558370
used_memory_dataset_perc:98.28%
allocator_allocated:27439958528
allocator_active:30561923072
allocator_resident:30887858176
total_system_memory:67430465536
total_system_memory_human:62.80G
used_memory_lua:71680
used_memory_lua_human:70.00K
used_memory_scripts:328
used_memory_scripts_human:328B
number_of_cached_scripts:1
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.11
allocator_frag_bytes:3121964544
allocator_rss_ratio:1.01
allocator_rss_bytes:325935104
rss_overhead_ratio:0.95
rss_overhead_bytes:-1639018496
mem_fragmentation_ratio:1.07
mem_fragmentation_bytes:1809122864
mem_not_counted_for_evict:2757318
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:3897320
mem_aof_buffer:2757318
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
# Persistence
loading:0
rdb_changes_since_last_save:243071
rdb_bgsave_in_progress:0
rdb_last_save_time:1586512576
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:176
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:1788354560
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:178
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:2703163392
aof_current_size:8631893606
aof_base_size:5356395646
aof_pending_rewrite:0
aof_buffer_length:2429702
aof_rewrite_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:0
# Stats
total_connections_received:87692211
total_commands_processed:7791229207
instantaneous_ops_per_sec:1028
total_net_input_bytes:1077885718213
total_net_output_bytes:2681176262270
instantaneous_input_kbps:2133.42
instantaneous_output_kbps:2011.65
rejected_connections:0
sync_full:22
sync_partial_ok:0
sync_partial_err:0
expired_keys:142669625
expired_stale_perc:11.97
expired_time_cap_reached_count:82
evicted_keys:0
keyspace_hits:1387798381
keyspace_misses:887619181
pubsub_channels:5
pubsub_patterns:2
latest_fork_usec:962972
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
# Replication
role:master
connected_slaves:0
master_replid:99a1639f42e9c3571e6091dc1614c02928f035b6
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:16371962814
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:16370914239
repl_backlog_histlen:1048576
# CPU
used_cpu_sys:222905.590899
used_cpu_user:151825.739929
used_cpu_sys_children:54078.278411
used_cpu_user_children:646080.199531
# Commandstats
cmdstat_select:calls=45601501,usec=76476636,usec_per_call=1.68
cmdstat_hincrby:calls=114617,usec=1597791,usec_per_call=13.94
cmdstat_zcount:calls=30609021,usec=242421868,usec_per_call=7.92
cmdstat_sdiffstore:calls=108525,usec=898780,usec_per_call=8.28
cmdstat_ltrim:calls=2117253,usec=2460414,usec_per_call=1.16
cmdstat_zrevrange:calls=2448424,usec=173150047,usec_per_call=70.72
cmdstat_hget:calls=146243930,usec=162148170,usec_per_call=1.11
cmdstat_watch:calls=11306,usec=47320,usec_per_call=4.19
cmdstat_mget:calls=295110331,usec=2942047537,usec_per_call=9.97
cmdstat_lpop:calls=10306127,usec=71787733,usec_per_call=6.97
cmdstat_zrangebyscore:calls=2802256,usec=72854894,usec_per_call=26.00
cmdstat_hmget:calls=754198571,usec=7021491099,usec_per_call=9.31
cmdstat_zremrangebyscore:calls=1150,usec=30060,usec_per_call=26.14
cmdstat_zrevrangebyscore:calls=2833668,usec=28241822,usec_per_call=9.97
cmdstat_exists:calls=5695429,usec=18698282,usec_per_call=3.28
cmdstat_getset:calls=1601,usec=6540,usec_per_call=4.08
cmdstat_keys:calls=2,usec=5145170,usec_per_call=2572585.00
cmdstat_hsetnx:calls=498445,usec=7340867,usec_per_call=14.73
cmdstat_flushdb:calls=5,usec=229658,usec_per_call=45931.60
cmdstat_sinter:calls=377158,usec=2210683,usec_per_call=5.86
cmdstat_zrem:calls=30359,usec=294056,usec_per_call=9.69
cmdstat_info:calls=261359,usec=27197045,usec_per_call=104.06
cmdstat_zincrby:calls=1562022,usec=53976783,usec_per_call=34.56
cmdstat_setex:calls=147322842,usec=769683329,usec_per_call=5.22
cmdstat_lpush:calls=2117734,usec=13259225,usec_per_call=6.26
cmdstat_echo:calls=992150,usec=1663922,usec_per_call=1.68
cmdstat_zrange:calls=2862296,usec=18485071,usec_per_call=6.46
cmdstat_slowlog:calls=130621,usec=95455616,usec_per_call=730.78
cmdstat_srem:calls=16840,usec=263764,usec_per_call=15.66
cmdstat_hkeys:calls=19416907,usec=71077665,usec_per_call=3.66
cmdstat_subscribe:calls=5,usec=157,usec_per_call=31.40
cmdstat_lrange:calls=12890615,usec=44372622,usec_per_call=3.44
cmdstat_unlink:calls=69130799,usec=353512311,usec_per_call=5.11
cmdstat_zscore:calls=1487,usec=21486,usec_per_call=14.45
cmdstat_hmset:calls=6019131,usec=290506821,usec_per_call=48.26
cmdstat_hlen:calls=76189588,usec=225516714,usec_per_call=2.96
cmdstat_setnx:calls=67493744,usec=645648492,usec_per_call=9.57
cmdstat_del:calls=1793260,usec=19592714,usec_per_call=10.93
cmdstat_zrank:calls=5604512,usec=34637707,usec_per_call=6.18
cmdstat_publish:calls=129049537,usec=265845144,usec_per_call=2.06
cmdstat_hgetall:calls=12766520,usec=1681172507,usec_per_call=131.69
cmdstat_hexists:calls=32000919,usec=72139859,usec_per_call=2.25
cmdstat_setbit:calls=73135718,usec=370674165,usec_per_call=5.07
cmdstat_llen:calls=92058,usec=199195,usec_per_call=2.16
cmdstat_sync:calls=22,usec=14624808,usec_per_call=664764.00
cmdstat_hdel:calls=5219930,usec=21077864,usec_per_call=4.04
cmdstat_smembers:calls=18094983,usec=134743447,usec_per_call=7.45
cmdstat_get:calls=145819631,usec=590210355,usec_per_call=4.05
cmdstat_set:calls=17662970,usec=136907399,usec_per_call=7.75
cmdstat_incr:calls=13440015,usec=116818158,usec_per_call=8.69
cmdstat_zadd:calls=133776149,usec=1838648949,usec_per_call=13.74
cmdstat_lrem:calls=2117253,usec=28154382,usec_per_call=13.30
cmdstat_command:calls=2,usec=1896,usec_per_call=948.00
cmdstat_eval:calls=139738,usec=10092178,usec_per_call=72.22
cmdstat_bitcount:calls=362994,usec=13898826,usec_per_call=38.29
cmdstat_rpush:calls=1620519,usec=16811332,usec_per_call=10.37
cmdstat_scard:calls=116454338,usec=351996535,usec_per_call=3.02
cmdstat_sismember:calls=110743119,usec=212351411,usec_per_call=1.92
cmdstat_psubscribe:calls=5,usec=22,usec_per_call=4.40
cmdstat_zcard:calls=2920770,usec=14005317,usec_per_call=4.80
cmdstat_zremrangebyrank:calls=134318838,usec=597480725,usec_per_call=4.45
cmdstat_multi:calls=35495436,usec=13635416,usec_per_call=0.38
cmdstat_hset:calls=1017888,usec=12570319,usec_per_call=12.35
cmdstat_scan:calls=4976040155,usec=82653796133,usec_per_call=16.61
cmdstat_incrby:calls=3261147,usec=21321273,usec_per_call=6.54
cmdstat_sadd:calls=3894473,usec=15280208,usec_per_call=3.92
cmdstat_type:calls=1567452,usec=4187875,usec_per_call=2.67
cmdstat_expire:calls=70368046,usec=118414278,usec_per_call=1.68
cmdstat_exec:calls=35495435,usec=4598426678,usec_per_call=129.55
cmdstat_config:calls=261251,usec=4249763,usec_per_call=16.27
# Cluster
cluster_enabled:0
# Keyspace
db0:keys=7600819,expires=2417312,avg_ttl=845729512
db9:keys=6875,expires=0,avg_ttl=0
db10:keys=10985,expires=10981,avg_ttl=110871
------ CLIENT LIST OUTPUT ------
id=87615875 addr=127.0.0.1:54364 fd=134 name= age=1328 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87608640 addr=127.0.0.1:59202 fd=121 name= age=1451 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87605061 addr=127.0.0.1:34284 fd=101 name= age=1511 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set
id=87606752 addr=127.0.0.1:45956 fd=127 name= age=1482 idle=20 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=47360096 addr=127.0.0.1:42496 fd=131 name= age=923168 idle=0 flags=P db=0 sub=5 psub=1 multi=-1 qbuf=0 qbuf-free=0 obl=16341 oll=41 omem=840664 events=r cmd=psubscribe
id=47360097 addr=127.0.0.1:42502 fd=133 name= age=923168 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hdel
id=87607519 addr=127.0.0.1:51216 fd=92 name= age=1470 idle=27 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=86802732 addr=127.0.0.1:39770 fd=123 name= age=17800 idle=17800 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87601500 addr=127.0.0.1:36830 fd=12 name= age=1575 idle=33 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87692212 addr=127.0.0.1:45332 fd=113 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sismember
id=87692213 addr=127.0.0.1:45334 fd=118 name= age=0 idle=0 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=select
id=87601111 addr=127.0.0.1:34392 fd=18 name= age=1581 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87600719 addr=127.0.0.1:59938 fd=28 name= age=1588 idle=39 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87603180 addr=127.0.0.1:49002 fd=40 name= age=1545 idle=25 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=83554131 addr=127.0.0.1:49206 fd=20 name= age=92119 idle=92119 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87692203 addr=127.0.0.1:45258 fd=120 name= age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=setnx
id=87692204 addr=127.0.0.1:45260 fd=122 name= age=1 idle=1 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=mget
id=87692205 addr=127.0.0.1:45266 fd=135 name= age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exists
id=87386420 addr=127.0.0.1:48190 fd=126 name= age=5178 idle=5178 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87604011 addr=127.0.0.1:54884 fd=50 name= age=1529 idle=21 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87602207 addr=127.0.0.1:41924 fd=32 name= age=1562 idle=6 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=85289127 addr=127.0.0.1:49480 fd=111 name= age=61819 idle=61819 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=85294149 addr=127.0.0.1:58380 fd=110 name= age=61712 idle=61049 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87604954 addr=127.0.0.1:33434 fd=57 name= age=1513 idle=6 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set
id=87603902 addr=127.0.0.1:54162 fd=10 name= age=1531 idle=14 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87601483 addr=127.0.0.1:36688 fd=47 name= age=1576 idle=7 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87605365 addr=127.0.0.1:36446 fd=74 name= age=1506 idle=43 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87601314 addr=127.0.0.1:35648 fd=15 name= age=1577 idle=90 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87604607 addr=127.0.0.1:59222 fd=97 name= age=1518 idle=29 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87605125 addr=127.0.0.1:34774 fd=80 name= age=1510 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set
id=87601309 addr=127.0.0.1:35614 fd=41 name= age=1578 idle=54 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87601104 addr=127.0.0.1:34344 fd=31 name= age=1581 idle=6 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87605044 addr=127.0.0.1:34134 fd=95 name= age=1511 idle=6 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set
id=84388751 addr=127.0.0.1:37886 fd=103 name= age=78686 idle=46411 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=86174039 addr=127.0.0.1:44454 fd=109 name= age=41164 idle=40428 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=85323094 addr=127.0.0.1:46426 fd=112 name= age=61101 idle=61101 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87603982 addr=127.0.0.1:54716 fd=46 name= age=1529 idle=16 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87602326 addr=127.0.0.1:42822 fd=16 name= age=1560 idle=125 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87604932 addr=127.0.0.1:33254 fd=89 name= age=1513 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set
id=87604081 addr=127.0.0.1:55338 fd=61 name= age=1528 idle=91 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=84300869 addr=127.0.0.1:58056 fd=73 name= age=80239 idle=80239 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87602180 addr=127.0.0.1:41602 fd=25 name= age=1563 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=85774640 addr=127.0.0.1:55428 fd=78 name= age=51436 idle=7268 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=83674736 addr=127.0.0.1:56928 fd=38 name= age=89890 idle=61109 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87273885 addr=127.0.0.1:55530 fd=117 name= age=7150 idle=7150 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=84850685 addr=127.0.0.1:59232 fd=87 name= age=70503 idle=42799 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=85288223 addr=127.0.0.1:42796 fd=55 name= age=61838 idle=61838 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=85293098 addr=127.0.0.1:51482 fd=85 name= age=61735 idle=60082 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87601274 addr=127.0.0.1:35412 fd=9 name= age=1578 idle=25 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87606627 addr=127.0.0.1:45214 fd=77 name= age=1484 idle=21 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87604918 addr=127.0.0.1:33162 fd=70 name= age=1513 idle=1509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hset
id=87605109 addr=127.0.0.1:34650 fd=58 name= age=1510 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lpop
id=87605032 addr=127.0.0.1:34012 fd=82 name= age=1511 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set
id=87604919 addr=127.0.0.1:33168 fd=75 name= age=1513 idle=1510 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=echo
id=87604556 addr=127.0.0.1:58926 fd=62 name= age=1518 idle=32 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=85337568 addr=127.0.0.1:40812 fd=107 name= age=60800 idle=59175 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=86070098 addr=127.0.0.1:49796 fd=124 name= age=44245 idle=2120 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87603104 addr=127.0.0.1:48568 fd=48 name= age=1545 idle=77 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87601452 addr=127.0.0.1:36492 fd=37 name= age=1576 idle=40 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87611281 addr=127.0.0.1:49546 fd=138 name= age=1405 idle=9 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=85290530 addr=127.0.0.1:60608 fd=14 name= age=61788 idle=61788 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=85270651 addr=127.0.0.1:51012 fd=53 name= age=62228 idle=1074 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=84313398 addr=127.0.0.1:43246 fd=19 name= age=79985 idle=78027 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=47355968 addr=127.0.0.1:59278 fd=147 name= age=923243 idle=0 flags=P db=0 sub=5 psub=1 multi=-1 qbuf=0 qbuf-free=0 obl=16341 oll=41 omem=840664 events=r cmd=psubscribe
id=87603018 addr=127.0.0.1:47886 fd=44 name= age=1547 idle=31 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87605083 addr=127.0.0.1:34484 fd=98 name= age=1510 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set
id=47355969 addr=127.0.0.1:59280 fd=148 name= age=923243 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hdel
id=87601258 addr=127.0.0.1:35298 fd=24 name= age=1578 idle=30 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=83691427 addr=127.0.0.1:60544 fd=36 name= age=89607 idle=65823 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87605008 addr=127.0.0.1:33848 fd=90 name= age=1512 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set
id=85270039 addr=127.0.0.1:46872 fd=108 name= age=62240 idle=5717 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87602134 addr=127.0.0.1:41310 fd=13 name= age=1564 idle=23 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=35518874 addr=127.0.0.1:53858 fd=26 name= age=1197179 idle=14 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=slowlog
id=87601157 addr=127.0.0.1:34648 fd=51 name= age=1580 idle=55 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87604896 addr=127.0.0.1:33030 fd=69 name= age=1514 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lpop
id=87603003 addr=127.0.0.1:47810 fd=30 name= age=1547 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87602777 addr=127.0.0.1:46146 fd=22 name= age=1551 idle=11 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87603676 addr=127.0.0.1:52464 fd=60 name= age=1536 idle=79 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=85291913 addr=127.0.0.1:42492 fd=105 name= age=61759 idle=61759 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87603595 addr=127.0.0.1:51984 fd=42 name= age=1537 idle=28 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87609987 addr=127.0.0.1:40564 fd=130 name= age=1427 idle=21 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=85323364 addr=127.0.0.1:48530 fd=116 name= age=61094 idle=41222 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87604220 addr=127.0.0.1:56382 fd=11 name= age=1525 idle=43 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=83696732 addr=127.0.0.1:41680 fd=59 name= age=89516 idle=88507 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87604287 addr=127.0.0.1:56916 fd=72 name= age=1524 idle=14 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=85322453 addr=127.0.0.1:41742 fd=86 name= age=61114 idle=40687 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=35518875 addr=127.0.0.1:53862 fd=34 name= age=1197179 idle=14 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=type
id=87609752 addr=127.0.0.1:38782 fd=119 name= age=1432 idle=131 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87604991 addr=127.0.0.1:33694 fd=52 name= age=1512 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=set
id=87602690 addr=127.0.0.1:45526 fd=27 name= age=1553 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87605077 addr=127.0.0.1:34418 fd=99 name= age=1511 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lpop
id=35518876 addr=127.0.0.1:53866 fd=76 name= age=1197179 idle=14 flags=N db=9 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=type
id=87603390 addr=127.0.0.1:50448 fd=56 name= age=1542 idle=76 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=85299135 addr=127.0.0.1:38270 fd=83 name= age=61603 idle=50357 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87609867 addr=127.0.0.1:39626 fd=125 name= age=1430 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87604453 addr=127.0.0.1:58140 fd=39 name= age=1520 idle=11 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87604879 addr=127.0.0.1:32900 fd=65 name= age=1514 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lpop
id=87605138 addr=127.0.0.1:34860 fd=100 name= age=1509 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=lpop
id=85153397 addr=127.0.0.1:44528 fd=94 name= age=64846 idle=61724 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87691925 addr=127.0.0.1:43196 fd=102 name= age=6 idle=0 flags=x db=0 sub=0 psub=0 multi=80573 qbuf=14 qbuf-free=32754 obl=16384 oll=6 omem=123024 events=r cmd=exec
id=87691926 addr=127.0.0.1:43198 fd=106 name= age=6 idle=6 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=select
id=83959106 addr=127.0.0.1:53816 fd=35 name= age=84918 idle=84918 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=84881726 addr=127.0.0.1:59638 fd=91 name= age=69969 idle=46764 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87602098 addr=127.0.0.1:41106 fd=8 name= age=1564 idle=9 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87604964 addr=127.0.0.1:33540 fd=45 name= age=1512 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=set
id=87602577 addr=127.0.0.1:44664 fd=23 name= age=1555 idle=131 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87604269 addr=127.0.0.1:56814 fd=33 name= age=1524 idle=56 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=35518877 addr=127.0.0.1:53872 fd=81 name= age=1197179 idle=14 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=type
id=87601797 addr=127.0.0.1:38932 fd=21 name= age=1570 idle=29 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87604870 addr=127.0.0.1:32834 fd=84 name= age=1514 idle=1513 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hset
id=84241028 addr=127.0.0.1:38288 fd=43 name= age=81314 idle=61830 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87604784 addr=127.0.0.1:60430 fd=64 name= age=1515 idle=44 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87602853 addr=127.0.0.1:46734 fd=17 name= age=1550 idle=131 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=87602854 addr=127.0.0.1:46744 fd=29 name= age=1550 idle=29 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
------ CURRENT CLIENT INFO ------
id=87691925 addr=127.0.0.1:43196 fd=102 name= age=6 idle=0 flags=x db=0 sub=0 psub=0 multi=80573 qbuf=14 qbuf-free=32754 obl=16384 oll=6 omem=123024 events=r cmd=exec
argv[0]: 'ZREMRANGEBYRANK'
argv[1]: 'prf:666633:feed'
argv[2]: '0'
argv[3]: '-41'
31416:M 10 Apr 2020 12:57:25.213 # key 'prf:666633:feed' found in DB containing the following object:
31416:M 10 Apr 2020 12:57:25.213 # Object type: 3
31416:M 10 Apr 2020 12:57:25.213 # Object encoding: 7
31416:M 10 Apr 2020 12:57:25.213 # Object refcount: 1
31416:M 10 Apr 2020 12:57:25.213 # Sorted set size: 41
31416:M 10 Apr 2020 12:57:25.213 # Skiplist level: 2
------ REGISTERS ------
31416:M 10 Apr 2020 12:57:25.213 #
RAX:8300000000000000 RBX:00007f79634541e0
RCX:0000000000000002 RDX:0000000000000000
RDI:00007f79634541e0 RSI:00007f78f8965301
RBP:0000000000000001 RSP:00007ffe63a8d8d8
R8 :00007f78f8965321 R9 :0000000000000002
R10:8300000000000000 R11:0000000000000001
R12:00007f78f8965301 R13:0000000000000001
R14:8300000000000000 R15:0000000000000001
RIP:000055767bdb88cd EFL:0000000000010286
CSGSFS:002b000000000033
31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8e7) -> 00007f7921a7fa20
31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8e6) -> 00007f7dc49c1730
31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8e5) -> 0000000000001000
31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8e4) -> d4ee6e5059e5cc00
31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8e3) -> 00007f7dc38009c0
31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8e2) -> 000055767be3fbb9
31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8e1) -> 00007ffe63a8d99f
31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8e0) -> 0000000000000028
31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8df) -> 00007f7dc49c1730
31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8de) -> 00007f7caa7a1300
31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8dd) -> 00007f7c4e97d600
31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8dc) -> 00007f7c8fd01300
31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8db) -> 00007f7c8fd01300
31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8da) -> 00007ffe63a8d8f0
31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8d9) -> 00007f7d153e42a0
31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8d8) -> 000055767bdb90a6
------ FAST MEMORY TEST ------
31416:M 10 Apr 2020 12:57:25.214 # Bio thread for job type #0 terminated
31416:M 10 Apr 2020 12:57:25.214 # Bio thread for job type #1 terminated
31416:M 10 Apr 2020 12:57:25.215 # Bio thread for job type #2 terminated
*** Preparing to test memory region 55767c0f2000 (2248704 bytes)
*** Preparing to test memory region 55767cc54000 (196608 bytes)
*** Preparing to test memory region 7f70c1600000 (55647928320 bytes)
*** Preparing to test memory region 7f7db64d3000 (83886080 bytes)
*** Preparing to test memory region 7f7dbb4d4000 (8388608 bytes)
*** Preparing to test memory region 7f7dbbcd5000 (8388608 bytes)
*** Preparing to test memory region 7f7dbc4d6000 (8388608 bytes)
*** Preparing to test memory region 7f7dc3200000 (8388608 bytes)
*** Preparing to test memory region 7f7dc3e80000 (20480 bytes)
*** Preparing to test memory region 7f7dc409d000 (16384 bytes)
*** Preparing to test memory region 7f7dc49c1000 (20480 bytes)
*** Preparing to test memory region 7f7dc49ce000 (8192 bytes)
*** Preparing to test memory region 7f7dc49d2000 (4096 bytes)
.O.O.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
------ DUMPING CODE AROUND EIP ------
Symbol: zslDeleteNode (base: 0x55767bdb8860)
Module: /usr/bin/redis-server 127.0.0.1:6379 (base 0x55767bd51000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x55767bdb8860 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
31416:M 10 Apr 2020 13:03:28.902 # dump of function (hexdump of 237 bytes):
448b4f184585c97e574989f031c9eb124883c10148836820014983c0104139c97e3e4863c148c1e004480304ca4839701875dd4d8b50204c8b58204883c1014983c0104f8d5413ff4c8950204d8b50084139c94c8950187fc90f1f8000000000488b46184885c0746f488b5610488950104183f9017e45418d51ff488b374863c24883c00148c1e00448837c060800741feb290f1f4400008d4aff4863c14883c00148c1e00448837c060800751a89ca83fa0175e3c747180100000048836f1001c3660f1f44000089571848836f1001c30f1f8000000000488b461048894708eb8f0f1f4000662e0f1f840000
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Comment From: antirez
Thanks, may you send us your Redis executable please?
On Fri, Apr 10, 2020, 12:07 Oleg Topchiy notifications@github.com wrote:
=== REDIS BUG REPORT START: Cut & paste starting from here === 31416:M 10 Apr 2020 12:57:25.209 # Redis 5.0.8 crashed by signal: 11 31416:M 10 Apr 2020 12:57:25.209 # Crashed running the instruction at: 0x55767bdb88cd 31416:M 10 Apr 2020 12:57:25.209 # Accessing address: (nil) 31416:M 10 Apr 2020 12:57:25.209 # Failed assertion:
( :0) ------ STACK TRACE ------ EIP: /usr/bin/redis-server 127.0.0.1:6379(zslDeleteNode+0x6d)[0x55767bdb88cd]
Backtrace: /usr/bin/redis-server 127.0.0.1:6379(logStackTrace+0x41)[0x55767bdd3c81] /usr/bin/redis-server 127.0.0.1:6379(sigsegvHandler+0xc9)[0x55767bdd4409] /lib64/libpthread.so.0(+0xf5f0)[0x7f7dc3e945f0] /usr/bin/redis-server 127.0.0.1:6379(zslDeleteNode+0x6d)[0x55767bdb88cd] /usr/bin/redis-server 127.0.0.1:6379(zslDeleteRangeByRank+0xd6)[0x55767bdb90a6] /usr/bin/redis-server 127.0.0.1:6379(zremrangeGenericCommand+0x35c)[0x55767bdbbd6c] /usr/bin/redis-server 127.0.0.1:6379(call+0x9d)[0x55767bd8de3d] /usr/bin/redis-server 127.0.0.1:6379(execCommand+0xd2)[0x55767bdd1082] /usr/bin/redis-server 127.0.0.1:6379(call+0x9d)[0x55767bd8de3d] /usr/bin/redis-server 127.0.0.1:6379(processCommand+0x39f)[0x55767bd8f1af] /usr/bin/redis-server 127.0.0.1:6379(processInputBuffer+0x19d)[0x55767bd9ee7d] /usr/bin/redis-server 127.0.0.1:6379(aeProcessEvents+0x2d8)[0x55767bd879b8] /usr/bin/redis-server 127.0.0.1:6379(aeMain+0x2b)[0x55767bd87c9b] /usr/bin/redis-server 127.0.0.1:6379(main+0x4e3)[0x55767bd848b3] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f7dc3ad9505] /usr/bin/redis-server 127.0.0.1:6379(+0x33b31)[0x55767bd84b31]
------ INFO OUTPUT ------
Server
redis_version:5.0.8 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:18ca9d7ea1c3e9cc redis_mode:standalone os:Linux 5.3.11-x86_64-linode131 x86_64 arch_bits:64 multiplexing_api:epoll atomicvar_api:atomic-builtin gcc_version:4.8.5 process_id:31416 run_id:dda7308c2de2756ecaf58c007843ac30445352b4 tcp_port:6379 uptime_in_seconds:1959552 uptime_in_days:22 hz:10 configured_hz:10 lru_clock:9454341 executable:/usr/bin/redis-server config_file:/etc/redis.conf
Clients
connected_clients:114 client_recent_max_input_buffer:152179 client_recent_max_output_buffer:0 blocked_clients:0
Memory
used_memory:27444284088 used_memory_human:25.56G used_memory_rss:29248839680 used_memory_rss_human:27.24G used_memory_peak:27733438728 used_memory_peak_human:25.83G used_memory_peak_perc:98.96% used_memory_overhead:471725718 used_memory_startup:791408 used_memory_dataset:26972558370 used_memory_dataset_perc:98.28% allocator_allocated:27439958528 allocator_active:30561923072 allocator_resident:30887858176 total_system_memory:67430465536 total_system_memory_human:62.80G used_memory_lua:71680 used_memory_lua_human:70.00K used_memory_scripts:328 used_memory_scripts_human:328B number_of_cached_scripts:1 maxmemory:0 maxmemory_human:0B maxmemory_policy:noeviction allocator_frag_ratio:1.11 allocator_frag_bytes:3121964544 allocator_rss_ratio:1.01 allocator_rss_bytes:325935104 rss_overhead_ratio:0.95 rss_overhead_bytes:-1639018496 mem_fragmentation_ratio:1.07 mem_fragmentation_bytes:1809122864 mem_not_counted_for_evict:2757318 mem_replication_backlog:0 mem_clients_slaves:0 mem_clients_normal:3897320 mem_aof_buffer:2757318 mem_allocator:jemalloc-5.1.0 active_defrag_running:0 lazyfree_pending_objects:0
Persistence
loading:0 rdb_changes_since_last_save:243071 rdb_bgsave_in_progress:0 rdb_last_save_time:1586512576 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:176 rdb_current_bgsave_time_sec:-1 rdb_last_cow_size:1788354560 aof_enabled:1 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:178 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok aof_last_cow_size:2703163392 aof_current_size:8631893606 aof_base_size:5356395646 aof_pending_rewrite:0 aof_buffer_length:2429702 aof_rewrite_buffer_length:0 aof_pending_bio_fsync:0 aof_delayed_fsync:0
Stats
total_connections_received:87692211 total_commands_processed:7791229207 instantaneous_ops_per_sec:1028 total_net_input_bytes:1077885718213 total_net_output_bytes:2681176262270 instantaneous_input_kbps:2133.42 instantaneous_output_kbps:2011.65 rejected_connections:0 sync_full:22 sync_partial_ok:0 sync_partial_err:0 expired_keys:142669625 expired_stale_perc:11.97 expired_time_cap_reached_count:82 evicted_keys:0 keyspace_hits:1387798381 keyspace_misses:887619181 pubsub_channels:5 pubsub_patterns:2 latest_fork_usec:962972 migrate_cached_sockets:0 slave_expires_tracked_keys:0 active_defrag_hits:0 active_defrag_misses:0 active_defrag_key_hits:0 active_defrag_key_misses:0
Replication
role:master connected_slaves:0 master_replid:99a1639f42e9c3571e6091dc1614c02928f035b6 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:16371962814 second_repl_offset:-1 repl_backlog_active:0 repl_backlog_size:1048576 repl_backlog_first_byte_offset:16370914239 repl_backlog_histlen:1048576
CPU
used_cpu_sys:222905.590899 used_cpu_user:151825.739929 used_cpu_sys_children:54078.278411 used_cpu_user_children:646080.199531
Commandstats
cmdstat_select:calls=45601501,usec=76476636,usec_per_call=1.68 cmdstat_hincrby:calls=114617,usec=1597791,usec_per_call=13.94 cmdstat_zcount:calls=30609021,usec=242421868,usec_per_call=7.92 cmdstat_sdiffstore:calls=108525,usec=898780,usec_per_call=8.28 cmdstat_ltrim:calls=2117253,usec=2460414,usec_per_call=1.16 cmdstat_zrevrange:calls=2448424,usec=173150047,usec_per_call=70.72 cmdstat_hget:calls=146243930,usec=162148170,usec_per_call=1.11 cmdstat_watch:calls=11306,usec=47320,usec_per_call=4.19 cmdstat_mget:calls=295110331,usec=2942047537,usec_per_call=9.97 cmdstat_lpop:calls=10306127,usec=71787733,usec_per_call=6.97 cmdstat_zrangebyscore:calls=2802256,usec=72854894,usec_per_call=26.00 cmdstat_hmget:calls=754198571,usec=7021491099,usec_per_call=9.31 cmdstat_zremrangebyscore:calls=1150,usec=30060,usec_per_call=26.14 cmdstat_zrevrangebyscore:calls=2833668,usec=28241822,usec_per_call=9.97 cmdstat_exists:calls=5695429,usec=18698282,usec_per_call=3.28 cmdstat_getset:calls=1601,usec=6540,usec_per_call=4.08 cmdstat_keys:calls=2,usec=5145170,usec_per_call=2572585.00 cmdstat_hsetnx:calls=498445,usec=7340867,usec_per_call=14.73 cmdstat_flushdb:calls=5,usec=229658,usec_per_call=45931.60 cmdstat_sinter:calls=377158,usec=2210683,usec_per_call=5.86 cmdstat_zrem:calls=30359,usec=294056,usec_per_call=9.69 cmdstat_info:calls=261359,usec=27197045,usec_per_call=104.06 cmdstat_zincrby:calls=1562022,usec=53976783,usec_per_call=34.56 cmdstat_setex:calls=147322842,usec=769683329,usec_per_call=5.22 cmdstat_lpush:calls=2117734,usec=13259225,usec_per_call=6.26 cmdstat_echo:calls=992150,usec=1663922,usec_per_call=1.68 cmdstat_zrange:calls=2862296,usec=18485071,usec_per_call=6.46 cmdstat_slowlog:calls=130621,usec=95455616,usec_per_call=730.78 cmdstat_srem:calls=16840,usec=263764,usec_per_call=15.66 cmdstat_hkeys:calls=19416907,usec=71077665,usec_per_call=3.66 cmdstat_subscribe:calls=5,usec=157,usec_per_call=31.40 cmdstat_lrange:calls=12890615,usec=44372622,usec_per_call=3.44 cmdstat_unlink:calls=69130799,usec=353512311,usec_per_call=5.11 cmdstat_zscore:calls=1487,usec=21486,usec_per_call=14.45 cmdstat_hmset:calls=6019131,usec=290506821,usec_per_call=48.26 cmdstat_hlen:calls=76189588,usec=225516714,usec_per_call=2.96 cmdstat_setnx:calls=67493744,usec=645648492,usec_per_call=9.57 cmdstat_del:calls=1793260,usec=19592714,usec_per_call=10.93 cmdstat_zrank:calls=5604512,usec=34637707,usec_per_call=6.18 cmdstat_publish:calls=129049537,usec=265845144,usec_per_call=2.06 cmdstat_hgetall:calls=12766520,usec=1681172507,usec_per_call=131.69 cmdstat_hexists:calls=32000919,usec=72139859,usec_per_call=2.25 cmdstat_setbit:calls=73135718,usec=370674165,usec_per_call=5.07 cmdstat_llen:calls=92058,usec=199195,usec_per_call=2.16 cmdstat_sync:calls=22,usec=14624808,usec_per_call=664764.00 cmdstat_hdel:calls=5219930,usec=21077864,usec_per_call=4.04 cmdstat_smembers:calls=18094983,usec=134743447,usec_per_call=7.45 cmdstat_get:calls=145819631,usec=590210355,usec_per_call=4.05 cmdstat_set:calls=17662970,usec=136907399,usec_per_call=7.75 cmdstat_incr:calls=13440015,usec=116818158,usec_per_call=8.69 cmdstat_zadd:calls=133776149,usec=1838648949,usec_per_call=13.74 cmdstat_lrem:calls=2117253,usec=28154382,usec_per_call=13.30 cmdstat_command:calls=2,usec=1896,usec_per_call=948.00 cmdstat_eval:calls=139738,usec=10092178,usec_per_call=72.22 cmdstat_bitcount:calls=362994,usec=13898826,usec_per_call=38.29 cmdstat_rpush:calls=1620519,usec=16811332,usec_per_call=10.37 cmdstat_scard:calls=116454338,usec=351996535,usec_per_call=3.02 cmdstat_sismember:calls=110743119,usec=212351411,usec_per_call=1.92 cmdstat_psubscribe:calls=5,usec=22,usec_per_call=4.40 cmdstat_zcard:calls=2920770,usec=14005317,usec_per_call=4.80 cmdstat_zremrangebyrank:calls=134318838,usec=597480725,usec_per_call=4.45 cmdstat_multi:calls=35495436,usec=13635416,usec_per_call=0.38 cmdstat_hset:calls=1017888,usec=12570319,usec_per_call=12.35 cmdstat_scan:calls=4976040155,usec=82653796133,usec_per_call=16.61 cmdstat_incrby:calls=3261147,usec=21321273,usec_per_call=6.54 cmdstat_sadd:calls=3894473,usec=15280208,usec_per_call=3.92 cmdstat_type:calls=1567452,usec=4187875,usec_per_call=2.67 cmdstat_expire:calls=70368046,usec=118414278,usec_per_call=1.68 cmdstat_exec:calls=35495435,usec=4598426678,usec_per_call=129.55 cmdstat_config:calls=261251,usec=4249763,usec_per_call=16.27
Cluster
cluster_enabled:0
Keyspace
db0:keys=7600819,expires=2417312,avg_ttl=845729512 db9:keys=6875,expires=0,avg_ttl=0 db10:keys=10985,expires=10981,avg_ttl=110871
------ CLIENT LIST OUTPUT ------ id=87615875 addr=127.0.0.1:54364 fd=134 name= age=1328 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87608640 addr=127.0.0.1:59202 fd=121 name= age=1451 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87605061 addr=127.0.0.1:34284 fd=101 name= age=1511 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set id=87606752 addr=127.0.0.1:45956 fd=127 name= age=1482 idle=20 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=47360096 addr=127.0.0.1:42496 fd=131 name= age=923168 idle=0 flags=P db=0 sub=5 psub=1 multi=-1 qbuf=0 qbuf-free=0 obl=16341 oll=41 omem=840664 events=r cmd=psubscribe id=47360097 addr=127.0.0.1:42502 fd=133 name= age=923168 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hdel id=87607519 addr=127.0.0.1:51216 fd=92 name= age=1470 idle=27 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=86802732 addr=127.0.0.1:39770 fd=123 name= age=17800 idle=17800 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87601500 addr=127.0.0.1:36830 fd=12 name= age=1575 idle=33 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87692212 addr=127.0.0.1:45332 fd=113 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sismember id=87692213 addr=127.0.0.1:45334 fd=118 name= age=0 idle=0 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=select id=87601111 addr=127.0.0.1:34392 fd=18 name= age=1581 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87600719 addr=127.0.0.1:59938 fd=28 name= age=1588 idle=39 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87603180 addr=127.0.0.1:49002 fd=40 name= age=1545 idle=25 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=83554131 addr=127.0.0.1:49206 fd=20 name= age=92119 idle=92119 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87692203 addr=127.0.0.1:45258 fd=120 name= age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=setnx id=87692204 addr=127.0.0.1:45260 fd=122 name= age=1 idle=1 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=mget id=87692205 addr=127.0.0.1:45266 fd=135 name= age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=exists id=87386420 addr=127.0.0.1:48190 fd=126 name= age=5178 idle=5178 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87604011 addr=127.0.0.1:54884 fd=50 name= age=1529 idle=21 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87602207 addr=127.0.0.1:41924 fd=32 name= age=1562 idle=6 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=85289127 addr=127.0.0.1:49480 fd=111 name= age=61819 idle=61819 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=85294149 addr=127.0.0.1:58380 fd=110 name= age=61712 idle=61049 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87604954 addr=127.0.0.1:33434 fd=57 name= age=1513 idle=6 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set id=87603902 addr=127.0.0.1:54162 fd=10 name= age=1531 idle=14 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87601483 addr=127.0.0.1:36688 fd=47 name= age=1576 idle=7 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87605365 addr=127.0.0.1:36446 fd=74 name= age=1506 idle=43 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87601314 addr=127.0.0.1:35648 fd=15 name= age=1577 idle=90 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87604607 addr=127.0.0.1:59222 fd=97 name= age=1518 idle=29 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87605125 addr=127.0.0.1:34774 fd=80 name= age=1510 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set id=87601309 addr=127.0.0.1:35614 fd=41 name= age=1578 idle=54 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87601104 addr=127.0.0.1:34344 fd=31 name= age=1581 idle=6 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87605044 addr=127.0.0.1:34134 fd=95 name= age=1511 idle=6 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set id=84388751 addr=127.0.0.1:37886 fd=103 name= age=78686 idle=46411 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=86174039 addr=127.0.0.1:44454 fd=109 name= age=41164 idle=40428 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=85323094 addr=127.0.0.1:46426 fd=112 name= age=61101 idle=61101 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87603982 addr=127.0.0.1:54716 fd=46 name= age=1529 idle=16 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87602326 addr=127.0.0.1:42822 fd=16 name= age=1560 idle=125 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87604932 addr=127.0.0.1:33254 fd=89 name= age=1513 idle=3 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set id=87604081 addr=127.0.0.1:55338 fd=61 name= age=1528 idle=91 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=84300869 addr=127.0.0.1:58056 fd=73 name= age=80239 idle=80239 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87602180 addr=127.0.0.1:41602 fd=25 name= age=1563 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=85774640 addr=127.0.0.1:55428 fd=78 name= age=51436 idle=7268 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=83674736 addr=127.0.0.1:56928 fd=38 name= age=89890 idle=61109 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87273885 addr=127.0.0.1:55530 fd=117 name= age=7150 idle=7150 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=84850685 addr=127.0.0.1:59232 fd=87 name= age=70503 idle=42799 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=85288223 addr=127.0.0.1:42796 fd=55 name= age=61838 idle=61838 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=85293098 addr=127.0.0.1:51482 fd=85 name= age=61735 idle=60082 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87601274 addr=127.0.0.1:35412 fd=9 name= age=1578 idle=25 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87606627 addr=127.0.0.1:45214 fd=77 name= age=1484 idle=21 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87604918 addr=127.0.0.1:33162 fd=70 name= age=1513 idle=1509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hset id=87605109 addr=127.0.0.1:34650 fd=58 name= age=1510 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lpop id=87605032 addr=127.0.0.1:34012 fd=82 name= age=1511 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set id=87604919 addr=127.0.0.1:33168 fd=75 name= age=1513 idle=1510 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=echo id=87604556 addr=127.0.0.1:58926 fd=62 name= age=1518 idle=32 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=85337568 addr=127.0.0.1:40812 fd=107 name= age=60800 idle=59175 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=86070098 addr=127.0.0.1:49796 fd=124 name= age=44245 idle=2120 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87603104 addr=127.0.0.1:48568 fd=48 name= age=1545 idle=77 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87601452 addr=127.0.0.1:36492 fd=37 name= age=1576 idle=40 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87611281 addr=127.0.0.1:49546 fd=138 name= age=1405 idle=9 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=85290530 addr=127.0.0.1:60608 fd=14 name= age=61788 idle=61788 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=85270651 addr=127.0.0.1:51012 fd=53 name= age=62228 idle=1074 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=84313398 addr=127.0.0.1:43246 fd=19 name= age=79985 idle=78027 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=47355968 addr=127.0.0.1:59278 fd=147 name= age=923243 idle=0 flags=P db=0 sub=5 psub=1 multi=-1 qbuf=0 qbuf-free=0 obl=16341 oll=41 omem=840664 events=r cmd=psubscribe id=87603018 addr=127.0.0.1:47886 fd=44 name= age=1547 idle=31 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87605083 addr=127.0.0.1:34484 fd=98 name= age=1510 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set id=47355969 addr=127.0.0.1:59280 fd=148 name= age=923243 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hdel id=87601258 addr=127.0.0.1:35298 fd=24 name= age=1578 idle=30 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=83691427 addr=127.0.0.1:60544 fd=36 name= age=89607 idle=65823 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87605008 addr=127.0.0.1:33848 fd=90 name= age=1512 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=set id=85270039 addr=127.0.0.1:46872 fd=108 name= age=62240 idle=5717 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87602134 addr=127.0.0.1:41310 fd=13 name= age=1564 idle=23 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=35518874 addr=127.0.0.1:53858 fd=26 name= age=1197179 idle=14 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=slowlog id=87601157 addr=127.0.0.1:34648 fd=51 name= age=1580 idle=55 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87604896 addr=127.0.0.1:33030 fd=69 name= age=1514 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lpop id=87603003 addr=127.0.0.1:47810 fd=30 name= age=1547 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87602777 addr=127.0.0.1:46146 fd=22 name= age=1551 idle=11 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87603676 addr=127.0.0.1:52464 fd=60 name= age=1536 idle=79 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=85291913 addr=127.0.0.1:42492 fd=105 name= age=61759 idle=61759 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87603595 addr=127.0.0.1:51984 fd=42 name= age=1537 idle=28 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87609987 addr=127.0.0.1:40564 fd=130 name= age=1427 idle=21 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=85323364 addr=127.0.0.1:48530 fd=116 name= age=61094 idle=41222 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87604220 addr=127.0.0.1:56382 fd=11 name= age=1525 idle=43 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=83696732 addr=127.0.0.1:41680 fd=59 name= age=89516 idle=88507 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87604287 addr=127.0.0.1:56916 fd=72 name= age=1524 idle=14 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=85322453 addr=127.0.0.1:41742 fd=86 name= age=61114 idle=40687 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=35518875 addr=127.0.0.1:53862 fd=34 name= age=1197179 idle=14 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=type id=87609752 addr=127.0.0.1:38782 fd=119 name= age=1432 idle=131 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87604991 addr=127.0.0.1:33694 fd=52 name= age=1512 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=set id=87602690 addr=127.0.0.1:45526 fd=27 name= age=1553 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=rpush id=87605077 addr=127.0.0.1:34418 fd=99 name= age=1511 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lpop id=35518876 addr=127.0.0.1:53866 fd=76 name= age=1197179 idle=14 flags=N db=9 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=type id=87603390 addr=127.0.0.1:50448 fd=56 name= age=1542 idle=76 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=85299135 addr=127.0.0.1:38270 fd=83 name= age=61603 idle=50357 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87609867 addr=127.0.0.1:39626 fd=125 name= age=1430 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87604453 addr=127.0.0.1:58140 fd=39 name= age=1520 idle=11 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87604879 addr=127.0.0.1:32900 fd=65 name= age=1514 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lpop id=87605138 addr=127.0.0.1:34860 fd=100 name= age=1509 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=lpop id=85153397 addr=127.0.0.1:44528 fd=94 name= age=64846 idle=61724 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87691925 addr=127.0.0.1:43196 fd=102 name= age=6 idle=0 flags=x db=0 sub=0 psub=0 multi=80573 qbuf=14 qbuf-free=32754 obl=16384 oll=6 omem=123024 events=r cmd=exec id=87691926 addr=127.0.0.1:43198 fd=106 name= age=6 idle=6 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=select id=83959106 addr=127.0.0.1:53816 fd=35 name= age=84918 idle=84918 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=84881726 addr=127.0.0.1:59638 fd=91 name= age=69969 idle=46764 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87602098 addr=127.0.0.1:41106 fd=8 name= age=1564 idle=9 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87604964 addr=127.0.0.1:33540 fd=45 name= age=1512 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=set id=87602577 addr=127.0.0.1:44664 fd=23 name= age=1555 idle=131 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87604269 addr=127.0.0.1:56814 fd=33 name= age=1524 idle=56 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=35518877 addr=127.0.0.1:53872 fd=81 name= age=1197179 idle=14 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=type id=87601797 addr=127.0.0.1:38932 fd=21 name= age=1570 idle=29 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87604870 addr=127.0.0.1:32834 fd=84 name= age=1514 idle=1513 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hset id=84241028 addr=127.0.0.1:38288 fd=43 name= age=81314 idle=61830 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87604784 addr=127.0.0.1:60430 fd=64 name= age=1515 idle=44 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87602853 addr=127.0.0.1:46734 fd=17 name= age=1550 idle=131 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush id=87602854 addr=127.0.0.1:46744 fd=29 name= age=1550 idle=29 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
------ CURRENT CLIENT INFO ------ id=87691925 addr=127.0.0.1:43196 fd=102 name= age=6 idle=0 flags=x db=0 sub=0 psub=0 multi=80573 qbuf=14 qbuf-free=32754 obl=16384 oll=6 omem=123024 events=r cmd=exec argv[0]: 'ZREMRANGEBYRANK' argv[1]: 'prf:666633:feed' argv[2]: '0' argv[3]: '-41' 31416:M 10 Apr 2020 12:57:25.213 # key 'prf:666633:feed' found in DB containing the following object: 31416:M 10 Apr 2020 12:57:25.213 # Object type: 3 31416:M 10 Apr 2020 12:57:25.213 # Object encoding: 7 31416:M 10 Apr 2020 12:57:25.213 # Object refcount: 1 31416:M 10 Apr 2020 12:57:25.213 # Sorted set size: 41 31416:M 10 Apr 2020 12:57:25.213 # Skiplist level: 2
------ REGISTERS ------ 31416:M 10 Apr 2020 12:57:25.213 # RAX:8300000000000000 RBX:00007f79634541e0 RCX:0000000000000002 RDX:0000000000000000 RDI:00007f79634541e0 RSI:00007f78f8965301 RBP:0000000000000001 RSP:00007ffe63a8d8d8 R8 :00007f78f8965321 R9 :0000000000000002 R10:8300000000000000 R11:0000000000000001 R12:00007f78f8965301 R13:0000000000000001 R14:8300000000000000 R15:0000000000000001 RIP:000055767bdb88cd EFL:0000000000010286 CSGSFS:002b000000000033 31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8e7) -> 00007f7921a7fa20 31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8e6) -> 00007f7dc49c1730 31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8e5) -> 0000000000001000 31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8e4) -> d4ee6e5059e5cc00 31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8e3) -> 00007f7dc38009c0 31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8e2) -> 000055767be3fbb9 31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8e1) -> 00007ffe63a8d99f 31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8e0) -> 0000000000000028 31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8df) -> 00007f7dc49c1730 31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8de) -> 00007f7caa7a1300 31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8dd) -> 00007f7c4e97d600 31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8dc) -> 00007f7c8fd01300 31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8db) -> 00007f7c8fd01300 31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8da) -> 00007ffe63a8d8f0 31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8d9) -> 00007f7d153e42a0 31416:M 10 Apr 2020 12:57:25.213 # (00007ffe63a8d8d8) -> 000055767bdb90a6
------ FAST MEMORY TEST ------ 31416:M 10 Apr 2020 12:57:25.214 # Bio thread for job type #0 terminated 31416:M 10 Apr 2020 12:57:25.214 # Bio thread for job type #1 terminated 31416:M 10 Apr 2020 12:57:25.215 # Bio thread for job type #2 terminated *** Preparing to test memory region 55767c0f2000 (2248704 bytes) *** Preparing to test memory region 55767cc54000 (196608 bytes) *** Preparing to test memory region 7f70c1600000 (55647928320 bytes) *** Preparing to test memory region 7f7db64d3000 (83886080 bytes) *** Preparing to test memory region 7f7dbb4d4000 (8388608 bytes) *** Preparing to test memory region 7f7dbbcd5000 (8388608 bytes) *** Preparing to test memory region 7f7dbc4d6000 (8388608 bytes) *** Preparing to test memory region 7f7dc3200000 (8388608 bytes) *** Preparing to test memory region 7f7dc3e80000 (20480 bytes) *** Preparing to test memory region 7f7dc409d000 (16384 bytes) *** Preparing to test memory region 7f7dc49c1000 (20480 bytes) *** Preparing to test memory region 7f7dc49ce000 (8192 bytes) *** Preparing to test memory region 7f7dc49d2000 (4096 bytes) .O.O.O.O.O.O.O.O.O.O.O.O.O Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
------ DUMPING CODE AROUND EIP ------ Symbol: zslDeleteNode (base: 0x55767bdb8860) Module: /usr/bin/redis-server 127.0.0.1:6379 (base 0x55767bd51000) $ xxd -r -p /tmp/dump.hex /tmp/dump.bin $ objdump --adjust-vma=0x55767bdb8860 -D -b binary -m i386:x86-64 /tmp/dump.bin
31416:M 10 Apr 2020 13:03:28.902 # dump of function (hexdump of 237 bytes): 448b4f184585c97e574989f031c9eb124883c10148836820014983c0104139c97e3e4863c148c1e004480304ca4839701875dd4d8b50204c8b58204883c1014983c0104f8d5413ff4c8950204d8b50084139c94c8950187fc90f1f8000000000488b46184885c0746f488b5610488950104183f9017e45418d51ff488b374863c24883c00148c1e00448837c060800741feb290f1f4400008d4aff4863c14883c00148c1e00448837c060800751a89ca83fa0175e3c747180100000048836f1001c3660f1f44000089571848836f1001c30f1f8000000000488b461048894708eb8f0f1f4000662e0f1f840000
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/antirez/redis/issues/7080, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAQAYEH4FJZYH3RUVEBGKLRL3VVZANCNFSM4MFLNX4A .
Comment From: thedotedge
Of course! redis-server.zip
Comment From: antirez
Thanks!
On Fri, Apr 10, 2020 at 12:28 PM Oleg Topchiy notifications@github.com wrote:
Of course! redis-server.zip https://github.com/antirez/redis/files/4461107/redis-server.zip
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/antirez/redis/issues/7080#issuecomment-611975672, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAQAYDGYN6AARNFH6ODI3DRL3YDJANCNFSM4MFLNX4A .
-- Salvatore 'antirez' Sanfilippo open source developer - Redis Labs https://redislabs.com
"If a system is to have conceptual integrity, someone must control the concepts." — Fred Brooks, "The Mythical Man-Month", 1975.
Comment From: itamarhaber
Hello @xvester - this appears to be a different issue as it involves RediSearch, please report it at https://github.com/redisearch/redisearch/issues
Thanks!
Comment From: antirez
@xvester Hi, your bug is unrelated, please submit to RediSearch module. I'm removing your comment.
Comment From: itamarhaber
@antirez lol - 4 seconds apart :)
Comment From: antirez
@oranagra please could you give me an hand with your low level programming skills? I disassembled the code, and this is what I see, but I may be very wrong:
So, the code is
if (x->level[0].forward) {
x->level[0].forward->backward = x->backward;
} else {
zsl->tail = x->backward;
}
And from disassembling I see:
void zslDeleteNode(zskiplist *zsl, zskiplistNode *x, ...
x is in %rsi (second argument):
55767bdb88c0: 48 8b 46 18 mov 0x18(%rsi),%rax
(this moved x->level[0].forward, 24 bytes offset, into %rax)
55767bdb88c4: 48 85 c0 test %rax,%rax
55767bdb88c7: 74 6f je 0x55767bdb8938
(is x->level[0].forward zero? Jump. But we don't jump since the
program crashed in 55767bdb88cd).
55767bdb88c9: 48 8b 56 10 mov 0x10(%rsi),%rdx
(Move x->backward into %rdx)
55767bdb88cd: 48 89 50 10 mov %rdx,0x10(%rax)
(Move %rdx, that is x->backward, into x->level[0].forward->backward)
However the program crashed because of a NULL deference. If the test immediately before saw that x->level[0].forward is not NULL, hence we entered the branch, then how it is possible that x->level[0].forward->backward is a NULL deference? I guess I'm just missing something.
Comment From: oranagra
@antirez it looks like x->level[0].forward = -16 i.e. the jump doesn't skip the code because it's not 0, and doing x[16] produces a NULL access.
Comment From: oranagra
that's actually not in line with the registers dump that says:
RAX:8300000000000000
i'll try to disassemble myself too and see if i find something else.
Comment From: antirez
@oranagra maybe it's not what you said (-16) because of the register, but still, smart idea! Didn't realize at all the pointer could be negative (or the equivalent unsigned, that is pointer_max-16), and wrap to 0 after the structure access offset.
Comment From: antirez
@thedotedge we see strange things, please could you open a ticket with Linode and ask if they are using ECC memory in this host, and if they are observing any hardware failure there?
Comment From: lcavallaro
Chipping in quickly @antirez and @oranagra - I didn't look at the dump, but if x->level[0].forward = -16, then -16 is 0xfffffffffffffff0 and the "mov %rdx,0x10(%rax)" simply does 0x10 + 0xfffffffffffffff0, which yields 0. So that insn is effectively trying to write to the address 0, hence the SIGSEGV due to a NULL access.
Now, why %rax gets -16, I don't know :-)
Comment From: oranagra
@lcavallaro isn't that repeating what i said? or am i missing something? anyway, the registers dump from the signal handler say that RAX is not 0xfffffffffffffff0
RAX:8300000000000000 RBX:00007f79634541e0
RCX:0000000000000002 RDX:0000000000000000
RDI:00007f79634541e0 RSI:00007f78f8965301
Comment From: lcavallaro
Oops, @oranagra I missed the RAX: 8300000000000000 thing and started with your x->level[0].forward = -16. Yes, that you already summed it up.
Comment From: oranagra
well, i don't see any jumps to into instructions between the test and that crash, so the only thing i can think of is that the crash long is lying.
@thedotedge can you have a look at your /usr/include/sys/ucontext.h and make sure RAX is indeed at offset 13 (or upload the file).
i seriously doubt this is this, since i don't see any other registers with values that may explain it. so maybe the print of the crash address si_addr is wrong?
Comment From: antirez
@oranagra hypothesis: there is a bug in the Hypervisor or the hardware the Hypervisor uses, so the process was suspended by the scheduler, and when the process was given CPU time again, the registers were not restored correctly.
Comment From: oranagra
@antirez that sounds extremely unlikely, and also, it still doesn't explain the contradiction between si_addr and the register dump. To me it sounds more likely that we have an issue in the crash report code (we usually don't look at the register dump, so that's more likely to be buggy). If that's true, it means we might be looking at two bugs, one that caused the corruption and crash, and one in the crash report.. We don't have any way to proceed about the possible corruption, but we can test and debug the crash report code. (although that might not be a high priority)
Comment From: thedotedge
@antirez
Yes, we utilize ECC memory. We are not currently aware of any hardware failure issues on your host.
Comment From: thedotedge
@oranagra yes, I think it's 13:
#ifdef __USE_GNU
/* Number of each register in the `gregset_t' array. */
enum
{
REG_R8 = 0,
# define REG_R8 REG_R8
REG_R9,
# define REG_R9 REG_R9
REG_R10,
# define REG_R10 REG_R10
REG_R11,
# define REG_R11 REG_R11
REG_R12,
# define REG_R12 REG_R12
REG_R13,
# define REG_R13 REG_R13
REG_R14,
# define REG_R14 REG_R14
REG_R15,
# define REG_R15 REG_R15
REG_RDI,
# define REG_RDI REG_RDI
REG_RSI,
# define REG_RSI REG_RSI
REG_RBP,
# define REG_RBP REG_RBP
REG_RBX,
# define REG_RBX REG_RBX
REG_RDX,
# define REG_RDX REG_RDX
REG_RAX,
# define REG_RAX REG_RAX
REG_RCX,
# define REG_RCX REG_RCX
REG_RSP,
# define REG_RSP REG_RSP
REG_RIP,
# define REG_RIP REG_RIP
REG_EFL,
# define REG_EFL REG_EFL
REG_CSGSFS, /* Actually short cs, gs, fs, __pad0. */
# define REG_CSGSFS REG_CSGSFS
REG_ERR,
# define REG_ERR REG_ERR
REG_TRAPNO,
# define REG_TRAPNO REG_TRAPNO
REG_OLDMASK,
# define REG_OLDMASK REG_OLDMASK
REG_CR2
# define REG_CR2 REG_CR2
};
#endif
Comment From: antirez
Update: I just modified Redis in order to trap a very similar error and generated a crash log. Basically in the crash log everything is correct but the address that caused the crash. So basically we can trust the registers here. It is very likely that in this case Redis crashed because RAX was set to 8300000000000000. How that's possible, and why now the crash code is no longer able to tell us the real address of the crash, is a different story.
Comment From: antirez
Btw this is what I changed in Redis in order to modify the behavior and trigger a similar crash:
diff --git a/src/t_zset.c b/src/t_zset.c
index 56ea396..c77dfab 100644
--- a/src/t_zset.c
+++ b/src/t_zset.c
@@ -459,6 +459,7 @@ unsigned long zslDeleteRangeByRank(zskiplist *zsl, unsigned int start, unsigned
x = x->level[0].forward;
while (x && traversed <= end) {
zskiplistNode *next = x->level[0].forward;
+ x->level[0].forward = (void*)0xaabbccddeeff0000UL;
zslDeleteNode(zsl,x,update);
dictDelete(dict,x->ele);
zslFreeNode(x);
And this is what I see in the crash:
11468:M 15 Apr 2020 08:46:28.743 # Redis 5.0.8 crashed by signal: 11
11468:M 15 Apr 2020 08:46:28.743 # Crashed running the instruction at: 0x45a6fd
11468:M 15 Apr 2020 08:46:28.743 # Accessing address: (nil)
11468:M 15 Apr 2020 08:46:28.743 # Failed assertion: <no assertion failed> (<no file>:0)
...
EIP:
./redis-server *:6379(zslDeleteNode+0x6d)[0x45a6fd]
Backtrace:
./redis-server *:6379(logStackTrace+0x45)[0x474545]
./redis-server *:6379(sigsegvHandler+0xb9)[0x474d09]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f3b61cf2390]
./redis-server *:6379(zslDeleteNode+0x6d)[0x45a6fd]
./redis-server *:6379(zslDeleteRangeByRank+0xde)[0x45af1e]
./redis-server *:6379(zremrangeGenericCommand+0x1ed)[0x45d90d]
./redis-server *:6379(call+0xab)[0x43026b]
./redis-server *:6379(processCommand+0x377)[0x430947]
./redis-server *:6379(processInputBuffer+0x175)[0x440db5]
./redis-server *:6379(aeProcessEvents+0x36d)[0x429e7d]
./redis-server *:6379(aeMain+0x2b)[0x42a0ab]
./redis-server *:6379(main+0x4bf)[0x426c4f]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f3b61937830]
./redis-server *:6379(_start+0x29)[0x426ec9]
...
RAX:aabbccddeeff0000 RBX:00007f3b6140e100
RCX:0000000000000003 RDX:00007f3b6143a140
RDI:00007f3b6140e100 RSI:00007f3b6143a168
RBP:00007f3b6143a190 RSP:00007fff3de83f48
R8 :0000000000000003 R9 :00007f3b6143a1b8
R10:aabbccddeeff0000 R11:0000000000000001
R12:0000000000000003 R13:00007f3b6143a168
R14:0000000000000000 R15:0000000000000003
RIP:000000000045a6fd EFL:0000000000010286
CSGSFS:0000000000000033
Note how, despite the different GCC version used, the crash offset inside the function is still +6D. So we can trust everything in the log but the crash address, that we know anyway.
Comment From: antirez
After further investigations I found that apparently the kernel only populates si_addr when it is within certain ranges. When it maps to certain out of range addresses, NULL is given instead. So there is apparently no bug in Redis itself (about the address reporting).
Comment From: oranagra
@antirez that's interesting, can you please post a reference to that info? So no bug in the crash log.. But there's still a bug that caused that corruption ( x->level[0].forward == 0x8300000000000000)
Comment From: antirez
@oranagra unfortunately I can't find any reference. I just tried: in my modification above, if I change the address to an address that is more in range of the process maps, it gets displayed correctly. Also if you use 0xfffff or just "DEBUG SEGFAULT", it works as expected, and so forth. But setting it to an intermediate very high address like 0xaabbccddaabbccdd fails to show anything useful. No idea why but the Linux man pages about sigaction report such limitation in a very generic way, sayings that it is possible that the informations in siginfo are not populate correctly in certain situations. Probably the sigaction crash "code" could be different in such cases, depending on the exact CPU fault produced trying to access such addresses.
About the bug, I just checked the history of Redis crashes, there are two related to sorted sets, one is clearly related to trying to load a broken sorted set. The other one is more obscure. Either we have a very very hard to trigger bug that only 1 or 2 users each year are able to trigger, or there is a bug in a non sorted-set place of the code (for instance quicklist is a potential source) that corrupts the heap, and will show up randomly everywhere. Or there is a jemalloc bug that we are able to hit from time to time since Redis is probably one of the most popular application running on Jemalloc, with such numbers of GB allocated (like in this case) and where bugs are publicly reported.
In the past I did a lot of fuzzing in the sorted set code, and was never able to find anything, but is impossible to exclude that Redis could have a bug somewhere causing this. At the same time it is almost impossible to act more on this bug. I performed a fuzzing activity with similar ranges and ZREMRANGEBYRANK, kept it running 24h, found nothing unfortunately.
At this point if @thedotedge happens to have a use case that is able to trigger this bug again, I hope he could help us in the future. Please if you incur again in such bug, report it here, compile Redis without jemalloc, to just use system malloc for the next run, and then report if it is crashing anyway or not.
Comment From: thedotedge
Not sure if it's related, but there was another crash today on the same instance:
=== REDIS BUG REPORT START: Cut & paste starting from here ===
27935:M 16 Apr 2020 19:48:34.965 # Redis 5.0.8 crashed by signal: 11
27935:M 16 Apr 2020 19:48:34.965 # Crashed running the instruction at: 0x55ff2e5af070
27935:M 16 Apr 2020 19:48:34.965 # Accessing address: 0x11
27935:M 16 Apr 2020 19:48:34.965 # Failed assertion: <no assertion failed> (<no file>:0)
------ STACK TRACE ------
EIP:
/usr/bin/redis-server 127.0.0.1:6379(dictNext+0x60)[0x55ff2e5af070]
Backtrace:
/usr/bin/redis-server 127.0.0.1:6379(logStackTrace+0x41)[0x55ff2e5f8c81]
/usr/bin/redis-server 127.0.0.1:6379(sigsegvHandler+0xc9)[0x55ff2e5f9409]
/lib64/libpthread.so.0(+0xf5f0)[0x7f4e189245f0]
/usr/bin/redis-server 127.0.0.1:6379(dictNext+0x60)[0x55ff2e5af070]
/usr/bin/redis-server 127.0.0.1:6379(rdbSaveObject+0x1f8)[0x55ff2e5d3568]
/usr/bin/redis-server 127.0.0.1:6379(rdbSaveKeyValuePair+0x114)[0x55ff2e5d3d34]
/usr/bin/redis-server 127.0.0.1:6379(rdbSaveRio+0x24a)[0x55ff2e5d439a]
/usr/bin/redis-server 127.0.0.1:6379(rdbSave+0xb4)[0x55ff2e5d48a4]
/usr/bin/redis-server 127.0.0.1:6379(prepareForShutdown+0x115)[0x55ff2e5b3385]
/usr/bin/redis-server 127.0.0.1:6379(shutdownCommand+0x3e)[0x55ff2e5c99ae]
/usr/bin/redis-server 127.0.0.1:6379(call+0x9d)[0x55ff2e5b2e3d]
/usr/bin/redis-server 127.0.0.1:6379(processCommand+0x39f)[0x55ff2e5b41af]
/usr/bin/redis-server 127.0.0.1:6379(processInputBuffer+0x19d)[0x55ff2e5c3e7d]
/usr/bin/redis-server 127.0.0.1:6379(aeProcessEvents+0x2d8)[0x55ff2e5ac9b8]
/usr/bin/redis-server 127.0.0.1:6379(aeMain+0x2b)[0x55ff2e5acc9b]
/usr/bin/redis-server 127.0.0.1:6379(main+0x4e3)[0x55ff2e5a98b3]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f4e18569505]
/usr/bin/redis-server 127.0.0.1:6379(+0x33b31)[0x55ff2e5a9b31]
------ INFO OUTPUT ------
# Server
redis_version:5.0.8
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:18ca9d7ea1c3e9cc
redis_mode:standalone
os:Linux 5.3.11-x86_64-linode131 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:4.8.5
process_id:27935
run_id:8be1ab0712a14d978d75c834e930eb1d717158ae
tcp_port:6379
uptime_in_seconds:542630
uptime_in_days:6
hz:10
configured_hz:10
lru_clock:9997401
executable:/usr/bin/redis-server
config_file:/etc/redis.conf
# Clients
connected_clients:125
client_recent_max_input_buffer:4
client_recent_max_output_buffer:0
blocked_clients:0
# Memory
used_memory:26342537760
used_memory_human:24.53G
used_memory_rss:26302595072
used_memory_rss_human:24.50G
used_memory_peak:26433592280
used_memory_peak_human:24.62G
used_memory_peak_perc:99.66%
used_memory_overhead:481481718
used_memory_startup:791408
used_memory_dataset:25861056042
used_memory_dataset_perc:98.18%
allocator_allocated:26342766128
allocator_active:27465166848
allocator_resident:27725070336
total_system_memory:67430465536
total_system_memory_human:62.80G
used_memory_lua:35840
used_memory_lua_human:35.00K
used_memory_scripts:328
used_memory_scripts_human:328B
number_of_cached_scripts:1
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.04
allocator_frag_bytes:1122400720
allocator_rss_ratio:1.01
allocator_rss_bytes:259903488
rss_overhead_ratio:0.95
rss_overhead_bytes:-1422475264
mem_fragmentation_ratio:1.00
mem_fragmentation_bytes:-40043480
mem_not_counted_for_evict:2920
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:3196734
mem_aof_buffer:2920
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
# Persistence
loading:0
rdb_changes_since_last_save:363584
rdb_bgsave_in_progress:1
rdb_last_save_time:1587054551
rdb_last_bgsave_status:err
rdb_last_bgsave_time_sec:199
rdb_current_bgsave_time_sec:76
rdb_last_cow_size:303042560
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:179
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:776445952
aof_current_size:11022791758
aof_base_size:5584884044
aof_pending_rewrite:0
aof_buffer_length:0
aof_rewrite_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:0
# Stats
total_connections_received:23586095
total_commands_processed:2109585952
instantaneous_ops_per_sec:400
total_net_input_bytes:286814558087
total_net_output_bytes:724402663290
instantaneous_input_kbps:51.70
instantaneous_output_kbps:96.42
rejected_connections:0
sync_full:6
sync_partial_ok:0
sync_partial_err:0
expired_keys:34463015
expired_stale_perc:1.00
expired_time_cap_reached_count:39
evicted_keys:0
keyspace_hits:413405050
keyspace_misses:231922232
pubsub_channels:5
pubsub_patterns:2
latest_fork_usec:779774
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
# Replication
role:master
connected_slaves:0
master_replid:31aec9cdf43f3b4a49800a41d013d99e9c0c6460
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:3973234867
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:3972186292
repl_backlog_histlen:1048576
# CPU
used_cpu_sys:60009.564314
used_cpu_user:40872.210204
used_cpu_sys_children:14389.025534
used_cpu_user_children:172306.491472
# Commandstats
cmdstat_smembers:calls=4068550,usec=33270308,usec_per_call=8.18
cmdstat_slowlog:calls=36164,usec=24716857,usec_per_call=683.47
cmdstat_zrangebyscore:calls=559514,usec=13145249,usec_per_call=23.49
cmdstat_bitcount:calls=114972,usec=4271965,usec_per_call=37.16
cmdstat_zrevrangebyscore:calls=569540,usec=4659182,usec_per_call=8.18
cmdstat_zscore:calls=339,usec=4457,usec_per_call=13.15
cmdstat_eval:calls=44153,usec=3157036,usec_per_call=71.50
cmdstat_zincrby:calls=444430,usec=14884068,usec_per_call=33.49
cmdstat_lrange:calls=3888584,usec=13407344,usec_per_call=3.45
cmdstat_srem:calls=1494,usec=16443,usec_per_call=11.01
cmdstat_select:calls=12080067,usec=19297183,usec_per_call=1.60
cmdstat_hkeys:calls=5172082,usec=18527196,usec_per_call=3.58
cmdstat_scard:calls=32147474,usec=96584412,usec_per_call=3.00
cmdstat_discard:calls=7,usec=17,usec_per_call=2.43
cmdstat_sdiffstore:calls=34962,usec=295649,usec_per_call=8.46
cmdstat_hset:calls=235494,usec=3083562,usec_per_call=13.09
cmdstat_hlen:calls=23497846,usec=68951668,usec_per_call=2.93
cmdstat_hmset:calls=1373757,usec=69329061,usec_per_call=50.47
cmdstat_rpush:calls=488441,usec=4972381,usec_per_call=10.18
cmdstat_hdel:calls=1314860,usec=5869077,usec_per_call=4.46
cmdstat_hmget:calls=236914933,usec=2035039118,usec_per_call=8.59
cmdstat_zremrangebyscore:calls=350,usec=8797,usec_per_call=25.13
cmdstat_hget:calls=41183721,usec=48371421,usec_per_call=1.17
cmdstat_exists:calls=1691621,usec=5616824,usec_per_call=3.32
cmdstat_getset:calls=175,usec=953,usec_per_call=5.45
cmdstat_type:calls=433968,usec=1144818,usec_per_call=2.64
cmdstat_mget:calls=73390183,usec=771773663,usec_per_call=10.52
cmdstat_hexists:calls=7039943,usec=17777579,usec_per_call=2.53
cmdstat_zremrangebyrank:calls=39982736,usec=169831703,usec_per_call=4.25
cmdstat_scan:calls=1296611482,usec=21751667674,usec_per_call=16.78
cmdstat_zadd:calls=39779027,usec=539835609,usec_per_call=13.57
cmdstat_lrem:calls=619658,usec=8445101,usec_per_call=13.63
cmdstat_zrank:calls=1119028,usec=6033363,usec_per_call=5.39
cmdstat_exec:calls=9660037,usec=1324685738,usec_per_call=137.13
cmdstat_setbit:calls=22192419,usec=107909729,usec_per_call=4.86
cmdstat_setnx:calls=16228104,usec=164562461,usec_per_call=10.14
cmdstat_hincrby:calls=31454,usec=440704,usec_per_call=14.01
cmdstat_echo:calls=332286,usec=550814,usec_per_call=1.66
cmdstat_zrevrange:calls=724838,usec=49846868,usec_per_call=68.77
cmdstat_set:calls=5340705,usec=41361464,usec_per_call=7.74
cmdstat_subscribe:calls=3,usec=93,usec_per_call=31.00
cmdstat_setex:calls=36209550,usec=194356621,usec_per_call=5.37
cmdstat_del:calls=541930,usec=5540574,usec_per_call=10.22
cmdstat_zrem:calls=8523,usec=80078,usec_per_call=9.40
cmdstat_unlink:calls=16461364,usec=83183638,usec_per_call=5.05
cmdstat_sync:calls=6,usec=5185056,usec_per_call=864176.00
cmdstat_hsetnx:calls=152328,usec=2256220,usec_per_call=14.81
cmdstat_config:calls=72339,usec=1160682,usec_per_call=16.05
cmdstat_flushdb:calls=1,usec=35383,usec_per_call=35383.00
cmdstat_zcard:calls=836524,usec=4096005,usec_per_call=4.90
cmdstat_ltrim:calls=619658,usec=726011,usec_per_call=1.17
cmdstat_hgetall:calls=3853623,usec=385706821,usec_per_call=100.09
cmdstat_publish:calls=39420774,usec=77936946,usec_per_call=1.98
cmdstat_llen:calls=24164,usec=56014,usec_per_call=2.32
cmdstat_sismember:calls=37090413,usec=78085869,usec_per_call=2.11
cmdstat_expire:calls=21138882,usec=34888392,usec_per_call=1.65
cmdstat_lpop:calls=3529341,usec=25622186,usec_per_call=7.26
cmdstat_incrby:calls=980065,usec=6739767,usec_per_call=6.88
cmdstat_psubscribe:calls=3,usec=15,usec_per_call=5.00
cmdstat_command:calls=1,usec=1681,usec_per_call=1681.00
cmdstat_lpush:calls=619753,usec=3832886,usec_per_call=6.18
cmdstat_incr:calls=3904031,usec=35261680,usec_per_call=9.03
cmdstat_get:calls=40127182,usec=163899291,usec_per_call=4.08
cmdstat_zrange:calls=576282,usec=2801484,usec_per_call=4.86
cmdstat_multi:calls=9660044,usec=3569655,usec_per_call=0.37
cmdstat_info:calls=72422,usec=7517445,usec_per_call=103.80
cmdstat_zcount:calls=8947404,usec=70556628,usec_per_call=7.89
cmdstat_sadd:calls=1236584,usec=4982273,usec_per_call=4.03
cmdstat_watch:calls=2217,usec=9541,usec_per_call=4.30
# Cluster
cluster_enabled:0
# Keyspace
db0:keys=7862220,expires=2581256,avg_ttl=842732797
db9:keys=6947,expires=0,avg_ttl=0
db10:keys=443,expires=439,avg_ttl=403125
------ CLIENT LIST OUTPUT ------
id=23586082 addr=127.0.0.1:42942 fd=63 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=setnx
id=23586084 addr=127.0.0.1:42946 fd=127 name= age=0 idle=0 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=mget
id=23586085 addr=127.0.0.1:42952 fd=129 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=setnx
id=23586086 addr=127.0.0.1:42954 fd=130 name= age=0 idle=0 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=mget
id=23586087 addr=127.0.0.1:42958 fd=131 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=setnx
id=23586088 addr=127.0.0.1:42964 fd=132 name= age=0 idle=0 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=mget
id=23586089 addr=127.0.0.1:42974 fd=99 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=setnx
id=23586090 addr=127.0.0.1:42976 fd=68 name= age=0 idle=0 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=mget
id=23586091 addr=127.0.0.1:42994 fd=101 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=setnx
id=23586092 addr=127.0.0.1:42996 fd=105 name= age=0 idle=0 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=mget
id=23586093 addr=127.0.0.1:43010 fd=111 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=23586094 addr=127.0.0.1:43012 fd=133 name= age=0 idle=0 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=select
id=23586060 addr=127.0.0.1:42832 fd=21 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=setnx
id=23586061 addr=127.0.0.1:42834 fd=38 name= age=0 idle=0 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=mget
id=23586062 addr=127.0.0.1:42838 fd=56 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=setnx
id=23586063 addr=127.0.0.1:42840 fd=65 name= age=0 idle=0 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=mget
id=23585920 addr=127.0.0.1:41854 fd=126 name= age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=23585922 addr=127.0.0.1:41878 fd=128 name= age=1 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=19868069 addr=127.0.0.1:34580 fd=78 name= age=79177 idle=504 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
id=19863769 addr=127.0.0.1:58616 fd=49 name= age=79247 idle=711 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19861562 addr=127.0.0.1:42072 fd=23 name= age=79285 idle=787 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=21571858 addr=127.0.0.1:34208 fd=102 name= age=32347 idle=32347 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=13002 addr=127.0.0.1:49196 fd=27 name= age=542459 idle=14 flags=N db=9 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=type
id=19864826 addr=127.0.0.1:38450 fd=54 name= age=79230 idle=670 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=22548433 addr=127.0.0.1:53042 fd=107 name= age=15557 idle=7789 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=13003 addr=127.0.0.1:49200 fd=33 name= age=542459 idle=14 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=type
id=19866369 addr=127.0.0.1:50176 fd=74 name= age=79205 idle=728 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19863508 addr=127.0.0.1:56676 fd=45 name= age=79251 idle=663 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19861655 addr=127.0.0.1:42710 fd=10 name= age=79284 idle=758 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19862631 addr=127.0.0.1:49946 fd=28 name= age=79267 idle=715 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19864925 addr=127.0.0.1:39240 fd=55 name= age=79228 idle=264 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
id=19838415 addr=127.0.0.1:58930 fd=77 name= age=79685 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=lpop
id=19838168 addr=127.0.0.1:57216 fd=47 name= age=79689 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=lpop
id=19864818 addr=127.0.0.1:38386 fd=8 name= age=79230 idle=673 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19838150 addr=127.0.0.1:57106 fd=64 name= age=79689 idle=79686 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=echo
id=19838147 addr=127.0.0.1:57088 fd=66 name= age=79689 idle=79685 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=hset
id=37 addr=127.0.0.1:44664 fd=35 name= age=542629 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=hdel
id=465 addr=127.0.0.1:47624 fd=42 name= age=542624 idle=14 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=slowlog
id=19865522 addr=127.0.0.1:43862 fd=69 name= age=79218 idle=24 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
id=19863617 addr=127.0.0.1:57450 fd=31 name= age=79250 idle=711 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19838388 addr=127.0.0.1:58788 fd=51 name= age=79685 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=lpop
id=19862546 addr=127.0.0.1:49256 fd=37 name= age=79269 idle=754 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=36 addr=127.0.0.1:44662 fd=34 name= age=542629 idle=660 flags=P db=0 sub=5 psub=1 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=psubscribe
id=19867944 addr=127.0.0.1:33592 fd=92 name= age=79179 idle=663 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19864504 addr=127.0.0.1:36000 fd=58 name= age=79235 idle=698 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19862886 addr=127.0.0.1:52030 fd=9 name= age=79262 idle=264 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
id=19862777 addr=127.0.0.1:51240 fd=22 name= age=79264 idle=594 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
id=22018445 addr=127.0.0.1:56482 fd=103 name= age=24303 idle=24303 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19861524 addr=127.0.0.1:41840 fd=7 name= age=79286 idle=840 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=21065211 addr=127.0.0.1:43268 fd=95 name= age=45600 idle=45600 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19862007 addr=127.0.0.1:45168 fd=16 name= age=79278 idle=867 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19861914 addr=127.0.0.1:44496 fd=13 name= age=79280 idle=504 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
id=21068615 addr=127.0.0.1:35186 fd=94 name= age=45485 idle=45485 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19987739 addr=127.0.0.1:50620 fd=30 name= age=76942 idle=23466 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=20043634 addr=127.0.0.1:51786 fd=87 name= age=75893 idle=75893 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=18808734 addr=127.0.0.1:51130 fd=110 name= age=98067 idle=660 flags=P db=0 sub=5 psub=1 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=psubscribe
id=22121095 addr=127.0.0.1:34366 fd=98 name= age=22610 idle=22610 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19861907 addr=127.0.0.1:44430 fd=11 name= age=79280 idle=679 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19863857 addr=127.0.0.1:59234 fd=40 name= age=79246 idle=669 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19865867 addr=127.0.0.1:46450 fd=80 name= age=79213 idle=504 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
id=19866165 addr=127.0.0.1:48758 fd=72 name= age=79208 idle=714 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19865192 addr=127.0.0.1:41294 fd=79 name= age=79224 idle=758 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19864653 addr=127.0.0.1:37158 fd=52 name= age=79233 idle=669 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19863459 addr=127.0.0.1:56308 fd=29 name= age=79252 idle=854 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19862062 addr=127.0.0.1:45686 fd=24 name= age=79277 idle=722 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=20036852 addr=127.0.0.1:57918 fd=71 name= age=76022 idle=76022 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19864977 addr=127.0.0.1:39622 fd=36 name= age=79227 idle=504 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
id=19862933 addr=127.0.0.1:52324 fd=32 name= age=79261 idle=764 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19865287 addr=127.0.0.1:42016 fd=18 name= age=79222 idle=751 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=18808735 addr=127.0.0.1:51134 fd=112 name= age=98067 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=hdel
id=21067574 addr=127.0.0.1:57068 fd=46 name= age=45523 idle=45523 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=20955459 addr=127.0.0.1:54404 fd=86 name= age=49397 idle=49397 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19863448 addr=127.0.0.1:56228 fd=43 name= age=79252 idle=865 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19861862 addr=127.0.0.1:44128 fd=14 name= age=79281 idle=723 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19864969 addr=127.0.0.1:39566 fd=57 name= age=79228 idle=738 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=22623467 addr=127.0.0.1:38208 fd=96 name= age=14328 idle=14328 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19864970 addr=127.0.0.1:39568 fd=60 name= age=79228 idle=767 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=22352180 addr=127.0.0.1:39736 fd=114 name= age=18839 idle=12711 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=22524787 addr=127.0.0.1:44284 fd=124 name= age=15947 idle=15947 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=13001 addr=127.0.0.1:49190 fd=26 name= age=542459 idle=14 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=type
id=19867810 addr=127.0.0.1:60878 fd=85 name= age=79181 idle=672 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19868130 addr=127.0.0.1:35034 fd=70 name= age=79175 idle=498 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
id=22269519 addr=127.0.0.1:48988 fd=100 name= age=20194 idle=20194 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19864881 addr=127.0.0.1:38888 fd=62 name= age=79229 idle=700 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19838433 addr=127.0.0.1:59050 fd=75 name= age=79685 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=lpop
id=19863795 addr=127.0.0.1:58792 fd=44 name= age=79247 idle=671 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19863055 addr=127.0.0.1:53164 fd=19 name= age=79260 idle=144 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
id=19863432 addr=127.0.0.1:56124 fd=39 name= age=79253 idle=384 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
id=19862654 addr=127.0.0.1:50122 fd=20 name= age=79267 idle=24 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=sadd
id=19865929 addr=127.0.0.1:47044 fd=84 name= age=79212 idle=663 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19864430 addr=127.0.0.1:35490 fd=59 name= age=79236 idle=204 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
id=19867793 addr=127.0.0.1:60756 fd=73 name= age=79181 idle=504 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
id=21143529 addr=127.0.0.1:34624 fd=91 name= age=43075 idle=7882 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=22058740 addr=127.0.0.1:46600 fd=90 name= age=23631 idle=13793 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19861692 addr=127.0.0.1:42950 fd=17 name= age=79283 idle=687 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19861492 addr=127.0.0.1:41588 fd=15 name= age=79286 idle=570 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
id=22622641 addr=127.0.0.1:60540 fd=118 name= age=14342 idle=14342 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=21295498 addr=127.0.0.1:35946 fd=89 name= age=38643 idle=38643 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=23098627 addr=127.0.0.1:39336 fd=116 name= age=7128 idle=7128 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=20040261 addr=127.0.0.1:55590 fd=53 name= age=75955 idle=26927 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=21315965 addr=127.0.0.1:58688 fd=97 name= age=38115 idle=38115 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19862297 addr=127.0.0.1:47438 fd=25 name= age=79274 idle=752 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=rpush
id=19866751 addr=127.0.0.1:52888 fd=41 name= age=79200 idle=504 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
id=19861553 addr=127.0.0.1:42040 fd=12 name= age=79285 idle=204 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=sadd
id=23586064 addr=127.0.0.1:42854 fd=61 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=setnx
id=23586065 addr=127.0.0.1:42856 fd=67 name= age=0 idle=0 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=mget
id=23586066 addr=127.0.0.1:42860 fd=76 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=set
id=23586067 addr=127.0.0.1:42862 fd=104 name= age=0 idle=0 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=select
id=23586068 addr=127.0.0.1:42870 fd=81 name= age=0 idle=0 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=mget
id=23586069 addr=127.0.0.1:42872 fd=82 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=setnx
id=23586070 addr=127.0.0.1:42880 fd=109 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=setnx
id=23586071 addr=127.0.0.1:42882 fd=115 name= age=0 idle=0 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=mget
id=23586072 addr=127.0.0.1:42886 fd=119 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=setnx
id=23586073 addr=127.0.0.1:42888 fd=120 name= age=0 idle=0 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=mget
id=23586074 addr=127.0.0.1:42908 fd=48 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=setnx
id=23586075 addr=127.0.0.1:42912 fd=50 name= age=0 idle=0 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=mget
id=23586076 addr=127.0.0.1:42910 fd=83 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=set
id=23586077 addr=127.0.0.1:42914 fd=88 name= age=0 idle=0 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=select
id=23586078 addr=127.0.0.1:42920 fd=121 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=setnx
id=23586079 addr=127.0.0.1:42922 fd=122 name= age=0 idle=0 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=mget
id=23586080 addr=127.0.0.1:42924 fd=123 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=hmset
id=23586081 addr=127.0.0.1:42926 fd=125 name= age=0 idle=0 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=select
id=23586095 addr=127.0.0.1:43024 fd=106 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=23586096 addr=127.0.0.1:43026 fd=108 name= age=0 idle=0 flags=N db=10 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=select
id=23586097 addr=127.0.0.1:43030 fd=134 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=18 qbuf-free=32750 obl=0 oll=0 omem=0 events=r cmd=shutdown
------ CURRENT CLIENT INFO ------
id=23586097 addr=127.0.0.1:43030 fd=134 name= age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=18 qbuf-free=32750 obl=0 oll=0 omem=0 events=r cmd=shutdown
argv[0]: 'shutdown'
27935:M 16 Apr 2020 19:48:34.967 # ------------------------------------------------
27935:M 16 Apr 2020 19:48:34.967 # !!! Software Failure. Press left mouse button to continue
27935:M 16 Apr 2020 19:48:34.967 # Guru Meditation: Unknown encoding type #object.c:526
27935:M 16 Apr 2020 19:48:34.967 # (forcing SIGSEGV in order to print the stack trace)
27935:M 16 Apr 2020 19:48:34.967 # ------------------------------------------------
28962:C 16 Apr 2020 19:48:38.120 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
28962:C 16 Apr 2020 19:48:38.120 # Redis version=5.0.8, bits=64, commit=00000000, modified=0, pid=28962, just started
28962:C 16 Apr 2020 19:48:38.120 # Configuration loaded
28962:C 16 Apr 2020 19:48:38.120 * supervised by systemd, will signal readiness
Comment From: oranagra
crashes in dictNext:
iter->nextEntry = iter->entry->next;
assembly:
0x0000000000039070 <+96>: mov 0x10(%rbx),%rax # read next from iter->entry
0x0000000000039074 <+100>: mov %rax,0x20(%rbp) # write into iter->nextEntry
seems like iter->entry == 1 so adding 0x10 (to access next member) causes an access to 0x11
Comment From: antirez
@oranagra @thedotedge yep but this is a completely unrelated crash. So there are two possibilities: @thedotedge is albe to trigger a memory corruption bug than has random effects, or there is faulty hardware there.
Comment From: antirez
@thedotedge is that the same VM as the previous one that crashed?
Comment From: antirez
@oranagra if the user can work with us, what do you think about making it compile Redis with Address Sanitizer and try to see what happens?
Comment From: thedotedge
@antirez it is the same VM, I am confirming whether it's the same hypervisor host.
Comment From: antirez
@thedotedge thanks, appreciate your help!
Comment From: antirez
@thedotedge I see that your level of usage of Redis is quite impressive, from the command stats you apparently are using almost every Redis feature out there :-D Congrats, it's cool to see Redis applied like that. However I wanted to ask if you are using Redis like that for a long time, and if you noticed that such crashes started once you started doing something new with Redis. Or if the set of features you are using is more or less constant for a long time, and such crashes are coming from nowhere.
Comment From: thedotedge
@antirez Redis is swiss-army knife indeed thanks to you (and the rest of contributors of course) 🙂 Data structures-wise it was used like that for many years now. The load has increased somewhat, but not dramatically:
Meanwhile, here's the response from Linode:
I can confirm that your Linode has not been migrated. Additionally, the host this Linode currently resides on is in great shape. I do not see any memory issues.