=== REDIS BUG REPORT START: Cut & paste starting from here ===
61053:C 31 Dec 02:07:03.746 # Redis 4.0.14 crashed by signal: 11
61053:C 31 Dec 02:07:03.746 # Crashed running the instruction at: 0x43308f
61053:C 31 Dec 02:07:03.746 # Accessing address: (nil)
61053:C 31 Dec 02:07:03.746 # Failed assertion: <no assertion failed> (<no file>:0)
------ STACK TRACE ------
EIP:
redis-rdb-bgsave 10.253.74.173:6381 [cluster](lzf_compress+0x3f)[0x43308f]
Backtrace:
redis-rdb-bgsave 10.253.74.173:6381 [cluster](logStackTrace+0x29)[0x468ad9]
redis-rdb-bgsave 10.253.74.173:6381 [cluster](sigsegvHandler+0xac)[0x46917c]
/lib64/libpthread.so.0(+0xf5f0)[0x7f1e09abb5f0]
redis-rdb-bgsave 10.253.74.173:6381 [cluster](lzf_compress+0x3f)[0x43308f]
redis-rdb-bgsave 10.253.74.173:6381 [cluster](rdbSaveLzfStringObject+0x4b)[0x447b8b]
redis-rdb-bgsave 10.253.74.173:6381 [cluster](rdbSaveRawString+0x17e)[0x447f3e]
redis-rdb-bgsave 10.253.74.173:6381 [cluster](rdbSaveObject+0x30f)[0x448eff]
redis-rdb-bgsave 10.253.74.173:6381 [cluster](rdbSaveKeyValuePair+0x94)[0x4492c4]
redis-rdb-bgsave 10.253.74.173:6381 [cluster](rdbSaveRio+0x286)[0x4497d6]
redis-rdb-bgsave 10.253.74.173:6381 [cluster](rdbSave+0x78)[0x449c68]
redis-rdb-bgsave 10.253.74.173:6381 [cluster](rdbSaveBackground+0x7c)[0x449e8c]
redis-rdb-bgsave 10.253.74.173:6381 [cluster](serverCron+0x6f0)[0x42d4a0]
redis-rdb-bgsave 10.253.74.173:6381 [cluster](aeProcessEvents+0x32f)[0x4268cf]
redis-rdb-bgsave 10.253.74.173:6381 [cluster](aeMain+0x2b)[0x426b0b]
redis-rdb-bgsave 10.253.74.173:6381 [cluster](main+0x49f)[0x42390f]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f1e09700505]
redis-rdb-bgsave 10.253.74.173:6381 [cluster][0x423c02]
------ INFO OUTPUT ------
# Server
redis_version:4.0.14
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:94c7b25a309bc889
redis_mode:cluster
os:Linux 4.7.1-1.el7.elrepo.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:4.8.5
process_id:61053
run_id:7606b913365382894ec13e04a906ae0547947ebe
tcp_port:6381
uptime_in_seconds:535755
uptime_in_days:6
hz:10
lru_clock:15515356
executable:/app/upay/apps/redis_6381/./src/redis-server
config_file:/app/upay/apps/redis_6381/./conf/redis.conf
# Clients
connected_clients:12
client_longest_output_list:0
client_biggest_input_buf:7
blocked_clients:0
# Memory
used_memory:41448619976
used_memory_human:38.60G
used_memory_rss:44859101184
used_memory_rss_human:41.78G
used_memory_peak:42884184360
used_memory_peak_human:39.94G
used_memory_peak_perc:96.65%
used_memory_overhead:9030279725
used_memory_startup:1517680
used_memory_dataset:32418340251
used_memory_dataset_perc:78.22%
total_system_memory:540893372416
total_system_memory_human:503.75G
used_memory_lua:43008
used_memory_lua_human:42.00K
maxmemory:53687091200
maxmemory_human:50.00G
maxmemory_policy:volatile-ttl
mem_fragmentation_ratio:1.08
mem_allocator:jemalloc-4.0.3
active_defrag_running:0
lazyfree_pending_objects:0
# Persistence
loading:0
rdb_changes_since_last_save:160210465
rdb_bgsave_in_progress:0
rdb_last_save_time:1608819958
rdb_last_bgsave_status:err
rdb_last_bgsave_time_sec:958
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:2462715904
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
# Stats
total_connections_received:64140
total_commands_processed:159482324
instantaneous_ops_per_sec:4639
total_net_input_bytes:25540664169
total_net_output_bytes:29925586
instantaneous_input_kbps:321.51
instantaneous_output_kbps:0.03
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:2164504
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
# Replication
role:slave
master_host:10.253.74.174
master_port:6383
master_link_status:up
master_last_io_seconds_ago:4
master_sync_in_progress:0
slave_repl_offset:208639444100
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:8c7fbf103cb8d00a2e549473267293f77b716cd3
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:208639444100
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:536870912
repl_backlog_first_byte_offset:208102573189
repl_backlog_histlen:536870912
# CPU
used_cpu_sys:138.57
used_cpu_user:606.79
used_cpu_sys_children:0.00
used_cpu_user_children:0.00
# Commandstats
cmdstat_set:calls=6,usec=97,usec_per_call=16.17
cmdstat_time:calls=32,usec=164,usec_per_call=5.12
cmdstat_pexpire:calls=17854780,usec=111097501,usec_per_call=6.22
cmdstat_exec:calls=70,usec=1703,usec_per_call=24.33
cmdstat_lpush:calls=18583640,usec=312910476,usec_per_call=16.84
cmdstat_del:calls=14137914,usec=547544060,usec_per_call=38.73
cmdstat_incrby:calls=25826698,usec=39064248,usec_per_call=1.51
cmdstat_sadd:calls=24,usec=338,usec_per_call=14.08
cmdstat_info:calls=13,usec=1813,usec_per_call=139.46
cmdstat_hset:calls=6,usec=63,usec_per_call=10.50
cmdstat_setnx:calls=12618273,usec=477222386,usec_per_call=37.82
cmdstat_multi:calls=35,usec=8,usec_per_call=0.23
cmdstat_select:calls=1,usec=2,usec_per_call=2.00
cmdstat_cluster:calls=1866,usec=1455788,usec_per_call=780.17
cmdstat_ping:calls=174689,usec=684171,usec_per_call=3.92
cmdstat_sdiffstore:calls=1,usec=140,usec_per_call=140.00
cmdstat_lrem:calls=70284247,usec=990254843,usec_per_call=14.09
cmdstat_command:calls=5,usec=9981,usec_per_call=1996.20
cmdstat_srem:calls=24,usec=67,usec_per_call=2.79
# Cluster
cluster_enabled:1
# Keyspace
db0:keys=133461287,expires=30721589,avg_ttl=0
------ CLIENT LIST OUTPUT ------
id=6 addr=10.253.74.174:6383 fd=53 name= age=532233 idle=4 flags=M db=0 sub=0 psub=0 multi=-1 qbuf=7 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=lrem
id=64138 addr=?:0 fd=56 name= age=26 idle=26 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=64141 addr=?:0 fd=60 name= age=20 idle=20 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=64142 addr=?:0 fd=58 name= age=12 idle=12 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=64143 addr=?:0 fd=61 name= age=12 idle=12 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=64135 addr=?:0 fd=41 name= age=70 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=64140 addr=?:0 fd=59 name= age=24 idle=24 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=64144 addr=?:0 fd=55 name= age=9 idle=9 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=64139 addr=?:0 fd=35 name= age=24 idle=24 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL
id=64136 addr=?:0 fd=42 name= age=52 idle=22 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=64133 addr=?:0 fd=54 name= age=89 idle=29 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=64137 addr=?:0 fd=57 name= age=40 idle=9 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
------ REGISTERS ------
61053:C 31 Dec 02:07:03.748 #
RAX:00007f1372c00000 RBX:8000007f341d64d6
RCX:000000001a064e3e RDX:00007f1372c00000
RDI:8000007f1a171696 RSI:00007f1372c00001
RBP:00007f138cc64e3e RSP:00007ffd90dd32a8
R8 :0000000000000000 R9 :0000000000000000
R10:0000000000000003 R11:0000000000000002
R12:8000007f341d64d8 R13:0000000000000108
R14:000000001a064e3e R15:0000000000000001
RIP:000000000043308f EFL:0000000000010202
CSGSFS:002b000000000033
61053:C 31 Dec 02:07:03.748 # (00007ffd90dd32b7) -> 0000000000000000
61053:C 31 Dec 02:07:03.748 # (00007ffd90dd32b6) -> 0000000000000000
61053:C 31 Dec 02:07:03.748 # (00007ffd90dd32b5) -> 0000000000000000
61053:C 31 Dec 02:07:03.748 # (00007ffd90dd32b4) -> 0000000000000000
61053:C 31 Dec 02:07:03.748 # (00007ffd90dd32b3) -> 0000000000000000
61053:C 31 Dec 02:07:03.748 # (00007ffd90dd32b2) -> 0000000000000000
61053:C 31 Dec 02:07:03.748 # (00007ffd90dd32b1) -> 0000000000000000
61053:C 31 Dec 02:07:03.748 # (00007ffd90dd32b0) -> 0000000000000000
61053:C 31 Dec 02:07:03.748 # (00007ffd90dd32af) -> 0000000000000000
61053:C 31 Dec 02:07:03.748 # (00007ffd90dd32ae) -> 0000000000000000
61053:C 31 Dec 02:07:03.748 # (00007ffd90dd32ad) -> 0000000000000000
61053:C 31 Dec 02:07:03.748 # (00007ffd90dd32ac) -> 0000000000000000
61053:C 31 Dec 02:07:03.748 # (00007ffd90dd32ab) -> 0000000000000000
61053:C 31 Dec 02:07:03.748 # (00007ffd90dd32aa) -> 0000000000000000
61053:C 31 Dec 02:07:03.748 # (00007ffd90dd32a9) -> 0000000000000000
61053:C 31 Dec 02:07:03.748 # (00007ffd90dd32a8) -> 0000000000000000
------ FAST MEMORY TEST ------
*** Preparing to test memory region 745000 (98304 bytes)
*** Preparing to test memory region 244f000 (135168 bytes)
*** Preparing to test memory region 7f1372c00000 (469762048 bytes)
*** Preparing to test memory region 7f138ec00000 (44862275584 bytes)
*** Preparing to test memory region 7f1e00dfe000 (8388608 bytes)
*** Preparing to test memory region 7f1e015ff000 (8388608 bytes)
*** Preparing to test memory region 7f1e01e00000 (14680064 bytes)
*** Preparing to test memory region 7f1e02c00000 (2097152 bytes)
*** Preparing to test memory region 7f1e09400000 (2097152 bytes)
*** Preparing to test memory region 7f1e09aa7000 (20480 bytes)
*** Preparing to test memory region 7f1e09cc4000 (16384 bytes)
*** Preparing to test memory region 7f1e0a3e4000 (16384 bytes)
*** Preparing to test memory region 7f1e0a3ec000 (8192 bytes)
*** Preparing to test memory region 7f1e0a3ee000 (4096 bytes)
*** Preparing to test memory region 7f1e0a3f1000 (4096 bytes)
.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
------ DUMPING CODE AROUND EIP ------
Symbol: lzf_compress (base: 0x433050)
Module: redis-rdb-bgsave 10.253.74.173:6381 [cluster] (base 0x400000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x433050 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
61053:C 31 Dec 02:10:59.846 # dump of function (hexdump of 191 bytes):
85f60f84c801000085c90f84c0010000415789c989f64531c94156415541bd0801000041544c8d2437488d720155488d2c0a53498d5c24fe4881ec90ff03000fb6070fb64f01c1e00809c84889f9eb2a4839ee0f83b40200004c8d41010fb6094183c1014183f9204c8d5601880e0f84a80200004c89d64c89c14839d90f835d010000440fb65902c1e0084989ce4189c04929fe410fb6c34409c0448d14804189c041c1e8084529d0450fb7c04e8d448488458b104589304989ce4901fa4d
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Please report the crash by opening an issue on github:
http://github.com/antirez/redis/issues
Suspect RAM error? Use redis-server --test-memory to verify it.
Comment From: oranagra
@EternalSpring can you please upload the executable that produced this crash, maybe i'll be able to tell which type of value was it that caused that crash.
Comment From: EternalSpring
redis-server.zip @oranagra The executable file has been uploaded,thanks for help
Comment From: oranagra
luckily we do have debug information and line numbers.
stack trace points to a list object, and it looks like the node->zl member is NULL.
0x43308f is in lzf_compress (lzf_c.c:139).
0x447b8b is in rdbSaveLzfStringObject (rdb.c:319).
0x447f3e is in rdbSaveRawString (rdb.c:392).
0x448eff is in rdbSaveObject (rdb.c:666).
0x4492c4 is in rdbSaveKeyValuePair (rdb.c:821).
0x4497d6 is in rdbSaveRio (rdb.c:930).
0x449c68 is in rdbSave (rdb.c:1024).
0x449e8c is in rdbSaveBackground (rdb.c:1080).
0x42d4a0 is in serverCron (server.c:1100).
0x4268cf is in aeProcessEvents (ae.c:333).
0x426b0b is in aeMain (ae.c:498).
i suppose such a thing could have happened when deleting elements from the list and forgetting to delete the quicklist node when it got empty, but i'm not currently aware of any such bug that was recently fixed or reported. since this version is quite old, i can only guess that maybe it was fixed since then (in more recent versions), but i'm not certain.
Comment From: EternalSpring
@oranagra Thanks for the answer, Is it possible that it is caused by a certain configuration ? Can this problem be avoided by adjusting the parameters ? (FYI : Linux version 4.7.1-1.el7.elrepo.x86_64 (mockbuild@Build64R7) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Wed Aug 17 12:55:37 EDT 2016)
Comment From: oranagra
@EternalSpring no, i don't think it could be configuration or OS. it's either a bug in redis, a module (if you used any), or a memory corruption.