=== REDIS BUG REPORT START: Cut & paste starting from here ===
15157:M 25 Nov 11:27:04.525 # Redis 3.2.8 crashed by signal: 11
15157:M 25 Nov 11:27:04.525 # Crashed running the instuction at: 0x41b82d
15157:M 25 Nov 11:27:04.525 # Accessing address: 0x7ff9df102398
15157:M 25 Nov 11:27:04.525 # Failed assertion:
------ STACK TRACE ------ EIP: /opt/redis-3.2.8/bin/redis-server *:6379(aeProcessEvents+0x13d)[0x41b82d]
Backtrace: /opt/redis-3.2.8/bin/redis-server :6379(logStackTrace+0x3c)[0x4580ac] /opt/redis-3.2.8/bin/redis-server :6379(sigsegvHandler+0xa3)[0x458f73] /lib64/libpthread.so.0[0x386b60f710] /opt/redis-3.2.8/bin/redis-server :6379(aeProcessEvents+0x13d)[0x41b82d] /opt/redis-3.2.8/bin/redis-server :6379(aeMain+0x2b)[0x41bb2b] /opt/redis-3.2.8/bin/redis-server :6379(main+0x370)[0x4232a0] /lib64/libc.so.6(__libc_start_main+0xfd)[0x386b21ed5d] /opt/redis-3.2.8/bin/redis-server :6379[0x418ff9]
------ INFO OUTPUT ------
Server
redis_version:3.2.8 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:6a26105ed9a327bd redis_mode:standalone os:Linux 2.6.32-504.el6.x86_64 x86_64 arch_bits:64 multiplexing_api:epoll gcc_version:4.4.7 process_id:15157 run_id:1c5dddad415261701d9db0dc01461fe34d8d45bf tcp_port:6379 uptime_in_seconds:18560099 uptime_in_days:214 hz:10 lru_clock:12439304 executable:/opt/redis-3.2.8/bin/redis-server config_file:/opt/redis-3.2.8/redis.conf
Clients
connected_clients:23 client_longest_output_list:0 client_biggest_input_buf:0 blocked_clients:0
Memory
used_memory:75210128 used_memory_human:71.73M used_memory_rss:113205248 used_memory_rss_human:107.96M used_memory_peak:149987024 used_memory_peak_human:143.04M total_system_memory:16726237184 total_system_memory_human:15.58G used_memory_lua:37888 used_memory_lua_human:37.00K maxmemory:0 maxmemory_human:0B maxmemory_policy:noeviction mem_fragmentation_ratio:1.51 mem_allocator:libc
Persistence
loading:0 rdb_changes_since_last_save:231 rdb_bgsave_in_progress:0 rdb_last_save_time:1606274823 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:1 rdb_current_bgsave_time_sec:-1 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok
Stats
total_connections_received:540 total_commands_processed:4866518147 instantaneous_ops_per_sec:723 total_net_input_bytes:658608073536 total_net_output_bytes:13290652036646 instantaneous_input_kbps:86.85 instantaneous_output_kbps:1463.49 rejected_connections:0 sync_full:0 sync_partial_ok:0 sync_partial_err:0 expired_keys:138302915 evicted_keys:0 keyspace_hits:4425514815 keyspace_misses:126793914 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:3106 migrate_cached_sockets:0
Replication
role:master connected_slaves:0 master_repl_offset:0 repl_backlog_active:0 repl_backlog_size:1048576 repl_backlog_first_byte_offset:0 repl_backlog_histlen:0
CPU
used_cpu_sys:115941.46 used_cpu_user:94415.22 used_cpu_sys_children:3484.79 used_cpu_user_children:22280.01
Commandstats
cmdstat_get:calls=4552530738,usec=19499074211,usec_per_call=4.28 cmdstat_set:calls=84364242,usec=838737131,usec_per_call=9.94 cmdstat_setex:calls=92012651,usec=487793394,usec_per_call=5.30 cmdstat_psetex:calls=930975,usec=6618533,usec_per_call=7.11 cmdstat_del:calls=46973754,usec=200956361,usec_per_call=4.28 cmdstat_exists:calls=89702467,usec=232400646,usec_per_call=2.59 cmdstat_pexpire:calls=3220,usec=16260,usec_per_call=5.05 cmdstat_keys:calls=66,usec=617089,usec_per_call=9349.83 cmdstat_auth:calls=1,usec=3,usec_per_call=3.00 cmdstat_ping:calls=1,usec=2,usec_per_call=2.00 cmdstat_info:calls=5,usec=59931,usec_per_call=11986.20 cmdstat_ttl:calls=14,usec=72,usec_per_call=5.14 cmdstat_config:calls=5,usec=12146,usec_per_call=2429.20 cmdstat_command:calls=7,usec=19428,usec_per_call=2775.43 cmdstat_host::calls=1,usec=2033,usec_per_call=2033.00
Cluster
cluster_enabled:0
Keyspace
db0:keys=37062,expires=37061,avg_ttl=40225156 hash_init_value: 1588542586
------ CLIENT LIST OUTPUT ------ id=442 addr=172.25.60.41:41608 fd=21 name= age=1746417 idle=17 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get id=540 addr=172.25.60.91:7041 fd=8 name= age=446 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=setex id=427 addr=172.25.60.26:51838 fd=13 name= age=1752086 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get id=453 addr=172.25.60.48:61837 fd=24 name= age=1737216 idle=61 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get id=441 addr=172.25.60.48:46315 fd=20 name= age=1746481 idle=60 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=setex id=491 addr=172.25.60.41:42834 fd=22 name= age=1387395 idle=93 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=setex id=492 addr=172.25.60.42:58725 fd=23 name= age=1387334 idle=157 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=setex id=532 addr=172.25.60.41:4378 fd=7 name= age=412659 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get id=499 addr=172.25.60.44:2319 fd=27 name= age=1265097 idle=1265097 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get id=493 addr=172.25.60.47:20350 fd=25 name= age=1387269 idle=92 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=setex id=533 addr=172.25.60.42:17289 fd=6 name= age=412522 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get id=536 addr=172.25.60.26:56883 fd=12 name= age=174403 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=setex id=534 addr=172.25.60.47:60894 fd=18 name= age=412423 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get id=438 addr=172.25.60.42:54924 fd=19 name= age=1748210 idle=98 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=setex id=443 addr=172.25.60.25:56225 fd=17 name= age=1743648 idle=12849 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get id=535 addr=172.25.60.25:34653 fd=10 name= age=174529 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get id=538 addr=172.25.60.92:39391 fd=26 name= age=173944 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=set id=439 addr=172.25.60.25:44338 fd=15 name= age=1747591 idle=2 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get id=528 addr=172.25.60.26:36419 fd=9 name= age=508638 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=get id=435 addr=172.25.60.47:13698 fd=14 name= age=1748819 idle=67 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=setex id=496 addr=172.25.60.48:19991 fd=11 name= age=1287849 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get id=541 addr=172.25.60.102:40684 fd=28 name= age=40 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=5 oll=0 omem=0 events=r cmd=config id=497 addr=172.25.60.92:21758 fd=16 name= age=1287726 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get
------ REGISTERS ------ 15157:M 25 Nov 11:27:04.535 # RAX:0000000006f4e9ed RBX:0000000001279268 RCX:0000000000000000 RDX:0000000000000002 RDI:000000386b58fe00 RSI:0000000000000000 RBP:00007ff9df102398 RSP:00007fff3e12f8d0 R8 :00000000ffffffff R9 :0000000000000000 R10:0000000000000020 R11:0000000000000000 R12:0000000000000000 R13:0000000000000000 R14:0000000000000001 R15:000000000000001c RIP:000000000041b82d EFL:0000000000010246 CSGSFS:0000000000000033 15157:M 25 Nov 11:27:04.535 # (00007fff3e12f8df) -> 00000000004232a0 15157:M 25 Nov 11:27:04.535 # (00007fff3e12f8de) -> 00007fff3e12fa88 15157:M 25 Nov 11:27:04.535 # (00007fff3e12f8dd) -> 000000000041bb2b 15157:M 25 Nov 11:27:04.535 # (00007fff3e12f8dc) -> 0000000000000000 15157:M 25 Nov 11:27:04.535 # (00007fff3e12f8db) -> 0000000001228a9b 15157:M 25 Nov 11:27:04.535 # (00007fff3e12f8da) -> 0000000000000000 15157:M 25 Nov 11:27:04.535 # (00007fff3e12f8d9) -> 0000000001220290 15157:M 25 Nov 11:27:04.535 # (00007fff3e12f8d8) -> 00000000ffffffff 15157:M 25 Nov 11:27:04.535 # (00007fff3e12f8d7) -> 0000000001279268 15157:M 25 Nov 11:27:04.535 # (00007fff3e12f8d6) -> 0000000001220290 15157:M 25 Nov 11:27:04.535 # (00007fff3e12f8d5) -> 0000000000000200 15157:M 25 Nov 11:27:04.535 # (00007fff3e12f8d4) -> 000000005fbdcf08 15157:M 25 Nov 11:27:04.535 # (00007fff3e12f8d3) -> 0000000300000001 15157:M 25 Nov 11:27:04.535 # (00007fff3e12f8d2) -> 0000000000000000 15157:M 25 Nov 11:27:04.535 # (00007fff3e12f8d1) -> 000000000042f288 15157:M 25 Nov 11:27:04.535 # (00007fff3e12f8d0) -> 0000000000000000
------ FAST MEMORY TEST ------ 15157:M 25 Nov 11:27:04.537 # Bio thread for job type #0 terminated 15157:M 25 Nov 11:27:04.537 # Bio thread for job type #1 terminated *** Preparing to test memory region 6da000 (86016 bytes) *** Preparing to test memory region 1220000 (286720 bytes) *** Preparing to test memory region 1266000 (124780544 bytes) *** Preparing to test memory region 386ac21000 (4096 bytes) *** Preparing to test memory region 386b58f000 (20480 bytes) *** Preparing to test memory region 386b819000 (16384 bytes) *** Preparing to test memory region 7ff9ddc62000 (647168 bytes) *** Preparing to test memory region 7ff9ddd01000 (10485760 bytes) *** Preparing to test memory region 7ff9de702000 (10485760 bytes) *** Preparing to test memory region 7ff9df151000 (4096 bytes) *** Preparing to test memory region 7ff9df152000 (16384 bytes) *** Preparing to test memory region 7ff9df160000 (4096 bytes) .O.O.O.O.O.O.O.O.O.O.O.O Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
------ DUMPING CODE AROUND EIP ------ Symbol: aeProcessEvents (base: 0x41b6f0) Module: /opt/redis-3.2.8/bin/redis-server *:6379 (base 0x400000) $ xxd -r -p /tmp/dump.hex /tmp/dump.bin $ objdump --adjust-vma=0x41b6f0 -D -b binary -m i386:x86-64 /tmp/dump.bin
15157:M 25 Nov 11:27:05.475 # dump of function (hexdump of 445 bytes): 41574156415541544531e455534889fb4883ec3840f6c6038974241c0f8436010000833fff0f84450100008b44241c83e00683f8020f8495020000f644241c040f85b7030000488b6b38b9ffffffff8b5304488b75088b7d00e8ead4ffff85c0894424180f8e1401000083e801488b7b20488b75084c8d04c50800000031d2908b0e89c883e00189c583cd02f6c1040f45c589c583cd02f6c1080f45c589c583cd0283e1108b4e040f45c54883c60c89441704890c174883c2084c39c275c14531ed4531e4eb41660f1f840000000000a802742185d20f846c010000488b4510483b4508740f488b55184489f14489fe4889dfffd04183c4014983c508443b6424180f8d500100004c89e84803432031d2448b38448b70044963ef48c1e50548036b188b45004421f0a80174a3488b55184489f14489fe4889dfff55088b4500ba010000004421f0eb86660f1f4400004501f40f1f4400004883c4384489e05b5d415c415d415e415fc3660f1f44000089f083e00683f8020f84520100004531e4f644241c0274d031ffe819d7ffff483b43100f8c17020000488b6b284531f64885ed488943104c8b6b0874a331d24c8d7c2428eb29662e0f1f840000
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Comment From: madolson
Hey, thanks for reporting this bug. Do you by chance have a core dump/more information about what you were doing when you ran into this issue?
3.2.8 is a very old version as well, this might have been fixed awhile ago, you should consider upgrading.
Comment From: ShooterIT
@708854835 did you config set maxclients before redis core dump? I noticed one of your client is as below and idle is 0.
id=541 addr=172.25.60.102:40684 fd=28 name= age=40 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=5 oll=0 omem=0 events=r cmd=config
If yes, maybe similar to the issue https://github.com/redis/redis/issues/6964, and has been fixed.
Comment From: 708854835
Hey, thanks for reporting this bug. Do you by chance have a core dump/more information about what you were doing when you ran into this issue?
3.2.8 is a very old version as well, this might have been fixed awhile ago, you should consider upgrading.
Thank you, this happened in a production environment without the core file being generated. We will upgrade to a higher version and avoid single points of failure.
Comment From: 708854835
Hey, thanks for reporting this bug. Do you by chance have a core dump/more information about what you were doing when you ran into this issue? 3.2.8 is a very old version as well, this might have been fixed awhile ago, you should consider upgrading.
Thank you, this happened in a production environment without the core file being generated. We will upgrade to a higher version and avoid single points of failure.
Thank you, 172.25.60.102 only made a connection to the redis server, but there is no actual business operation. After a comprehensive analysis, we decided to upgrade redis.
Comment From: oranagra
closing this since it's likely to be already solved, and redis version was too old to justify further debugging. @708854835 thanks for reporting, feel free to share more info if you'll have it.