Crash report
Paste the complete crash log between the quotes below. Please include a few lines from the log preceding the crash report to provide some context.
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 3.2.12 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
| `-._ `._ / _.-' | PID: 1347
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
1347:M 13 Jan 19:18:09.103 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1347:M 13 Jan 19:18:09.103 # Server started, Redis version 3.2.12
1347:M 13 Jan 19:18:09.103 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and the
n reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1347:M 13 Jan 19:18:09.103 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the com
mand 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disable
d.
1347:M 13 Jan 19:18:09.103 * The server is now ready to accept connections on port 6379
=== REDIS BUG REPORT START: Cut & paste starting from here ===
1347:M 14 Jan 17:20:23.097 # Redis 3.2.12 crashed by signal: 11
1347:M 14 Jan 17:20:23.097 # Accessing address: 0xffff7dffea00
1347:M 14 Jan 17:20:23.097 # Failed assertion: <no assertion failed> (<no file>:0)
------ STACK TRACE ------
/usr/bin/redis-server *:6379(logStackTrace+0x40)[0xaaaac0433ba8]
/usr/bin/redis-server *:6379(sigsegvHandler+0x9c)[0xaaaac04342e4]
linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0xffff8767a668]
/usr/bin/redis-server *:6379(dictFind+0x68)[0xaaaac03f7340]
/usr/bin/redis-server *:6379(hashTypeGetFromHashTable+0x24)[0xaaaac0424eb8]
/usr/bin/redis-server *:6379(hashTypeGetObject+0x3c)[0xaaaac042505c]
/usr/bin/redis-server *:6379(hincrbyCommand+0x84)[0xaaaac04262b8]
/usr/bin/redis-server *:6379(call+0x7c)[0xaaaac03facbc]
/usr/bin/redis-server *:6379(processCommand+0x3e4)[0xaaaac03fdbf8]
/usr/bin/redis-server *:6379(processInputBuffer+0x118)[0xaaaac040b000]
/usr/bin/redis-server *:6379(aeProcessEvents+0x34c)[0xaaaac03f4fd0]
/usr/bin/redis-server *:6379(aeMain+0x30)[0xaaaac03f51c4]
/usr/bin/redis-server *:6379(main+0x340)[0xaaaac03f1e44]
/lib64/libc.so.6(__libc_start_main+0xe4)[0xffff8737cda4]
/usr/bin/redis-server *:6379(+0x21100)[0xaaaac03f2100]
------ INFO OUTPUT ------
# Server
redis_version:3.2.12
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:7dfac9c7a0dd06b5
redis_mode:standalone
os:Linux 4.14.301-224.520.amzn2.aarch64 aarch64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.8.5
process_id:1347
run_id:76358d803f21bf61f9d594e2a08e66a15c590423
tcp_port:6379
uptime_in_seconds:79334
uptime_in_days:0
hz:10
lru_clock:12772439
executable:/usr/bin/redis-server
config_file:/etc/redis.conf
# Clients
connected_clients:5
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:2
# Memory
used_memory:4269928
used_memory_human:4.07M
used_memory_rss:8261632
used_memory_rss_human:7.88M
used_memory_peak:4389536
used_memory_peak_human:4.19M
total_system_memory:2028507136
total_system_memory_human:1.89G
used_memory_lua:37888
used_memory_lua_human:37.00K
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
mem_fragmentation_ratio:1.93
mem_allocator:jemalloc-3.6.0
# Persistence
loading:0
rdb_changes_since_last_save:2115028
rdb_bgsave_in_progress:0
rdb_last_save_time:1673637489
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
# Stats
total_connections_received:156928
total_commands_processed:3437935
instantaneous_ops_per_sec:45
total_net_input_bytes:289216487
total_net_output_bytes:116334312
instantaneous_input_kbps:3.84
instantaneous_output_kbps:0.44
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:35823
evicted_keys:0
keyspace_hits:378184
keyspace_misses:181322
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
migrate_cached_sockets:0
# Replication
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:102.17
used_cpu_user:62.34
used_cpu_sys_children:0.00
used_cpu_user_children:0.00
# Commandstats
cmdstat_get:calls=326527,usec=566347,usec_per_call=1.73
cmdstat_setnx:calls=200,usec=527,usec_per_call=2.63
cmdstat_setex:calls=8839,usec=26291,usec_per_call=2.97
cmdstat_del:calls=209,usec=447,usec_per_call=2.14
cmdstat_exists:calls=141427,usec=332477,usec_per_call=2.35
cmdstat_incr:calls=29336,usec=69643,usec_per_call=2.37
cmdstat_rpush:calls=310,usec=2163,usec_per_call=6.98
cmdstat_blpop:calls=29098,usec=121848,usec_per_call=4.19
cmdstat_hset:calls=39349,usec=83836,usec_per_call=2.13
cmdstat_hsetnx:calls=136080,usec=218760,usec_per_call=1.61
cmdstat_hget:calls=219506,usec=341094,usec_per_call=1.55
cmdstat_hmset:calls=256,usec=2022,usec_per_call=7.90
cmdstat_hincrby:calls=833671,usec=3589081,usec_per_call=4.31
cmdstat_hincrbyfloat:calls=140062,usec=1700608,usec_per_call=12.14
cmdstat_hdel:calls=502,usec=2308,usec_per_call=4.60
cmdstat_hlen:calls=136,usec=268,usec_per_call=1.97
cmdstat_hvals:calls=10594,usec=65206,usec_per_call=6.15
cmdstat_hgetall:calls=548,usec=3066,usec_per_call=5.59
cmdstat_expire:calls=1008539,usec=919200,usec_per_call=0.91
cmdstat_keys:calls=3,usec=342,usec_per_call=114.00
cmdstat_scan:calls=506341,usec=3074507,usec_per_call=6.07
cmdstat_multi:calls=1316,usec=243,usec_per_call=0.18
cmdstat_exec:calls=3948,usec=13331,usec_per_call=3.38
cmdstat_persist:calls=256,usec=247,usec_per_call=0.96
cmdstat_dump:calls=880,usec=5998,usec_per_call=6.82
cmdstat_command:calls=2,usec=946,usec_per_call=473.00
# Cluster
cluster_enabled:0
# Keyspace
db0:keys=932,expires=928,avg_ttl=24455372
hash_init_value: 1673582029
------ CLIENT LIST OUTPUT ------
id=568 addr=127.0.0.1:53262 fd=7 name= age=77321 idle=77317 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=keys
id=156875 addr=172.30.0.205:34278 fd=8 name= age=22 idle=2 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=blpop
id=156872 addr=127.0.0.1:49302 fd=11 name= age=22 idle=2 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=blpop
id=156928 addr=127.0.0.1:49674 fd=6 name= age=2 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=hincrby
id=156929 addr=127.0.0.1:49680 fd=9 name= age=2 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=expire
------ CURRENT CLIENT INFO ------
id=156928 addr=127.0.0.1:49674 fd=6 name= age=2 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=hincrby
argv[0]: 'HINCRBY'
argv[1]: 'stats:page:views:by_anon:in_day:19371'
argv[2]: '3612bdd5525ef2bca9ffbd338480557c'
argv[3]: '1'
1347:M 14 Jan 17:20:23.098 # key 'stats:page:views:by_anon:in_day:19371' found in DB containing the following object:
1347:M 14 Jan 17:20:23.098 # Object type: 4
1347:M 14 Jan 17:20:23.098 # Object encoding: 2
1347:M 14 Jan 17:20:23.098 # Object refcount: 1
1347:M 14 Jan 17:20:23.098 # Hash size: 8204
------ REGISTERS ------
------ FAST MEMORY TEST ------
1347:M 14 Jan 17:20:23.098 # Bio thread for job type #0 terminated
1347:M 14 Jan 17:20:23.098 # Bio thread for job type #1 terminated
*** Preparing to test memory region aaaac04d6000 (86016 bytes)
*** Preparing to test memory region ffff7d800000 (8380416 bytes)
*** Preparing to test memory region ffff7dfff000 (10485760 bytes)
*** Preparing to test memory region ffff7ea00000 (18874368 bytes)
*** Preparing to test memory region ffff86800000 (8388608 bytes)
*** Preparing to test memory region ffff874df000 (16384 bytes)
*** Preparing to test memory region ffff87514000 (16384 bytes)
*** Preparing to test memory region ffff87569000 (4096 bytes)
*** Preparing to test memory region ffff8766c000 (16384 bytes)
.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Additional information
-
OS distribution and version Amazon AMI2 4.14.301-224.520.amzn2.aarch64
-
Steps to reproduce (if any) Unknown. This has happened randomly twice in the last couple of days -- the redis server just crashes for no obvious reason.
Comment From: modernjess
This is the default redis available on Amazon AMI2 running ARM. I didn't realize it at the time, but there is a newer version hiding in linux-extras (redis 6) which I have enabled and upgraded to.
Therefore, I don't have any vested interest in resolving this particular crash.
Comment From: oranagra
seems like a memory corruption issue, but considering it's a very old version, and not sure it was tested on ARM, i rather assume it was resolved with a later version, please report if you see it with the recent one. p.s. not sure the data from that redis can be trusted, if the memory was corrupted and we take the data to the newer version, the old corruption can still cause odd crashes.
Comment From: modernjess
seems like a memory corruption issue, but considering it's a very old version, and not sure it was tested on ARM, i rather assume it was resolved with a later version, please report if you see it with the recent one.
Will do.
p.s. not sure the data from that redis can be trusted, if the memory was corrupted and we take the data to the newer version, the old corruption can still cause odd crashes.
Thanks for the tip. After the first crash, I disabled all persistence, as I am using redis strictly for cacheing and information that can be recreated (although at a time/processing cost) from elsewhere. I did have another crash with that version, even without using any of the previous data. Which was weird, because it had been running continuously in production without issue for approximately a year.