Notice! - If a Redis module was involved, please open an issue in the module's repo instead! - If you're using docker on Apple M1, please make sure the image you're using was compiled for ARM!
Crash report
Paste the complete crash log between the quotes below. Please include a few lines from the log preceding the crash report to provide some context.
7059:M 29 Apr 23:59:21.081 * Background saving started by pid 7644
7644:C 30 Apr 00:00:05.893 # Out Of Memory allocating 292806299210713325 bytes!
=== REDIS BUG REPORT START: Cut & paste starting from here ===
7644:C 30 Apr 00:00:05.942 # ------------------------------------------------
7644:C 30 Apr 00:00:05.942 # !!! Software Failure. Press left mouse button to continue
7644:C 30 Apr 00:00:05.944 # Guru Meditation: "Redis aborting for OUT OF MEMORY" #server.c:3847
7644:C 30 Apr 00:00:05.944 # (forcing SIGSEGV in order to print the stack trace)
7644:C 30 Apr 00:00:05.944 # ------------------------------------------------
7644:C 30 Apr 00:00:05.946 # Redis 3.2.12 crashed by signal: 11
7644:C 30 Apr 00:00:05.946 # Crashed running the instuction at: 0x45f24a
7644:C 30 Apr 00:00:05.946 # Accessing address: 0xffffffffffffffff
7644:C 30 Apr 00:00:05.946 # Failed assertion: <no assertion failed> (<no file>:0)
------ STACK TRACE ------
EIP:
redis-rdb-bgsave 0.0.0.0:6379(_serverPanic+0x7a)[0x45f24a]
Backtrace:
redis-rdb-bgsave 0.0.0.0:6379(logStackTrace+0x45)[0x460ac5]
redis-rdb-bgsave 0.0.0.0:6379(sigsegvHandler+0xb9)[0x461289]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f0f09a7e390]
redis-rdb-bgsave 0.0.0.0:6379(_serverPanic+0x7a)[0x45f24a]
redis-rdb-bgsave 0.0.0.0:6379(redisOutOfMemoryHandler+0x2c)[0x42467c]
redis-rdb-bgsave 0.0.0.0:6379(zmalloc+0x69)[0x42e299]
redis-rdb-bgsave 0.0.0.0:6379[0x441de7]
redis-rdb-bgsave 0.0.0.0:6379(rdbSaveRawString+0x4f)[0x44205f]
redis-rdb-bgsave 0.0.0.0:6379(rdbSaveStringObject+0x40)[0x4423b0]
redis-rdb-bgsave 0.0.0.0:6379(rdbSaveObject+0xc5)[0x442d75]
redis-rdb-bgsave 0.0.0.0:6379(rdbSaveKeyValuePair+0xc3)[0x443173]
redis-rdb-bgsave 0.0.0.0:6379(rdbSaveRio+0x304)[0x443714]
redis-rdb-bgsave 0.0.0.0:6379(rdbSave+0x8f)[0x443a8f]
redis-rdb-bgsave 0.0.0.0:6379(rdbSaveBackground+0x67)[0x443cb7]
redis-rdb-bgsave 0.0.0.0:6379(serverCron+0x71a)[0x427b1a]
redis-rdb-bgsave 0.0.0.0:6379(aeProcessEvents+0x31f)[0x420b7f]
redis-rdb-bgsave 0.0.0.0:6379(aeMain+0x2b)[0x420d7b]
redis-rdb-bgsave 0.0.0.0:6379(main+0x417)[0x41daa7]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f0f096c3840]
redis-rdb-bgsave 0.0.0.0:6379(_start+0x29)[0x41dd19]
------ INFO OUTPUT ------
# Server
redis_version:3.2.12
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:82061ca9aff2896b
redis_mode:standalone
os:Linux 4.4.0-87-generic x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:5.4.0
process_id:7644
run_id:c2e4f5f147b0f4db7bb98f2f984f8a0368f1b502
tcp_port:6379
uptime_in_seconds:43068142
uptime_in_days:498
hz:10
lru_clock:3130329
executable:/data/redis-3.2.12/bin/redis-server
config_file:/data/redis-3.2.12/config/redis.conf
# Clients
connected_clients:304
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
# Memory
used_memory:7876254192
used_memory_human:7.34G
used_memory_rss:8671993856
used_memory_rss_human:8.08G
used_memory_peak:8463846008
used_memory_peak_human:7.88G
total_system_memory:25282473984
total_system_memory_human:23.55G
used_memory_lua:36864
used_memory_lua_human:36.00K
maxmemory:12884901888
maxmemory_human:12.00G
maxmemory_policy:noeviction
mem_fragmentation_ratio:1.10
mem_allocator:jemalloc-4.0.3
# Persistence
loading:0
rdb_changes_since_last_save:43471
rdb_bgsave_in_progress:0
rdb_last_save_time:1714406300
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:56
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
# Stats
total_connections_received:14766412
total_commands_processed:17776078552
instantaneous_ops_per_sec:955
total_net_input_bytes:4132045762099
total_net_output_bytes:45638486524
instantaneous_input_kbps:74.20
instantaneous_output_kbps:189.82
rejected_connections:0
sync_full:1
sync_partial_ok:0
sync_partial_err:0
expired_keys:5772
evicted_keys:0
keyspace_hits:1913717096
keyspace_misses:3275813
pubsub_channels:3
pubsub_patterns:0
latest_fork_usec:50782
migrate_cached_sockets:0
# Replication
role:master
connected_slaves:0
master_repl_offset:7715014
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:6666439
repl_backlog_histlen:1048576
# CPU
used_cpu_sys:4.00
used_cpu_user:40.78
used_cpu_sys_children:0.00
used_cpu_user_children:0.00
# Commandstats
cmdstat_get:calls=1916876385,usec=6524872960,usec_per_call=3.40
cmdstat_set:calls=1922354555,usec=75615228962,usec_per_call=39.33
cmdstat_setnx:calls=8985626,usec=189690821,usec_per_call=21.11
cmdstat_setex:calls=1466807401,usec=66704765306,usec_per_call=45.48
cmdstat_del:calls=3145945623,usec=80620599209,usec_per_call=25.63
cmdstat_exists:calls=7284,usec=58067,usec_per_call=7.97
cmdstat_incr:calls=28100114,usec=185659835,usec_per_call=6.61
cmdstat_rpush:calls=50228189,usec=43088541971,usec_per_call=857.86
cmdstat_lpush:calls=15875009,usec=467243667,usec_per_call=29.43
cmdstat_llen:calls=1086,usec=9875,usec_per_call=9.09
cmdstat_lrange:calls=523,usec=21578,usec_per_call=41.26
cmdstat_lrem:calls=22377,usec=3202281,usec_per_call=143.11
cmdstat_sadd:calls=94526843,usec=7370389124,usec_per_call=77.97
cmdstat_srem:calls=32060915,usec=597381831,usec_per_call=18.63
cmdstat_sismember:calls=1701,usec=129899,usec_per_call=76.37
cmdstat_smembers:calls=63,usec=26726,usec_per_call=424.22
cmdstat_zadd:calls=29331208,usec=6560231595,usec_per_call=223.66
cmdstat_zrem:calls=1784,usec=434130,usec_per_call=243.35
cmdstat_zrevrange:calls=1031,usec=86991,usec_per_call=84.38
cmdstat_hset:calls=5445143459,usec=383744965323,usec_per_call=70.47
cmdstat_hsetnx:calls=5953,usec=655452,usec_per_call=110.10
cmdstat_hget:calls=47117,usec=1274915,usec_per_call=27.06
cmdstat_hmset:calls=5754116,usec=3669201730,usec_per_call=637.67
cmdstat_hincrby:calls=11697350,usec=1130512293,usec_per_call=96.65
cmdstat_hdel:calls=28750913,usec=6722283595,usec_per_call=233.81
cmdstat_hgetall:calls=252,usec=13468,usec_per_call=53.44
cmdstat_hexists:calls=17105,usec=453256,usec_per_call=26.50
cmdstat_incrby:calls=660184750,usec=13533226422,usec_per_call=20.50
cmdstat_select:calls=100905304,usec=85235782,usec_per_call=0.84
cmdstat_expire:calls=28259866,usec=258220815,usec_per_call=9.14
cmdstat_pexpire:calls=812165861,usec=8578953736,usec_per_call=10.56
cmdstat_auth:calls=14766391,usec=292011698,usec_per_call=19.78
cmdstat_ping:calls=4272938,usec=1706200,usec_per_call=0.40
cmdstat_psync:calls=1,usec=60008,usec_per_call=60008.00
cmdstat_replconf:calls=2,usec=5596,usec_per_call=2798.00
cmdstat_info:calls=13275985,usec=2559772009,usec_per_call=192.81
cmdstat_ttl:calls=49602,usec=158295,usec_per_call=3.19
cmdstat_slaveof:calls=19,usec=42849,usec_per_call=2255.21
cmdstat_config:calls=2204057,usec=880367554,usec_per_call=399.43
cmdstat_subscribe:calls=30,usec=258,usec_per_call=8.60
cmdstat_publish:calls=15874859,usec=13312572,usec_per_call=0.84
cmdstat_client:calls=712153,usec=3851188,usec_per_call=5.41
cmdstat_eval:calls=1885438248,usec=55623958615,usec_per_call=29.50
cmdstat_evalsha:calls=31362509,usec=2352490439,usec_per_call=75.01
cmdstat_slowlog:calls=1424306,usec=116764601,usec_per_call=81.98
cmdstat_command:calls=1,usec=15259,usec_per_call=15259.00
cmdstat_pfadd:calls=1587955,usec=166073173,usec_per_call=104.58
cmdstat_pfcount:calls=337570,usec=30986381,usec_per_call=91.79
cmdstat_latency:calls=712163,usec=14279911,usec_per_call=20.05
# Cluster
cluster_enabled:0
# Keyspace
db0:keys=29711,expires=0,avg_ttl=0
db2:keys=68923,expires=0,avg_ttl=0
db3:keys=2,expires=0,avg_ttl=0
db4:keys=11570,expires=0,avg_ttl=0
db8:keys=398534,expires=101479,avg_ttl=247702119
db10:keys=2208407,expires=851081,avg_ttl=132617183
db15:keys=650775,expires=0,avg_ttl=0
hash_init_value: 1670625308
RAX:0000000000000000 RBX:00000000004ee9b8
RCX:0000000000000000 RDX:0000000000000000
RDI:00007f0f09a67b20 RSI:0000000000000000
RBP:00000000004f1098 RSP:00007ffe25ae4d10
R8 :000000000163ed60 R9 :0000000000000001
R10:0000000000000030 R11:0000000000000246
R12:0000000000000f07 R13:00007ffe25ae4f60
R14:00007f0ec4ec6c45 R15:041041c71c7388ec
RIP:000000000045f24a EFL:0000000000010206
CSGSFS:0000000000000033
7644:C 30 Apr 00:00:06.116 # (00007ffe25ae4d1f) -> 041041c71c7388f0
7644:C 30 Apr 00:00:06.116 # (00007ffe25ae4d1e) -> 00007f0ec4ec6c45
7644:C 30 Apr 00:00:06.116 # (00007ffe25ae4d1d) -> 00007ffe25ae4f60
7644:C 30 Apr 00:00:06.116 # (00007ffe25ae4d1c) -> 0000000001bc5d5d
7644:C 30 Apr 00:00:06.116 # (00007ffe25ae4d1b) -> 00007ffe25ae4f60
7644:C 30 Apr 00:00:06.116 # (00007ffe25ae4d1a) -> 00007ffe25ae4d80
7644:C 30 Apr 00:00:06.116 # (00007ffe25ae4d19) -> 0000000000441de7
7644:C 30 Apr 00:00:06.116 # (00007ffe25ae4d18) -> 0000000001bc5d5d
7644:C 30 Apr 00:00:06.116 # (00007ffe25ae4d17) -> 041041c71c7388f0
7644:C 30 Apr 00:00:06.116 # (00007ffe25ae4d16) -> 0000000000000005
7644:C 30 Apr 00:00:06.116 # (00007ffe25ae4d15) -> 000000000042e299
7644:C 30 Apr 00:00:06.116 # (00007ffe25ae4d14) -> 041041c71c7388ec
7644:C 30 Apr 00:00:06.116 # (00007ffe25ae4d13) -> 000000000042467c
7644:C 30 Apr 00:00:06.116 # (00007ffe25ae4d12) -> 0000000000000000
7644:C 30 Apr 00:00:06.116 # (00007ffe25ae4d11) -> 041041c71c7388ed
7644:C 30 Apr 00:00:06.116 # (00007ffe25ae4d10) -> 0000000000000000
------ FAST MEMORY TEST ------
*** Preparing to test memory region 737000 (94208 bytes)
*** Preparing to test memory region 162e000 (135168 bytes)
*** Preparing to test memory region 7f0d00c00000 (8705277952 bytes)
*** Preparing to test memory region 7f0f07bff000 (8388608 bytes)
*** Preparing to test memory region 7f0f08400000 (12582912 bytes)
*** Preparing to test memory region 7f0f09400000 (2097152 bytes)
*** Preparing to test memory region 7f0f09a69000 (16384 bytes)
*** Preparing to test memory region 7f0f09c86000 (16384 bytes)
*** Preparing to test memory region 7f0f0a3b0000 (20480 bytes)
*** Preparing to test memory region 7f0f0a3be000 (4096 bytes)
.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
------ DUMPING CODE AROUND EIP ------
Symbol: _serverPanic (base: 0x45f1d0)
Module: redis-rdb-bgsave 0.0.0.0:6379 (base 0x400000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x45f1d0 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
7644:C 30 Apr 00:05:49.184 # dump of function (hexdump of 250 bytes):
8b05a6962d0041544189d4554889f5534889fb85c07505e8c4f8ffffbee0944f00bf0300000031c0e84353fcffbe18954f00bf0300000031c0e83253fcff4589e04889e94889dabe1aa54f00bf0300000031c0e81853fcffbe58954f00bf0300000031c0e80753fcffbee0944f00bf0300000031c0e8f652fcffc60425ffffffff785b5d415cc3660f1f840000000000415741564155415455534881ec6801000048c7070000000048c7470800000000c74710000000008b05c3912d0064488b1c252800000048899c245801000031db48897c241085c00f8ecf06000048c744242800000000c744242400000000488d9c24d0000000eb230f1f
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Please report the crash by opening an issue on github:
http://github.com/antirez/redis/issues
Suspect RAM error? Use redis-server --test-memory to verify it.
17059:M 30 Apr 00:06:49.733 # Background saving terminated by signal 11
17059:M 30 Apr 00:06:49.869 * 10 changes in 300 seconds. Saving...
17059:M 30 Apr 00:06:49.938 * Background saving started by pid 13804
13804:C 30 Apr 00:07:53.781 # Out Of Memory allocating 292806299210713325 bytes!
Additional information
- OS distribution and version ubuntu 16
- Steps to reproduce (if any)
Comment From: zzoujinn
I don't know what action triggered Redis to allocate such a large amount of memory: 292806299210713325.
Comment From: sundb
the version you are using is so old, can you try the new version? 6.x or newer?
Comment From: zzoujinn
We want to resolve this issue in this version. if cant reslove, we are considering upgrading the version.
Comment From: sundb
sorry, since it's no longer a LTS version, i suggest you check the release node after this version to see if there are any clues.