Crash report
Paste the complete crash log between the quotes below. Please include a few lines from the log preceding the crash report to provide some context.
772733:C 03 Jun 2022 09:01:17.323 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
772733:C 03 Jun 2022 09:01:17.323 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=772733, just started
772733:C 03 Jun 2022 09:01:17.323 # Configuration loaded
772733:M 03 Jun 2022 09:01:17.324 * monotonic clock: POSIX clock_gettime
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 6.2.6 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
| `-._ `._ / _.-' | PID: 772733
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | https://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
772733:M 03 Jun 2022 09:01:17.324 # Server initialized
772733:M 03 Jun 2022 09:01:17.327 * <graph> Starting up RedisGraph version 99.99.99.
772733:M 03 Jun 2022 09:01:17.327 * <graph> Thread pool created, using 16 threads.
772733:M 03 Jun 2022 09:01:17.327 * <graph> Maximum number of OpenMP threads set to 16
772733:M 03 Jun 2022 09:01:17.328 * Module 'graph' loaded from /usr/local/lib/redis/redisgraph.so
772733:M 03 Jun 2022 09:01:17.328 * Loading RDB produced by version 6.2.6
772733:M 03 Jun 2022 09:01:17.328 * RDB age 57664 seconds
772733:M 03 Jun 2022 09:01:17.328 * RDB memory usage when created 4940.56 Mb
=== REDIS BUG REPORT START: Cut & paste starting from here ===
772733:M 03 Jun 2022 09:01:51.588 # Redis 6.2.6 crashed by signal: 11, si_code: 1
772733:M 03 Jun 2022 09:01:51.588 # Accessing address: 0x20
772733:M 03 Jun 2022 09:01:51.588 # Crashed running the instruction at: 0x7f6d4cf05169
----- STACK TRACE ------
EIP:
/usr/local/lib/redis/redisgraph.so(RG_Matrix_setElement_UINT64+0x39)[0x7f6d4cf05169]
Backtrace:
/lib/x86_64-linux-gnu/libpthread.so.0(+0x143c0)[0x7f6d4f4943c0]
/usr/local/lib/redis/redisgraph.so(RG_Matrix_setElement_UINT64+0x39)[0x7f6d4cf05169]
/usr/local/lib/redis/redisgraph.so(Graph_FormConnection+0x56)[0x7f6d4ceff696]
/usr/local/lib/redis/redisgraph.so(RdbLoadEdges_v10+0xac)[0x7f6d4cf087fc]
/usr/local/lib/redis/redisgraph.so(RdbLoadGraphContext_v10+0x3a3)[0x7f6d4cf08ca3]
/usr/local/lib/redis/redisgraph.so(+0x2168e5)[0x7f6d4cf068e5]
/usr/local/bin/redis-server *:6379(rdbLoadObject+0x18d)[0x55db9ed7b08d]
/usr/local/bin/redis-server *:6379(rdbLoadRio+0x2b5)[0x55db9ed7da05]
/usr/local/bin/redis-server *:6379(rdbLoad+0x69)[0x55db9ed7e619]
/usr/local/bin/redis-server *:6379(loadDataFromDisk+0xa0)[0x55db9ed50550]
/usr/local/bin/redis-server *:6379(main+0x2c1)[0x55db9ed3fea1]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0x7f6d4f2b20b3]
/usr/local/bin/redis-server *:6379(_start+0x2e)[0x55db9ed4045e]
------ REGISTERS ------
772733:M 03 Jun 2022 09:01:51.590 #
RAX:0000000000000000 RBX:000000000000001a
RCX:0000000000000019 RDX:0000000000000019
RDI:0000000000000000 RSI:0000000000000018
RBP:0000000000000000 RSP:00007fffa401f840
R8 :0000000000000002 R9 :0000000000000001
R10:0000000000000024 R11:0000000000000018
R12:0000000000000019 R13:0000000000000019
R14:0000000000000017 R15:0000000000000018
RIP:00007f6d4cf05169 EFL:0000000000010246
CSGSFS:002b000000000033
772733:M 03 Jun 2022 09:01:51.590 # (00007fffa401f84f) -> 00007fffa401fac0
772733:M 03 Jun 2022 09:01:51.590 # (00007fffa401f84e) -> 00007fffa401fac0
772733:M 03 Jun 2022 09:01:51.590 # (00007fffa401f84d) -> 00007f6d4ceff696
772733:M 03 Jun 2022 09:01:51.590 # (00007fffa401f84c) -> 0000000000000017
772733:M 03 Jun 2022 09:01:51.590 # (00007fffa401f84b) -> 0000000000000000
772733:M 03 Jun 2022 09:01:51.590 # (00007fffa401f84a) -> 0000000000000019
772733:M 03 Jun 2022 09:01:51.590 # (00007fffa401f849) -> 0000000000000018
772733:M 03 Jun 2022 09:01:51.590 # (00007fffa401f848) -> 00007f6ab97d68c0
772733:M 03 Jun 2022 09:01:51.590 # (00007fffa401f847) -> 000000000000001a
772733:M 03 Jun 2022 09:01:51.590 # (00007fffa401f846) -> 0000000000000000
772733:M 03 Jun 2022 09:01:51.590 # (00007fffa401f845) -> f5b48f4a432eba00
772733:M 03 Jun 2022 09:01:51.590 # (00007fffa401f844) -> 00007fffa401f8a6
772733:M 03 Jun 2022 09:01:51.590 # (00007fffa401f843) -> 00007f6ab987b300
772733:M 03 Jun 2022 09:01:51.590 # (00007fffa401f842) -> 0000000000000001
772733:M 03 Jun 2022 09:01:51.590 # (00007fffa401f841) -> 00007f6d4cf04908
772733:M 03 Jun 2022 09:01:51.590 # (00007fffa401f840) -> 0000000000000019
------ INFO OUTPUT ------
# Server
redis_version:6.2.6
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:5736d487eb15e73f
redis_mode:standalone
os:Linux 5.13.0-1025-aws x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:9.3.0
process_id:772733
process_supervised:no
run_id:e4353cf12f512791636852c771a1eddb0b637df5
tcp_port:6379
server_time_usec:1654246911572137
uptime_in_seconds:34
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:10079709
executable:/usr/local/bin/redis-server
config_file:/etc/redis/redis.conf
io_threads_active:0
# Clients
connected_clients:1
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:0
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
# Memory
used_memory:3619048048
used_memory_human:3.37G
used_memory_rss:3372019712
used_memory_rss_human:3.14G
used_memory_peak:3619048048
used_memory_peak_human:3.37G
used_memory_peak_perc:100.17%
used_memory_overhead:850696
used_memory_startup:843816
used_memory_dataset:3618197352
used_memory_dataset_perc:100.00%
allocator_allocated:3613058184
allocator_active:3618553856
allocator_resident:3652702208
total_system_memory:66597511168
total_system_memory_human:62.02G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.00
allocator_frag_bytes:5495672
allocator_rss_ratio:1.01
allocator_rss_bytes:34148352
rss_overhead_ratio:0.92
rss_overhead_bytes:-280682496
mem_fragmentation_ratio:0.93
mem_fragmentation_bytes:-241004824
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:0
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
# Persistence
loading:1
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1654246877
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
loading_start_time:1654246877
loading_total_bytes:1348589522
loading_rdb_used_mem:5180555768
loading_loaded_bytes:1073741823
loading_loaded_perc:79.62
loading_eta_seconds:8
# Stats
total_connections_received:1
total_commands_processed:3
instantaneous_ops_per_sec:0
total_net_input_bytes:69
total_net_output_bytes:13435
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:3
total_writes_processed:3
io_threaded_reads_processed:0
io_threaded_writes_processed:0
# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:52071240411ff027f25a4bbdf9f4d04b48a3e4a9
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:3.283807
used_cpu_user:112.240365
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:1.451543
used_cpu_user_main_thread:32.441754
# Modules
module:name=graph,ver=999999,api=1,filters=0,usedby=[],using=[],options=[]
# Commandstats
cmdstat_info:calls=3,usec=268,usec_per_call=89.33,rejected_calls=0,failed_calls=0
# Errorstats
# Cluster
cluster_enabled:0
# Keyspace
db0:keys=120,expires=0,avg_ttl=0
------ CLIENT LIST OUTPUT ------
id=5 addr=127.0.0.1:51750 laddr=127.0.0.1:6379 fd=9 name= age=21 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61456 events=r cmd=info user=default redir=-1
------ MODULES INFO OUTPUT ------
# graph_executing commands
------ FAST MEMORY TEST ------
772733:M 03 Jun 2022 09:01:51.590 # Bio thread for job type #0 terminated
772733:M 03 Jun 2022 09:01:51.590 # Bio thread for job type #1 terminated
772733:M 03 Jun 2022 09:01:51.590 # Bio thread for job type #2 terminated
*** Preparing to test memory region 55db9ef0f000 (2277376 bytes)
*** Preparing to test memory region 55db9fd10000 (135168 bytes)
*** Preparing to test memory region 55db9fd31000 (667648 bytes)
*** Preparing to test memory region 7f6a44000000 (2416054272 bytes)
*** Preparing to test memory region 7f6ad8000000 (135168 bytes)
*** Preparing to test memory region 7f6adc000000 (135168 bytes)
*** Preparing to test memory region 7f6ae0000000 (135168 bytes)
*** Preparing to test memory region 7f6ae4000000 (135168 bytes)
*** Preparing to test memory region 7f6ae8000000 (135168 bytes)
*** Preparing to test memory region 7f6aec000000 (135168 bytes)
*** Preparing to test memory region 7f6af0000000 (135168 bytes)
....
*** Preparing to test memory region 7f6d4f0f1000 (24576 bytes)
*** Preparing to test memory region 7f6d4f282000 (8192 bytes)
*** Preparing to test memory region 7f6d4f47c000 (16384 bytes)
*** Preparing to test memory region 7f6d4f49f000 (16384 bytes)
*** Preparing to test memory region 7f6d4f551000 (4096 bytes)
*** Preparing to test memory region 7f6d4f6a7000 (8192 bytes)
*** Preparing to test memory region 7f6d4f6df000 (4096 bytes)
Additional information
- OS distribution and version : Linux acc-redis 5.13.0-1025-aws #27~20.04.1-Ubuntu SMP Thu May 19 15:17:13 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux3.
- Steps to reproduce (if any) Using Redis + redis graph with persistence enabled, after restarting the system, redis is stuck in a loop with crashes. Tried changing instances several times (on AWS) after running memory test, we are still getting the error.
Comment From: oranagra
@vijaykiran please open in https://github.com/RedisGraph/RedisGraph @AviAvni FYI