The .aof file is not huge (~54MB) The command used is: redis-6.0.8/bin/redis-server --appendonly yes --appendfilename test_bdc.aof --loglevel debug
Help appreciated!!!
=== REDIS BUG REPORT START: Cut & paste starting from here ===
23506:M 30 Sep 2020 06:30:43.434 # Redis 6.0.8 crashed by signal: 11
23506:M 30 Sep 2020 06:30:43.434 # Crashed running the instruction at: 0x4480b5
23506:M 30 Sep 2020 06:30:43.434 # Accessing address: (nil)
23506:M 30 Sep 2020 06:30:43.434 # Failed assertion:
------ STACK TRACE ------
EIP:
/home/utils/redis-6.0.8/bin/redis-server *:6379(protectClient+0x15)[0x4480b5]
Backtrace:
/home/utils/redis-6.0.8/bin/redis-server :6379(logStackTrace+0x32)[0x47bd92]
/home/utils/redis-6.0.8/bin/redis-server :6379(sigsegvHandler+0xa6)[0x47c466]
/lib64/libpthread.so.0(+0xf7e0)[0x7ffff753f7e0]
/home/utils/redis-6.0.8/bin/redis-server :6379(protectClient+0x15)[0x4480b5]
/home/utils/redis-6.0.8/bin/redis-server :6379(luaMaskCountHook+0x91)[0x489781]
/home/utils/redis-6.0.8/bin/redis-server :6379[0x4d0cf0]
/home/utils/redis-6.0.8/bin/redis-server :6379[0x4da451]
/home/utils/redis-6.0.8/bin/redis-server :6379[0x4d14fd]
/home/utils/redis-6.0.8/bin/redis-server :6379[0x4d0848]
/home/utils/redis-6.0.8/bin/redis-server :6379[0x4d165b]
/home/utils/redis-6.0.8/bin/redis-server :6379(lua_pcall+0x46)[0x4cee66]
/home/utils/redis-6.0.8/bin/redis-server :6379(evalGenericCommand+0x18e)[0x48aebe]
/home/utils/redis-6.0.8/bin/redis-server :6379(loadAppendOnlyFile+0x537)[0x4772f7]
/home/utils/redis-6.0.8/bin/redis-server :6379(loadDataFromDisk+0xcc)[0x43e96c]
/home/utils/redis-6.0.8/bin/redis-server :6379(main+0x3b8)[0x430488]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7ffff71bad1d]
/home/utils/redis-6.0.8/bin/redis-server *:6379[0x43084d]
------ INFO OUTPUT ------
Server
redis_version:6.0.8
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:240e25d97efd0115
redis_mode:standalone
os:Linux 4.1.15.pnotify x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:5.5.0
process_id:23506
run_id:ea7429b07e9403177cb648c7193157b9d8941f9f
tcp_port:6379
uptime_in_seconds:5
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:7637118
executable:/home/utils/redis-6.0.8/bin/redis-server
config_file:
io_threads_active:0
Clients
connected_clients:0
client_recent_max_input_buffer:0
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
Memory
used_memory:1018552
used_memory_human:994.68K
used_memory_rss:0
used_memory_rss_human:0B
used_memory_peak:1018552
used_memory_peak_human:994.68K
used_memory_peak_perc:101.18%
used_memory_overhead:869128
used_memory_startup:786680
used_memory_dataset:149424
used_memory_dataset_perc:64.44%
allocator_allocated:0
allocator_active:0
allocator_resident:0
total_system_memory:135069868032
total_system_memory_human:125.79G
used_memory_lua:67202048
used_memory_lua_human:64.09M
used_memory_scripts:80016
used_memory_scripts_human:78.14K
number_of_cached_scripts:34
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:-nan
allocator_frag_bytes:0
allocator_rss_ratio:-nan
allocator_rss_bytes:0
rss_overhead_ratio:-nan
rss_overhead_bytes:0
mem_fragmentation_ratio:-nan
mem_fragmentation_bytes:0
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:0
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
Persistence
loading:1
rdb_changes_since_last_save:1541
rdb_bgsave_in_progress:0
rdb_last_save_time:1601472638
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
aof_current_size:0
aof_base_size:0
aof_pending_rewrite:0
aof_buffer_length:0
aof_rewrite_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:0
loading_start_time:1601472638
loading_total_bytes:56626161
loading_loaded_bytes:1155844
loading_loaded_perc:2.04
loading_eta_seconds:239
Stats
total_connections_received:0
total_commands_processed:3956699
instantaneous_ops_per_sec:0
total_net_input_bytes:0
total_net_output_bytes:0
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
keyspace_hits:3955433
keyspace_misses:180
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:2
total_reads_processed:0
total_writes_processed:0
io_threaded_reads_processed:0
io_threaded_writes_processed:0
Replication
role:master
connected_slaves:0
master_replid:3d32f9cca76797332a33a0ee857b2e965d75a84b
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
CPU
used_cpu_sys:0.024000 used_cpu_user:4.992000 used_cpu_sys_children:0.000000 used_cpu_user_children:0.000000
Modules
Commandstats
Cluster
cluster_enabled:0
Keyspace
db0:keys=48,expires=0,avg_ttl=0
------ CLIENT LIST OUTPUT ------
------ REGISTERS ------
23506:M 30 Sep 2020 06:30:43.435 #
RAX:0000000000000000 RBX:00007ffff6d18b00
RCX:000000000043741a RDX:0000000000000000
RDI:0000000000000000 RSI:0000000000000000
RBP:0000000000000170 RSP:00007fffffffc8f0
R8 :00007ffff7fb1f40 R9 :64373866203a7369
R10:6332323164333537 R11:0000000000000000
R12:0000000000000170 R13:00000000004896f0
R14:00000000009fa5b0 R15:0000000000a0977c
RIP:00000000004480b5 EFL:0000000000010246
CSGSFS:0000000000000033
23506:M 30 Sep 2020 06:30:43.435 # (00007fffffffc8ff) -> 00000000004d8f7d
23506:M 30 Sep 2020 06:30:43.435 # (00007fffffffc8fe) -> 0000000000a1fa90
23506:M 30 Sep 2020 06:30:43.435 # (00007fffffffc8fd) -> 0000000000000064
23506:M 30 Sep 2020 06:30:43.435 # (00007fffffffc8fc) -> 0000000000a1fab0
23506:M 30 Sep 2020 06:30:43.435 # (00007fffffffc8fb) -> 0000000000a1faa0
23506:M 30 Sep 2020 06:30:43.435 # (00007fffffffc8fa) -> 00000000009fa5b0
23506:M 30 Sep 2020 06:30:43.435 # (00007fffffffc8f9) -> 00000000ffffffff
23506:M 30 Sep 2020 06:30:43.435 # (00007fffffffc8f8) -> 0000000000a1faa0
23506:M 30 Sep 2020 06:30:43.435 # (00007fffffffc8f7) -> 00000000004deff5
23506:M 30 Sep 2020 06:30:43.435 # (00007fffffffc8f6) -> 00000000009fa5b0
23506:M 30 Sep 2020 06:30:43.435 # (00007fffffffc8f5) -> 00000000009fa5b0
23506:M 30 Sep 2020 06:30:43.435 # (00007fffffffc8f4) -> 0000000000000003
23506:M 30 Sep 2020 06:30:43.435 # (00007fffffffc8f3) -> 00000000004d0cf0
23506:M 30 Sep 2020 06:30:43.435 # (00007fffffffc8f2) -> 00000000009fa5b0
23506:M 30 Sep 2020 06:30:43.435 # (00007fffffffc8f1) -> 0000000000489781
23506:M 30 Sep 2020 06:30:43.435 # (00007fffffffc8f0) -> 00000000009fa5b0
------ MODULES INFO OUTPUT ------
------ FAST MEMORY TEST ------ 23506:M 30 Sep 2020 06:30:43.435 # Bio thread for job type #0 terminated 23506:M 30 Sep 2020 06:30:43.435 # Bio thread for job type #1 terminated 23506:M 30 Sep 2020 06:30:43.435 # Bio thread for job type #2 terminated *** Preparing to test memory region 7ca000 (2633728 bytes) *** Preparing to test memory region 7fffe6912000 (67112960 bytes) *** Preparing to test memory region 7ffff0753000 (33787904 bytes) *** Preparing to test memory region 7ffff6800000 (8388608 bytes) *** Preparing to test memory region 7ffff752c000 (16384 bytes) *** Preparing to test memory region 7ffff7749000 (16384 bytes) *** Preparing to test memory region 7ffff7fb0000 (24576 bytes) *** Preparing to test memory region 7ffff7fed000 (45056 bytes) *** Preparing to test memory region 7ffff7ffe000 (4096 bytes) .O.O.O.O.O.O.O.O.O Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
------ DUMPING CODE AROUND EIP ------ Symbol: protectClient (base: 0x4480a0) Module: /home/utils/redis-6.0.8/bin/redis-server *:6379 (base 0x400000) $ xxd -r -p /tmp/dump.hex /tmp/dump.bin $ objdump --adjust-vma=0x4480a0 -D -b binary -m i386:x86-64 /tmp/dump.bin
23506:M 30 Sep 2020 06:30:43.749 # dump of function (hexdump of 149 bytes): 534889fb48818fb000000000000010488b7f0831f6488b07ff5038488b7b0831d231f65b488b07488b4030ffe00f1f00488b87b0000000a9000000107502c3904825ffffffef534889fb488987b0000000488b7f08bed0c54400488b07ff50388b835402000085c075164889dfe8aee5ffff85c0750a5bc30f1f8400000000004889df5be9b7e9ffff0f1f800000000055534889fb
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Comment From: oranagra
@ambarsarkar looks like a serious bug in 6.0 when loading long running Lua scripts from AOF.
you can probably work around it by starting redis with --lua-time-limit 100000 or alike, and then change that with CONFIG SET after the file is loaded.
i'll add a fix and a test soon
Comment From: oranagra
@ambarsarkar are you using lua-replicate-commands no?
can you share why?
Comment From: ambarsarkar
@ambarsarkar are you using
lua-replicate-commands no? can you share why?
The aof file was created with 4.0.6 and I am not sure if that was an option even. Not using it now. Any guidelines/suggestions when using it? Thanks!
Comment From: oranagra
ok, so it was an old file, that makes sense.
AFAIK there's no good reason to use lua-replicate-commands no, and we're hoping to deprecate it some day soon.. which is why i asked why are you using it (hoping to lean of a reason).
but that's not the case.
I assume the workaround i provided will solve your problem while we're taking the time to fix the bug and decide on the release urgency. thanks for reporting.
Comment From: ambarsarkar
Interestingly, a timeout of 100000 still crashed, but 1000000(1M) succeeded, with similar errors(assuming for each lua script stored). This is with 6.0.8.
Thank you so much, and lmk if I can help testing! -Ambar