Crash report
Paste the complete crash log between the quotes below. Please include a few lines from the log preceding the crash report to provide some context.
airflow-webserver_1 | 172.20.0.1 - - [02/Jul/2021:20:01:22 +0000] "GET /object/task_instances?dag_id=annotate_dug&execution_date=2021-06-30T12%3A38%3A50.695362%2B00%3A00 HTTP/1.1" 200 4554 "http://localhost:8080/graph?dag_id=annotate_dug" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36"
airflow-webserver_1 | 127.0.0.1 - - [02/Jul/2021:20:01:24 +0000] "GET /health HTTP/1.1" 200 187 "-" "curl/7.64.0"
airflow-webserver_1 | 172.20.0.1 - - [02/Jul/2021:20:01:25 +0000] "GET /object/task_instances?dag_id=annotate_dug&execution_date=2021-06-30T12%3A38%3A50.695362%2B00%3A00 HTTP/1.1" 200 4554 "http://localhost:8080/graph?dag_id=annotate_dug" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36"
airflow-webserver_1 | 172.20.0.1 - - [02/Jul/2021:20:01:28 +0000] "GET /object/task_instances?dag_id=annotate_dug&execution_date=2021-06-30T12%3A38%3A50.695362%2B00%3A00 HTTP/1.1" 200 4554 "http://localhost:8080/graph?dag_id=annotate_dug" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36"
redis_1 |
redis_1 |
redis_1 | === REDIS BUG REPORT START: Cut & paste starting from here ===
redis_1 | 1:M 02 Jul 2021 20:01:30.320 # Redis 6.2.3 crashed by signal: 11, si_code: 1
redis_1 | 1:M 02 Jul 2021 20:01:30.320 # Accessing address: 0x8c944b4f028c946e
redis_1 | 1:M 02 Jul 2021 20:01:30.321 # Crashed running the instruction at: 0x400004812c
redis_1 |
redis_1 | ------ STACK TRACE ------
redis_1 | EIP:
redis_1 | /usr/local/bin/redis-server *:6379(dictFind+0x5c)[0x400004812c]
redis_1 |
redis_1 | Backtrace:
redis_1 | /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x400217d730]
redis_1 | /usr/local/bin/redis-server *:6379(dictFind+0x5c)[0x400004812c]
redis_1 | /usr/local/bin/redis-server *:6379(hashTypeGetFromHashTable+0x17)[0x400008e9e7]
redis_1 | /usr/local/bin/redis-server *:6379(+0x8ea3e)[0x400008ea3e]
redis_1 | /usr/local/bin/redis-server *:6379(call+0xad)[0x400004cc6d]
redis_1 | /usr/local/bin/redis-server *:6379(processCommand+0x5b3)[0x400004e7a3]
redis_1 | /usr/local/bin/redis-server *:6379(processCommandAndResetClient+0x1c)[0x4000061f3c]
redis_1 | /usr/local/bin/redis-server *:6379(processInputBuffer+0xea)[0x40000645aa]
redis_1 | /usr/local/bin/redis-server *:6379(+0xf3bd8)[0x40000f3bd8]
redis_1 | /usr/local/bin/redis-server *:6379(aeProcessEvents+0x2a1)[0x4000045ce1]
redis_1 | /usr/local/bin/redis-server *:6379(aeMain+0x1d)[0x4000045f3d]
redis_1 | /usr/local/bin/redis-server *:6379(main+0x314)[0x40000426f4]
redis_1 | /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)[0x40021b009b]
redis_1 | /usr/local/bin/redis-server *:6379(_start+0x2a)[0x4000042bda]
redis_1 |
redis_1 | ------ REGISTERS ------
redis_1 | 1:M 02 Jul 2021 20:01:30.333 #
redis_1 | RAX:000000400a938400 RBX:8c944b4f028c946e
redis_1 | RCX:0000000000003640 RDX:00000000000002b1
redis_1 | RDI:000000400a78d1f2 RSI:000000400a1a2552
redis_1 | RBP:0000004002a5d500 RSP:0000004001c2d9a0
redis_1 | R8 :000000000000004f R9 :3261623930663961
redis_1 | R10:000000400a78d1eb R11:0000004002316220
redis_1 | R12:000000400a78d1a3 R13:0000004002a5d530
redis_1 | R14:3b00c4b6c1df62b1 R15:0000004002a5d550
redis_1 | RIP:000000400004812c EFL:0000000000000282
redis_1 | CSGSFS:002b000000000033
redis_1 | 1:M 02 Jul 2021 20:01:30.333 # (0000004001c2d9af) -> 000000400004cc6d
redis_1 | 1:M 02 Jul 2021 20:01:30.333 # (0000004001c2d9ae) -> 0000004009e3b600
redis_1 | 1:M 02 Jul 2021 20:01:30.333 # (0000004001c2d9ad) -> 0000004009e3b600
redis_1 | 1:M 02 Jul 2021 20:01:30.333 # (0000004001c2d9ac) -> 0000000000000bd8
redis_1 | 1:M 02 Jul 2021 20:01:30.333 # (0000004001c2d9ab) -> 00000040000905be
redis_1 | 1:M 02 Jul 2021 20:01:30.333 # (0000004001c2d9aa) -> 000000000000000f
redis_1 | 1:M 02 Jul 2021 20:01:30.333 # (0000004001c2d9a9) -> 000000400008ea3e
redis_1 | 1:M 02 Jul 2021 20:01:30.333 # (0000004001c2d9a8) -> 000000400a9495f2
redis_1 | 1:M 02 Jul 2021 20:01:30.333 # (0000004001c2d9a7) -> 000000400008e9e7
redis_1 | 1:M 02 Jul 2021 20:01:30.333 # (0000004001c2d9a6) -> 00000040001f72e8
redis_1 | 1:M 02 Jul 2021 20:01:30.333 # (0000004001c2d9a5) -> 00000000b4baad34
redis_1 | 1:M 02 Jul 2021 20:01:30.333 # (0000004001c2d9a4) -> 0000000000000000
redis_1 | 1:M 02 Jul 2021 20:01:30.333 # (0000004001c2d9a3) -> 0000000000000000
redis_1 | 1:M 02 Jul 2021 20:01:30.333 # (0000004001c2d9a2) -> 000000000000000f
redis_1 | 1:M 02 Jul 2021 20:01:30.333 # (0000004001c2d9a1) -> 0000004009e3b600
redis_1 | 1:M 02 Jul 2021 20:01:30.333 # (0000004001c2d9a0) -> 0000004002a57000
redis_1 |
redis_1 | ------ INFO OUTPUT ------
redis_1 | # Server
redis_1 | redis_version:6.2.3
redis_1 | redis_git_sha1:00000000
redis_1 | redis_git_dirty:0
redis_1 | redis_build_id:33cdf76a1080ed11
redis_1 | redis_mode:standalone
redis_1 | os:Linux 5.10.25-linuxkit x86_64
redis_1 | arch_bits:64
redis_1 | multiplexing_api:epoll
redis_1 | atomicvar_api:c11-builtin
redis_1 | gcc_version:8.3.0
redis_1 | process_id:1
redis_1 | process_supervised:no
redis_1 | run_id:da954554367f78c55a1e5bbbd55245210a1bed3b
redis_1 | tcp_port:6379
redis_1 | server_time_usec:1625256090319012
redis_1 | uptime_in_seconds:809
redis_1 | uptime_in_days:0
redis_1 | hz:10
redis_1 | configured_hz:10
redis_1 | lru_clock:14643354
redis_1 | executable:/usr/local/bin/redis-server
redis_1 | config_file:
redis_1 | io_threads_active:0
redis_1 |
redis_1 | # Clients
redis_1 | connected_clients:36
redis_1 | cluster_connections:0
redis_1 | maxclients:10000
redis_1 | client_recent_max_input_buffer:99
redis_1 | client_recent_max_output_buffer:0
redis_1 | blocked_clients:1
redis_1 | tracking_clients:0
redis_1 | clients_in_timeout_table:1
redis_1 |
redis_1 | # Memory
redis_1 | used_memory:13152200
redis_1 | used_memory_human:12.54M
redis_1 | used_memory_rss:0
redis_1 | used_memory_rss_human:0B
redis_1 | used_memory_peak:13154552
redis_1 | used_memory_peak_human:12.55M
redis_1 | used_memory_peak_perc:99.98%
redis_1 | used_memory_overhead:1582083
redis_1 | used_memory_startup:842976
redis_1 | used_memory_dataset:11570117
redis_1 | used_memory_dataset_perc:94.00%
redis_1 | allocator_allocated:13284592
redis_1 | allocator_active:13795328
redis_1 | allocator_resident:16527360
redis_1 | total_system_memory:10434994176
redis_1 | total_system_memory_human:9.72G
redis_1 | used_memory_lua:43008
redis_1 | used_memory_lua_human:42.00K
redis_1 | used_memory_scripts:296
redis_1 | used_memory_scripts_human:296B
redis_1 | number_of_cached_scripts:1
redis_1 | maxmemory:0
redis_1 | maxmemory_human:0B
redis_1 | maxmemory_policy:noeviction
redis_1 | allocator_frag_ratio:1.04
redis_1 | allocator_frag_bytes:510736
redis_1 | allocator_rss_ratio:1.20
redis_1 | allocator_rss_bytes:2732032
redis_1 | rss_overhead_ratio:0.00
redis_1 | rss_overhead_bytes:-16527360
redis_1 | mem_fragmentation_ratio:0.00
redis_1 | mem_fragmentation_bytes:-13150232
redis_1 | mem_not_counted_for_evict:0
redis_1 | mem_replication_backlog:0
redis_1 | mem_clients_slaves:0
redis_1 | mem_clients_normal:738475
redis_1 | mem_aof_buffer:0
redis_1 | mem_allocator:jemalloc-5.1.0
redis_1 | active_defrag_running:0
redis_1 | lazyfree_pending_objects:0
redis_1 | lazyfreed_objects:0
redis_1 |
redis_1 | # Persistence
redis_1 | loading:0
redis_1 | current_cow_size:0
redis_1 | current_cow_size_age:0
redis_1 | current_fork_perc:0.00
redis_1 | current_save_keys_processed:0
redis_1 | current_save_keys_total:0
redis_1 | rdb_changes_since_last_save:1300
redis_1 | rdb_bgsave_in_progress:0
redis_1 | rdb_last_save_time:1625255883
redis_1 | rdb_last_bgsave_status:ok
redis_1 | rdb_last_bgsave_time_sec:0
redis_1 | rdb_current_bgsave_time_sec:-1
redis_1 | rdb_last_cow_size:2568192
redis_1 | aof_enabled:0
redis_1 | aof_rewrite_in_progress:0
redis_1 | aof_rewrite_scheduled:0
redis_1 | aof_last_rewrite_time_sec:-1
redis_1 | aof_current_rewrite_time_sec:-1
redis_1 | aof_last_bgrewrite_status:ok
redis_1 | aof_last_write_status:ok
redis_1 | aof_last_cow_size:0
redis_1 | module_fork_in_progress:0
redis_1 | module_fork_last_cow_size:0
redis_1 |
redis_1 | # Stats
redis_1 | total_connections_received:193
redis_1 | total_commands_processed:37294
redis_1 | instantaneous_ops_per_sec:128
redis_1 | total_net_input_bytes:14912920
redis_1 | total_net_output_bytes:52906653
redis_1 | instantaneous_input_kbps:26.49
redis_1 | instantaneous_output_kbps:246.36
redis_1 | rejected_connections:0
redis_1 | sync_full:0
redis_1 | sync_partial_ok:0
redis_1 | sync_partial_err:0
redis_1 | expired_keys:0
redis_1 | expired_stale_perc:0.00
redis_1 | expired_time_cap_reached_count:0
redis_1 | expire_cycle_cpu_milliseconds:28
redis_1 | evicted_keys:0
redis_1 | keyspace_hits:27192
redis_1 | keyspace_misses:4527
redis_1 | pubsub_channels:0
redis_1 | pubsub_patterns:3
redis_1 | latest_fork_usec:1460
redis_1 | total_forks:2
redis_1 | migrate_cached_sockets:0
redis_1 | slave_expires_tracked_keys:0
redis_1 | active_defrag_hits:0
redis_1 | active_defrag_misses:0
redis_1 | active_defrag_key_hits:0
redis_1 | active_defrag_key_misses:0
redis_1 | tracking_total_keys:0
redis_1 | tracking_total_items:0
redis_1 | tracking_total_prefixes:0
redis_1 | unexpected_error_replies:0
redis_1 | total_error_replies:158
redis_1 | dump_payload_sanitizations:0
redis_1 | total_reads_processed:37426
redis_1 | total_writes_processed:38139
redis_1 | io_threaded_reads_processed:0
redis_1 | io_threaded_writes_processed:0
redis_1 |
redis_1 | # Replication
redis_1 | role:master
redis_1 | connected_slaves:0
redis_1 | master_failover_state:no-failover
redis_1 | master_replid:70b7396c9972d56847c235418f27bc9c8bfca030
redis_1 | master_replid2:0000000000000000000000000000000000000000
redis_1 | master_repl_offset:0
redis_1 | second_repl_offset:-1
redis_1 | repl_backlog_active:0
redis_1 | repl_backlog_size:1048576
redis_1 | repl_backlog_first_byte_offset:0
redis_1 | repl_backlog_histlen:0
redis_1 |
redis_1 | # CPU
redis_1 | used_cpu_sys:3.766021
redis_1 | used_cpu_user:5.687902
redis_1 | used_cpu_sys_children:0.039034
redis_1 | used_cpu_user_children:0.148486
redis_1 | used_cpu_sys_main_thread:3.739147
redis_1 | used_cpu_user_main_thread:5.615179
redis_1 |
redis_1 | # Modules
redis_1 | module:name=graph,ver=20406,api=1,filters=0,usedby=[],using=[],options=[]
redis_1 |
redis_1 | # Commandstats
redis_1 | cmdstat_get:calls=10,usec=159,usec_per_call=15.90,rejected_calls=0,failed_calls=0
redis_1 | cmdstat_set:calls=14,usec=4739,usec_per_call=338.50,rejected_calls=0,failed_calls=0
redis_1 | cmdstat_multi:calls=33,usec=181,usec_per_call=5.48,rejected_calls=0,failed_calls=0
redis_1 | cmdstat_psubscribe:calls=3,usec=333,usec_per_call=111.00,rejected_calls=0,failed_calls=0
redis_1 | cmdstat_lpush:calls=1,usec=2013,usec_per_call=2013.00,rejected_calls=0,failed_calls=0
redis_1 | cmdstat_llen:calls=92,usec=704,usec_per_call=7.65,rejected_calls=0,failed_calls=0
redis_1 | cmdstat_zadd:calls=1,usec=2821,usec_per_call=2821.00,rejected_calls=0,failed_calls=0
redis_1 | cmdstat_sadd:calls=14,usec=2719,usec_per_call=194.21,rejected_calls=0,failed_calls=0
redis_1 | cmdstat_ping:calls=26,usec=110,usec_per_call=4.23,rejected_calls=157,failed_calls=0
redis_1 | cmdstat_srem:calls=9,usec=1043,usec_per_call=115.89,rejected_calls=0,failed_calls=0
redis_1 | cmdstat_brpop:calls=727,usec=65579,usec_per_call=90.20,rejected_calls=0,failed_calls=0
redis_1 | cmdstat_zrevrangebyscore:calls=10,usec=9039,usec_per_call=903.90,rejected_calls=0,failed_calls=0
redis_1 | cmdstat_auth:calls=36,usec=1504,usec_per_call=41.78,rejected_calls=0,failed_calls=0
redis_1 | cmdstat_del:calls=46,usec=579,usec_per_call=12.59,rejected_calls=0,failed_calls=0
redis_1 | cmdstat_hset:calls=4100,usec=186941,usec_per_call=45.60,rejected_calls=0,failed_calls=0
redis_1 | cmdstat_evalsha:calls=11,usec=6465,usec_per_call=587.73,rejected_calls=0,failed_calls=1
redis_1 | cmdstat_script:calls=1,usec=10292,usec_per_call=10292.00,rejected_calls=0,failed_calls=0
redis_1 | cmdstat_exec:calls=33,usec=6098,usec_per_call=184.79,rejected_calls=0,failed_calls=0
redis_1 | cmdstat_publish:calls=521,usec=19593,usec_per_call=37.61,rejected_calls=0,failed_calls=0
redis_1 | cmdstat_hget:calls=31606,usec=454245,usec_per_call=14.37,rejected_calls=0,failed_calls=0
redis_1 |
redis_1 | # Errorstats
redis_1 | errorstat_NOAUTH:count=157
redis_1 | errorstat_NOSCRIPT:count=1
redis_1 |
redis_1 | # Cluster
redis_1 | cluster_enabled:0
redis_1 |
redis_1 | # Keyspace
redis_1 | db0:keys=6,expires=0,avg_ttl=0
redis_1 |
redis_1 | ------ CLIENT LIST OUTPUT ------
redis_1 | id=44 addr=172.20.0.9:44528 laddr=172.20.0.6:6379 fd=32 name= age=729 idle=39 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20528 events=r cmd=evalsha user=default redir=-1
redis_1 | id=34 addr=172.20.0.12:42924 laddr=172.20.0.6:6379 fd=23 name= age=732 idle=730 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
redis_1 | id=35 addr=172.20.0.12:42926 laddr=172.20.0.6:6379 fd=24 name= age=732 idle=730 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
redis_1 | id=20 addr=172.20.0.12:42888 laddr=172.20.0.6:6379 fd=9 name= age=734 idle=0 flags=P db=0 sub=0 psub=1 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=psubscribe user=default redir=-1
redis_1 | id=21 addr=172.20.0.12:42890 laddr=172.20.0.6:6379 fd=10 name= age=734 idle=734 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=sadd user=default redir=-1
redis_1 | id=22 addr=172.20.0.12:42898 laddr=172.20.0.6:6379 fd=11 name= age=732 idle=731 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20536 events=r cmd=brpop user=default redir=-1
redis_1 | id=23 addr=172.20.0.12:42902 laddr=172.20.0.6:6379 fd=12 name= age=732 idle=731 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20536 events=r cmd=brpop user=default redir=-1
redis_1 | id=24 addr=172.20.0.12:42904 laddr=172.20.0.6:6379 fd=13 name= age=732 idle=731 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20536 events=r cmd=brpop user=default redir=-1
redis_1 | id=19 addr=172.20.0.12:42886 laddr=172.20.0.6:6379 fd=8 name= age=734 idle=734 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=ping user=default redir=-1
redis_1 | id=40 addr=172.20.0.12:42936 laddr=172.20.0.6:6379 fd=29 name= age=732 idle=4 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=publish user=default redir=-1
redis_1 | id=41 addr=172.20.0.12:42938 laddr=172.20.0.6:6379 fd=30 name= age=732 idle=732 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=publish user=default redir=-1
redis_1 | id=25 addr=172.20.0.12:42906 laddr=172.20.0.6:6379 fd=14 name= age=732 idle=731 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20536 events=r cmd=brpop user=default redir=-1
redis_1 | id=46 addr=172.20.0.9:44534 laddr=172.20.0.6:6379 fd=34 name= age=729 idle=729 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=ping user=default redir=-1
redis_1 | id=47 addr=172.20.0.9:44536 laddr=172.20.0.6:6379 fd=35 name= age=729 idle=729 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=publish user=default redir=-1
redis_1 | id=52 addr=172.20.0.9:44546 laddr=172.20.0.6:6379 fd=40 name= age=728 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=61464 events=r cmd=publish user=default redir=-1
redis_1 | id=26 addr=172.20.0.12:42908 laddr=172.20.0.6:6379 fd=15 name= age=732 idle=731 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20536 events=r cmd=brpop user=default redir=-1
redis_1 | id=37 addr=172.20.0.12:42930 laddr=172.20.0.6:6379 fd=26 name= age=732 idle=732 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=publish user=default redir=-1
redis_1 | id=38 addr=172.20.0.12:42932 laddr=172.20.0.6:6379 fd=27 name= age=732 idle=732 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=ping user=default redir=-1
redis_1 | id=50 addr=172.20.0.9:44542 laddr=172.20.0.6:6379 fd=38 name= age=728 idle=4 flags=P db=0 sub=0 psub=1 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=psubscribe user=default redir=-1
redis_1 | id=51 addr=172.20.0.9:44544 laddr=172.20.0.6:6379 fd=39 name= age=728 idle=728 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=sadd user=default redir=-1
redis_1 | id=39 addr=172.20.0.12:42934 laddr=172.20.0.6:6379 fd=28 name= age=732 idle=731 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20536 events=r cmd=brpop user=default redir=-1
redis_1 | id=27 addr=172.20.0.12:42910 laddr=172.20.0.6:6379 fd=16 name= age=732 idle=731 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20536 events=r cmd=brpop user=default redir=-1
redis_1 | id=79 addr=172.20.0.10:42318 laddr=172.20.0.6:6379 fd=41 name= age=595 idle=595 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=ping user=default redir=-1
redis_1 | id=80 addr=172.20.0.10:42320 laddr=172.20.0.6:6379 fd=42 name= age=595 idle=595 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=lpush user=default redir=-1
redis_1 | id=28 addr=172.20.0.12:42912 laddr=172.20.0.6:6379 fd=17 name= age=732 idle=732 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=ping user=default redir=-1
redis_1 | id=48 addr=172.20.0.9:44538 laddr=172.20.0.6:6379 fd=36 name= age=728 idle=0 flags=P db=0 sub=0 psub=1 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=psubscribe user=default redir=-1
redis_1 | id=49 addr=172.20.0.9:44540 laddr=172.20.0.6:6379 fd=37 name= age=728 idle=728 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=sadd user=default redir=-1
redis_1 | id=29 addr=172.20.0.12:42914 laddr=172.20.0.6:6379 fd=18 name= age=732 idle=731 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20536 events=r cmd=brpop user=default redir=-1
redis_1 | id=30 addr=172.20.0.12:42916 laddr=172.20.0.6:6379 fd=19 name= age=732 idle=730 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
redis_1 | id=36 addr=172.20.0.12:42928 laddr=172.20.0.6:6379 fd=25 name= age=732 idle=730 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
redis_1 | id=45 addr=172.20.0.9:44530 laddr=172.20.0.6:6379 fd=33 name= age=729 idle=729 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=ping user=default redir=-1
redis_1 | id=31 addr=172.20.0.12:42918 laddr=172.20.0.6:6379 fd=20 name= age=732 idle=730 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
redis_1 | id=32 addr=172.20.0.12:42920 laddr=172.20.0.6:6379 fd=21 name= age=732 idle=730 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
redis_1 | id=33 addr=172.20.0.12:42922 laddr=172.20.0.6:6379 fd=22 name= age=732 idle=730 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=exec user=default redir=-1
redis_1 | id=83 addr=172.20.0.9:45140 laddr=172.20.0.6:6379 fd=43 name= age=584 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=126 qbuf-free=40828 argv-mem=102 obl=0 oll=0 omem=0 tot-mem=61566 events=r cmd=hget user=default redir=-1
redis_1 | id=43 addr=172.20.0.9:44526 laddr=172.20.0.6:6379 fd=31 name= age=729 idle=1 flags=b db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=40954 argv-mem=43 obl=0 oll=0 omem=0 tot-mem=61531 events=r cmd=brpop user=default redir=-1
redis_1 |
redis_1 | ------ CURRENT CLIENT INFO ------
redis_1 | id=83 addr=172.20.0.9:45140 laddr=172.20.0.6:6379 fd=43 name= age=584 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=126 qbuf-free=40828 argv-mem=102 obl=0 oll=0 omem=0 tot-mem=61566 events=r cmd=hget user=default redir=-1
redis_1 | argv[0]: 'HGET'
redis_1 | argv[1]: 'annotator:responses'
redis_1 | argv[2]: '��D'
redis_1 | 1:M 02 Jul 2021 20:01:30.340 # key 'annotator:responses' found in DB containing the following object:
redis_1 | 1:M 02 Jul 2021 20:01:30.340 # Object type: 4
redis_1 | 1:M 02 Jul 2021 20:01:30.340 # Object encoding: 2
redis_1 | 1:M 02 Jul 2021 20:01:30.340 # Object refcount: 1
redis_1 |
redis_1 | ------ MODULES INFO OUTPUT ------
redis_1 | # graph_executing commands
redis_1 |
redis_1 | ------ FAST MEMORY TEST ------
redis_1 | 1:M 02 Jul 2021 20:01:30.343 # Bio thread for job type #0 terminated
redis_1 | 1:M 02 Jul 2021 20:01:30.344 # Bio thread for job type #1 terminated
redis_1 | 1:M 02 Jul 2021 20:01:30.344 # Bio thread for job type #2 terminated
redis_1 | *** Preparing to test memory region 4000201000 (2547712 bytes)
redis_1 | *** Preparing to test memory region 4001c57000 (12288 bytes)
redis_1 | *** Preparing to test memory region 4001ddf000 (8192 bytes)
redis_1 | *** Preparing to test memory region 4002167000 (16384 bytes)
redis_1 | *** Preparing to test memory region 4002188000 (16384 bytes)
redis_1 | *** Preparing to test memory region 4002349000 (40960 bytes)
redis_1 | *** Preparing to test memory region 4002600000 (2097152 bytes)
redis_1 | *** Preparing to test memory region 4002953000 (2097152 bytes)
redis_1 | *** Preparing to test memory region 4003000000 (4194304 bytes)
redis_1 | *** Preparing to test memory region 4003dd1000 (12288 bytes)
redis_1 | *** Preparing to test memory region 4003e08000 (8388608 bytes)
redis_1 | *** Preparing to test memory region 4004609000 (8388608 bytes)
redis_1 | *** Preparing to test memory region 4004e0a000 (8388608 bytes)
redis_1 | *** Preparing to test memory region 400560b000 (8388608 bytes)
redis_1 | *** Preparing to test memory region 4005e0c000 (8388608 bytes)
redis_1 | *** Preparing to test memory region 400660d000 (8388608 bytes)
redis_1 | *** Preparing to test memory region 4006e0e000 (8388608 bytes)
redis_1 | *** Preparing to test memory region 400760f000 (8388608 bytes)
redis_1 | *** Preparing to test memory region 4007e10000 (8388608 bytes)
redis_1 | *** Preparing to test memory region 4008611000 (8388608 bytes)
redis_1 | *** Preparing to test memory region 4008e12000 (8388608 bytes)
redis_1 | *** Preparing to test memory region 4009613000 (8388608 bytes)
redis_1 | *** Preparing to test memory region 4009e13000 (13631488 bytes)
redis_1 | .O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
redis_1 | Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
redis_1 |
redis_1 | ------ DUMPING CODE AROUND EIP ------
redis_1 | Symbol: dictFind (base: 0x40000480d0)
redis_1 | Module: /usr/local/bin/redis-server *:6379 (base 0x4000000000)
redis_1 | $ xxd -r -p /tmp/dump.hex /tmp/dump.bin
redis_1 | $ objdump --adjust-vma=0x40000480d0 -D -b binary -m i386:x86-64 /tmp/dump.bin
redis_1 | ------
redis_1 | 1:M 02 Jul 2021 20:01:31.090 # dump of function (hexdump of 220 bytes):
redis_1 | 415741564155415455534883ec08488b4748480347280f848f00000048837f50ff4989f44889fd740b66837f58000f848c000000488b45004c89e74c8d6d104c8d7d50ff104989c6498b5510498b45004c21f2488b1cd04885db743f488b134939d47514eb47662e0f1f840000000000488b134c39e27435488b4500488b40184885c0740d488b7d084c89e6ffd085c0751b488b5b104885db75d548837d50ff74094983c5204d39ef759d31db4883c4084889d85b5d415c415d415e415fc390be01000000e8e6f4ffffe965ffffff904883ec08e827ffffff4885c0
redis_1 | Function at 0x4000047680 is dictRehash
redis_1 | Function at 0x40000480d0 is dictFind
redis_1 |
redis_1 | === REDIS BUG REPORT END. Make sure to include from START to END. ===
redis_1 |
redis_1 | Please report the crash by opening an issue on github:
redis_1 |
redis_1 | http://github.com/redis/redis/issues
redis_1 |
redis_1 | Suspect RAM error? Use redis-server --test-memory to verify it.
redis_1 |
redis_1 | qemu: uncaught target signal 11 (Segmentation fault) - core dumped
airflow-webserver_1 | 172.20.0.1 - - [02/Jul/2021:20:01:31 +0000] "GET /object/task_instances?dag_id=annotate_dug&execution_date=2021-06-30T12%3A38%3A50.695362%2B00%3A00 HTTP/1.1" 200 4554 "http://localhost:8080/graph?dag_id=annotate_dug" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36"
airflow-webserver_1 | 127.0.0.1 - - [02/Jul/2021:20:01:34 +0000] "GET /health HTTP/1.1" 200 187 "-" "curl/7.64.0"
Additional information
- OS distribution and version
Mac M1 chip laptop. Big Sur 11.4
- Steps to reproduce (if any)
An airflow pipeline. I am the only M1 laptop user on my project at RENCI HeLx platform.
Comment From: madolson
Just to confirm, this consistently crashes in the same place when you run your workflow (I assume that is what an airflow pipeline is?)
Comment From: madolson
The crash seems to be on the hash entry being garbage, (note 5c = 92 and %rbx is the hash entry), so when we dereference the hash entry is crashes. It looks like he is completely garbage, since it's way out of the heap space.
490 if (key==he->key || dictCompareKeys(d, key, he->key))
0x00000000004308ac <+92>: mov (%rbx),%rdx
0x00000000004308af <+95>: cmp %rdx,%r12
0x00000000004308b2 <+98>: jne 0x4308be <dictFind+110>
0x00000000004308b4 <+100>: jmp 0x4308f3 <dictFind+163>
0x00000000004308b6 <+102>: mov (%rbx),%rdx
0x00000000004308b9 <+105>: cmp %r12,%rdx
0x00000000004308bc <+108>: je 0x4308f3 <dictFind+163>
0x00000000004308be <+110>: mov 0x0(%rbp),%rax
0x00000000004308c2 <+114>: mov 0x18(%rax),%rax
0x00000000004308c6 <+118>: test %rax,%rax
0x00000000004308c9 <+121>: je 0x4308d8 <dictFind+136>
0x00000000004308cb <+123>: mov 0x8(%rbp),%rdi
0x00000000004308cf <+127>: mov %r12,%rsi
0x00000000004308d2 <+130>: callq *%rax
0x00000000004308d4 <+132>: test %eax,%eax
0x00000000004308d6 <+134>: jne 0x4308f3 <dictFind+163>
Comment From: warrenstephens
"It" has been crashing for a few days under previous versions -- maybe the same issue, but am not sure -- meaning the airflow pipeline.
I had just upgraded redis to 6.2.3 a few hours ago in the docker-compose yaml, and was paying more attention to what was going on, when I spotted this crash.
I have cleaned out some docker things, and am going to restart the whole thing now.
Comment From: warrenstephens
My airflow pipeline had been crashing after about 12 minutes of doing stuff, with redis probably being one of the causes, but on this last run redis did not crash, although something in our code still crashed. So at this point, I may not be able to re-create any further redis crashes like the one posted -- but I hope that you got something to go on with that hash entry!
Comment From: madolson
Scanning through the code base I wasn't able to find any clear culprits. If it does crash again, would love to better understand the workload you ran that caused this.
Comment From: warrenstephens
It is crashing still -- at the same instruction 0x400004812c
Comment From: madolson
@warrenstephens Is there anyway you can get something that is semi-reducible that we can test on?
Comment From: warrenstephens
I have upgraded to RedisGraph 2.4.7, and the problem has gone away.
That version includes the comment "updating to the latest redis docker" which may be the source of the solution? --although the Redis version still looks like 6.2.3.
I am running this on a new Apple M1 chip based laptop.
Comment From: madolson
I wonder if RedisGraph was causing the issue (Maybe they were corrupting the hash somehow). Would you mind opening an issue to them, https://github.com/RedisGraph/RedisGraph, to see if they have any input?
Comment From: sundb
It also looks like it is similar to #10341.
Comment From: oranagra
following the discussion in the other ticket, i think we can now conclude that using jemalloc on docker on arm (qemu) is unsupported (unresolved jemalloc bug), and we can close all these related tickets.
Comment From: oranagra
closing this, see https://github.com/redis/redis/issues/10162#issuecomment-1072375709