Describe the bug
There is an error on redis:6.0 using docker on Mac M1 (8gb RAM for docker)
To reproduce
Start docker container, run application writting/reading redis db.
See redis crashed, lsee redis logs.
Expected behavior
No Redis crash should be
Additional information
redis | === REDIS BUG REPORT START: Cut & paste starting from here ===
redis | 1:M 29 Sep 2021 10:44:10.559 # Redis 6.0.15 crashed by signal: 11, si_code: 1
redis | 1:M 29 Sep 2021 10:44:10.560 # Crashed running the instruction at: 0x4000042dab
redis | 1:M 29 Sep 2021 10:44:10.560 # Accessing address: 0x3a226c6562616c
redis | 1:M 29 Sep 2021 10:44:10.560 # Failed assertion: (:0)
redis |
redis | ------ STACK TRACE ------
redis | EIP:
redis | /usr/local/bin/redis-server :6379(dictAddRaw+0xcb)[0x4000042dab]
redis |
redis | Backtrace:
redis | /usr/local/bin/redis-server :6379(logStackTrace+0x32)[0x400008d3c2]
redis | /usr/local/bin/redis-server :6379(sigsegvHandler+0xb0)[0x400008dad0]
redis | /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x4002158730]
redis | /usr/local/bin/redis-server :6379(dictAddRaw+0xcb)[0x4000042dab]
redis | /usr/local/bin/redis-server :6379(setTypeAdd+0xd8)[0x4000074c58]
redis | /usr/local/bin/redis-server :6379(saddCommand+0x65)[0x4000074d85]
redis | /usr/local/bin/redis-server :6379(call+0xaa)[0x400004735a]
redis | /usr/local/bin/redis-server :6379(processCommand+0x465)[0x4000047db5]
redis | /usr/local/bin/redis-server :6379(processCommandAndResetClient+0x10)[0x4000055fd0]
redis | /usr/local/bin/redis-server :6379(+0x5a9ca)[0x400005a9ca]
redis | /usr/local/bin/redis-server :6379(+0xda318)[0x40000da318]
redis | /usr/local/bin/redis-server :6379(aeProcessEvents+0x2e7)[0x4000040cd7]
redis | /usr/local/bin/redis-server :6379(aeMain+0x1d)[0x400004101d]
redis | /usr/local/bin/redis-server :6379(main+0x476)[0x400003d816]
redis | /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)[0x400218b09b]
redis | /usr/local/bin/redis-server :6379(_start+0x2a)[0x400003db5a]
redis |
redis | ------ INFO OUTPUT ------
redis | # Server
redis | redis_version:6.0.15
redis | redis_git_sha1:00000000
redis | redis_git_dirty:0
redis | redis_build_id:bc06c43d1c123d61
redis | redis_mode:standalone
redis | os:Linux 5.10.47-linuxkit x86_64
redis | arch_bits:64
redis | multiplexing_api:epoll
redis | atomicvar_api:atomic-builtin
redis | gcc_version:8.3.0
redis | process_id:1
redis | run_id:2accdfa1b348e6bd55f7b6551d44055cf2031143
redis | tcp_port:6379
redis | uptime_in_seconds:117
redis | uptime_in_days:0
redis | hz:10
redis | configured_hz:10
redis | lru_clock:5522298
redis | executable:/usr/local/bin/redis-server
redis | config_file:
redis | io_threads_active:0
redis |
redis | # Clients
redis | connected_clients:16
redis | client_recent_max_input_buffer:40960
redis | client_recent_max_output_buffer:0
redis | blocked_clients:0
redis | tracking_clients:0
redis | clients_in_timeout_table:0
redis |
redis | # Memory
redis | used_memory:13555520
redis | used_memory_human:12.93M
redis | used_memory_rss:0
redis | used_memory_rss_human:0B
redis | used_memory_peak:14397512
redis | used_memory_peak_human:13.73M
redis | used_memory_peak_perc:94.15%
redis | used_memory_overhead:1192080
redis | used_memory_startup:803304
redis | used_memory_dataset:12363440
redis | used_memory_dataset_perc:96.95%
redis | allocator_allocated:13676288
redis | allocator_active:14196736
redis | allocator_resident:16764928
redis | total_system_memory:8337653760
redis | total_system_memory_human:7.77G
redis | used_memory_lua:37888
redis | used_memory_lua_human:37.00K
redis | used_memory_scripts:0
redis | used_memory_scripts_human:0B
redis | number_of_cached_scripts:0
redis | maxmemory:524288000
redis | maxmemory_human:500.00M
redis | maxmemory_policy:allkeys-lru
redis | allocator_frag_ratio:1.04
redis | allocator_frag_bytes:520448
redis | allocator_rss_ratio:1.18
redis | allocator_rss_bytes:2568192
redis | rss_overhead_ratio:0.00
redis | rss_overhead_bytes:-16764928
redis | mem_fragmentation_ratio:0.00
redis | mem_fragmentation_bytes:-13506728
redis | mem_not_counted_for_evict:0
redis | mem_replication_backlog:0
redis | mem_clients_slaves:0
redis | mem_clients_normal:328136
redis | mem_aof_buffer:0
redis | mem_allocator:jemalloc-5.1.0
redis | active_defrag_running:0
redis | lazyfree_pending_objects:0
redis |
redis | # Persistence
redis | loading:0
redis | rdb_changes_since_last_save:4368
redis | rdb_bgsave_in_progress:0
redis | rdb_last_save_time:1632912133
redis | rdb_last_bgsave_status:ok
redis | rdb_last_bgsave_time_sec:-1
redis | rdb_current_bgsave_time_sec:-1
redis | rdb_last_cow_size:0
redis | aof_enabled:0
redis | aof_rewrite_in_progress:0
redis | aof_rewrite_scheduled:0
redis | aof_last_rewrite_time_sec:-1
redis | aof_current_rewrite_time_sec:-1
redis | aof_last_bgrewrite_status:ok
redis | aof_last_write_status:ok
redis | aof_last_cow_size:0
redis | module_fork_in_progress:0
redis | module_fork_last_cow_size:0
redis |
redis | # Stats
redis | total_connections_received:16
redis | total_commands_processed:9502
redis | instantaneous_ops_per_sec:1227
redis | total_net_input_bytes:14499900
redis | total_net_output_bytes:17974966
redis | instantaneous_input_kbps:226.18
redis | instantaneous_output_kbps:118.62
redis | rejected_connections:0
redis | sync_full:0
redis | sync_partial_ok:0
redis | sync_partial_err:0
redis | expired_keys:0
redis | expired_stale_perc:0.00
redis | expired_time_cap_reached_count:0
redis | expire_cycle_cpu_milliseconds:8
redis | evicted_keys:0
redis | keyspace_hits:1135
redis | keyspace_misses:2860
redis | pubsub_channels:0
redis | pubsub_patterns:0
redis | latest_fork_usec:0
redis | migrate_cached_sockets:0
redis | slave_expires_tracked_keys:0
redis | active_defrag_hits:0
redis | active_defrag_misses:0
redis | active_defrag_key_hits:0
redis | active_defrag_key_misses:0
redis | tracking_total_keys:0
redis | tracking_total_items:0
redis | tracking_total_prefixes:0
redis | unexpected_error_replies:0
redis | total_reads_processed:5980
redis | total_writes_processed:5187
redis | io_threaded_reads_processed:0
redis | io_threaded_writes_processed:0
redis |
redis | # Replication
redis | role:master
redis | connected_slaves:0
redis | master_replid:671826bf1909ec1eb0d37cf4020c61acd302c69d
redis | master_replid2:0000000000000000000000000000000000000000
redis | master_repl_offset:0
redis | second_repl_offset:-1
redis | repl_backlog_active:0
redis | repl_backlog_size:1048576
redis | repl_backlog_first_byte_offset:0
redis | repl_backlog_histlen:0
redis |
redis | # CPU
redis | used_cpu_sys:1.278236
redis | used_cpu_user:1.494135
redis | used_cpu_sys_children:0.010561
redis | used_cpu_user_children:0.049518
redis |
redis | # Modules
redis |
redis | # Commandstats
redis | cmdstat_hmset:calls=1102,usec=72362,usec_per_call=65.66
redis | cmdstat_hincrby:calls=42,usec=4351,usec_per_call=103.60
redis | cmdstat_select:calls=10,usec=416,usec_per_call=41.60
redis | cmdstat_dbsize:calls=1,usec=43,usec_per_call=43.00
redis | cmdstat_module:calls=1,usec=96,usec_per_call=96.00
redis | cmdstat_scan:calls=1,usec=674,usec_per_call=674.00
redis | cmdstat_info:calls=2,usec=5684,usec_per_call=2842.00
redis | cmdstat_ping:calls=4,usec=169,usec_per_call=42.25
redis | cmdstat_expire:calls=7,usec=212,usec_per_call=30.29
redis | cmdstat_hget:calls=3993,usec=86605,usec_per_call=21.69
redis | cmdstat_hmget:calls=2,usec=150,usec_per_call=75.00
redis | cmdstat_sadd:calls=4336,usec=125110,usec_per_call=28.85
redis | cmdstat_config:calls=1,usec=1399,usec_per_call=1399.00
redis |
redis | # Cluster
redis | cluster_enabled:0
redis |
redis | # Keyspace
redis | db0:keys=1097,expires=5,avg_ttl=19754895
redis | db2:keys=2,expires=2,avg_ttl=21582131
redis |
redis | ------ CLIENT LIST OUTPUT ------
redis | id=11 addr=172.23.0.7:40560 fd=15 name= age=50 idle=50 flags=N db=1 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=select user=default
redis | id=12 addr=172.23.0.7:40562 fd=16 name= age=50 idle=7 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=hget user=default
redis | id=13 addr=172.23.0.7:40564 fd=17 name= age=50 idle=50 flags=N db=1 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=select user=default
redis | id=14 addr=172.23.0.7:40566 fd=18 name= age=50 idle=20 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=hget user=default
redis | id=15 addr=172.23.0.7:40568 fd=19 name= age=50 idle=50 flags=N db=1 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=select user=default
redis | id=18 addr=172.23.0.7:40580 fd=22 name= age=20 idle=20 flags=N db=2 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default
redis | id=19 addr=172.23.0.7:40582 fd=23 name= age=20 idle=0 flags=N db=2 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20520 events=r cmd=hincrby user=default
redis | id=16 addr=172.23.0.7:40570 fd=20 name= age=50 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=872 qbuf-free=31896 argv-mem=70 obl=13 oll=0 omem=0 tot-mem=61534 events=r cmd=sadd user=default
redis | id=9 addr=172.23.0.7:40552 fd=13 name= age=79 idle=79 flags=N db=1 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=select user=default
redis | id=10 addr=172.23.0.7:40554 fd=14 name= age=79 idle=22 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=hget user=default
redis | id=4 addr=172.23.0.1:61370 fd=8 name= age=103 idle=13 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20496 events=r cmd=ping user=default
redis | id=5 addr=172.23.0.7:40482 fd=9 name= age=97 idle=97 flags=N db=1 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=select user=default
redis | id=17 addr=172.23.0.7:40578 fd=21 name= age=23 idle=23 flags=N db=2 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=expire user=default
redis | id=6 addr=172.23.0.7:40484 fd=10 name= age=97 idle=22 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=hget user=default
redis | id=7 addr=172.23.0.7:40486 fd=11 name= age=97 idle=97 flags=N db=1 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20504 events=r cmd=select user=default
redis | id=8 addr=172.23.0.7:40488 fd=12 name= age=97 idle=23 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=20512 events=r cmd=hget user=default
redis |
redis | ------ CURRENT CLIENT INFO ------
redis | id=16 addr=172.23.0.7:40570 fd=20 name= age=50 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=872 qbuf-free=31896 argv-mem=70 obl=13 oll=0 omem=0 tot-mem=61534 events=r cmd=sadd user=default
redis | argv[0]: 'SADD'
redis | argv[1]: 'zc:ti:d19_DB_DDL'
redis | argv[2]: 'd19_DB_PDO_MYSQL_DDL_VERTEX_ORDER_INVOICE_STATUS_2'
redis | 1:M 29 Sep 2021 10:44:10.577 # key 'zc:ti:d19_DB_DDL' found in DB containing the following object:
redis | 1:M 29 Sep 2021 10:44:10.577 # Object type: 2
redis | 1:M 29 Sep 2021 10:44:10.577 # Object encoding: 2
redis | 1:M 29 Sep 2021 10:44:10.577 # Object refcount: 1
redis | 1:M 29 Sep 2021 10:44:10.577 # Set size: 1025
redis |
redis | ------ REGISTERS ------
redis | 1:M 29 Sep 2021 10:44:10.577 #
redis | RAX:0000004005708640 RBX:0000004002aa7903
redis | RCX:a9f34c6b5448e5ea RDX:a1bb64e34cefb119
redis | RDI:000000000000005f RSI:7aa43cb24c280a95
redis | RBP:0848288818a754f3 RSP:0000004001c08860
redis | R8 :000000400018b038 R9 :0000004002a33050
redis | R10:0000004002aa7933 R11:0000000000000002
redis | R12:223a226c6562616c R13:0000000000000000
redis | R14:0000004002a33020 R15:00000000000004f3
redis | RIP:0000004000042dab EFL:0000000000000206
redis | CSGSFS:002b000000000033
redis | 1:M 29 Sep 2021 10:44:10.577 # (0000004001c0886f) -> 0000000000000002
redis | 1:M 29 Sep 2021 10:44:10.578 # (0000004001c0886e) -> 000000000000000f
redis | 1:M 29 Sep 2021 10:44:10.578 # (0000004001c0886d) -> 0005cd2002a4b2c2
redis | 1:M 29 Sep 2021 10:44:10.578 # (0000004001c0886c) -> 0000000000000000
redis | 1:M 29 Sep 2021 10:44:10.578 # (0000004001c0886b) -> 0000004000074c58
redis | 1:M 29 Sep 2021 10:44:10.578 # (0000004001c0886a) -> 000000000000000f
redis | 1:M 29 Sep 2021 10:44:10.578 # (0000004001c08869) -> 0000004002a902e0
redis | 1:M 29 Sep 2021 10:44:10.578 # (0000004001c08868) -> 0000000000000000
redis | 1:M 29 Sep 2021 10:44:10.578 # (0000004001c08867) -> 0000004002aa7903
redis | 1:M 29 Sep 2021 10:44:10.578 # (0000004001c08866) -> 0000000000000000
redis | 1:M 29 Sep 2021 10:44:10.578 # (0000004001c08865) -> 0000004002a33020
redis | 1:M 29 Sep 2021 10:44:10.578 # (0000004001c08864) -> 0000004002947cc8
redis | 1:M 29 Sep 2021 10:44:10.578 # (0000004001c08863) -> 0000004002a33030
redis | 1:M 29 Sep 2021 10:44:10.578 # (0000004001c08862) -> 0000004002a33070
redis | 1:M 29 Sep 2021 10:44:10.579 # (0000004001c08861) -> 0000000000002798
redis | 1:M 29 Sep 2021 10:44:10.579 # (0000004001c08860) -> 0000000000000000
redis |
redis | ------ MODULES INFO OUTPUT ------
redis |
redis | ------ FAST MEMORY TEST ------
redis | 1:M 29 Sep 2021 10:44:10.583 # Bio thread for job type #0 terminated
redis | 1:M 29 Sep 2021 10:44:10.583 # Bio thread for job type #1 terminated
redis | 1:M 29 Sep 2021 10:44:10.584 # Bio thread for job type #2 terminated
redis | *** Preparing to test memory region 40001dc000 (2547712 bytes)
redis | *** Preparing to test memory region 4001c32000 (12288 bytes)
redis | *** Preparing to test memory region 4001dba000 (8192 bytes)
redis | *** Preparing to test memory region 4002142000 (16384 bytes)
redis | *** Preparing to test memory region 4002163000 (16384 bytes)
redis | *** Preparing to test memory region 4002324000 (40960 bytes)
redis | *** Preparing to test memory region 4002600000 (2097152 bytes)
redis | *** Preparing to test memory region 400292e000 (2097152 bytes)
redis | *** Preparing to test memory region 4003000000 (4194304 bytes)
redis | *** Preparing to test memory region 400352e000 (8388608 bytes)
redis | *** Preparing to test memory region 4003d2f000 (8388608 bytes)
redis | *** Preparing to test memory region 4004530000 (8388608 bytes)
redis | *** Preparing to test memory region 4004d31000 (8388608 bytes)
redis | *** Preparing to test memory region 4005531000 (18874368 bytes)
redis | .O.O.O.O.O.O.O.O.O.O.O.O.O.O
redis | Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
redis |
redis | ------ DUMPING CODE AROUND EIP ------
redis | Symbol: dictAddRaw (base: 0x4000042ce0)
redis | Module: /usr/local/bin/redis-server :6379 (base 0x4000000000)
redis | $ xxd -r -p /tmp/dump.hex /tmp/dump.bin
redis | $ objdump --adjust-vma=0x4000042ce0 -D -b binary -m i386:x86-64 /tmp/dump.bin
redis | ------
redis | 1:M 29 Sep 2021 10:44:10.972 # dump of function (hexdump of 331 bytes):
redis | 415741564989fe41554989d5415455534889f34883ec2848837f50ff740b48837f58000f84cf000000498b064889dfff104889c54d85ed740849c745000000000049837e50ff7520498b7618498b7e284885f60f84370100004839fe0f8656010000660f1f440000498d461048894424184989c1498d465048894424104d8b79104921ef4a8d04fd000000004889442408498b014e8b24f84d85e4752eeb7190498b06488b40184885c074154c890c24498b7e084889deffd04c8b0c2485c075134d8b6424104d85e47445498b14244839d375cc4d85ed74044d8965004531ed4883c4284c89e85b5d415c415d415e415fc3660f1f440000be01000000e8befaffffe922ffffff660f1f84000000000049837e50ff0f84d00000004983c1204c394c24100f8553ffffff4983ffff74ad498d6e30bf18000000e812c600004c8b6424084c0365004989c549
redis | Function at 0x40000428a0 is dictRehash
redis | Function at 0x400004f430 is zmalloc
redis |
redis | === REDIS BUG REPORT END. Make sure to include from START to END. ===
Comment From: yoav-steinberg
According to the report this happened in a docker container on an M1 Mac, but I see that the platform was x86_64:
redis | os:Linux 5.10.47-linuxkit x86_64
Unless I'm missing something this should either be an ARM build (in if it's been run on an M1, docker or no docker) or an x86_64 build, but not on M1.
Also if this was indeed run in a container, can you point to what container it was? This can help recreate.
Can you provide some minimal Redis command sequence to recreate this? Looks like you were working with Redis sets.
@oranagra I see we got a SIGSEGV and I noticed the invalid address isn't 8 byte aligned. Do ARM alignment issues cause SIGSEGV or some other signal?
Comment From: oranagra
I think it's using binary translation https://en.m.wikipedia.org/wiki/Rosetta_(software)
Which is supposed to work now after we resolved #8062 in Redis 6.0.10
Regarding alignment issues I think all modern ARM processor handle that properly, and the ones who didn't used to generate a SIGBUS.
Comment From: sundb
@yoav-steinberg I've reproduced the problem on m1's docker, using redis-benchmark -t zadd -l -r 10000 and it crashes, but I haven't found the crashing problem yet.
Also, I found out that m1 docker is emulated using qemu, and I'm not sure if the crash was caused by qemu.
@oranagra I ran the unstable code on top of https://hub.docker.com/_/redis and it still crashed.
Comment From: sundb
@deivua Can you share your docker start command or docker-compose.yml?
Is your platform parameter linux/x86_64, if so you can try removing it and test again.
Comment From: mojtabacazi
I'm using bitnami/redis:6.2-debian-10 without any modifications using qemu and it's crashing similarly to what op described here.
Comment From: sundb
@mojtabacazi Can you share your docker start command or docker_compose.yml?
Comment From: sundb
Related to #10341, so we can close it.
Comment From: oranagra
closing, see https://github.com/redis/redis/issues/10162#issuecomment-1072375709