Crash report
Paste the complete crash log between the quotes below. Please include a few lines from the log preceding the crash report to provide some context.
49510:M 21 Oct 2022 14:07:45.466 # === ASSERTION FAILED OBJECT CONTEXT ===
49510:M 21 Oct 2022 14:07:45.467 # Object type: 3
49510:M 21 Oct 2022 14:07:45.467 # Object encoding: 5
49510:M 21 Oct 2022 14:07:45.467 # Object refcount: 1
49510:M 21 Oct 2022 14:07:45.467 # === ASSERTION FAILED ===
49510:M 21 Oct 2022 14:07:45.468 # ==> t_zset.c:1195 'eptr != NULL' is not true
------ STACK TRACE ------
Backtrace:
0 redis-server 0x0000000104250d9c zsetConvert.cold.2 + 40
1 redis-server 0x00000001041ad0e8 zsetConvert + 768
2 redis-server 0x00000001041ad3d8 zsetAdd + 488
3 redis-server 0x00000001041b381c zrangeResultEmitCBufferForStore + 76
4 redis-server 0x00000001041b0d44 genericZrangebyscoreCommand + 552
5 redis-server 0x00000001041b09c0 zrangeGenericCommand + 1008
6 redis-server 0x00000001041b05c4 zrangestoreCommand + 100
7 redis-server 0x0000000104170d48 call + 260
8 redis-server 0x0000000104171ce8 processCommand + 2632
9 redis-server 0x0000000104186410 processInputBuffer + 308
10 redis-server 0x000000010421c818 connSocketEventHandler + 288
11 redis-server 0x0000000104168390 aeProcessEvents + 792
12 redis-server 0x00000001041685fc aeMain + 32
13 redis-server 0x0000000104175530 main + 1432
14 dyld 0x000000010466d08c start + 520
------ INFO OUTPUT ------
# Server
redis_version:6.2.6
redis_git_sha1:190b28cb
redis_git_dirty:1
redis_build_id:b262be297d7fe9e5
redis_mode:standalone
os:Darwin 21.6.0 arm64
arch_bits:64
multiplexing_api:kqueue
atomicvar_api:c11-builtin
gcc_version:4.2.1
process_id:49510
process_supervised:no
run_id:044dd95453bbe4169b9a960e7267fadd2a82139a
tcp_port:6379
server_time_usec:1666354065466490
uptime_in_seconds:93
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:5409681
executable:/Users/mpaluch/git/data/spring-data-redis/work/redis/bin/redis-server
config_file:/Users/mpaluch/git/data/spring-data-redis/work/redis-6379.conf
io_threads_active:0
# Clients
connected_clients:9
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:48
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
# Memory
used_memory:2347584
used_memory_human:2.24M
used_memory_rss:4358144
used_memory_rss_human:4.16M
used_memory_peak:2607552
used_memory_peak_human:2.49M
used_memory_peak_perc:90.03%
used_memory_overhead:2269096
used_memory_startup:1028560
used_memory_dataset:78488
used_memory_dataset_perc:5.95%
allocator_allocated:2277872
allocator_active:4320256
allocator_resident:4320256
total_system_memory:34359738368
total_system_memory_human:32.00G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.90
allocator_frag_bytes:2042384
allocator_rss_ratio:1.00
allocator_rss_bytes:0
rss_overhead_ratio:1.01
rss_overhead_bytes:37888
mem_fragmentation_ratio:1.91
mem_fragmentation_bytes:2080272
mem_not_counted_for_evict:0
mem_replication_backlog:1048576
mem_clients_slaves:34912
mem_clients_normal:156976
mem_aof_buffer:0
mem_allocator:libc
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
# Persistence
loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:4
rdb_bgsave_in_progress:0
rdb_last_save_time:1666353972
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
# Stats
total_connections_received:11
total_commands_processed:641
instantaneous_ops_per_sec:6
total_net_input_bytes:33326
total_net_output_bytes:226138
instantaneous_input_kbps:0.36
instantaneous_output_kbps:1.24
rejected_connections:0
sync_full:1
sync_partial_ok:1
sync_partial_err:1
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:4
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:1
pubsub_patterns:0
latest_fork_usec:239
total_forks:1
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:628
total_writes_processed:1140
io_threaded_reads_processed:0
io_threaded_writes_processed:0
# Replication
role:master
connected_slaves:2
slave0:ip=127.0.0.1,port=6380,state=online,offset=20651,lag=1
slave1:ip=127.0.0.1,port=6381,state=online,offset=20651,lag=1
master_failover_state:no-failover
master_replid:800c2c9bc1dcfadb32136aa5e30fad707db8f940
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:20651
second_repl_offset:-1
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:20651
# CPU
used_cpu_sys:0.211528
used_cpu_user:0.175698
used_cpu_sys_children:0.000478
used_cpu_user_children:0.000079
# Modules
# Commandstats
cmdstat_subscribe:calls=3,usec=5,usec_per_call=1.67,rejected_calls=0,failed_calls=0
cmdstat_client:calls=6,usec=3,usec_per_call=0.50,rejected_calls=0,failed_calls=0
cmdstat_replconf:calls=186,usec=244,usec_per_call=1.31,rejected_calls=0,failed_calls=0
cmdstat_psync:calls=2,usec=839,usec_per_call=419.50,rejected_calls=0,failed_calls=0
cmdstat_ping:calls=272,usec=320,usec_per_call=1.18,rejected_calls=0,failed_calls=0
cmdstat_flushall:calls=1,usec=4,usec_per_call=4.00,rejected_calls=0,failed_calls=0
cmdstat_publish:calls=135,usec=1449,usec_per_call=10.73,rejected_calls=0,failed_calls=0
cmdstat_zadd:calls=3,usec=40,usec_per_call=13.33,rejected_calls=0,failed_calls=0
cmdstat_info:calls=30,usec=4380,usec_per_call=146.00,rejected_calls=0,failed_calls=0
cmdstat_hello:calls=3,usec=19,usec_per_call=6.33,rejected_calls=0,failed_calls=0
# Errorstats
# Cluster
cluster_enabled:0
# Keyspace
db0:keys=1,expires=0,avg_ttl=0
------ CLIENT LIST OUTPUT ------
id=8 addr=127.0.0.1:57199 laddr=127.0.0.1:6379 fd=15 name=sentinel-ffe3bce9-pubsub age=93 idle=1 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=17440 events=r cmd=subscribe user=default redir=-1
id=3 addr=127.0.0.1:57194 laddr=127.0.0.1:6379 fd=8 name= age=93 idle=1 flags=S db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=17456 events=r cmd=replconf user=default redir=-1
id=9 addr=127.0.0.1:57200 laddr=127.0.0.1:6379 fd=9 name=sentinel-464fde9b-cmd age=93 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=17440 events=r cmd=ping user=default redir=-1
id=10 addr=127.0.0.1:57201 laddr=127.0.0.1:6379 fd=10 name=sentinel-464fde9b-pubsub age=93 idle=1 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=17440 events=r cmd=subscribe user=default redir=-1
id=4 addr=127.0.0.1:57195 laddr=127.0.0.1:6379 fd=11 name= age=93 idle=1 flags=S db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=17456 events=r cmd=replconf user=default redir=-1
id=13 addr=127.0.0.1:57254 laddr=127.0.0.1:6379 fd=18 name= age=84 idle=84 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=17440 events=r cmd=hello user=default redir=-1
id=11 addr=127.0.0.1:57252 laddr=127.0.0.1:6379 fd=16 name= age=84 idle=84 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=17440 events=r cmd=flushall user=default redir=-1
id=5 addr=127.0.0.1:57196 laddr=127.0.0.1:6379 fd=12 name=sentinel-1754c9fc-cmd age=93 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=17440 events=r cmd=ping user=default redir=-1
id=6 addr=127.0.0.1:57197 laddr=127.0.0.1:6379 fd=13 name=sentinel-1754c9fc-pubsub age=93 idle=1 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=17440 events=r cmd=subscribe user=default redir=-1
id=7 addr=127.0.0.1:57198 laddr=127.0.0.1:6379 fd=14 name=sentinel-ffe3bce9-cmd age=93 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=0 omem=0 tot-mem=17440 events=r cmd=ping user=default redir=-1
id=12 addr=127.0.0.1:57253 laddr=127.0.0.1:6379 fd=17 name= age=84 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=873 qbuf-free=64657 argv-mem=828 obl=0 oll=0 omem=0 tot-mem=83820 events=r cmd=zrangestore user=default redir=-1
------ CURRENT CLIENT INFO ------
id=12 addr=127.0.0.1:57253 laddr=127.0.0.1:6379 fd=17 name= age=84 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=873 qbuf-free=64657 argv-mem=828 obl=0 oll=0 omem=0 tot-mem=83820 events=r cmd=zrangestore user=default redir=-1
argv[0]: 'ZRANGESTORE'
argv[1]: '��'
argv[2]: '��'
argv[3]: '1.0'
argv[4]: '2.0'
argv[5]: 'BYSCORE'
------ MODULES INFO OUTPUT ------
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Please report the crash by opening an issue on github:
http://github.com/redis/redis/issues
Suspect RAM error? Use redis-server --test-memory to verify it.
Additional information
- OS distribution and version: OSX 12, Mac M1
- Steps to reproduce (if any)
Invoke
ZRANGESTOREwith binary keys
Comment From: sundb
It looks like it may not relate to the binary key, the crash is due to converting an empty ziplist to a skiplist. Do you have any reproduction steps?
Comment From: sundb
@mp911de Is your zset-max-ziplist-entries config 0?
Comment From: oranagra
seems like the crash that was reported and fixed in #10767.
@mp911de let us know if we're wrong and zset-max-ziplist-entries isn't 0 (in which case we need a way to reproduce it)
Comment From: mp911de
zset-max-ziplist-entries is 128
127.0.0.1:6379> config get zset-max-ziplist-entries
1) "zset-max-ziplist-entries"
2) "128"
The commands (via redis-cli) to reproduce the issue (tested against 6.2.7/e6f67092) are:
127.0.0.1:6379> "ZADD" "\xac\xed\x00\x05t\x00Lorg.springframework.data.redis.support.collections.RedisZSetIntegrationTests" "1.0" "\xac\xed\x00\x05sr\x00%org.springframework.data.redis.Person\x01I\x199\xf2\xd4>m\x02\x00\x04L\x00\aaddresst\x00(Lorg/springframework/data/redis/Address;L\x00\x03aget\x00\x13Ljava/lang/Integer;L\x00\tfirstNamet\x00\x12Ljava/lang/String;L\x00\blastNameq\x00~\x00\x03xpsr\x00&org.springframework.data.redis.AddressDU\xb9?\x9d\x9b\x85{\x02\x00\x02L\x00\x06numberq\x00~\x00\x02L\x00\x06streetq\x00~\x00\x03xpsr\x00\x11java.lang.Integer\x12\xe2\xa0\xa4\xf7\x81\x878\x02\x00\x01I\x00\x05valuexr\x00\x10java.lang.Number\x86\xac\x95\x1d\x0b\x94\xe0\x8b\x02\x00\x00xp\x00\x00\x00\x01t\x00$77ea4e70-f8bd-4341-862d-f5ca89a37533q\x00~\x00\tq\x00~\x00\nq\x00~\x00\n"
(integer) 1
127.0.0.1:6379> "ZADD" "\xac\xed\x00\x05t\x00Lorg.springframework.data.redis.support.collections.RedisZSetIntegrationTests" "2.0" "\xac\xed\x00\x05sr\x00%org.springframework.data.redis.Person\x01I\x199\xf2\xd4>m\x02\x00\x04L\x00\aaddresst\x00(Lorg/springframework/data/redis/Address;L\x00\x03aget\x00\x13Ljava/lang/Integer;L\x00\tfirstNamet\x00\x12Ljava/lang/String;L\x00\blastNameq\x00~\x00\x03xpsr\x00&org.springframework.data.redis.AddressDU\xb9?\x9d\x9b\x85{\x02\x00\x02L\x00\x06numberq\x00~\x00\x02L\x00\x06streetq\x00~\x00\x03xpsr\x00\x11java.lang.Integer\x12\xe2\xa0\xa4\xf7\x81\x878\x02\x00\x01I\x00\x05valuexr\x00\x10java.lang.Number\x86\xac\x95\x1d\x0b\x94\xe0\x8b\x02\x00\x00xp\x00\x00\x00\x02t\x00$f5d65152-6db6-4648-bde8-3dbf2d0b169bq\x00~\x00\tq\x00~\x00\nq\x00~\x00\n"
(integer) 1
127.0.0.1:6379> "ZADD" "\xac\xed\x00\x05t\x00Lorg.springframework.data.redis.support.collections.RedisZSetIntegrationTests" "3.0" "\xac\xed\x00\x05sr\x00%org.springframework.data.redis.Person\x01I\x199\xf2\xd4>m\x02\x00\x04L\x00\aaddresst\x00(Lorg/springframework/data/redis/Address;L\x00\x03aget\x00\x13Ljava/lang/Integer;L\x00\tfirstNamet\x00\x12Ljava/lang/String;L\x00\blastNameq\x00~\x00\x03xpsr\x00&org.springframework.data.redis.AddressDU\xb9?\x9d\x9b\x85{\x02\x00\x02L\x00\x06numberq\x00~\x00\x02L\x00\x06streetq\x00~\x00\x03xpsr\x00\x11java.lang.Integer\x12\xe2\xa0\xa4\xf7\x81\x878\x02\x00\x01I\x00\x05valuexr\x00\x10java.lang.Number\x86\xac\x95\x1d\x0b\x94\xe0\x8b\x02\x00\x00xp\x00\x00\x00\x03t\x00$1eb24dc5-c911-422a-8128-5b3baee0c5d9q\x00~\x00\tq\x00~\x00\nq\x00~\x00\n"
(integer) 1
127.0.0.1:6379> ZRANGESTORE dest "\xac\xed\x00\x05t\x00Lorg.springframework.data.redis.support.collections.RedisZSetIntegrationTests" 2 3 BYSCORE
Error: Server closed the connection
Comment From: mp911de
I can also confirm that Redis 7.0.5 (1571907e/0) doesn't reproduce the issue.
Comment From: sundb
Sure, this crash was fixed in #10767.
Your zset-max-ziplist-entries should also be the default value of 128, ZRANGESTORE command triggers the conversion when inserting 402 bytes of value into dest(empty listpack).
Comment From: oranagra
so the crash is (also) when the first key being inserted by ZRANGESTORE tnto the zset is over the limit of zset-max-ziplist-value.
i'll add that detail to the top comment of the PR that fixed it (will be backported to the next 6.2 release), but i guess we can close this one.