Crash report
Paste the complete crash log between the quotes below. Please include a few lines from the log preceding the crash report to provide some context.
=== REDIS BUG REPORT START: Cut & paste starting from here ===
51320:M 05 Dec 2022 08:10:35.109 # ------------------------------------------------
51320:M 05 Dec 2022 08:10:35.109 # !!! Software Failure. Press left mouse button to continue
51320:M 05 Dec 2022 08:10:35.109 # Guru Meditation: Unknown set encoding #t_zset.c:2279
------ STACK TRACE ------
Backtrace:
/home/hbina/git/redis/src/redis-server 127.0.0.1:6379(+0xee79f)[0x55555564279f]
/home/hbina/git/redis/src/redis-server 127.0.0.1:6379(zunionInterDiffGenericCommand+0xbda)[0x55555564347a]
/home/hbina/git/redis/src/redis-server 127.0.0.1:6379(call+0xe2)[0x5555555dcae2]
/home/hbina/git/redis/src/redis-server 127.0.0.1:6379(processCommand+0xb10)[0x5555555de020]
/home/hbina/git/redis/src/redis-server 127.0.0.1:6379(processInputBuffer+0x107)[0x5555555febd7]
/home/hbina/git/redis/src/redis-server 127.0.0.1:6379(readQueryFromClient+0x348)[0x5555555ff138]
/home/hbina/git/redis/src/redis-server 127.0.0.1:6379(+0x1ade9c)[0x555555701e9c]
/home/hbina/git/redis/src/redis-server 127.0.0.1:6379(aeMain+0xf1)[0x5555555d2ab1]
/home/hbina/git/redis/src/redis-server 127.0.0.1:6379(main+0x3d4)[0x5555555c8254]
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7ffff7c29d90]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7ffff7c29e40]
/home/hbina/git/redis/src/redis-server 127.0.0.1:6379(_start+0x25)[0x5555555c89b5]
------ INFO OUTPUT ------
# Server
redis_version:255.255.255
redis_git_sha1:61c85a2b
redis_git_dirty:0
redis_build_id:ce6e622df897e510
redis_mode:standalone
os:Linux 6.0.9-060009-generic x86_64
arch_bits:64
monotonic_clock:POSIX clock_gettime
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:11.3.0
process_id:51320
process_supervised:no
run_id:bfb2eb674e0616154f7e79c17e39fbfc7c5b340f
tcp_port:6379
server_time_usec:1670199035109087
uptime_in_seconds:117
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:9254651
executable:/home/hbina/git/redis/src/redis-server
config_file:/home/hbina/git/redis/redis.conf
io_threads_active:0
listener0:name=tcp,bind=127.0.0.1,bind=-::1,port=6379
# Clients
connected_clients:1
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:20480
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
# Memory
used_memory:990144
used_memory_human:966.94K
used_memory_rss:7516160
used_memory_rss_human:7.17M
used_memory_peak:1112776
used_memory_peak_human:1.06M
used_memory_peak_perc:88.98%
used_memory_overhead:867368
used_memory_startup:865272
used_memory_dataset:122776
used_memory_dataset_perc:98.32%
allocator_allocated:1526896
allocator_active:1904640
allocator_resident:6438912
total_system_memory:33047990272
total_system_memory_human:30.78G
used_memory_lua:31744
used_memory_vm_eval:31744
used_memory_lua_human:31.00K
used_memory_scripts_eval:0
number_of_cached_scripts:0
number_of_functions:0
number_of_libraries:0
used_memory_vm_functions:32768
used_memory_vm_total:64512
used_memory_vm_total_human:63.00K
used_memory_functions:184
used_memory_scripts:184
used_memory_scripts_human:184B
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.25
allocator_frag_bytes:377744
allocator_rss_ratio:3.38
allocator_rss_bytes:4534272
rss_overhead_ratio:1.17
rss_overhead_bytes:1077248
mem_fragmentation_ratio:7.61
mem_fragmentation_bytes:6528912
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_total_replication_buffers:0
mem_clients_slaves:0
mem_clients_normal:1800
mem_cluster_links:0
mem_aof_buffer:0
mem_allocator:jemalloc-5.2.1
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
# Persistence
loading:0
async_loading:0
current_cow_peak:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:7
rdb_bgsave_in_progress:0
rdb_last_save_time:1670199029
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_saves:0
rdb_last_cow_size:0
rdb_last_load_keys_expired:0
rdb_last_load_keys_loaded:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_rewrites:0
aof_rewrites_consecutive_failures:0
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
# Stats
total_connections_received:1
total_commands_processed:4
instantaneous_ops_per_sec:0
total_net_input_bytes:212
total_net_output_bytes:200352
total_net_repl_input_bytes:0
total_net_repl_output_bytes:0
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
instantaneous_input_repl_kbps:0.00
instantaneous_output_repl_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:3
evicted_keys:0
evicted_clients:0
total_eviction_exceeded_time:0
current_eviction_exceeded_time:0
keyspace_hits:2
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
pubsubshard_channels:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
total_active_defrag_time:0
current_active_defrag_time:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:5
total_writes_processed:6
io_threaded_reads_processed:0
io_threaded_writes_processed:0
reply_buffer_shrinks:1
reply_buffer_expands:0
acl_access_denied_auth:0
acl_access_denied_cmd:0
acl_access_denied_key:0
acl_access_denied_channel:0
# Replication
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:acfdf4ca028008dfe4493fd8694c70987c2b735d
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:0.132816
used_cpu_user:0.070558
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:0.132267
used_cpu_user_main_thread:0.070267
# Modules
# Commandstats
cmdstat_sadd:calls=1,usec=15,usec_per_call=15.00,rejected_calls=0,failed_calls=0
cmdstat_command|docs:calls=1,usec=1588,usec_per_call=1588.00,rejected_calls=0,failed_calls=0
cmdstat_zadd:calls=1,usec=41,usec_per_call=41.00,rejected_calls=0,failed_calls=0
cmdstat_flushall:calls=1,usec=19022,usec_per_call=19022.00,rejected_calls=0,failed_calls=0
# Errorstats
# Latencystats
latency_percentiles_usec_sadd:p50=15.039,p99=15.039,p99.9=15.039
latency_percentiles_usec_command|docs:p50=1589.247,p99=1589.247,p99.9=1589.247
latency_percentiles_usec_zadd:p50=41.215,p99=41.215,p99.9=41.215
latency_percentiles_usec_flushall:p50=19136.511,p99=19136.511,p99.9=19136.511
# Cluster
cluster_enabled:0
# Keyspace
db0:keys=2,expires=0,avg_ttl=0
------ CLIENT LIST OUTPUT ------
id=3 addr=127.0.0.1:56792 laddr=127.0.0.1:6379 fd=8 name= age=7 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=41 qbuf-free=20433 argv-mem=13 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=22317 events=r cmd=zdiff user=default redir=-1 resp=2
------ CURRENT CLIENT INFO ------
id=3 addr=127.0.0.1:56792 laddr=127.0.0.1:6379 fd=8 name= age=7 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=41 qbuf-free=20433 argv-mem=13 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=22317 events=r cmd=zdiff user=default redir=-1 resp=2
argv[0]: '"zdiff"'
argv[1]: '"2"'
argv[2]: '"key"'
argv[3]: '"key2"'
------ MODULES INFO OUTPUT ------
------ CONFIG DEBUG OUTPUT ------
client-query-buffer-limit 1gb
lazyfree-lazy-eviction no
io-threads-do-reads no
replica-read-only yes
lazyfree-lazy-server-del no
activedefrag no
list-compress-depth 0
sanitize-dump-payload no
repl-diskless-sync yes
lazyfree-lazy-expire no
proto-max-bulk-len 512mb
lazyfree-lazy-user-del no
slave-read-only yes
lazyfree-lazy-user-flush no
repl-diskless-load disabled
io-threads 1
------ FAST MEMORY TEST ------
51320:M 05 Dec 2022 08:10:35.116 # Bio thread for job type #0 terminated
51320:M 05 Dec 2022 08:10:35.117 # Bio thread for job type #1 terminated
51320:M 05 Dec 2022 08:10:35.117 # Bio thread for job type #2 terminated
*** Preparing to test memory region 5555558c2000 (2441216 bytes)
*** Preparing to test memory region 7ffff4b7c000 (2621440 bytes)
*** Preparing to test memory region 7ffff4dfd000 (8388608 bytes)
*** Preparing to test memory region 7ffff55fe000 (8388608 bytes)
*** Preparing to test memory region 7ffff5dff000 (8388608 bytes)
*** Preparing to test memory region 7ffff6600000 (8388608 bytes)
*** Preparing to test memory region 7ffff7400000 (8388608 bytes)
*** Preparing to test memory region 7ffff7e1b000 (53248 bytes)
*** Preparing to test memory region 7ffff7eb9000 (16384 bytes)
*** Preparing to test memory region 7ffff7fbb000 (8192 bytes)
.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Please report the crash by opening an issue on github:
http://github.com/redis/redis/issues
If a Redis module was involved, please open in the module's repo instead.
Suspect RAM error? Use redis-server --test-memory to verify it.
Some other issues could be detected by redis-server --check-system
Additional information
- Screenfetch
hbina@akarin ~> screenfetch -n
hbina@akarin
OS: Ubuntu 22.04 jammy
Kernel: x86_64 Linux 6.0.9-060009-generic
Uptime: 1h 24m
Packages: 2535
Shell: fish 3.3.1
Resolution: 1920x1080
DE: GNOME 41.7
WM: Mutter
WM Theme: Adwaita
GTK Theme: Yaru-red-dark [GTK2/3]
Icon Theme: Yaru-red
Font: Ubuntu 11
Disk: 290G / 436G (71%)
CPU: AMD Ryzen 7 4800U with Radeon Graphics @ 16x 1.8GHz
GPU: AMD/ATI
RAM: 6676MiB / 31517MiB
```
## Reproduction Steps (works every time!)
1. Build from source (61c85a2b2081dffaf45eb8c6b7754b8d9a80c60d)
2. Run the following commands:
```shell
hbina@akarin ~/g/redis (unstable)> redis-cli -p 6379
127.0.0.1:6379> zadd key 2 a 7 b 9 d 12 c
(integer) 4
127.0.0.1:6379> sadd key2 a b c
(integer) 3
127.0.0.1:6379> zdiff 2 key key2
Error: Server closed the connection
(231.17s)
not connected>
- Boom
I tried to look into the source code and the only thing I can worked out is that key2 is being stored as a OBJ_ENCODING_LISTPACK but the diff algorithm doesn't consider this possibility and just panics. Adding the following simple change
else if (op->encoding == OBJ_ENCODING_LISTPACK) {
zuiSdsFromValue(val);
if (zzlFind(op->subject->ptr,val->ele,score) != NULL) {
/* Score is already set by zzlFind. */
return 1;
} else {
return 0;
}
}
does not fix the issue becase zzlFind will also crash. I tried to debug zzFind but debugger doesn't really work (lots of things get optimized out) and its honestly such a mess of code it'll take me a very long time to decipher it. I'd be very interested to know how you fix it!
Comment From: enjoy-binbin
thanks for the report. and i must say, zdiff acts on a set object, which is a bit strange but ok
taking a look, few ways i can think: - the easiest one is that, we can change the SET from listpack encoding to a hashtable encoding - the other one is support it in zuiFind, this is the right one we need to fix
Comment From: hbina
thanks for the report. and i must say, zdiff acts on a set object, which is a bit strange but ok
Doesn't seem all that strange to me. a SET is just a ZSET with all of its member having a score of 1.
Comment From: enjoy-binbin
yep, i know, it's just that I don't use it very often, and I'm guessing that's why our tests don't cover it all I have found a fix, trying to test it, btw, you should not use zzlFind, you need to use lpfind instread
Comment From: enjoy-binbin
@hbina Hi, i made a quick PR, you can see my fix in #11581