Describe the bug It crashes out of nowhere. To reproduce No idea, it just does out of nowhere, i really don't know how it happens. Steps to reproduce the behavior and/or a minimal code sample.
Expected behavior
A description of what you expected to happen.
Additional information
Any additional information that is relevant to the problem.
I have the logs of the crash
------ FAST MEMORY TEST ------ 45622:C 20 Dec 2023 09:07:38.510 # Bio worker thread #0 terminated 77690:M 20 Dec 2023 09:07:38.603 # Background saving terminated by signal 11 77690:M 20 Dec 2023 09:07:44.023 * 1 changes in 3600 seconds. Saving... 77690:M 20 Dec 2023 09:07:44.043 * Background saving started by pid 45710
=== REDIS BUG REPORT START: Cut & paste starting from here === 45710:C 20 Dec 2023 09:07:44.471 # Redis 7.2.1 crashed by signal: 11, si_code: 128 45710:C 20 Dec 2023 09:07:44.471 # Accessing address: (nil) 45710:C 20 Dec 2023 09:07:44.471 # Crashed running the instruction at: 0x557b62873aaf
------ STACK TRACE ------ EIP: redis-rdb-bgsave 0.0.0.0:6379(dictNext+0x8f)[0x557b62873aaf]
Backtrace: /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7f3e654c8520] redis-rdb-bgsave 0.0.0.0:6379(dictNext+0x8f)[0x557b62873aaf] redis-rdb-bgsave 0.0.0.0:6379(rdbSaveObject+0x5b3)[0x557b628bf393] redis-rdb-bgsave 0.0.0.0:6379(rdbSaveKeyValuePair+0x8e)[0x557b628bfc9e] redis-rdb-bgsave 0.0.0.0:6379(rdbSaveDb+0x21a)[0x557b628c005a] redis-rdb-bgsave 0.0.0.0:6379(rdbSaveRio+0x1f4)[0x557b628c03c4] redis-rdb-bgsave 0.0.0.0:6379(+0xd54fd)[0x557b628c04fd] redis-rdb-bgsave 0.0.0.0:6379(rdbSave+0x9e)[0x557b628c0c1e] redis-rdb-bgsave 0.0.0.0:6379(rdbSaveBackground+0xf8)[0x557b628c0e98] redis-rdb-bgsave 0.0.0.0:6379(serverCron+0x952)[0x557b6287ae72] redis-rdb-bgsave 0.0.0.0:6379(aeProcessEvents+0x2a5)[0x557b628779c5] redis-rdb-bgsave 0.0.0.0:6379(aeMain+0x1d)[0x557b62877bbd] redis-rdb-bgsave 0.0.0.0:6379(main+0x39b)[0x557b6286debb] /lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7f3e654afd90] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7f3e654afe40] redis-rdb-bgsave 0.0.0.0:6379(_start+0x25)[0x557b6286e555]
------ REGISTERS ------ 45710:C 20 Dec 2023 09:07:44.472 # RAX:0008000000000000 RBX:00007f3e516557a0 RCX:0000000000000000 RDX:0000000000000000 RDI:00007f3e516557a0 RSI:00007f3e572c9c63 RBP:0000000000000000 RSP:00007ffd1b0f98b0 R8 :00007f3e572c9c87 R9 :00000000000000b0 R10:0000000000000034 R11:00007f3e64c01f40 R12:00007f3e566927a0 R13:000000000002d073 R14:0000557b62a697d8 R15:00007f3e572c9c63 RIP:0000557b62873aaf EFL:0000000000010293 CSGSFS:002b000000000033 45710:C 20 Dec 2023 09:07:44.472 # (00007ffd1b0f98bf) -> 0000000000000000 45710:C 20 Dec 2023 09:07:44.472 # (00007ffd1b0f98be) -> 0000000000000000 45710:C 20 Dec 2023 09:07:44.472 # (00007ffd1b0f98bd) -> 0000000000000000 45710:C 20 Dec 2023 09:07:44.472 # (00007ffd1b0f98bc) -> 0000000000000000 45710:C 20 Dec 2023 09:07:44.472 # (00007ffd1b0f98bb) -> 0000000000000000 45710:C 20 Dec 2023 09:07:44.472 # (00007ffd1b0f98ba) -> 0000000000000000 45710:C 20 Dec 2023 09:07:44.472 # (00007ffd1b0f98b9) -> 0000000000000000 45710:C 20 Dec 2023 09:07:44.472 # (00007ffd1b0f98b8) -> 0000000000000000 45710:C 20 Dec 2023 09:07:44.472 # (00007ffd1b0f98b7) -> 00000000007bf9db 45710:C 20 Dec 2023 09:07:44.472 # (00007ffd1b0f98b6) -> 0000000000000000 45710:C 20 Dec 2023 09:07:44.472 # (00007ffd1b0f98b5) -> 0000557b628bf393 45710:C 20 Dec 2023 09:07:44.472 # (00007ffd1b0f98b4) -> 00007f3e516557a0 45710:C 20 Dec 2023 09:07:44.472 # (00007ffd1b0f98b3) -> 0000557b62a697b0 45710:C 20 Dec 2023 09:07:44.472 # (00007ffd1b0f98b2) -> 00007ffd1b0f9d10 45710:C 20 Dec 2023 09:07:44.472 # (00007ffd1b0f98b1) -> 00000000007bfa00 45710:C 20 Dec 2023 09:07:44.472 # (00007ffd1b0f98b0) -> 00007f3e516557a0
------ INFO OUTPUT ------
Server
redis_version:7.2.1 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:95712a67f5005c28 redis_mode:standalone os:Linux 5.15.0-84-generic x86_64 arch_bits:64 monotonic_clock:POSIX clock_gettime multiplexing_api:epoll atomicvar_api:c11-builtin gcc_version:11.4.0 process_id:45710 process_supervised:systemd run_id:eed7cfd59e7c010b0ace94a744675c5c12bec044 tcp_port:6379 server_time_usec:1703063264023764 uptime_in_seconds:171793 uptime_in_days:1 hz:10 configured_hz:10 lru_clock:8564448 executable:/usr/bin/redis-server config_file:/etc/redis/redis.conf io_threads_active:0 listener0:name=tcp,bind=0.0.0.0,port=6379
Clients
connected_clients:343 cluster_connections:0 maxclients:65503 client_recent_max_input_buffer:20480 client_recent_max_output_buffer:0 blocked_clients:0 tracking_clients:0 clients_in_timeout_table:0 total_blocking_keys:0 total_blocking_keys_on_nokey:0
Memory
used_memory:281650984 used_memory_human:268.60M used_memory_rss:406597632 used_memory_rss_human:387.76M used_memory_peak:13953735392 used_memory_peak_human:13.00G used_memory_peak_perc:2.02% used_memory_overhead:5363536 used_memory_startup:4519664 used_memory_dataset:276287448 used_memory_dataset_perc:99.70% allocator_allocated:281930744 allocator_active:282726400 allocator_resident:407941120 total_system_memory:134950866944 total_system_memory_human:125.68G used_memory_lua:31744 used_memory_vm_eval:31744 used_memory_lua_human:31.00K used_memory_scripts_eval:0 number_of_cached_scripts:0 number_of_functions:0 number_of_libraries:0 used_memory_vm_functions:32768 used_memory_vm_total:64512 used_memory_vm_total_human:63.00K used_memory_functions:184 used_memory_scripts:184 used_memory_scripts_human:184B maxmemory:0 maxmemory_human:0B maxmemory_policy:noeviction allocator_frag_ratio:1.00 allocator_frag_bytes:795656 allocator_rss_ratio:1.44 allocator_rss_bytes:125214720 rss_overhead_ratio:1.00 rss_overhead_bytes:-1343488 mem_fragmentation_ratio:1.44 mem_fragmentation_bytes:124948616 mem_not_counted_for_evict:0 mem_replication_backlog:0 mem_total_replication_buffers:0 mem_clients_slaves:0 mem_clients_normal:841144 mem_cluster_links:0 mem_aof_buffer:0 mem_allocator:jemalloc-5.3.0 active_defrag_running:0 lazyfree_pending_objects:0 lazyfreed_objects:0
Persistence
loading:0 async_loading:0 current_cow_peak:0 current_cow_size:0 current_cow_size_age:0 current_fork_perc:0.00 current_save_keys_processed:0 current_save_keys_total:0 rdb_changes_since_last_save:12425 rdb_bgsave_in_progress:0 rdb_last_save_time:1703047069 rdb_last_bgsave_status:err rdb_last_bgsave_time_sec:0 rdb_current_bgsave_time_sec:-1 rdb_saves:5120 rdb_last_cow_size:8056832 rdb_last_load_keys_expired:0 rdb_last_load_keys_loaded:49 aof_enabled:0 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:-1 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_rewrites:0 aof_rewrites_consecutive_failures:0 aof_last_write_status:ok aof_last_cow_size:0 module_fork_in_progress:0 module_fork_last_cow_size:0
Stats
total_connections_received:16765074 total_commands_processed:517731645 instantaneous_ops_per_sec:1919 total_net_input_bytes:23914278262 total_net_output_bytes:114731546598 total_net_repl_input_bytes:0 total_net_repl_output_bytes:0 instantaneous_input_kbps:92.81 instantaneous_output_kbps:327.62 instantaneous_input_repl_kbps:0.00 instantaneous_output_repl_kbps:0.00 rejected_connections:0 sync_full:0 sync_partial_ok:0 sync_partial_err:0 expired_keys:0 expired_stale_perc:0.00 expired_time_cap_reached_count:0 expire_cycle_cpu_milliseconds:1258 evicted_keys:0 evicted_clients:0 total_eviction_exceeded_time:0 current_eviction_exceeded_time:0 keyspace_hits:95596155 keyspace_misses:57570659 pubsub_channels:117 pubsub_patterns:0 pubsubshard_channels:0 latest_fork_usec:19803 total_forks:5119 migrate_cached_sockets:0 slave_expires_tracked_keys:0 active_defrag_hits:0 active_defrag_misses:0 active_defrag_key_hits:0 active_defrag_key_misses:0 total_active_defrag_time:0 current_active_defrag_time:0 tracking_total_keys:0 tracking_total_items:0 tracking_total_prefixes:0 unexpected_error_replies:0 total_error_replies:16444642 dump_payload_sanitizations:0 total_reads_processed:432160086 total_writes_processed:1164103301 io_threaded_reads_processed:0 io_threaded_writes_processed:0 reply_buffer_shrinks:670415 reply_buffer_expands:344378 eventloop_cycles:435162358 eventloop_duration_sum:8995321934 eventloop_duration_cmd_sum:734420265 instantaneous_eventloop_cycles_per_sec:3699 instantaneous_eventloop_duration_usec:12 acl_access_denied_auth:0 acl_access_denied_cmd:0 acl_access_denied_key:0 acl_access_denied_channel:0
Replication
role:master connected_slaves:0 master_failover_state:no-failover master_replid:8a8a9e95e53156f4d4921e12b0a69a7cf6c150e9 master_replid2:0000000000000000000000000000000000000000 master_repl_offset:0 second_repl_offset:-1 repl_backlog_active:0 repl_backlog_size:1048576 repl_backlog_first_byte_offset:0 repl_backlog_histlen:0
CPU
used_cpu_sys:0.044057 used_cpu_user:0.384497 used_cpu_sys_children:0.000000 used_cpu_user_children:0.000000 used_cpu_sys_main_thread:0.044057 used_cpu_user_main_thread:0.384497
Modules
Commandstats
cmdstat_zscore:calls=55839492,usec=12911192,usec_per_call=0.23,rejected_calls=0,failed_calls=0 cmdstat_keys:calls=1,usec=42,usec_per_call=42.00,rejected_calls=1,failed_calls=0 cmdstat_zrevrange:calls=24749,usec=330292,usec_per_call=13.35,rejected_calls=0,failed_calls=0 cmdstat_zrevrank:calls=86317,usec=239768,usec_per_call=2.78,rejected_calls=0,failed_calls=0 cmdstat_bgsave:calls=3,usec=59766,usec_per_call=19922.00,rejected_calls=1,failed_calls=0 cmdstat_publish:calls=39043759,usec=252584750,usec_per_call=6.47,rejected_calls=0,failed_calls=0 cmdstat_hdel:calls=627,usec=4047,usec_per_call=6.45,rejected_calls=0,failed_calls=0 cmdstat_quit:calls=16764165,usec=3421518,usec_per_call=0.20,rejected_calls=0,failed_calls=0 cmdstat_hget:calls=97213214,usec=57990646,usec_per_call=0.60,rejected_calls=0,failed_calls=0 cmdstat_hset:calls=1875269,usec=7223666,usec_per_call=3.85,rejected_calls=0,failed_calls=0 cmdstat_zadd:calls=14571205,usec=51016666,usec_per_call=3.50,rejected_calls=0,failed_calls=0 cmdstat_command:calls=0,usec=0,usec_per_call=0.00,rejected_calls=3,failed_calls=0 cmdstat_command|docs:calls=0,usec=0,usec_per_call=0.00,rejected_calls=3,failed_calls=0 cmdstat_zincrby:calls=36155570,usec=260582242,usec_per_call=7.21,rejected_calls=0,failed_calls=0 cmdstat_client|setname:calls=466283,usec=414531,usec_per_call=0.89,rejected_calls=0,failed_calls=0 cmdstat_lastsave:calls=16,usec=13,usec_per_call=0.81,rejected_calls=0,failed_calls=0 cmdstat_zrange:calls=17,usec=315,usec_per_call=18.53,rejected_calls=0,failed_calls=0 cmdstat_zcount:calls=3025,usec=24591,usec_per_call=8.13,rejected_calls=0,failed_calls=0 cmdstat_info:calls=1,usec=99,usec_per_call=99.00,rejected_calls=4,failed_calls=0 cmdstat_zrem:calls=91987,usec=246057,usec_per_call=2.67,rejected_calls=0,failed_calls=0 cmdstat_auth:calls=16765071,usec=25101634,usec_per_call=1.50,rejected_calls=0,failed_calls=0 cmdstat_subscribe:calls=671,usec=2691,usec_per_call=4.01,rejected_calls=0,failed_calls=0 cmdstat_ping:calls=238830203,usec=62265739,usec_per_call=0.26,rejected_calls=16444629,failed_calls=0
Errorstats
errorstat_ERR:count=2 errorstat_LOADING:count=621 errorstat_MISCONF:count=16444008 errorstat_NOAUTH:count=11
Latencystats
latency_percentiles_usec_zscore:p50=0.001,p99=1.003,p99.9=4.015 latency_percentiles_usec_keys:p50=42.239,p99=42.239,p99.9=42.239 latency_percentiles_usec_zrevrange:p50=13.055,p99=34.047,p99.9=63.231 latency_percentiles_usec_zrevrank:p50=3.007,p99=10.047,p99.9=23.039 latency_percentiles_usec_bgsave:p50=19529.727,p99=20971.519,p99.9=20971.519 latency_percentiles_usec_publish:p50=4.015,p99=63.231,p99.9=93.183 latency_percentiles_usec_hdel:p50=6.015,p99=16.063,p99.9=22.015 latency_percentiles_usec_quit:p50=0.001,p99=1.003,p99.9=3.007 latency_percentiles_usec_hget:p50=0.001,p99=3.007,p99.9=11.007 latency_percentiles_usec_hset:p50=3.007,p99=14.015,p99.9=29.055 latency_percentiles_usec_zadd:p50=2.007,p99=19.071,p99.9=35.071 latency_percentiles_usec_zincrby:p50=6.015,p99=22.015,p99.9=40.191 latency_percentiles_usec_client|setname:p50=1.003,p99=2.007,p99.9=12.031 latency_percentiles_usec_lastsave:p50=1.003,p99=2.007,p99.9=2.007 latency_percentiles_usec_zrange:p50=16.063,p99=46.079,p99.9=46.079 latency_percentiles_usec_zcount:p50=8.031,p99=21.119,p99.9=36.095 latency_percentiles_usec_info:p50=99.327,p99=99.327,p99.9=99.327 latency_percentiles_usec_zrem:p50=2.007,p99=8.031,p99.9=22.015 latency_percentiles_usec_auth:p50=1.003,p99=4.015,p99.9=16.063 latency_percentiles_usec_subscribe:p50=3.007,p99=17.023,p99.9=42.239 latency_percentiles_usec_ping:p50=0.001,p99=1.003,p99.9=5.023
Cluster
cluster_enabled:0
Keyspace
db0:keys=50,expires=0,avg_ttl=0
Comment From: hpatro
@PedroMPagani Would it be possible to share the dataset or if it was generated via benchmark tool? From the latency stats, it looks like it has bunch of sorted sets/hash.
How frequently does it happen? The crash happens during bgsave (snapshot). Not sure, if it happens on each bgsave operation.
Comment From: PedroMPagani
Hey hpatro, it does not happen on every save, my logs have 100s of saves before, my config for save is the default every minute.
I can share the dataset there's no sensible data on it(there is a small one but its ipv4s of our machines - so if possiblke i'd like to share this privately), yes it's a lot of sorted sets and hashsets, thisi s a minecraft network and its all leaderboards and player skins texture. where should I upload the file, here?
@hpatro
Comment From: oranagra
@PedroMPagani is there anything in dmesg when it happens? (asking because of si_code: 128)
how was redis built? anything that could be unusual?
somewhat similar to https://github.com/redis/redis/issues/9551
Comment From: oranagra
did you have the DUMPING CODE AROUND EIP section in the crash report? can you attach it?
Comment From: PedroMPagani
I don't see anything on dmesg on the redis port, is there a way to make it fast reading? what exactly would I be looking for there
Comment From: oranagra
i recently saw something like this:
[901206.570455] traps: redis-server[1203536] general protection fault
it was due to misaligned memory and MMX instructions.
Comment From: MeirShpilraien
@oranagra the dmesg will only show this error if you run with --crash-log-enabled no option.
Comment From: oranagra
ohh, right.