Crash report
Paste the complete crash log between the quotes below. Please include a few lines from the log preceding the crash report to provide some context.
redis_1 | === REDIS BUG REPORT START: Cut & paste starting from here ===
redis_1 | 55:C 26 Sep 2021 00:40:33.286 # Redis 6.2.5 crashed by signal: 11, si_code: 128
redis_1 | 55:C 26 Sep 2021 00:40:33.286 # Accessing address: (nil)
redis_1 | 55:C 26 Sep 2021 00:40:33.286 # Crashed running the instruction at: 0x55e3fac2d804
redis_1 |
redis_1 | ------ STACK TRACE ------
redis_1 | EIP:
redis_1 | redis-rdb-bgsave *:6379(rdbSaveObjectType+0x4)[0x55e3fac2d804]
redis_1 |
redis_1 | Backtrace:
redis_1 | /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730)[0x7fba174ef730]
redis_1 | redis-rdb-bgsave *:6379(rdbSaveObjectType+0x4)[0x55e3fac2d804]
redis_1 | redis-rdb-bgsave *:6379(rdbSaveKeyValuePair+0x47)[0x55e3fac2e5c7]
redis_1 | redis-rdb-bgsave *:6379(rdbSaveRio+0x229)[0x55e3fac2ecb9]
redis_1 | redis-rdb-bgsave *:6379(rdbSave+0x94)[0x55e3fac31364]
redis_1 | redis-rdb-bgsave *:6379(rdbSaveBackground+0xd3)[0x55e3fac31673]
redis_1 | redis-rdb-bgsave *:6379(serverCron+0x595)[0x55e3fac01f55]
redis_1 | redis-rdb-bgsave *:6379(aeProcessEvents+0xfd)[0x55e3fabf9b3d]
redis_1 | redis-rdb-bgsave *:6379(aeMain+0x1d)[0x55e3fabf9f3d]
redis_1 | redis-rdb-bgsave *:6379(main+0x314)[0x55e3fabf66f4]
redis_1 | /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)[0x7fba1733e09b]
redis_1 | redis-rdb-bgsave *:6379(_start+0x2a)[0x55e3fabf6bda]
redis_1 |
redis_1 | ------ REGISTERS ------
redis_1 | 55:C 26 Sep 2021 00:40:33.287 #
redis_1 | RAX:ffffffffffffffff RBX:00007ffcab9d6380
redis_1 | RCX:ffffffffffffffff RDX:4b424b4867496b52
redis_1 | RDI:00007ffcab9d6380 RSI:4b424b4867496b52
redis_1 | RBP:0000000000000000 RSP:00007ffcab9d6250
redis_1 | R8 :00007fba16e0fb70 R9 :0000000000015f08
redis_1 | R10:0000000000000000 R11:00007fba16e0b430
redis_1 | R12:4b424b4867496b52 R13:4b424b4867496b52
redis_1 | R14:0000000000000000 R15:00007ffcab9d6320
redis_1 | RIP:000055e3fac2d804 EFL:0000000000010206
redis_1 | CSGSFS:002b000000000033
redis_1 | 55:C 26 Sep 2021 00:40:33.287 # (00007ffcab9d625f) -> 000055e3fac2ecb9
redis_1 | 55:C 26 Sep 2021 00:40:33.287 # (00007ffcab9d625e) -> 00007ffcab9d6320
redis_1 | 55:C 26 Sep 2021 00:40:33.287 # (00007ffcab9d625d) -> 0000000000000000
redis_1 | 55:C 26 Sep 2021 00:40:33.287 # (00007ffcab9d625c) -> 00007fba16f04000
redis_1 | 55:C 26 Sep 2021 00:40:33.287 # (00007ffcab9d625b) -> 4b424b4867496b52
redis_1 | 55:C 26 Sep 2021 00:40:33.287 # (00007ffcab9d625a) -> 000000000000bfc8
redis_1 | 55:C 26 Sep 2021 00:40:33.287 # (00007ffcab9d6259) -> 00007ffcab9d6380
redis_1 | 55:C 26 Sep 2021 00:40:33.287 # (00007ffcab9d6258) -> 00007ffcab9d6380
redis_1 | 55:C 26 Sep 2021 00:40:33.287 # (00007ffcab9d6257) -> 00007ffcab9e87b5
redis_1 | 55:C 26 Sep 2021 00:40:33.287 # (00007ffcab9d6256) -> 000000042a1695b8
redis_1 | 55:C 26 Sep 2021 00:40:33.287 # (00007ffcab9d6255) -> 0000000000083000
redis_1 | 55:C 26 Sep 2021 00:40:33.287 # (00007ffcab9d6254) -> 0000000000000001
redis_1 | 55:C 26 Sep 2021 00:40:33.287 # (00007ffcab9d6253) -> 000055e3fac2e5c7
redis_1 | 55:C 26 Sep 2021 00:40:33.287 # (00007ffcab9d6252) -> 00007ffcab9d6320
redis_1 | 55:C 26 Sep 2021 00:40:33.287 # (00007ffcab9d6251) -> 0000000000000000
redis_1 | 55:C 26 Sep 2021 00:40:33.287 # (00007ffcab9d6250) -> 00007fba16e5f780
redis_1 |
redis_1 | ------ INFO OUTPUT ------
redis_1 | # Server
redis_1 | redis_version:6.2.5
redis_1 | redis_git_sha1:00000000
redis_1 | redis_git_dirty:0
redis_1 | redis_build_id:2a56d1e8c2635a70
redis_1 | redis_mode:standalone
redis_1 | os:Linux 5.13.16-100.fc33.x86_64 x86_64
redis_1 | arch_bits:64
redis_1 | multiplexing_api:epoll
redis_1 | atomicvar_api:c11-builtin
redis_1 | gcc_version:8.3.0
redis_1 | process_id:55
redis_1 | process_supervised:no
redis_1 | run_id:05579e70dbb10edde095aec6ad9e87a7590d5fa8
redis_1 | tcp_port:6379
redis_1 | server_time_usec:1632616833046453
redis_1 | uptime_in_seconds:6341
redis_1 | uptime_in_days:0
redis_1 | hz:10
redis_1 | configured_hz:10
redis_1 | lru_clock:5226881
redis_1 | executable:/data/redis-server
redis_1 | config_file:
redis_1 | io_threads_active:0
redis_1 |
redis_1 | # Clients
redis_1 | connected_clients:6
redis_1 | cluster_connections:0
redis_1 | maxclients:10000
redis_1 | client_recent_max_input_buffer:32
redis_1 | client_recent_max_output_buffer:0
redis_1 | blocked_clients:0
redis_1 | tracking_clients:0
redis_1 | clients_in_timeout_table:0
redis_1 |
redis_1 | # Memory
redis_1 | used_memory:98098664
redis_1 | used_memory_human:93.55M
redis_1 | used_memory_rss:107307008
redis_1 | used_memory_rss_human:102.34M
redis_1 | used_memory_peak:181952832
redis_1 | used_memory_peak_human:173.52M
redis_1 | used_memory_peak_perc:53.91%
redis_1 | used_memory_overhead:8757832
redis_1 | used_memory_startup:809920
redis_1 | used_memory_dataset:89340832
redis_1 | used_memory_dataset_perc:91.83%
redis_1 | allocator_allocated:98248520
redis_1 | allocator_active:98553856
redis_1 | allocator_resident:105357312
redis_1 | total_system_memory:67333484544
redis_1 | total_system_memory_human:62.71G
redis_1 | used_memory_lua:37888
redis_1 | used_memory_lua_human:37.00K
redis_1 | used_memory_scripts:0
redis_1 | used_memory_scripts_human:0B
redis_1 | number_of_cached_scripts:0
redis_1 | maxmemory:0
redis_1 | maxmemory_human:0B
redis_1 | maxmemory_policy:noeviction
redis_1 | allocator_frag_ratio:1.00
redis_1 | allocator_frag_bytes:305336
redis_1 | allocator_rss_ratio:1.07
redis_1 | allocator_rss_bytes:6803456
redis_1 | rss_overhead_ratio:1.02
redis_1 | rss_overhead_bytes:1949696
redis_1 | mem_fragmentation_ratio:1.09
redis_1 | mem_fragmentation_bytes:9169232
redis_1 | mem_not_counted_for_evict:1274
redis_1 | mem_replication_backlog:0
redis_1 | mem_clients_slaves:0
redis_1 | mem_clients_normal:123040
redis_1 | mem_aof_buffer:1280
redis_1 | mem_allocator:jemalloc-5.1.0
redis_1 | active_defrag_running:0
redis_1 | lazyfree_pending_objects:0
redis_1 | lazyfreed_objects:0
redis_1 |
redis_1 | # Persistence
redis_1 | loading:0
redis_1 | current_cow_size:0
redis_1 | current_cow_size_age:0
redis_1 | current_fork_perc:0.00
redis_1 | current_save_keys_processed:0
redis_1 | current_save_keys_total:0
redis_1 | rdb_changes_since_last_save:8268
redis_1 | rdb_bgsave_in_progress:0
redis_1 | rdb_last_save_time:1632616514
redis_1 | rdb_last_bgsave_status:err
redis_1 | rdb_last_bgsave_time_sec:3
redis_1 | rdb_current_bgsave_time_sec:-1
redis_1 | rdb_last_cow_size:1679360
redis_1 | aof_enabled:1
redis_1 | aof_rewrite_in_progress:0
redis_1 | aof_rewrite_scheduled:0
redis_1 | aof_last_rewrite_time_sec:1
redis_1 | aof_current_rewrite_time_sec:-1
redis_1 | aof_last_bgrewrite_status:ok
redis_1 | aof_last_write_status:ok
redis_1 | aof_last_cow_size:1740800
redis_1 | module_fork_in_progress:0
redis_1 | module_fork_last_cow_size:0
redis_1 | aof_current_size:102191584
redis_1 | aof_base_size:62117139
redis_1 | aof_pending_rewrite:0
redis_1 | aof_buffer_length:0
redis_1 | aof_rewrite_buffer_length:0
redis_1 | aof_pending_bio_fsync:0
redis_1 | aof_delayed_fsync:0
redis_1 |
redis_1 | # Stats
redis_1 | total_connections_received:76
redis_1 | total_commands_processed:4283803
redis_1 | instantaneous_ops_per_sec:18
redis_1 | total_net_input_bytes:2543071051
redis_1 | total_net_output_bytes:2536549102
redis_1 | instantaneous_input_kbps:14.46
redis_1 | instantaneous_output_kbps:9.53
redis_1 | rejected_connections:0
redis_1 | sync_full:0
redis_1 | sync_partial_ok:0
redis_1 | sync_partial_err:0
redis_1 | expired_keys:0
redis_1 | expired_stale_perc:0.00
redis_1 | expired_time_cap_reached_count:0
redis_1 | expire_cycle_cpu_milliseconds:86
redis_1 | evicted_keys:0
redis_1 | keyspace_hits:3734518
redis_1 | keyspace_misses:154
redis_1 | pubsub_channels:1
redis_1 | pubsub_patterns:0
redis_1 | latest_fork_usec:3263
redis_1 | total_forks:34
redis_1 | migrate_cached_sockets:0
redis_1 | slave_expires_tracked_keys:0
redis_1 | active_defrag_hits:0
redis_1 | active_defrag_misses:0
redis_1 | active_defrag_key_hits:0
redis_1 | active_defrag_key_misses:0
redis_1 | tracking_total_keys:0
redis_1 | tracking_total_items:0
redis_1 | tracking_total_prefixes:0
redis_1 | unexpected_error_replies:0
redis_1 | total_error_replies:77
redis_1 | dump_payload_sanitizations:0
redis_1 | total_reads_processed:4283950
redis_1 | total_writes_processed:7164156
redis_1 | io_threaded_reads_processed:0
redis_1 | io_threaded_writes_processed:0
redis_1 |
redis_1 | # Replication
redis_1 | role:master
redis_1 | connected_slaves:0
redis_1 | master_failover_state:no-failover
redis_1 | master_replid:4b298a3aea2f704df93322b8c8726f52809b2e09
redis_1 | master_replid2:0000000000000000000000000000000000000000
redis_1 | master_repl_offset:0
redis_1 | second_repl_offset:-1
redis_1 | repl_backlog_active:0
redis_1 | repl_backlog_size:1048576
redis_1 | repl_backlog_first_byte_offset:0
redis_1 | repl_backlog_histlen:0
redis_1 |
redis_1 | # CPU
redis_1 | used_cpu_sys:0.021971
redis_1 | used_cpu_user:0.214888
redis_1 | used_cpu_sys_children:0.000000
redis_1 | used_cpu_user_children:0.000000
redis_1 | used_cpu_sys_main_thread:0.021971
redis_1 | used_cpu_user_main_thread:0.214888
redis_1 |
redis_1 | # Modules
redis_1 |
redis_1 | # Commandstats
redis_1 | cmdstat_subscribe:calls=18,usec=93,usec_per_call=5.17,rejected_calls=0,failed_calls=0
redis_1 | cmdstat_publish:calls=274581,usec=2965924,usec_per_call=10.80,rejected_calls=0,failed_calls=0
redis_1 | cmdstat_keys:calls=27,usec=1233810,usec_per_call=45696.67,rejected_calls=0,failed_calls=0
redis_1 | cmdstat_get:calls=3734672,usec=5210970,usec_per_call=1.40,rejected_calls=0,failed_calls=0
redis_1 | cmdstat_flushdb:calls=1,usec=64823,usec_per_call=64823.00,rejected_calls=0,failed_calls=0
redis_1 | cmdstat_set:calls=274504,usec=1890179,usec_per_call=6.89,rejected_calls=77,failed_calls=0
redis_1 |
redis_1 | # Errorstats
redis_1 | errorstat_MISCONF:count=77
redis_1 |
redis_1 | # Cluster
redis_1 | cluster_enabled:0
redis_1 |
redis_1 | # Keyspace
redis_1 | db0:keys=143161,expires=0,avg_ttl=0
redis_1 |
redis_1 | ------ CLIENT LIST OUTPUT ------
redis_1 | id=45 addr=10.8.165.115:37876 laddr=172.20.0.2:6379 fd=32 name= age=2044 idle=1 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 ob
l=0 oll=0 omem=0 tot-mem=20504 events=r cmd=subscribe user=default redir=-1
redis_1 | id=46 addr=10.8.165.115:37878 laddr=172.20.0.2:6379 fd=33 name= age=2044 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 ob
l=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
redis_1 | id=40 addr=10.8.165.115:37866 laddr=172.20.0.2:6379 fd=27 name= age=2046 idle=1 flags=P db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 ob
l=0 oll=0 omem=0 tot-mem=20504 events=r cmd=subscribe user=default redir=-1
redis_1 | id=41 addr=10.8.165.115:37868 laddr=172.20.0.2:6379 fd=28 name= age=2046 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 ob
l=0 oll=0 omem=0 tot-mem=20512 events=r cmd=publish user=default redir=-1
redis_1 | id=42 addr=10.8.165.115:37870 laddr=172.20.0.2:6379 fd=29 name= age=2046 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 ob
l=0 oll=0 omem=0 tot-mem=20504 events=r cmd=get user=default redir=-1
redis_1 | id=44 addr=10.8.165.115:37874 laddr=172.20.0.2:6379 fd=31 name= age=2044 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 ob
l=0 oll=0 omem=0 tot-mem=20512 events=r cmd=publish user=default redir=-1
redis_1 |
redis_1 | ------ MODULES INFO OUTPUT ------
redis_1 |
redis_1 | ------ FAST MEMORY TEST ------
redis_1 | *** Preparing to test memory region 55e3fadb7000 (2277376 bytes)
redis_1 | *** Preparing to test memory region 55e3fb530000 (270336 bytes)
redis_1 | *** Preparing to test memory region 7fb9f8000000 (135168 bytes)
redis_1 | *** Preparing to test memory region 7fb9fda00000 (8388608 bytes)
redis_1 | *** Preparing to test memory region 7fb9fe200000 (2097152 bytes)
redis_1 | *** Preparing to test memory region 7fb9fe600000 (8388608 bytes)
redis_1 | *** Preparing to test memory region 7fb9fee00000 (8388608 bytes)
redis_1 | *** Preparing to test memory region 7fb9ff6fc000 (355467264 bytes)
redis_1 | *** Preparing to test memory region 7fba149fd000 (8388608 bytes)
redis_1 | *** Preparing to test memory region 7fba151fe000 (8388608 bytes)
redis_1 | *** Preparing to test memory region 7fba159ff000 (8388608 bytes)
redis_1 | *** Preparing to test memory region 7fba16200000 (8388608 bytes)
redis_1 | *** Preparing to test memory region 7fba16a00000 (8388608 bytes)
redis_1 | *** Preparing to test memory region 7fba17316000 (16384 bytes)
redis_1 | *** Preparing to test memory region 7fba174d7000 (24576 bytes)
redis_1 | *** Preparing to test memory region 7fba174fa000 (16384 bytes)
redis_1 | *** Preparing to test memory region 7fba177e3000 (16384 bytes)
redis_1 | *** Preparing to test memory region 7fba17a0b000 (8192 bytes)
redis_1 | *** Preparing to test memory region 7fba17a38000 (4096 bytes)
redis_1 | .O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
redis_1 | Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
redis_1 |
redis_1 | ------ DUMPING CODE AROUND EIP ------
redis_1 | Symbol: rdbSaveObjectType (base: 0x55e3fac2d800)
redis_1 | Module: redis-rdb-bgsave *:6379 (base 0x55e3fabb3000)
redis_1 | $ xxd -r -p /tmp/dump.hex /tmp/dump.bin
redis_1 | $ objdump --adjust-vma=0x55e3fac2d800 -D -b binary -m i386:x86-64 /tmp/dump.bin
redis_1 | ------
redis_1 | 55:C 26 Sep 2021 00:40:34.825 # dump of function (hexdump of 132 bytes):
redis_1 | 4883ec180fb61689d083e00f3c060f8795010000488d0d29e311000fb6c0486304814801c8ffe0660f1f840000000000488d74240fba01000000c644240f07e86ce6ffff4883c418c30
f1f8000000000488d74240fba01000000c644240f0fe84ce6ffff4883c418c30f1f8000000000488d74240fba01000000c644240f00e82ce6ffff
redis_1 |
redis_1 | === REDIS BUG REPORT END. Make sure to include from START to END. ===
Additional information
- OS distribution and version Fedora 33, using redis 6.2.5 in Docker
- Steps to reproduce (if any) Hard to say, start up the docker container with ports published, and hit it with about a dozen-sh clients Set()'ing and publishing. Keys are about approx 580 chars. each thread is setting and publishing about 8x2 keys a second.
Comment From: nathanhack
Following the above crash my AOF seems to be corrupt:
redis_1 | 1:C 26 Sep 2021 02:13:27.673 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis_1 | 1:C 26 Sep 2021 02:13:27.673 # Redis version=6.2.5, bits=64, commit=00000000, modified=0, pid=1, just started
redis_1 | 1:C 26 Sep 2021 02:13:27.673 # Configuration loaded
redis_1 | 1:M 26 Sep 2021 02:13:27.673 * monotonic clock: POSIX clock_gettime
redis_1 | 1:M 26 Sep 2021 02:13:27.674 * Running mode=standalone, port=6379.
redis_1 | 1:M 26 Sep 2021 02:13:27.674 # Server initialized
redis_1 | 1:M 26 Sep 2021 02:13:27.674 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis_1 | 1:M 26 Sep 2021 02:13:27.674 * Reading RDB preamble from AOF file...
redis_1 | 1:M 26 Sep 2021 02:13:27.674 * Loading RDB produced by version 6.2.5
redis_1 | 1:M 26 Sep 2021 02:13:27.674 * RDB age 6508 seconds
redis_1 | 1:M 26 Sep 2021 02:13:27.674 * RDB memory usage when created 71.54 Mb
redis_1 | 1:M 26 Sep 2021 02:13:27.674 * RDB has an AOF tail
redis_1 | 1:M 26 Sep 2021 02:13:27.807 # Wrong RDB checksum expected: (37309b0d67726672) but got (de6e363aed78836f). Aborting now.
redis_1 | 1:M 26 Sep 2021 02:13:27.807 # Internal error in RDB reading offset 60817112, function at rdb.c:2614 -> RDB CRC error
redis_1 | [offset 0] Checking RDB file appendonly.aof
redis_1 | [offset 26] AUX FIELD redis-ver = '6.2.5'
redis_1 | [offset 40] AUX FIELD redis-bits = '64'
redis_1 | [offset 52] AUX FIELD ctime = '1632615899'
redis_1 | [offset 67] AUX FIELD used-mem = '75010328'
redis_1 | [offset 83] AUX FIELD aof-preamble = '1'
redis_1 | [offset 85] Selecting DB ID 0
redis_1 | --- RDB ERROR DETECTED ---
redis_1 | [offset 62067761] RDB CRC error
redis_1 | [additional info] While doing: check-sum
redis_1 | [info] 107596 keys read
redis_1 | [info] 0 expires
redis_1 | [info] 0 already expired
search_redis_1 exited with code 1
Is there any way to recover it? I've tried using redis-check-aof --fix but that didn't work (see below).
root@7650cf1bb563:/data# redis-check-aof --fix appendonly.aof.fix
The AOF appears to start with an RDB preamble.
Checking the RDB preamble to start:
[offset 0] Checking RDB file --fix
[offset 26] AUX FIELD redis-ver = '6.2.5'
[offset 40] AUX FIELD redis-bits = '64'
[offset 52] AUX FIELD ctime = '1632615899'
[offset 67] AUX FIELD used-mem = '75010328'
[offset 83] AUX FIELD aof-preamble = '1'
[offset 85] Selecting DB ID 0
--- RDB ERROR DETECTED ---
[offset 62067761] RDB CRC error
[additional info] While doing: check-sum
[info] 107596 keys read
[info] 0 expires
[info] 0 already expired
RDB preamble of AOF file is not sane, aborting.
Comment From: oranagra
smells like some kind of memory corruption. the crash was in the background save process (didn't cause the main server to stop) while generating an RDB, the fact the AOF is also corrupt, suggests memory corruption too. do you happen to know if your hardware is using ECC memory?
Comment From: oranagra
p.s. signal 11 is SIGSGV, and AFAIK the only valid values for SI_CODE are:
#define SEGV_MAPERR 1 /* address not mapped to object */
#define SEGV_ACCERR 2 /* invalid permissions for mapped object */
#define SEGV_BNDERR 3 /* failed address bound checks */
#define SEGV_PKUERR 4 /* failed protection key checks */
so a value of 128 seems odd, @yossigo do you have any insight into that?
Comment From: yossigo
@oranagra si_code value of 0x80 is SI_KERNEL - could be hardware issues, kernel bugs, etc.
Comment From: oranagra
@nathanhack can you please see if you have anything in your dmsg/syslog/kern.log etc if you still have them..
Comment From: nathanhack
smells like some kind of memory corruption. the crash was in the background save process (didn't cause the main server to stop) while generating an RDB, the fact the AOF is also corrupt, suggests memory corruption too. do you happen to know if your hardware is using ECC memory?
Sadly all 64GB of ram is non ECC. The only artifact I still have is the AOF file, do you think there is any way to recover it?
Comment From: nathanhack
p.s. signal 11 is SIGSGV, and AFAIK the only valid values for SI_CODE are:
```
define SEGV_MAPERR 1 / address not mapped to object /
define SEGV_ACCERR 2 / invalid permissions for mapped object /
define SEGV_BNDERR 3 / failed address bound checks /
define SEGV_PKUERR 4 / failed protection key checks /
```
so a value of 128 seems odd, @yossigo do you have any insight into that?
The situation was weird. After redis crashed I quickly grabbed the log entry about the crash and my AOF file. Right after the system became unresponsive (it required a reboot). What ever happened, it took the whole system down. If this is a memory issue it's the first noticed incident. Since the reboot it seems stable. I've been hammering away at it with no problem this time. One thing to note (or it at least I think I saw in htop) was half of swap was being used. Swap is on an SSD, about 3 years old, and has probably been put through above average use. If there was a weak point that might have been it.
Comment From: nathanhack
@nathanhack can you please see if you have anything in your dmsg/syslog/kern.log etc if you still have them..
I did a quick look, but didn't find anything (I'm also not an expert with docker yet). The container was set with --rm, do you know if there's any places the logs would be hiding?
Comment From: oranagra
Considering what you wrote about the whole system being unresponsive and what Yossi said about kernel triggered signal, I don't think this was memory corruption inside redis anymore, so I don't think ECC has anything to do with it. Sounds like some system issue.
Regarding the recovery of data, I see that the aof failed on rdb checksum check, maybe if you disable that (edit redis code to skip it) you'll be able to somehow recover the data. Another alternative is to try to load it with some other tools e.g. Redis-rdb-tools or others.