Describe the bug
Getting fast memory issue.
To reproduce
kubectl create -f redis-deployment-definition.yaml
What special configuration is required to run redis pod on an arm64 worker node.
I modified the deployment config as shown below: below is my redis-deployment-definition.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
component: redis
name: redis
spec:
selector:
matchLabels:
component: redis
replicas: 1
template:
metadata:
labels:
component: redis
spec:
containers:
- name: redis
image: redis:rc-buster
ports:
- containerPort: 6379
resources:
limits:
memory: 400Mi
cpu: 1
requests:
memory: 200Mi
cpu: 500m
restartPolicy: Always
nodeSelector:
boardType: rockpro64-arm64
A description of what you expected to happen.
Additional information
Any additional information that is relevant to the problem.
Error I am getting
1:C 17 Jan 2021 20:34:14.005 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 17 Jan 2021 20:34:14.005 # Redis version=6.1.241, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 17 Jan 2021 20:34:14.005 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
1:M 17 Jan 2021 20:34:14.010 * monotonic clock: POSIX clock_gettime
1:M 17 Jan 2021 20:34:14.012 * Running mode=standalone, port=6379.
1:M 17 Jan 2021 20:34:14.013 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 17 Jan 2021 20:34:14.013 # Server initialized
=== REDIS BUG REPORT START: Cut & paste starting from here ===
1:M 17 Jan 2021 20:34:14.013 # === ASSERTION FAILED ===
1:M 17 Jan 2021 20:34:14.013 # ==> server.c:5239 '!ret' is not true
------ STACK TRACE ------
Backtrace:
redis-server *:6379(linuxMadvFreeForkBugCheck+0x2e0)[0x555f99caa0]
redis-server *:6379(main+0x3ac)[0x555f98d924]
/lib/aarch64-linux-gnu/libc.so.6(__libc_start_main+0xe4)[0x7fa836dd24]
redis-server *:6379(+0x42cd0)[0x555f98dcd0]
------ INFO OUTPUT ------
# Server
redis_version:6.1.241
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:f73a8ce7d18e1a7e
redis_mode:standalone
os:Linux 4.4.190-1233-rockchip-ayufan-gd3f1be0ed310 aarch64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:8.3.0
process_id:1
process_supervised:no
run_id:d7bd1367ce4f031c1084bc823155a1da71674c8a
tcp_port:6379
server_time_usec:1610915653995871
uptime_in_seconds:-1
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:302917
executable:/data/redis-server
config_file:
io_threads_active:0
# Clients
connected_clients:0
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:0
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0
# Memory
used_memory:817576
used_memory_human:798.41K
used_memory_rss:0
used_memory_rss_human:0B
used_memory_peak:817576
used_memory_peak_human:798.41K
used_memory_peak_perc:inf%
used_memory_overhead:0
used_memory_startup:0
used_memory_dataset:817576
used_memory_dataset_perc:100.00%
allocator_allocated:0
allocator_active:0
allocator_resident:0
total_system_memory:2085765120
total_system_memory_human:1.94G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:nan
allocator_frag_bytes:0
allocator_rss_ratio:nan
allocator_rss_bytes:0
rss_overhead_ratio:nan
rss_overhead_bytes:0
mem_fragmentation_ratio:nan
mem_fragmentation_bytes:0
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:0
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0
# Persistence
loading:0
current_cow_size:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1610915654
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0
module_fork_in_progress:0
module_fork_last_cow_size:0
# Stats
total_connections_received:0
total_commands_processed:0
instantaneous_ops_per_sec:0
total_net_input_bytes:0
total_net_output_bytes:0
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
total_forks:0
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:0
dump_payload_sanitizations:0
total_reads_processed:0
total_writes_processed:0
io_threaded_reads_processed:0
io_threaded_writes_processed:0
# Replication
role:master
connected_slaves:0
master_replid:948654150bf6d84c72889c5480580b6e60dfc866
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
# CPU
used_cpu_sys:0.044000
used_cpu_user:0.056000
used_cpu_sys_children:0.008000
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:0.040000
used_cpu_user_main_thread:0.048000
# Modules
# Commandstats
# Errorstats
# Cluster
cluster_enabled:0
# Keyspace
------ CLIENT LIST OUTPUT ------
------ MODULES INFO OUTPUT ------
------ FAST MEMORY TEST ------
*** Preparing to test memory region 555fb3d000 (708608 bytes)
*** Preparing to test memory region 5576a55000 (270336 bytes)
*** Preparing to test memory region 7fa7500000 (13631488 bytes)
*** Preparing to test memory region 7fa84bb000 (16384 bytes)
*** Preparing to test memory region 7fa84ea000 (16384 bytes)
*** Preparing to test memory region 7fa8791000 (16384 bytes)
*** Preparing to test memory region 7fa8873000 (4096 bytes)
*** Preparing to test memory region 7fa8952000 (4096 bytes)
*** Preparing to test memory region 7fa8954000 (32768 bytes)
.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Please report the crash by opening an issue on github:
http://github.com/redis/redis/issues
Suspect RAM error? Use redis-server --test-memory to verify it.
Comment From: yossigo
Because of a serious bug in some versions of the Linux kernel on arm64, Redis attempts to test kernel behavior and produce a warning/refuse to start. In your case it seems like the test itself fails for some reason (perhaps due to container restrictions?).
To get some idea of what happens, can you please apply the following patch and see what error is produced in the logs (as a side effect Redis will also start):
diff --git a/src/server.c b/src/server.c
index 90b9045ea..13678036c 100644
--- a/src/server.c
+++ b/src/server.c
@@ -5236,7 +5236,10 @@ int linuxMadvFreeForkBugCheck(void) {
#define MADV_FREE 8
#endif
ret = madvise(q, 4096, MADV_FREE);
- serverAssert(!ret);
+ if (ret < 0) {
+ serverLog(LL_WARNING, "madvise failed: errno=%d", errno);
+ return 0;
+ }
/* Write to the page after being marked for freeing, this is supposed to take
* ownership of that page again. */
Comment From: yossigo
Update: have been trying to reproduce this on an arm64 board (Linux pine64 5.10.4-sunxi64 #20.11.7 SMP Tue Jan 5 23:28:09 CET 2021 aarch64 aarch64 aarch64 GNU/Linux) with both docker and microk8s (running the above deployment) without success.
I assume this might have something to do with the specific kernel in use, but can't tell for sure. Unless we learn something new I think 6.2 should treat such errors as test failure (which can be manually ignored if necessary).
Comment From: oranagra
@sanfx can you please apply the change Yossi suggested and share the results?