hi, I'm trying to create a redis cluster on microk8s, describe environment below: o/s: windows 11 pro virtual pc engine: hyper-v vm o/s: ubuntu server 22.04.3 k8s: microk8s v1.27.5 rev 5891, install by snap redis image: v7.2.1 pull from docker hub with tag: latest
the problem is I get error message Error: invalid CLUSTER NODES reply command terminated with exit code 1 when executed redis-cli --cluster create redis-app-0-ip:6379 ... redis-app-5-ip:6379 --replicas 1.
kubectl exec -n redis-cluster redis-app-0 -- redis-cli cluster nodes
4abeadb5d70db4f5392315205573cba644a2ab79 10.1.172.253:6379@16379 myself,master - 0 0 0 connected
kubectl exec -n redis-cluster redis-app-5 -- redis-cli cluster nodes
9e3a9bbfee4d854f5aadbae8abc5702382f9ea91 10.1.172.206:6379@16379 myself,master - 0 0 0 connected
please help me to figure out the problem. thank you~
here are configurations:
redis config
apiVersion: v1
kind: ConfigMap
metadata:
name: redis-conf
namespace: redis-cluster
data:
redis.conf: |
appendonly yes
cluster-enabled yes
cluster-config-file /var/lib/redis/nodes.conf
dir /var/lib/redis
port 6379
cluster-announce-bus-port 16379
cluster-node-timeout 5000
persistent volumes
apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-pv-0
namespace: redis-cluster
spec:
storageClassName: "microk8s-hostpath"
capacity:
storage: 0.5Gi
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
hostPath:
path: "/tmp/redis-pv-0"
type: "DirectoryOrCreate"
---
(skip)
---
apiVersion: v1
kind: PersistentVolume
metadata:
name: redis-pv-7
namespace: redis-cluster
spec:
storageClassName: "microk8s-hostpath"
capacity:
storage: 0.5Gi
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
hostPath:
path: "/tmp/redis-pv-7"
type: "DirectoryOrCreate"
statefulset
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: redis-app
namespace: redis-cluster
spec:
selector:
matchLabels:
app: redis-cluster-app
serviceName: "redis-cluster-app"
replicas: 6
template:
metadata:
labels:
app: redis-cluster-app
spec:
containers:
- name: redis
image: "redis:latest"
imagePullPolicy: Never
command:
- "redis-server"
args:
- "/etc/redis/redis.conf"
- "--protected-mode"
- "no"
- "--cluster-announce-ip"
- " $(MY_POD_IP)"
ports:
- name: redis
containerPort: 6379
protocol: "TCP"
- name: cluster
containerPort: 16379
protocol: "TCP"
volumeMounts:
- name: "redis-conf"
mountPath: "/etc/redis"
- name: "redis-data"
mountPath: "/var/lib/redis"
env:
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
volumes:
- name: "redis-conf"
configMap:
name: "redis-conf"
items:
- key: "redis.conf"
path: "redis.conf"
volumeClaimTemplates:
- metadata:
name: redis-data
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "microk8s-hostpath"
resources:
requests:
storage: 0.5Gi
headless
apiVersion: v1
kind: Service
metadata:
name: redis-headless-service
namespace: redis-cluster
labels:
app: redis-cluster-app
spec:
ports:
- name: redis-port
port: 6379
clusterIP: None
selector:
app: redis-cluster-app
service
apiVersion: v1
kind: Service
metadata:
name: redis-headless-service
namespace: redis-cluster
labels:
app: redis-cluster-app
spec:
ports:
- name: redis-port
port: 6379
clusterIP: None
selector:
app: redis-cluster-app
kelvin@ubuntu22043:~/microk8s/redis-cluster$ cat service.yml
apiVersion: v1
kind: Service
metadata:
name: redis-service
namespace: redis-cluster
labels:
app: redis-cluster-app
spec:
ports:
- name: redis-port
protocol: "TCP"
port: 6379
targetPort: 6379
selector:
app: redis-cluster-app
here runtime status: kubectl get pods,pv,pvc,configMap -n redis-cluster
NAME READY STATUS RESTARTS AGE
pod/redis-app-0 1/1 Running 0 102m
pod/redis-app-1 1/1 Running 0 102m
pod/redis-app-2 1/1 Running 0 102m
pod/redis-app-3 1/1 Running 0 102m
pod/redis-app-4 1/1 Running 0 102m
pod/redis-app-5 1/1 Running 0 102m
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/redis-pv-3 512Mi RWO Retain Available microk8s-hostpath 102m
persistentvolume/redis-pv-7 512Mi RWO Retain Available microk8s-hostpath 102m
persistentvolume/redis-pv-0 512Mi RWO Retain Bound redis-cluster/redis-data-redis-app-0 microk8s-hostpath 102m
persistentvolume/redis-pv-1 512Mi RWO Retain Bound redis-cluster/redis-data-redis-app-1 microk8s-hostpath 102m
persistentvolume/redis-pv-5 512Mi RWO Retain Bound redis-cluster/redis-data-redis-app-2 microk8s-hostpath 102m
persistentvolume/redis-pv-4 512Mi RWO Retain Bound redis-cluster/redis-data-redis-app-3 microk8s-hostpath 102m
persistentvolume/redis-pv-6 512Mi RWO Retain Bound redis-cluster/redis-data-redis-app-4 microk8s-hostpath 102m
persistentvolume/redis-pv-2 512Mi RWO Retain Bound redis-cluster/redis-data-redis-app-5 microk8s-hostpath 102m
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/redis-data-redis-app-0 Bound redis-pv-0 512Mi RWO microk8s-hostpath 102m
persistentvolumeclaim/redis-data-redis-app-1 Bound redis-pv-1 512Mi RWO microk8s-hostpath 102m
persistentvolumeclaim/redis-data-redis-app-2 Bound redis-pv-5 512Mi RWO microk8s-hostpath 102m
persistentvolumeclaim/redis-data-redis-app-3 Bound redis-pv-4 512Mi RWO microk8s-hostpath 102m
persistentvolumeclaim/redis-data-redis-app-4 Bound redis-pv-6 512Mi RWO microk8s-hostpath 102m
persistentvolumeclaim/redis-data-redis-app-5 Bound redis-pv-2 512Mi RWO microk8s-hostpath 102m
NAME DATA AGE
configmap/kube-root-ca.crt 1 6d21h
configmap/redis-conf 1 102m
pod ip list
redis-app-0 10.1.172.253
redis-app-1 10.1.172.200
redis-app-2 10.1.172.210
redis-app-3 10.1.172.212
redis-app-4 10.1.172.211
redis-app-5 10.1.172.206
kubectl logs -n redis-cluster redis-app-0
1:C 14 Oct 2023 00:30:19.042 # WARNING: Changing databases number from 16 to 1 since we are in cluster mode
1:C 14 Oct 2023 00:30:19.042 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 14 Oct 2023 00:30:19.042 * Redis version=7.2.1, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 14 Oct 2023 00:30:19.042 * Configuration loaded
1:M 14 Oct 2023 00:30:19.042 * monotonic clock: POSIX clock_gettime
1:M 14 Oct 2023 00:30:19.044 * Running mode=cluster, port=6379.
1:M 14 Oct 2023 00:30:19.044 * No cluster configuration found, I'm 4abeadb5d70db4f5392315205573cba644a2ab79
1:M 14 Oct 2023 00:30:19.049 * Server initialized
1:M 14 Oct 2023 00:30:19.052 * Creating AOF base file appendonly.aof.1.base.rdb on server start
1:M 14 Oct 2023 00:30:19.057 * Creating AOF incr file appendonly.aof.1.incr.aof on server start
1:M 14 Oct 2023 00:30:19.057 * Ready to accept connections tcp
1:M 14 Oct 2023 01:10:59.167 # Bad message length or signature received on the Cluster bus from 10.1.172.206:32886
kubectl exec -n redis-cluster redis-app-0 -- redis-cli cluster info
cluster_state:fail
cluster_slots_assigned:0
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:1
cluster_size:0
cluster_current_epoch:0
cluster_my_epoch:0
cluster_stats_messages_sent:0
cluster_stats_messages_received:0
total_cluster_links_buffer_limit_exceeded:0
Comment From: neonlai
I hv encountered the same issue. @huangchinlin did you resolve this issue yet ? Thanks
Comment From: huangchinlin
@neonlai not yet, but I changed the way to install redis cluster by helm, it works for me, FYI.