Describe the bug
I am trying to run redis cluster in nomad. If I assign bus port increment by 10000 to the data port everything is fine.
However if I leave Nomad to assign the ports (it uses range 20000-32000) or I assign something that is not increment by 10000 the cluster create command is stuck on Waiting for the cluster to join and cluster cant be initialized. This is a problem because we can't really run more than one cluster. So it seems to me that cluster-bus in the config file does not overwrite the default behavior as described in the documentation. And the cluster expects the bus port to be +10000 of the data port.
Tested with official docker images, versions 7.0.2, 7.0, 6.2.7, 5.0.14
I have cluster-port defined in the configuration file.
Working with: data port: 2164{0..6} bus port: 3164{0..6}
Does not work: data port: 2664{0..6} bus port: 3164{0..6}
To reproduce
Create 6 redis instances in Nomad without setting static port and create cluster with them. You can use the following Nomad file (needs to be run through Levant template) Static port allocations are commented, if you remove the comments the cluster will work.
job "redis-cluster" {
datacenters = ["dc1"]
type = "service"
[[ range $idx := loop 6 ]]
group "redis-[[ $idx ]]" {
network {
port "db" {
//static = 638[[ $idx ]]
}
port "gossip" {
//static = 1638[[ $idx ]]
}
}
task "redis" {
driver = "docker"
config {
image = "redis:7.0.2"
ports = ["db", "gossip"]
args = [
"redis-server", "/etc/redis/redis.conf"
]
volumes = [
"local/redis.conf:/etc/redis/redis.conf"
]
}
template {
destination = "local/redis.conf"
change_mode = "restart"
data = <<EOF
#bind {{ env "NOMAD_IP_db" }}
bind 0.0.0.0
port {{ env "NOMAD_PORT_db"}}
########################## CLUSTER DOCKER/NAT support ########################
# In certain deployments, Redis Cluster nodes address discovery fails, because
# addresses are NAT-ted or because ports are forwarded (the typical case is
# Docker and other containers).
#
# In order to make Redis Cluster working in such environments, a static
# configuration where each node knows its public address is needed. The
# following two options are used for this scope, and are:
#
# * cluster-announce-ip
# * cluster-announce-port
# * cluster-announce-bus-port
cluster-announce-ip {{ env "NOMAD_IP_db" }}
cluster-announce-port {{ env "NOMAD_PORT_db" }}
cluster-announce-bus-port {{ env "NOMAD_PORT_gossip" }}
cluster-port {{ env "NOMAD_PORT_gossip"}}
cluster-node-timeout 15000
cluster-enabled yes
protected-mode no
cluster-config-file {{ env "NOMAD_TASK_DIR" }}/nodes.conf
cluster-node-timeout 5000
appendonly yes
EOF
}
service {
name = "redis"
port = "db"
check {
port = "db"
type = "tcp"
interval = "5s"
timeout = "3s"
#initial_status = "passing"
}
}
resources {
cpu = 100
memory = 100
}
}
}
[[ end ]]
}
Expected behavior
To be able to run cluster with any data and bus port range from 20000 to 32000
Comment From: enjoy-binbin
if i remember correctly, cluster-port was added in 7.0, yes, see #9389, i think the doc should mention the version
how do you create the cluster, by running redis-cli --cluster create? if so, currently redis-cli doesn't support instances with customized cluster-port, see #10344 (this may be part of the 7.0.3)
Comment From: madolson
To extend the comment, it seems likely that it's just using CLUSTER MEET {port} which assumes that the cluster bus port is {port} + 10000.
Comment From: idenkov
if i remember correctly, cluster-port was added in 7.0, yes, see #9389, i think the doc should mention the version
how do you create the cluster, by running redis-cli --cluster create? if so, currently redis-cli doesn't support instances with customized cluster-port, see #10344 (this may be part of the 7.0.3)
Yes I am using the the redis-cli --cluster create, but I do not pass the bus port because it accept only the data port. AFAIK the bus port can be used only with cluster meet. Bus port is defined in the config, but it does not seems like redis take it into consideration and it just does +10000 on the data port.
Comment From: enjoy-binbin
yes, like i said, currently redis-cli --cluster create dose not support instances with customized cluster-port. it (redis-cli) just send CLUSTER MEET without the bus_port, by default, redis-server just dose +10000, so the redis-server (clusters) cannot communicate on the real bus_port port
Comment From: idenkov
yes, like i said, currently redis-cli --cluster create dose not support instances with customized cluster-port. it (redis-cli) just send CLUSTER MEET without the bus_port, by default, redis-server just dose +10000, so the redis-server (clusters) cannot communicate on the real bus_port port
Redis does not take the cluster-port from the config. Even if I send it with cluster meet the nodes can't connect. The issue is not that I can not send custom bus port with cluster create, the issue is that is not using any other port defined in the config. cluster-port in redis.conf does not do anything. Sending it with cluster meet still marks nodes as disconnected. So simply it does not work at all.
Comment From: enjoy-binbin
Redis does not take the cluster-port from the config
no way. can you try config get cluster-port? and see the ouput. i just try it, it work
[root@binblog redis]# src/redis-cli -p 7004 config get cluster-port
1) "cluster-port"
2) "8888"
[root@binblog redis]# src/redis-cli -p 7003 cluster meet 127.0.0.1 7004 8888
OK
[root@binblog redis]# src/redis-cli -p 7003 cluster nodes
b46620805ec7bfed67e565bf3b0eed23913d6520 127.0.0.1:7004@8004 slave,fail 78ffa3569ca0d34c524155972e35bb61c3c78141 1657385359455 1657385358453 1 disconnected
78ffa3569ca0d34c524155972e35bb61c3c78141 127.0.0.1:7003@8003 myself,master - 0 1657385779000 1 connected 3300 15495
461d7307c6506ae1959b076c2862846a56b69c72 127.0.0.1:7004@8888 slave 78ffa3569ca0d34c524155972e35bb61c3c78141 0 1657385783778 1 connected
Even if I send it with cluster meet the nodes can't connect
CLUSTER MEET ip port cluster-port, did you try this way? i know the doc (https://redis.io/commands/cluster-meet/ it will be fix in next release i guess) is missing the cluster-bus arg, but you can see the usage in cluster help