Comment From: wilkinsona

Using Kafka with Docker Compose when you want to use ephemeral ports is quite hard.

It doesn't appear to be possible with the official image (confluentinc/cp-kafka) as Kafka cannot be configured within the compose YAML to advertise the ephemeral port. Testcontainers works around this by dynamically exporting the KAFKA_ADVERTISED_LISTENERS in the container when it's starting up and after the port mapping has been performed. I don't think such an approach is possible with Docker Compose.

The wurstmeister/kafka image works around this by including scripts in the image that use a PORT_COMMAND environment variable to provide a command that should be run in the container to determine the port that Kafka should advertise. This works well with Docker Compose but the latest version of Kafka supported by the image is 2.8.1 which was released in September 2021. At the time of writing, the latest version of Kafka is 3.4.0 which was released in February 2023.

Comment From: wilkinsona

Based on the problems described above, we've decided to decline this for now at least.

Comment From: trajano

Assuming the following compose.yml

services:
  zookeeper:
    image: confluentinc/cp-zookeeper
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_LOG4J_ROOT_LOGLEVEL: WARN
    deploy:
      resources:
        limits:
          memory: 256m
    healthcheck:
      test: zookeeper-shell localhost:2181 ls /

  kafka:
    image: confluentinc/cp-kafka
    environment:
      KAFKA_ADVERTISED_HOST_PORT: kafka:49092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: COMPOSE://0.0.0.0:49092,LOCALHOST://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: COMPOSE://kafka:49092,LOCALHOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: COMPOSE:PLAINTEXT,LOCALHOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: COMPOSE
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_LOG4J_ROOT_LOGLEVEL: WARN
      KAFKA_LOG4J_LOGGERS: kafka=WARN,kafka.controller=WARN
      KAFKA_OPTS: -XX:MaxRAMPercentage=80
    healthcheck:
      test: kafka-log-dirs --bootstrap-server kafka:49092 --describe
    labels:
      org.springframework.boot.service-connection: kafka
    ports:
      - '9092'

The problem is making these two be set as part of the environment BEFORE the container is instantiated.

  KAFKA_LISTENERS: LOCALHOST://0.0.0.0:9092
  KAFKA_ADVERTISED_LISTENERS:  LOCALHOST://localhost:9092

The only way I can think of how to do this would be to allow something like

  KAFKA_LISTENERS: COMPOSE://0.0.0.0:49092,LOCALHOST://0.0.0.0:${KAFKA_EPHEMERAL_PORT:-9092}
  KAFKA_ADVERTISED_LISTENERS: COMPOSE://kafka:49092,LOCALHOST://localhost:${KAFKA_EPHEMERAL_PORT:-9092}

Where EPHEMERAL_PORT is computed by Spring before sending to Docker CLI