Not Able To Access Messages From Confluent Kafka On Ec2

Confluent Kafka 5.0.0 has been installed on AWS EC2 which has Public IP say 54.XX.XX.XX Opened port 9092 on the EC2 machine with 0.0.0.0 In /etc/kafka/server.properties I have adv

Solution 1:

Following steps seem to work.

On both, local and EC2 machine, in /etc/kakfa/server.properties set

listeners=PLAINTEXT://0.0.0.0:9092advertised.listeners=PLAINTEXT://54.XX.XX.XX:9092

On local machine, in /etc/kakfa/producer.properties set

bootstrap.servers=0.0.0.0:9092

On EC2 machine, in /etc/kakfa/producer.properties set

bootstrap.servers=localhost:9092

On both local and EC2 machine, in /etc/kakfa/consumer.properties set

bootstrap.servers=0.0.0.0:9092group.id=mygroup

Use 'confluent-start' to start all necessary daemons on remote EC2 machine. On local machine, Confluent is NOT made running.

On local machine (for ip hiding, optional):

export KAFKA_PRODUCER_IP=54.XX.XX.XX

With this, producer from local machine,can put messages on remote EC2 Kafka by following:

broker = os.environ['KAFKA_PRODUCER_IP'] + ':9092'topic = 'mytopic'p = Producer({'bootstrap.servers': broker})

From local machine, messages could be fetched from remote EC2 kafka by following:

broker = os.environ['KAFKA_PRODUCER_IP'] + ':9092'
topic = 'mytopic'group = 'mygroup'
     c = Consumer({
         'bootstrap.servers': broker,
         'group.id': group,
         'session.timeout.ms': 6000,
         'default.topic.config': {
             'auto.offset.reset': 'smallest'
         }
     })

These steps seems to work. There could be some redundancies, if so, do point out.