Kafka tutorial : My First kafka Program | how to use Kafka

My First Kafka program to start as beginner ,Before start you need to setup Kafka server and Start it resources.

Apache Kafka is a distributed streaming platform that can be used for building real-time streaming and creating a data pipelines and applications. A Kafka consumer is a client that reads records from a Kafka cluster.A Kafka Producer is store data and send to Topic.

We can see kafka consumer and producer in action.In this example we are using console .we have csv file and send to Kafka consumer .

Step 1 : Creating Kafka Topic 

Apache Kafka producers are clients that send records to Kafka topic,Here we are seeing two factor which affect a creation of topic as follow

1. Replication factor : for backup purpose
2. Partition : depend on parellel processing and storage 

Below is command you need to run on console after starting of Kafka server.

bin\windows\kafka-topics.bat - create - topic test - partitions 1 
- replication-factor 1 - bootstrap-server localhost:9092

Step 2 : Sending data to Kafka producer using Kafka console

We need to define broker list and path of file , where our data is store (MysampleFile.csv).Here we can see Producer console and run below command 

bin\windows\kafka-console-producer.bat - topic test - broker-list 
localhost:9092 < C:\Users\Admin\Documents\kafka\data\data\MysampleFile.csv

Step 3: Reading Kafka topic using Consumer using kafka console

Now we can read topic from consumer and we need define parameter as beginning

bin\windows\kafka-console-consumer.bat - topic test - bootstrap-server
 localhost:9092 - from-beginning

We can say ” A group of consumers that work together to consume records from a topic. Each consumer in the group is assigned a subset of the partitions”.

Conclusion:

In this tutorial, we explored the fundamentals of Apache Kafka by setting up a Kafka server, creating topics, and using producers and consumers to handle real-time data streaming. By following these steps, you now understand how to send data to a Kafka topic and retrieve it efficiently using the console.

Kafka is a powerful tool for building scalable data pipelines and streaming applications. As you progress, consider exploring advanced concepts like consumer groups, offsets, and partitioning strategies to optimize your Kafka workflows.

Stay tuned for more Kafka tutorials to enhance your expertise in real-time data streaming! 🚀