{"id":704,"date":"2018-07-11T09:31:25","date_gmt":"2018-07-11T09:31:25","guid":{"rendered":"http:\/\/shijuvarghese.com\/?p=704"},"modified":"2018-09-06T09:55:16","modified_gmt":"2018-09-06T09:55:16","slug":"kafka-basics-terminologies","status":"publish","type":"post","link":"http:\/\/shijuvarghese.com\/?p=704","title":{"rendered":"Kafka: Basics – terminologies"},"content":{"rendered":"

Kafka is a middle-ware application developed by Apache Foundation, used for building real-time data pipelines and streaming apps, which is scalable and has fault-tolerant capabilities.<\/p>\n

 <\/p>\n

Posted below are the basic terminologies used by Kafka, and a simple diagram explaining the process:<\/p>\n

\"Kafka_dig1\"<\/a><\/p>\n

Producer:<\/strong> A application that sends messages to KAFKA
\nMessage:<\/strong> A small peice of data, a simple array of bites. Each row in a file can be send as a message.
\nConsumer:<\/strong> A application that reads data from KAFKA
\nKafka Server:<\/strong> Received messages from Producer, and gives it to Consumer.
\nKafka Broker:<\/strong> A Kafka Server that stand as a middle man to exchange messages.
\nKafka Cluster:<\/strong> A group of Kafka Brokers that works in a distributed environment to get the job done.
\nTopic:<\/strong> Arbitrary name given to a data set so that consumers can as for the correct data from the Broker. Similarly when producer sends a message to Kafka, it needs to be send to specific Topic.
\nPartition:<\/strong> Break up of topics to smaller chunks. The number of partitions for each topic is decided by the person who created the partition.
\nOffset:<\/strong> A sequence ID assigned to messages that arrives in a partition. Message IDs start from 0.
\nA message’s full address:<\/strong> A unique combination of Topic name – Partition number – Offset ID.
\nConsumer Group:<\/strong> It is a group of consumers working as it clustered to one.<\/p>\n

Scaling Kafka:<\/strong> Kafka cluster, partitioning and consumer group is a the path to take.<\/p>\n

Kafka do not allow more than 2 consumers to access a single partition.<\/p>\n","protected":false},"excerpt":{"rendered":"

Kafka is a middle-ware application developed by Apache Foundation, used for building real-time data pipelines and streaming apps, which is scalable and has fault-tolerant capabilities. […]<\/a><\/p>\n<\/div>","protected":false},"author":1,"featured_media":706,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[21,24],"tags":[],"class_list":["post-704","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops","category-kafka"],"_links":{"self":[{"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=\/wp\/v2\/posts\/704","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=704"}],"version-history":[{"count":4,"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=\/wp\/v2\/posts\/704\/revisions"}],"predecessor-version":[{"id":710,"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=\/wp\/v2\/posts\/704\/revisions\/710"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=\/wp\/v2\/media\/706"}],"wp:attachment":[{"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=704"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=704"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/shijuvarghese.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=704"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}