Fast Data Architectures for Streaming Applications by Dean Wampler

Author:Dean Wampler
Language: eng
Format: epub, mobi
Publisher: O'Reilly Media, Inc.
Published: 2016-10-14T04:00:00+00:00

Why Kafka?

Kafka’s current popularity is because it is ideally suited as the backbone of fast data architectures. It combines the benefits of event logs as the fundamental abstraction for streaming with the benefits of message queues. The Kafka documentation describes it as “a distributed, partitioned, replicated commit log service.” Note the emphasis on logging, which is why Kafka doesn’t delete messages once they’ve been read. Instead, multiple consumers can see the whole log and process it as they see fit (or even reprocess the log when an analysis task fails). The quote also hints that Kafka topics are partitioned for greater scalability and the partitions can be replicated across the cluster for greater durability and resiliency.

Kafka has also benefited from years of production use and development at LinkedIn, where it started. A year ago, LinkedIn’s Kafka infrastructure surpassed 1.1 trillion messages a day, and it’s still growing.

With the Kafka backbone and persistence options like distributed filesystems and databases, the third key element is the processing engine. For the last several years, Kafka, Cassandra, and Spark Streaming have been a very popular combination for streaming implementations.4 However, our thinking about stream processing semantics is evolving, too, which has fueled the emergence of Spark competitors.

Download

Copyright Disclaimer:
This site does not store any files on its server. We only index and link to content provided by other sites. Please contact the content providers to delete copyright contents if any and email us, we'll remove relevant links or contents immediately.