Receiver Based Reliable Low Level Kafka-Spark Consumer for Spark Streaming . Built-in Back-Pressure Controller . ZK Based offset Management . WAL Less Recovery . Custom Message Interceptor
@dibbhatt / (6)
This utility will help to pull messages from Kafka using Spark Streaming and have better handling of the Kafka Offsets and failures. This Consumer have implemented a Custom Reliable Receiver which uses low level Kafka Consumer API to fetch messages from Kafka and store every received block in Spark BlockManager. Receiver commit the Kafka offsets of Received blocks once store to Spark BlockManager is successful.
Low Level Consumer API is compatible across Kafka versions 0.8, 0.9 and 0.10 and All Spark Versions
Salient Features of Kafka Spark Consumer
-- This Consumer uses Zookeeper for storing the consumed and processed offset for each Kafka partition, which will help to recover in case of failure
-- Spark streaming job using this Consumer does not require WAL for recovery from Driver or Executor failures. Consumer has capability to store the processed offset after every Batch interval, in case of any failure, Consumer can start from last Processed offset.
-- This Consumer has implemented PID (Proportional , Integral , Derivative ) based Rate Controller for controlling Back-Pressure by altering the Spark Block Size
-- This consumer have capability to use Message Interceptor which can be used to preprocess kafka messages before writing to Spark Block Manager
For more details about how to use this consumer , please refer to https://github.com/dibbhatt/kafka-spark-consumer/blob/master/README.md
Include this package in your Spark Applications using:
spark-shell, pyspark, or spark-submit
> $SPARK_HOME/bin/spark-shell --packages dibbhatt:kafka-spark-consumer:1.0.8
If you use the sbt-spark-package plugin, in your sbt build file, add:
spDependencies += "dibbhatt/kafka-spark-consumer:1.0.8"
resolvers += "Spark Packages Repo" at "http://dl.bintray.com/spark-packages/maven" libraryDependencies += "dibbhatt" % "kafka-spark-consumer" % "1.0.8"
MavenIn your pom.xml, add:
<dependencies> <!-- list of dependencies --> <dependency> <groupId>dibbhatt</groupId> <artifactId>kafka-spark-consumer</artifactId> <version>1.0.8</version> </dependency> </dependencies> <repositories> <!-- list of other repositories --> <repository> <id>SparkPackagesRepo</id> <url>http://dl.bintray.com/spark-packages/maven</url> </repository> </repositories>