kafka-spark-consumer (homepage)

High Performance Kafka Consumer for Spark Streaming.Supports Multi Topic Fetch, Kafka Security. Reliable offset management in Zookeeper. No Data-loss. No dependency on HDFS and WAL. In-built PID rate controller. Support Message Handler . Offset Lag checker

@dibbhatt / (7)

Version 2.1.0 : Spark verison 2.x and above. Kafka version 0.10 and above. Support for handling Multi Topic .
Version 2.0.0 : Spark verison 2.x and above. Kafka version 0.10 and above. Support for Kafka Security . Used New Kafka Consumer API
Version 1.0.9 : Spark Version earlier to 2.0 ( i.e. 1.6 and prior). All Kafka version ( 0.8.x, 0.9.x, 0.10.x, 0.11.x). No support for Kafka Security. Used Low Level SimpleConsumer API


Salient Feature of Kafka-Spark-Consumer
User latest Kafka Consumer API. Support for Kafka Security
Support for consuming from multiple topics
Zookeeper for storing the offset for each Kafka partition, which will help to recover in case of failure
Spark streaming job using this Consumer does not require WAL for recovery from Driver or Executor failures. As this consumer has capability to store the processed offset after every Batch interval, in case of any failure, Consumer can start from the correct offset from the last Processed offset.
This Consumer has implemented PID (Proportional , Integral , Derivative ) based Rate Controller for controlling Back-Pressure.
This consumer have capability to use Message Interceptor which can be used to preprocess kafka messages before writing to Spark Block Manager
Support for Consumer Lag Checker (like ConsumerOffsetChecker) tool to find Consumer Lag
 

Tags

  • 4|streaming
  • 3|kafka

How to

Include this package in your Spark Applications using:

spark-shell, pyspark, or spark-submit

> $SPARK_HOME/bin/spark-shell --packages dibbhatt:kafka-spark-consumer:2.1.0

sbt

If you use the sbt-spark-package plugin, in your sbt build file, add:

spDependencies += "dibbhatt/kafka-spark-consumer:2.1.0"

Otherwise,

resolvers += "Spark Packages Repo" at "https://repos.spark-packages.org/"

libraryDependencies += "dibbhatt" % "kafka-spark-consumer" % "2.1.0"

Maven

In your pom.xml, add: 
<dependencies>
  <!-- list of dependencies -->
  <dependency>
    <groupId>dibbhatt</groupId>
    <artifactId>kafka-spark-consumer</artifactId>
    <version>2.1.0</version>
  </dependency>
</dependencies>
<repositories>
  <!-- list of other repositories -->
  <repository>
    <id>SparkPackagesRepo</id>
    <url>https://repos.spark-packages.org/</url>
  </repository>
</repositories>

Releases

Version: 2.1.0 ( 154c36 | zip | jar ) / Date: 2019-08-28 / License: Apache-2.0 / Scala version: 2.11

Version: 2.0.0 ( 5c4ddd | zip | jar ) / Date: 2019-08-12 / License: Apache-2.0 / Scala version: 2.11

Version: 1.0.19 ( 78b273 | zip | jar ) / Date: 2019-08-05 / License: Apache-2.0 / Scala version: 2.11

Version: 1.0.18 ( 94b111 | zip | jar ) / Date: 2019-06-12 / License: Apache-2.0 / Scala version: 2.11

Version: 1.0.17 ( 3d4ea8 | zip | jar ) / Date: 2019-06-03 / License: Apache-2.0 / Scala version: 2.11

Version: 1.0.16 ( b69082 | zip | jar ) / Date: 2019-06-01 / License: Apache-2.0 / Scala version: 2.11

Version: 1.0.15 ( 88099d | zip | jar ) / Date: 2018-09-21 / License: Apache-2.0 / Scala version: 2.11

Version: 1.0.14 ( fdba55 | zip | jar ) / Date: 2017-12-03 / License: Apache-2.0 / Scala version: 2.11

Version: 1.0.13 ( 2688a2 | zip | jar ) / Date: 2017-11-30 / License: Apache-2.0 / Scala version: 2.11

Version: 1.0.12 ( 6284f8 | zip | jar ) / Date: 2017-07-27 / License: Apache-2.0 / Scala version: 2.11

Version: 1.0.11 ( 26d02b | zip | jar ) / Date: 2017-07-18 / License: Apache-2.0 / Scala version: 2.11

Version: 1.0.10 ( 978cf6 | zip | jar ) / Date: 2017-02-02 / License: Apache-2.0 / Scala version: 2.11

Version: 1.0.9 ( 117f98 | zip | jar ) / Date: 2017-01-25 / License: Apache-2.0 / Scala version: 2.10

Version: 1.0.8 ( 3028f6 | zip | jar ) / Date: 2016-08-25 / License: Apache-2.0 / Scala version: 2.10

Version: 1.0.7 ( e2dd9e | zip | jar ) / Date: 2016-08-25 / License: Apache-2.0 / Scala version: 2.10

Version: 1.0.6 ( 928fe9 | zip | jar ) / Date: 2016-01-07 / License: Apache-2.0 / Scala version: 2.10

Spark Scala/Java API compatibility: - 88% , - 55% , - 99% , - 81% , - 99% , - 54%

Version: 1.0.5 ( 3eeda6 | zip | jar ) / Date: 2015-10-08 / License: Apache-2.0 / Scala version: 2.10

Spark Scala/Java API compatibility: - 56% , - 100% , - 82% , - 100% , - 54% , - 88%

Version: 1.0.4 ( b256e7 | zip | jar ) / Date: 2015-08-26 / License: Apache-2.0 / Scala version: 2.10

Spark Scala/Java API compatibility: - 56% , - 100% , - 82% , - 54% , - 88%

Version: 1.0.3 ( aeaf15 | zip | jar ) / Date: 2015-06-10 / License: Apache-2.0 / Scala version: 2.10

Spark Scala/Java API compatibility: - 100% , - 64% , - 99% , - 66%

Version: 1.0.2 ( 95a4cf | zip | jar ) / Date: 2015-06-02 / License: Apache-2.0 / Scala version: 2.10

Spark Scala/Java API compatibility: - 66% , - 100% , - 64% , - 99%

Version: 1.0.1 ( cbc500 | zip | jar ) / Date: 2015-04-30 / License: Apache-2.0 / Scala version: 2.10

Spark Scala/Java API compatibility: - 100% , - 64% , - 99% , - 66%

Version: 1.0.0 ( fc82ea | zip | jar ) / Date: 2015-04-26 / License: Apache-2.0 / Scala version: 2.10

Spark Scala/Java API compatibility: - 64% , - 99% , - 66% , - 100%