spark-redshift (homepage)

Redshift Data Source for Apache Spark

@databricks / (3)

A library to load data into Spark SQL DataFrames from Amazon Redshift, and write them back to Redshift tables. Amazon S3 is used to efficiently transfer data in and out of Redshift, and JDBC is used to automatically trigger the appropriate COPY and UNLOAD commands on Redshift.

Tags

  • 2|sql
  • 2|redshift
  • 2|data source
  • 1|input

How to

Include this package in your Spark Applications using:

spark-shell, pyspark, or spark-submit

> $SPARK_HOME/bin/spark-shell --packages com.databricks:spark-redshift_2.10:3.0.0-preview1

sbt

If you use the sbt-spark-package plugin, in your sbt build file, add:

spDependencies += "databricks/spark-redshift:3.0.0-preview1"

Otherwise,

libraryDependencies += "com.databricks" % "spark-redshift_2.10" % "3.0.0-preview1"

Maven

In your pom.xml, add: 
<dependencies>
  <!-- list of dependencies -->
  <dependency>
    <groupId>com.databricks</groupId>
    <artifactId>spark-redshift_2.10</artifactId>
    <version>3.0.0-preview1</version>
  </dependency>
</dependencies>

Releases

Version: 3.0.0-preview1 ( a28832 | zip | jar ) / Date: 2016-11-01 / License: Apache-2.0 / Scala version: 2.10

Version: 1.1.0 ( 33fa62 | zip | jar ) / Date: 2016-08-21 / License: Apache-2.0 / Scala version: 2.10

Version: 2.0.1 ( f0fa36 | zip | jar ) / Date: 2016-08-20 / License: Apache-2.0 / Scala version: 2.10

Version: 2.0.0 ( 9b9479 | zip | jar ) / Date: 2016-08-02 / License: Apache-2.0 / Scala version: 2.10

Version: 1.0.0 ( 626270 | zip | jar ) / Date: 2016-07-11 / License: Apache-2.0 / Scala version: 2.10

Version: 0.6.0 ( 5b4c4d | zip | jar ) / Date: 2016-01-06 / License: Apache-2.0 / Scala version: 2.10

Spark Scala/Java API compatibility: - 44% , - 40% , - 10% , - 78% , - 35% , - 100%

Version: 0.5.2 ( 3e3274 | zip | jar ) / Date: 2015-10-23 / License: Apache-2.0 / Scala version: 2.10

Spark Scala/Java API compatibility: - 10% , - 78% , - 35% , - 100% , - 44% , - 40%

Version: 0.5.1 ( f4a636 | zip | jar ) / Date: 2015-10-05 / License: Apache-2.0 / Scala version: 2.10

Spark Scala/Java API compatibility: - 35% , - 100% , - 44% , - 40% , - 10% , - 78%

Version: 0.5.0 ( 7e1aca | zip | jar ) / Date: 2015-09-11 / License: Apache-2.0 / Scala version: 2.10

Spark Scala/Java API compatibility: - 100% , - 44% , - 40% , - 10% , - 78% , - 35%

Version: 0.4.0-hadoop2 ( f3afe9 | zip | jar ) / Date: 2015-05-20 / License: Apache-2.0 / Scala version: 2.10

Spark Scala/Java API compatibility: - 22% , - 100% , - 64% , - 70%

Version: 0.4.0-hadoop1 ( f3afe9 | zip | jar ) / Date: 2015-05-20 / License: Apache-2.0 / Scala version: 2.10

Spark Scala/Java API compatibility: - 70% , - 22% , - 100% , - 64%

Version: 0.3 ( ce6ff8 | zip | jar ) / Date: 2015-03-31 / License: Apache-2.0 / Scala version: 2.10

Spark Scala/Java API compatibility: - 22% , - 100% , - 64% , - 70%