spark-bigquery (homepage)

Google BigQuery support for Spark, Structured Streaming, SQL, and DataFrames with easy Databricks integration.

@samelamin / (1)

Easy integration with Databricks. 
Support to stream to and from BQ tables using structured streaming 
Create Time Partitioned Tables
Can update schemas per write
Read BigQuery tables and SELECT query results as DataFrames 
Write DataFrames to BigQuery tables.

Tags

  • 1|streaming
  • 1|sql
  • 1|core
  • 1|BigQuery

How to

Include this package in your Spark Applications using:

spark-shell, pyspark, or spark-submit

> $SPARK_HOME/bin/spark-shell --packages samelamin:spark-bigquery:0.2.5

sbt

If you use the sbt-spark-package plugin, in your sbt build file, add:

spDependencies += "samelamin/spark-bigquery:0.2.5"

Otherwise,

resolvers += "Spark Packages Repo" at "https://repos.spark-packages.org/"

libraryDependencies += "samelamin" % "spark-bigquery" % "0.2.5"

Maven

In your pom.xml, add: 
<dependencies>
  <!-- list of dependencies -->
  <dependency>
    <groupId>samelamin</groupId>
    <artifactId>spark-bigquery</artifactId>
    <version>0.2.5</version>
  </dependency>
</dependencies>
<repositories>
  <!-- list of other repositories -->
  <repository>
    <id>SparkPackagesRepo</id>
    <url>https://repos.spark-packages.org/</url>
  </repository>
</repositories>

Releases

Version: 0.2.5 ( 623190 | zip | jar ) / Date: 2018-08-08 / License: Apache-2.0 / Scala version: 2.11

Version: 0.2.4 ( d45c73 | zip | jar ) / Date: 2018-03-14 / License: Apache-2.0 / Scala version: 2.11

Version: 0.2.3 ( 18af7e | zip | jar ) / Date: 2017-11-25 / License: Apache-2.0 / Scala version: 2.11

Version: 0.2.2 ( ed716c | zip | jar ) / Date: 2017-08-13 / License: Apache-2.0 / Scala version: 2.11

Version: 0.2.1 ( e02833 | zip | jar ) / Date: 2017-07-28 / License: Apache-2.0 / Scala version: 2.11

Version: 0.2.0 ( 76a95c | zip | jar ) / Date: 2017-07-27 / License: Apache-2.0 / Scala version: 2.11

Version: 0.1.9 ( 184215 | zip | jar ) / Date: 2017-07-16 / License: Apache-2.0 / Scala version: 2.11

Version: 0.1.8 ( 5c93d4 | zip | jar ) / Date: 2017-07-07 / License: Apache-2.0 / Scala version: 2.11

Version: 0.1.7 ( ded6f3 | zip | jar ) / Date: 2017-06-22 / License: Apache-2.0 / Scala version: 2.11

Version: 0.1.6 ( e615a7 | zip | jar ) / Date: 2017-05-10 / License: Apache-2.0 / Scala version: 2.11

Version: 0.1.5 ( bb1873 | zip | jar ) / Date: 2017-04-24 / License: Apache-2.0 / Scala version: 2.11

Version: 0.1.4 ( df3431 | zip | jar ) / Date: 2017-03-23 / License: Apache-2.0 / Scala version: 2.11

Version: 0.1.3 ( 101add | zip | jar ) / Date: 2017-02-23 / License: Apache-2.0 / Scala version: 2.11

Version: 0.1.2 ( b6ff20 | zip | jar ) / Date: 2017-02-15 / License: Apache-2.0 / Scala version: 2.11