spark-crossdata (homepage)

SparkSQL extension as a library for Apache Spark extending and improving its capabilities for a data federation system.

@Stratio / (6)

Crossdata is an extension of SparkSQL through the XDContext and acting as a data federation system. It also provides extended capabilities for some datasources such as Cassandra (Datastax), MongoDB (Stratio) and Elasticsearch (elastic). Moreover, it extends the SparkSQL language with custom SQL-like sentences to add metadata discovery operations and creation of external tables. These extensions also allow to resolve queries natively, that is, Crossdata goes directly to the persistence layer. This native access speed-ups some queries and avoids the usage of the Spark cluster resources, relieving the pressure of the memory when possible. In addition, Crossdata supports batch and streaming processing so that you can mix data from both incoming origins using, again, a SQL-like language. These extensions of SparkSQL also includes persistent metadata and logical views.

Tags

  • 3|sql
  • 3|SparkSQL
  • 2|library
  • 2|scala
  • 2|core
  • 1|streaming
  • 1|application

How to

Include this package in your Spark Applications using:

spark-shell, pyspark, or spark-submit

> $SPARK_HOME/bin/spark-shell --packages Stratio:spark-crossdata:1.4.0

sbt

If you use the sbt-spark-package plugin, in your sbt build file, add:

spDependencies += "Stratio/spark-crossdata:1.4.0"

Otherwise,

resolvers += "Spark Packages Repo" at "https://repos.spark-packages.org/"

libraryDependencies += "Stratio" % "spark-crossdata" % "1.4.0"

Maven

In your pom.xml, add: 
<dependencies>
  <!-- list of dependencies -->
  <dependency>
    <groupId>Stratio</groupId>
    <artifactId>spark-crossdata</artifactId>
    <version>1.4.0</version>
  </dependency>
</dependencies>
<repositories>
  <!-- list of other repositories -->
  <repository>
    <id>SparkPackagesRepo</id>
    <url>https://repos.spark-packages.org/</url>
  </repository>
</repositories>

Releases

Version: 1.4.0 ( 5e1aa1 | zip | jar ) / Date: 2016-07-06 / License: Apache-2.0 / Scala version: 2.10

Version: 1.3.0 ( 295fd8 | zip | jar ) / Date: 2016-06-28 / License: Apache-2.0 / Scala version: 2.10

Version: 1.2.2 ( 28a514 | zip | jar ) / Date: 2016-06-02 / License: Apache-2.0 / Scala version: 2.10

Version: 1.2.1 ( 2e7cc2 | zip | jar ) / Date: 2016-04-22 / License: Apache-2.0 / Scala version: 2.10