spark-utils (homepage)

Basic framework utilities to quickly start writing production ready Apache Spark applications

@tupol / (0)

This project contains some basic utilities that can help setting up a Spark application project.

The SparkRunnable and SparkApp together with the configuration framework provide for easy Spark application creation with configuration that can be managed through configuration files or application parameters.

The IO frameworks for reading and writing data frames add extra convenience for setting up batch jobs that transform various types of files.

Last but not least, there are many utility functions that provide convenience for loading resources, dealing with schemas and so on.

Most of the common features are also implemented as decorators to main Spark classes, like SparkContext, DataFrame and StructType and they are conveniently available by importing the org.tupol.spark.implicits._ package.

The main utilities and frameworks available:

SparkApp & SparkRunnable
DataSource Framework
DataSink Framework

Tags

  • 1|library
  • 1|scala
  • 1|tools
  • 1|tools utilities framework quick-start
  • 1|utilities
  • 1|quick-start
  • 1|framework

How to

Include this package in your Spark Applications using:

spark-shell, pyspark, or spark-submit

> $SPARK_HOME/bin/spark-shell --packages org.tupol:spark-utils_2.11:0.4.2

sbt

In your sbt build file, add:

libraryDependencies += "org.tupol" % "spark-utils_2.11" % "0.4.2"

Maven

In your pom.xml, add: 
<dependencies>
  <!-- list of dependencies -->
  <dependency>
    <groupId>org.tupol</groupId>
    <artifactId>spark-utils_2.11</artifactId>
    <version>0.4.2</version>
  </dependency>
</dependencies>

Releases

Version: 0.6.1 ( 7d13e5 | zip ) / Date: 2021-10-18 / License: MIT

Version: 0.4.2-s_2.11 ( 5c786e | zip | jar ) / Date: 2020-09-12 / License: MIT / Scala version: 2.11

Version: 0.4.2-s_2.12 ( 5c786e | zip | jar ) / Date: 2020-09-12 / License: MIT / Scala version: 2.12

Version: 0.4.2 ( 5c786e | zip | jar ) / Date: 2020-09-12 / License: MIT / Scala version: 2.11

Version: 0.4.1 ( 38b926 | zip | jar ) / Date: 2019-08-28 / License: MIT / Scala version: 2.11

Version: 0.4.0 ( b493e7 | zip | jar ) / Date: 2019-05-08 / License: MIT / Scala version: 2.11

Version: 0.3.2 ( baaf95 | zip | jar ) / Date: 2019-05-06 / License: MIT / Scala version: 2.11

Version: 0.3.1 ( 642225 | zip | jar ) / Date: 2019-04-10 / License: MIT / Scala version: 2.11