drunken-data-quality (homepage)

Some utility classes for checking data quality in Spark

@FRosner / (1)

DDQ is a small library for checking constraints on Spark data structures. It can be used to assure a certain data quality, especially when continuous imports happen.

Tags

  • 1|data quality
  • 1|data frames

How to

Include this package in your Spark Applications using:

spark-shell, pyspark, or spark-submit

> $SPARK_HOME/bin/spark-shell --packages FRosner:drunken-data-quality:5.0.0-s_2.11

sbt

If you use the sbt-spark-package plugin, in your sbt build file, add:

spDependencies += "FRosner/drunken-data-quality:5.0.0-s_2.11"

Otherwise,

resolvers += "Spark Packages Repo" at "https://repos.spark-packages.org/"

libraryDependencies += "FRosner" % "drunken-data-quality" % "5.0.0-s_2.11"

Maven

In your pom.xml, add: 
<dependencies>
  <!-- list of dependencies -->
  <dependency>
    <groupId>FRosner</groupId>
    <artifactId>drunken-data-quality</artifactId>
    <version>5.0.0-s_2.11</version>
  </dependency>
</dependencies>
<repositories>
  <!-- list of other repositories -->
  <repository>
    <id>SparkPackagesRepo</id>
    <url>https://repos.spark-packages.org/</url>
  </repository>
</repositories>

Releases

Version: 5.0.0-s_2.11 ( 0cb73a | zip | jar ) / Date: 2020-03-21 / License: Apache-2.0 / Scala version: 2.11

Version: 4.1.1-s_2.11 ( b79565 | zip | jar ) / Date: 2017-04-12 / License: Apache-2.0 / Scala version: 2.11

Version: 4.1.1-s_2.10 ( b79565 | zip | jar ) / Date: 2017-04-12 / License: Apache-2.0 / Scala version: 2.10

Version: 4.1.0-s_2.11 ( bd019e | zip | jar ) / Date: 2017-01-31 / License: Apache-2.0 / Scala version: 2.11

Version: 4.1.0-s_2.10 ( bd019e | zip | jar ) / Date: 2017-01-31 / License: Apache-2.0 / Scala version: 2.10

Version: 4.0.0-s_2.11 ( 84f184 | zip | jar ) / Date: 2017-01-24 / License: Apache-2.0 / Scala version: 2.11

Version: 4.0.0-s_2.10 ( 84f184 | zip | jar ) / Date: 2017-01-24 / License: Apache-2.0 / Scala version: 2.10

Version: 3.2.1-s_2.11 ( adc54a | zip | jar ) / Date: 2016-08-16 / License: Apache-2.0 / Scala version: 2.11

Version: 3.2.1-s_2.10 ( adc54a | zip | jar ) / Date: 2016-08-16 / License: Apache-2.0 / Scala version: 2.10

Version: 3.2.0-s_2.11 ( d29abc | zip | jar ) / Date: 2016-08-03 / License: Apache-2.0 / Scala version: 2.11

Version: 3.2.0-s_2.10 ( d29abc | zip | jar ) / Date: 2016-08-03 / License: Apache-2.0 / Scala version: 2.10

Version: 3.1.0-s_2.11 ( b5948e | zip | jar ) / Date: 2016-04-06 / License: Apache-2.0 / Scala version: 2.11

Version: 3.1.0-s_2.10 ( b5948e | zip | jar ) / Date: 2016-04-06 / License: Apache-2.0 / Scala version: 2.10

Version: 3.0.0-s_2.11 ( 729efd | zip | jar ) / Date: 2016-03-24 / License: Apache-2.0 / Scala version: 2.11

Version: 3.0.0-s_2.10 ( 729efd | zip | jar ) / Date: 2016-03-24 / License: Apache-2.0 / Scala version: 2.10

Version: 2.1.0-s_2.11 ( 7406fb | zip | jar ) / Date: 2016-02-15 / License: Apache-2.0 / Scala version: 2.11

Version: 2.1.0-s_2.10 ( 7406fb | zip | jar ) / Date: 2016-02-15 / License: Apache-2.0 / Scala version: 2.10