spark-xml-utils (homepage)

Spark-xml-utils provides the ability to filter documents based on an xpath expression, return specific nodes for an xpath/xquery expression, or transform documents using a xslt stylesheet.

The spark-xml-utils library was developed because there is a large amount of xml in some big datasets and I felt this data could be better served by providing some helpful xml utilities. This includes the ability to filter documents based on an xpath expression, return specific nodes for an xpath/xquery expression, or transform documents using a xslt stylesheet. By providing some basic wrappers to Saxon, the spark-xml-utils library exposes some basic XPath, XSLT, and XQuery functionality that can readily be leveraged by any Spark application.

Spark-xml-utils is not meant for processing one large single GBs xml record. However, if you have many xml records (we have millions)in the MBs (or less) then this should be a handy tool.

Tags

  • 1|xml
  • 1|xpath
  • 1|xquery
  • 1|xslt

How to

Include this package in your Spark Applications using:

spark-shell, pyspark, or spark-submit

> $SPARK_HOME/bin/spark-shell --packages elsevierlabs-os:spark-xml-utils:1.10.0

sbt

If you use the sbt-spark-package plugin, in your sbt build file, add:

spDependencies += "elsevierlabs-os/spark-xml-utils:1.10.0"

Otherwise,

resolvers += "Spark Packages Repo" at "https://repos.spark-packages.org/"

libraryDependencies += "elsevierlabs-os" % "spark-xml-utils" % "1.10.0"

Maven

In your pom.xml, add: 
<dependencies>
  <!-- list of dependencies -->
  <dependency>
    <groupId>elsevierlabs-os</groupId>
    <artifactId>spark-xml-utils</artifactId>
    <version>1.10.0</version>
  </dependency>
</dependencies>
<repositories>
  <!-- list of other repositories -->
  <repository>
    <id>SparkPackagesRepo</id>
    <url>https://repos.spark-packages.org/</url>
  </repository>
</repositories>

Releases

Version: 1.10.0 ( bd24eb | zip | jar ) / Date: 2021-12-08 / License: Apache-2.0

Version: 1.9.0 ( e9ee74 | zip | jar ) / Date: 2021-05-04 / License: Apache-2.0

Version: 1.8.0 ( 079029 | zip | jar ) / Date: 2018-03-07 / License: Apache-2.0

Version: 1.7.0 ( 980394 | zip | jar ) / Date: 2017-12-15 / License: Apache-2.0

Version: 1.6.0 ( 1bd2f1 | zip | jar ) / Date: 2017-04-20 / License: Apache-2.0

Version: 1.5.0 ( 5b2ef9 | zip | jar ) / Date: 2017-03-09 / License: Apache-2.0

Version: 1.4.0 ( 777824 | zip | jar ) / Date: 2017-02-06 / License: Apache-2.0

Version: 1.3.0 ( bd2a87 | zip | jar ) / Date: 2015-12-04 / License: Apache-2.0

Version: 1.2.0 ( 40ff1d | zip | jar ) / Date: 2015-09-28 / License: Apache-2.0