sparklens (homepage)

Qubole Sparklens tool for performance tuning Apache Spark

@qubole / (2)

Sparklens is a profiling tool for Spark with built-in Spark Scheduler simulator. Its primary goal is to make it easy to understand the scalability limits of spark applications. It helps in understanding how efficiently is a given spark application using the compute resources provided to it. May be your application will run faster with more executors and may be it wont. Sparklens can answer this question by looking at a single run of your application.
It helps you narrow down to few stages (or driver, or skew or lack of tasks) which are limiting your application from scaling out and provides contextual information about what could be going wrong with these stages. Primarily it helps you approach spark application tuning as a well defined method/process instead of something you learn by trial and error, saving both developer and compute time.

Tags (No tags yet, login to add one. )

How to

Include this package in your Spark Applications using:

spark-shell, pyspark, or spark-submit

> $SPARK_HOME/bin/spark-shell --packages qubole:sparklens:0.3.2-s_2.11

sbt

If you use the sbt-spark-package plugin, in your sbt build file, add:

spDependencies += "qubole/sparklens:0.3.2-s_2.11"

Otherwise,

resolvers += "Spark Packages Repo" at "https://repos.spark-packages.org/"

libraryDependencies += "qubole" % "sparklens" % "0.3.2-s_2.11"

Maven

In your pom.xml, add: 
<dependencies>
  <!-- list of dependencies -->
  <dependency>
    <groupId>qubole</groupId>
    <artifactId>sparklens</artifactId>
    <version>0.3.2-s_2.11</version>
  </dependency>
</dependencies>
<repositories>
  <!-- list of other repositories -->
  <repository>
    <id>SparkPackagesRepo</id>
    <url>https://repos.spark-packages.org/</url>
  </repository>
</repositories>

Releases

Version: 0.3.2-s_2.11 ( 61c0bd | zip | jar ) / Date: 2020-05-04 / License: Apache-2.0 / Scala version: 2.11

Version: 0.3.1-s_2.11 ( e15a34 | zip | jar ) / Date: 2019-10-14 / License: Apache-2.0 / Scala version: 2.11

Version: 0.3.1-s_2.10 ( e15a34 | zip | jar ) / Date: 2019-10-14 / License: Apache-2.0 / Scala version: 2.10

Version: 0.3.0-s_2.11 ( f3fac1 | zip | jar ) / Date: 2019-05-09 / License: Apache-2.0 / Scala version: 2.11

Version: 0.2.2-s_2.11 ( db7046 | zip | jar ) / Date: 2019-05-09 / License: Apache-2.0 / Scala version: 2.11

Version: 0.2.1-s_2.10 ( 312b4d | zip | jar ) / Date: 2019-01-11 / License: Apache-2.0 / Scala version: 2.10

Version: 0.2.1-s_2.11 ( 312b4d | zip | jar ) / Date: 2019-01-10 / License: Apache-2.0 / Scala version: 2.11

Version: 0.2.0-s_2.11 ( 5f7ba5 | zip | jar ) / Date: 2018-08-07 / License: Apache-2.0 / Scala version: 2.11

Version: 0.2.0-s_2.10 ( 5f7ba5 | zip | jar ) / Date: 2018-08-07 / License: Apache-2.0 / Scala version: 2.10

Version: 0.1.2-s_2.11 ( 45e19c | zip | jar ) / Date: 2018-04-06 / License: Apache-2.0 / Scala version: 2.11

Version: 0.1.2-s_2.10 ( 45e19c | zip | jar ) / Date: 2018-04-06 / License: Apache-2.0 / Scala version: 2.10

Version: 0.1.1-s_2.11 ( 792313 | zip | jar ) / Date: 2018-04-06 / License: Apache-2.0 / Scala version: 2.11

Version: 0.1.1-s_2.10 ( 792313 | zip | jar ) / Date: 2018-04-06 / License: Apache-2.0 / Scala version: 2.10