In the standard distribution, Spark includes RDD join operators that must collect all values across all rows on both sides of the join for any given key into an in-memory buffer before it can output the joined values for that key. Because of this, you might experience scalability issues when joining large datasets or datasets that exhibit a low-cardinality or skewed distribution of values.  This library solves these issues by spilling to disk (only) when necessary to avoid OOM yet still performs very comparable to the standard join operators in situations were no spilling is necessary.

For example, this will fail (assuming -Xmx512M) because the standard join operators collect all values for a particular key in memory and you'll get an OutOfMemoryError:

val rdd1 = sc.parallelize(0 to 10000000).map(v => "key1" -> v)
val rdd2 = sc.parallelize(0 to 10000000).map(v => "key1" -> v)
rdd1.join(rdd2).take(1)

Here is the same join using the "mergeJoin" operator provided by this package (there's also leftOuterJoin, rightOuterJoin, fullOuterJoin):

val rdd1 = sc.parallelize(0 to 10000000).map(v => "key1" -> v)
val rdd2 = sc.parallelize(0 to 10000000).map(v => "key1" -> v)
rdd1.mergeJoin(rdd2).take(1)

This will succeed because it will spill to disk when necessary, but will only do so when necessary.  Because of this, performance should be comparable to the standard join operators, except in the spill case, you'll still get a successful join rather than OutOfMemoryError. 

See the homepage link for more detailed information.