datafu-spark contains a number of spark API's and a "Scala-Python bridge" that makes calling Scala code from Python, and vice-versa, easier.

Here are some examples of things you can do with it:

"Dedup" a table - remove duplicates based on a key and ordering (typically a date updated field, to get only the mostly recently updated record).

Join a table with a numeric field with a table with a range

Do a skewed join between tables (where the small table is still too big to fit in memory)

Count distinct up to - an efficient implementation when you just want to verify that a certain minimum of distinct rows appear in a table