GraphXwithGPU (homepage)

Spark-based graph processing system demo using GPU

@Kamosphere / (3)

This project is a demo of spark-based graph processing system.
The purpose of this project is to fully utilize the computing ability of emerging hardware and to accelerate heavy computation tasks such graph processing for distributed system, as well as to achieve well-behaved workload balancing and task scheduling for GPU processing kernel by using managers built in distributed system.


In order to running demo,  extra C++ library is required at https://github.com/thoh-testarossa/Graph_Algo,
and please use spark 2.4.8-USTC-NODB version https://github.com/Kamosphere/spark-GPUGraphX

The project is developed by the following members:
@thoh-testarossa Thoh Testarossa
@Kamosphere Kamosphere
@cave-g-f cave-g-f
Thanks Prof.Xike Xie for the guidance.

Tags

  • 1|graph
  • 1|GPU
  • 1|CUDA

How to

Include this package in your Spark Applications using:

spark-shell, pyspark, or spark-submit

> $SPARK_HOME/bin/spark-shell --packages Kamosphere:GraphXwithGPU:1.1

sbt

If you use the sbt-spark-package plugin, in your sbt build file, add:

spDependencies += "Kamosphere/GraphXwithGPU:1.1"

Otherwise,

resolvers += "Spark Packages Repo" at "https://repos.spark-packages.org/"

libraryDependencies += "Kamosphere" % "GraphXwithGPU" % "1.1"

Maven

In your pom.xml, add: 
<dependencies>
  <!-- list of dependencies -->
  <dependency>
    <groupId>Kamosphere</groupId>
    <artifactId>GraphXwithGPU</artifactId>
    <version>1.1</version>
  </dependency>
</dependencies>
<repositories>
  <!-- list of other repositories -->
  <repository>
    <id>SparkPackagesRepo</id>
    <url>https://repos.spark-packages.org/</url>
  </repository>
</repositories>

Releases

Version: 1.1 ( 6ade30 | zip | jar ) / Date: 2020-12-22 / License: Apache-2.0