Spark-based graph processing system demo using GPU
@Kamosphere / (3)
This project is a demo of spark-based graph processing system.
The purpose of this project is to fully utilize the computing ability of emerging hardware and to accelerate heavy computation tasks such graph processing for distributed system, as well as to achieve well-behaved workload balancing and task scheduling for GPU processing kernel by using managers built in distributed system.
In order to running demo, extra C++ library is required at https://github.com/thoh-testarossa/Graph_Algo,
and please use spark 2.4.8-USTC-NODB version https://github.com/Kamosphere/spark-GPUGraphX
The project is developed by the following members:
@thoh-testarossa Thoh Testarossa
Thanks Prof.Xike Xie for the guidance.
Include this package in your Spark Applications using:
spark-shell, pyspark, or spark-submit
> $SPARK_HOME/bin/spark-shell --packages Kamosphere:GraphXwithGPU:1.1
If you use the sbt-spark-package plugin, in your sbt build file, add:
spDependencies += "Kamosphere/GraphXwithGPU:1.1"
resolvers += "Spark Packages Repo" at "http://dl.bintray.com/spark-packages/maven" libraryDependencies += "Kamosphere" % "GraphXwithGPU" % "1.1"
MavenIn your pom.xml, add:
<dependencies> <!-- list of dependencies --> <dependency> <groupId>Kamosphere</groupId> <artifactId>GraphXwithGPU</artifactId> <version>1.1</version> </dependency> </dependencies> <repositories> <!-- list of other repositories --> <repository> <id>SparkPackagesRepo</id> <url>http://dl.bintray.com/spark-packages/maven</url> </repository> </repositories>