中文
注册
我要评分
文档获取效率
文档正确性
内容完整性
文档易理解
在线提单
论坛求助

获取图分析算法加速库开发依赖JAR包

  • 构建图分析算法加速库适配代码Spark-graph-algo-lib过程如下。此过程以适配Spark 3.3.1代码的构建为例。
  • 以下操作请在Linux环境下操作,该章节仅供参考。

构建适配代码

图分析算法加速库开源仓的代码是基于Spark 3.3.1和图算法开源patch组成,构建过程:

  1. 下载Spark 3.3.1源码zip包到“/opt/”目录并解压,得到Spark源码目录“/opt/spark-3.3.1”
    1
    2
    wget https://github.com/apache/spark/archive/v3.3.1.zip
    unzip spark-3.3.1.zip
    
  2. “/opt/”目录下建立工程Spark-graph-algo-lib及相关目录。
    1
    2
    3
    cd /opt/
    mkdir -p Spark-graph-algo-lib/graph-accelerator/src/main/scala/org/apache/spark/graphx
    mkdir -p Spark-graph-algo-lib/graph-core/src/main/scala/spark/mllib/feature
    
  3. 按照表1的对应关系将Spark 3.3.1中的对应原文件复制到Spark-graph-algo-lib目录,表格左边两列是目标目录和文件名,右边两列的是需要移动的原文件目录及文件名,示例如下。
    操作命令示例:
    1
    2
    3
    4
    cp /opt/spark-3.3.1/graphx/src/main/scala/org/apache/spark/graphx/GraphOps.scala /opt/
    Spark-graph-algo-lib/graph-accelerator/src/main/scala/org/apache/spark/graphx
    cp /opt/spark-3.3.1/mllib/src/main/scala/org/apache/spark/mllib/feature/Word2Vec.scala /opt/
    Spark-graph-algo-lib/graph-core/src/main/scala/spark/mllib/feature
    
    表1 Spark中需要放入Spark-graph-algo-lib工程的文件

    Spark-graph-algo-lib工程目录

    Spark-graph-algo-lib工程文件名

    Spark原文件所在目录

    Spark原文件名

    Spark-graph-algo-lib/graph-accelerator/src/main/scala/org/apache/spark/graphx

    GraphOps.scala

    spark-3.3.1/graphx/src/main/scala/org/apache/spark/graphx

    GraphOps.scala

    Spark-graph-algo-lib/graph-core/src/main/scala/spark/mllib/feature

    Word2Vec.scala

    spark-3.3.1/mllib/src/main/scala/org/apache/spark/mllib/feature

    Word2Vec.scala

  4. 下载并解压patch到“/opt/”目录下,得到完整的图分析算法加速库适配代码Spark-graph-algo-lib。
    1
    2
    3
    4
    5
    cd /opt/
    wget https://github.com/kunpengcompute/Spark-graph-algo-lib/releases/download/v3.0.0-spark3.3.1/Spark-graph-algo-lib-Spark3.3.1.patch
    mv Spark-graph-algo-lib-Spark3.3.1.patch Spark-graph-algo-lib
    cd Spark-graph-algo-lib
    patch -p1 < Spark-graph-algo-lib-Spark3.3.1.patch