Spark 源码除了用 sbt/sbt assembly 编译，也可用Maven进行编译，具体步骤如下：
1、配置Maven 的参数
|
1 |
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m" |
如果不配置Maven参数，编译时会看到类似如下的错误信息：
|
1 2 3 4 5 |
[INFO] Compiling 203 Scala sources and 9 Java sources to /Users/me/Development/spark/core/target/scala-2.10/classes... [ERROR] PermGen space -> [Help 1] [INFO] Compiling 203 Scala sources and 9 Java sources to /Users/me/Development/spark/core/target/scala-2.10/classes... [ERROR] Java heap space -> [Help 1] |
2、执行编译命令
对于Apache Hadoop versions 1.x, Cloudera CDH MRv1, and other Hadoop versions without YARN等版本用命令：
|
1 2 3 4 5 |
# Apache Hadoop 1.2.1 $ mvn -Dhadoop.version=1.2.1 -DskipTests clean package # Cloudera CDH 4.2.0 with MapReduce v1 $ mvn -Dhadoop.version=2.0.0-mr1-cdh4.2.0 -DskipTests clean package |
对于Apache Hadoop 2.x, 0.23.x, Cloudera CDH MRv2, and other Hadoop versions with YARN 等版本需要开启 “yarn-alpha” or “yarn” 属性配置，并且要指定 “hadoop.version”, “yarn.version” 的属性值：
|
1 2 3 4 5 6 7 8 |
# Apache Hadoop 2.0.5-alpha $ mvn -Pyarn-alpha -Dhadoop.version=2.0.5-alpha -Dyarn.version=2.0.5-alpha -DskipTests clean package # Cloudera CDH 4.2.0 with MapReduce v2 $ mvn -Pyarn-alpha -Dhadoop.version=2.0.0-cdh4.2.0 -Dyarn.version=2.0.0-chd4.2.0 -DskipTests clean package # Apache Hadoop 2.2.X ( e.g. 2.2.0 as below ) and newer $ mvn -Pyarn -Dhadoop.version=2.2.0 -Dyarn.version=2.2.0 -DskipTests clean package |
我们以Hadoop2.2.0为例,详细过程如下：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
micmiu-mbp:spark-0.9.1 micmiu$ export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m" micmiu-mbp:spark-0.9.1 micmiu$ mvn -Pyarn -Dhadoop.version=2.2.0 -Dyarn.version=2.2.0 -DskipTests clean package [INFO] Scanning for projects... ...... 省略若干内容 by micmiu.com [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Spark Project Parent POM .......................... SUCCESS [57.441s] [INFO] Spark Project Core ................................ SUCCESS [12:06.394s] [INFO] Spark Project Bagel ............................... SUCCESS [24.019s] [INFO] Spark Project GraphX .............................. SUCCESS [2:05.960s] [INFO] Spark Project ML Library .......................... SUCCESS [2:33.265s] [INFO] Spark Project Streaming ........................... SUCCESS [2:06.185s] [INFO] Spark Project Tools ............................... SUCCESS [14.450s] [INFO] Spark Project REPL ................................ SUCCESS [1:06.779s] [INFO] Spark Project YARN Parent POM ..................... SUCCESS [1.799s] [INFO] Spark Project YARN Stable API ..................... SUCCESS [38.236s] [INFO] Spark Project Assembly ............................ SUCCESS [1:06.514s] [INFO] Spark Project External Twitter .................... SUCCESS [22.277s] [INFO] Spark Project External Kafka ...................... SUCCESS [1:06.591s] [INFO] Spark Project External Flume ...................... SUCCESS [30.070s] [INFO] Spark Project External ZeroMQ ..................... SUCCESS [40.599s] [INFO] Spark Project External MQTT ....................... SUCCESS [2:07.303s] [INFO] Spark Project Examples ............................ SUCCESS [7:35.495s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 35:44.003s [INFO] Finished at: Mon Apr 14 17:16:23 CST 2014 [INFO] Final Memory: 67M/441M [INFO] ------------------------------------------------------------------------ micmiu-mbp:spark-0.9.1 micmiu$ |
参考：
https://spark.apache.org/docs/latest/building-with-maven.html
—————– EOF @Michael Sun —————–
原创文章，转载请注明： 转载自micmiu – 软件开发+生活点滴[ http://www.micmiu.com/ ]
本文链接地址: http://www.micmiu.com/bigdata/spark/spark-building-with-maven/
0 条评论。