Hadoop2.2.0源码编译

基本环境参数:

  • Mac OSX 10.9.1
  • Maven 3.0.4
  • Java 1.6.0_65
  • Hadoop 2.2.0

源码编译的基本步骤:

1、checkout 源码

ps: hadoop2.2.0的源码有个bug :修改 <源码跟目录>/hadoop-common-project/hadoop-auth/pom.xml中添加如下依赖:

2、安装 protoc

到 Protocol Buffers 的官网 https://code.google.com/p/protobuf/ 下载2.5.0的安装源文件进行安装:

安装完成后,执行 protoc –vresion 验证是否安装成功。

3. 安装cmake

4. 执行mvn 命令进行编译了

在Mac OSX系统下compile、package 都能成功,如果指定-Pdist,native 生成native lib 不成功,查阅有关官方介绍发现:Hadoop本地库只支持*nix平台,已经广泛使用在GNU/Linux平台上,但是不支持 Cygwin  和 Mac OS X 。当然有人已经给出了Mac OSX系统编译本地库的patch ,具体的方法可以参见文章:

Hadoop 2.x build native library on Mac os x (http://www.micmiu.com/bigdata/hadoop/hadoop-build-native-library-on-mac-os-x/ )

5.常见错误及解决方法

ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.0.0-SNAPSHOT:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: ‘protoc –version’ did not return a version -> [Help 1]

上面的错误原因是:没有安装protoc

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-common: An Ant BuildException has occured: Execute failed: java.io.IOException: Cannot run program “cmake” (in directory “/Users/micmiu/no_sync/opensource_code/hadoop/trunk/hadoop-common-project/hadoop-common/target/native”): error=2, No such file or directory
[ERROR] around Ant part …<exec dir=”/Users/micmiu/no_sync/opensource_code/hadoop/trunk/hadoop-common-project/hadoop-common/target/native” executable=”cmake” failonerror=”true”>… @ 4:154 in /Users/micmiu/no_sync/opensource_code/hadoop/trunk/hadoop-common-project/hadoop-common/target/antrun/build-main.xml

上面的错误原因是: 没有安装 cmake

ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-common: An Ant BuildException has occured: exec returned: 2
[ERROR] around Ant part …<exec dir=”/Users/micmiu/no_sync/opensource_code/hadoop/trunk/hadoop-common-project/hadoop-common/target/native” executable=”make” failonerror=”true”>… @ 7:153 in /Users/micmiu/no_sync/opensource_code/hadoop/trunk/hadoop-common-project/hadoop-common/target/antrun/build-main.xml
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-common: An Ant BuildException has occured: exec returned: 2
around Ant part …<exec dir=”/Users/micmiu/no_sync/opensource_code/hadoop/trunk/hadoop-common-project/hadoop-common/target/native” executable=”make” failonerror=”true”>… @ 7:153 in /Users/micmiu/no_sync/opensource_code/hadoop/trunk/hadoop-common-project/hadoop-common/target/antrun/build-main.xml
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:317)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:152)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:555)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:214)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:158)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: An Ant BuildException has occured: exec returned: 2
around Ant part …<exec dir=”/Users/micmiu/no_sync/opensource_code/hadoop/trunk/hadoop-common-project/hadoop-common/target/native” executable=”make” failonerror=”true”>… @ 7:153 in /Users/micmiu/no_sync/opensource_code/hadoop/trunk/hadoop-common-project/hadoop-common/target/antrun/build-main.xml
at org.apache.maven.plugin.antrun.AntRunMojo.execute(AntRunMojo.java:355)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:106)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
… 19 more
Caused by: /Users/micmiu/no_sync/opensource_code/hadoop/trunk/hadoop-common-project/hadoop-common/target/antrun/build-main.xml:7: exec returned: 2

在Mac OSX 系统下编译本地库会由提示上面的错误信息,具体解决办法看上面的 第 【4】小节的介绍。

6、参考:

—————–  EOF @Michael Sun —————–

原创文章,转载请注明: 转载自micmiu – 软件开发+生活点滴[ http://www.micmiu.com/ ]

本文链接地址: http://www.micmiu.com/bigdata/hadoop/hadoop-build-source-2-2-0/

发表评论?

2 条评论。

  1. 文章写的很好,找了好久才找到。我按照教程一步一步在mac上搭建hadoop环境。cmake 已经安装好,但是还是报了错

    • [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-common: An Ant BuildException has occured: exec returned: 1
      [ERROR] around Ant part …… @ 4:143 in /Users/Howie/Downloads/hadoop-2.5.0-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml

发表评论


注意 - 你可以用以下 HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">