在mapper处理阶段中有时候我们需要获取当前正在处理的HDFS文件名/HDFS目录名,其实我们可以通过 Context 来获取相关参数,代码类似如下:
[crayon-675a74bf58fbb6[……]
标签存档: Hadoop
Hadoop在Map阶段获取当前split的文件名
hadoop多硬盘配置注意点
Hadoop修改配置PID文件路径
[一]、 概述
Hadoop启动后的PID文件默认配置是保存在 /tmp
目录下的,而linux下 /tmp 目录会定时清理,所以在集群运行一段时间后如果在停Hadoop相关服务是会出现类似:n[......]
Hadoop安装配置snappy压缩
本文的主要详细介绍hadoop2.x中snappy压缩算法安装配置的详细步骤。
[一]、 实验环境
- CentOS 6.3 64位
- Hadoop 2.6.0
- JDK 1.7.0_[……]
Hadoop2分布式及NN和RM实现HA的实验
Hadoop2.x在Ubuntu系统中编译源码
本文主要记录Hadoop2.x在Ubuntu 12.04下编译源码的主要步骤和注意点:
[一]、基础环境
- Ubuntu 12.04 LTS 64 位
- Hadoop2.2.0
- J[……]
eclipse中开发Hadoop2.x的Map/Reduce项目
java.net.ConnectException: to 0.0.0.0:10020 failed on connection exception
在DataNode节点中的Hive CLI中执行 select count(*) from table_name 查询时报错:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
java.io.IOException: java.net.ConnectException: Call From Slave7.Hadoop/192.168.8.207 to 0.0.0.0:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:331) at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:416) at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:522) at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:183) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:580) at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:578) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:416) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:578) at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:596) at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:288) at org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:547) at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:426) at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:136) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1472) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1239) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1057) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:884) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:874) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) |
从[……]
Sqoop安装配置及演示
Sqoop是一个用来将Hadoop(Hive、HBase)和关系型数据库中的数据相互转移的工具,可以将一个关系型数据库(例如:MySQL ,Oracle ,Postgres等)中的数据导入到Hadoo[……]
hadoop 2.2.0 集群模式安装配置和测试
本文详细记录Hadoop 2.2.0 集群安装配置的步骤,并运行演示一个简单的job。基本目录结构如下:
- 环境准备
- Hadoop安装配置
- 启动及演示
[一]、环境准备
本文所有集群节点的操[……]
近期评论