博客
关于我
强烈建议你试试无所不能的chatGPT，快点击我
Hadoop3.1.1运行自带例子wordcount发生的错误
阅读量：6413 次
发布时间：2019-06-23

本文共 10694 字，大约阅读时间需要 35 分钟。

　错误 1

Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMasterPlease check whether your etc/hadoop/mapred-site.xml contains the below configuration:
       
      
       yarn.app.mapreduce.am.env
        
      
       HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}
      
     
       
      
       mapreduce.map.env
        
      
       HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}
      
     
       
      
       mapreduce.reduce.env
        
      
       HADOOP_MAPRED_HOME=${full path of your hadoop distribution directory}

　解决方法

　　在配置文件中添加hadoop的根目录即可，如：

     
     
     
     
     
          
       
        mapreduce.framework.name
           
       
        yarn
       
      
        
       
        yarn.app.mapreduce.am.env
         
       
        HADOOP_MAPRED_HOME=/usr/local/hadoop/hadoop-3.1.1
       
      
        
       
        mapreduce.map.env
         
       
        HADOOP_MAPRED_HOME=/usr/local/hadoop/hadoop-3.1.1
       
      
        
       
        mapreduce.reduce.env
         
       
        HADOOP_MAPRED_HOME=/usr/local/hadoop/hadoop-3.1.1

　错误 2

2018-12-09 22:01:31,884 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:80322018-12-09 22:01:32,752 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1544410258618_00052018-12-09 22:01:33,056 INFO input.FileInputFormat: Total input files to process : 12018-12-09 22:01:33,621 INFO mapreduce.JobSubmitter: number of splits:12018-12-09 22:01:33,664 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled2018-12-09 22:01:34,306 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1544410258618_00052018-12-09 22:01:34,307 INFO mapreduce.JobSubmitter: Executing with tokens: []2018-12-09 22:01:34,595 INFO conf.Configuration: resource-types.xml not found2018-12-09 22:01:34,595 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.2018-12-09 22:01:34,715 INFO impl.YarnClientImpl: Submitted application application_1544410258618_00052018-12-09 22:01:34,772 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1544410258618_0005/2018-12-09 22:01:34,773 INFO mapreduce.Job: Running job: job_1544410258618_00052018-12-09 22:01:43,122 INFO mapreduce.Job: Job job_1544410258618_0005 running in uber mode : false2018-12-09 22:01:43,123 INFO mapreduce.Job: map 0% reduce 0%2018-12-09 22:01:57,528 INFO mapreduce.Job: Task Id : attempt_1544410258618_0005_m_000000_0, Status : FAILED[2018-12-09 22:01:54.252]Container [pid=10353,containerID=container_1544410258618_0005_01_000002] is running 523131392B beyond the 'VIRTUAL' memory limit. Current usage: 257.3 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container.Dump of the process-tree for container_1544410258618_0005_01_000002 :|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE|- 10353 10352 10353 10353 (bash) 0 0 115896320 173 /bin/bash -c /usr/local/java/jdk1.8.0_191/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1544410258618_0005/container_1544410258618_0005_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 38435 attempt_1544410258618_0005_m_000000_0 2 1>/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000002/stdout 2>/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000002/stderr|- 10364 10353 10353 10353 (java) 624 174 2662092800 65693 /usr/local/java/jdk1.8.0_191/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1544410258618_0005/container_1544410258618_0005_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 38435 attempt_1544410258618_0005_m_000000_0 2[2018-12-09 22:01:55.961]Container killed on request. Exit code is 143[2018-12-09 22:01:55.961]Container exited with a non-zero exit code 143.2018-12-09 22:02:07,613 INFO mapreduce.Job: Task Id : attempt_1544410258618_0005_m_000000_1, Status : FAILED[2018-12-09 22:02:06.339]Container [pid=10421,containerID=container_1544410258618_0005_01_000003] is running 521972224B beyond the 'VIRTUAL' memory limit. Current usage: 136.2 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container.Dump of the process-tree for container_1544410258618_0005_01_000003 :|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE|- 10430 10421 10421 10421 (java) 346 100 2660933632 34552 /usr/local/java/jdk1.8.0_191/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1544410258618_0005/container_1544410258618_0005_01_000003/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 38435 attempt_1544410258618_0005_m_000000_1 3|- 10421 10419 10421 10421 (bash) 0 0 115896320 306 /bin/bash -c /usr/local/java/jdk1.8.0_191/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1544410258618_0005/container_1544410258618_0005_01_000003/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 38435 attempt_1544410258618_0005_m_000000_1 3 1>/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000003/stdout 2>/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000003/stderr[2018-12-09 22:02:06.409]Container killed on request. Exit code is 143[2018-12-09 22:02:06.420]Container exited with a non-zero exit code 143.2018-12-09 22:02:19,533 INFO mapreduce.Job: map 100% reduce 0%2018-12-09 22:02:19,534 INFO mapreduce.Job: Task Id : attempt_1544410258618_0005_m_000000_2, Status : FAILED[2018-12-09 22:02:16.854]Container [pid=10477,containerID=container_1544410258618_0005_01_000004] is running 551004672B beyond the 'VIRTUAL' memory limit. Current usage: 272.7 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container.Dump of the process-tree for container_1544410258618_0005_01_000004 :|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE|- 10486 10477 10477 10477 (java) 412 63 2689966080 69764 /usr/local/java/jdk1.8.0_191/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1544410258618_0005/container_1544410258618_0005_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 38435 attempt_1544410258618_0005_m_000000_2 4|- 10477 10475 10477 10477 (bash) 0 0 115896320 50 /bin/bash -c /usr/local/java/jdk1.8.0_191/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/usr/local/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1544410258618_0005/container_1544410258618_0005_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 127.0.0.1 38435 attempt_1544410258618_0005_m_000000_2 4 1>/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000004/stdout 2>/usr/local/hadoop/hadoop-3.1.1/logs/userlogs/application_1544410258618_0005/container_1544410258618_0005_01_000004/stderr[2018-12-09 22:02:18.409]Container killed on request. Exit code is 143[2018-12-09 22:02:18.410]Container exited with a non-zero exit code 143.2018-12-09 22:02:20,541 INFO mapreduce.Job: map 0% reduce 0%2018-12-09 22:02:33,619 INFO mapreduce.Job: map 100% reduce 100%2018-12-09 22:02:35,639 INFO mapreduce.Job: Job job_1544410258618_0005 failed with state FAILED due to: Task failed task_1544410258618_0005_m_000000Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 02018-12-09 22:02:35,794 INFO mapreduce.Job: Counters: 13Job CountersFailed map tasks=4Killed reduce tasks=1Launched map tasks=4Other local map tasks=3Data-local map tasks=1Total time spent by all maps in occupied slots (ms)=40350Total time spent by all reduces in occupied slots (ms)=0Total time spent by all map tasks (ms)=40350Total vcore-milliseconds taken by all map tasks=40350Total megabyte-milliseconds taken by all map tasks=41318400Map-Reduce FrameworkCPU time spent (ms)=0Physical memory (bytes) snapshot=0Virtual memory (bytes) snapshot=0

　解决方法

　　添加内存

转载于:https://www.cnblogs.com/xiaoerjun/p/10100665.html

你可能感兴趣的文章
又一年...
查看>>
Linux VFS
查看>>
ext不能选中复制属性_如何实现Extjs的grid单元格只让选择（即可以复制单元格内容）但是不让修改？...
查看>>
python中print的作用*8、不能+8_在 Python 3.x 中语句 print(*[1,2,3]) 不能正确执行。 (1.0分)_学小易找答案...
查看>>
python 生成html代码_使用Python Markdown 生成 html
查看>>
axure如何导出原件_Axure 教程：轻松导出图标字体所有图标
查看>>
laravel input值必须不等于0_框架不提供，动手造一个：Laravel表单验证自定义用法...
查看>>
cad填充图案乱理石_太快了吧！原来大神是这样用CAD图案填充的
查看>>
activator.createinstance 需要垃圾回收么_在垃圾回收器中有哪几种判断是否需要被回收的方法...
查看>>
rocketmq 消息指定_RocketMQ入坑系列（一）角色介绍及基本使用
查看>>
redis zset转set 反序列化失败_掌握好Redis的数据类型，面试心里有底了
查看>>
p图软件pⅰc_娱乐圈最塑料的夫妻，P图永远只P自己，太精彩了吧！
查看>>
怎么判断冠词用a还是an_葡语干货 | 葡萄牙语冠词用法整理大全
查看>>
js传参不是数字_JS的Reflect学习和应用
查看>>
三个不等_数学一轮复习05，从函数观点看方程与不等式，记住口诀与联系
查看>>
卡尺测量的最小范围_汽车维修工具-测量用具
查看>>
网优5g前景_5G网络优化师前景怎么样？
查看>>
竞态条件的赋值_[译] part25: golang Mutex互斥锁
查看>>
delmatch oracle_完美完全卸载（清除）oracle数据库的方式（方法）
查看>>
pyqt 滚动条 美化_Pyqt5 关于流式布局和滚动条的综合使用示例代码
查看>>
喝酒易醉，品茶养心，人生如梦，品茶悟道，何以解忧？唯有杜康！-- 愿君每日到此一游！
当前时间: 2025-01-15 14:10:10   当前IP: 3.21.104.16   联系邮箱:javaeecc@qq.com     Copyright © 2020 - 2022 baihongyu.com 京ICP备2021015314号-2
强烈建议你试试无所不能的CHAT-GPT，快点击我
强烈建议你试试无所不能的CHAT-GPT，快点击我