Tag Archives: MapReduce

MapReduce instance – an error is reported when the map side join uses the addcachefile() method, and the file cannot be found

Solution: put the file that cannot be found into the root directory of the project

The reason is unclear.

But I found that one record in the log was abnormal, so I thought of putting the file in the root directory

2021-11-18 23:20:36,770 WARN  [main] mapred.LocalDistributedCacheManager (LocalDistributedCacheManager.java:symlink(201)) - Failed to create symlink: \tmp\hadoop-wuhao\mapred\local\job_local1290197105_0001_a1814cc0-1378-4a60-bfef-7c0838160efe\orders <- D:\IntelliJ IDEA 2020.3.1\IdeaProjects\MapreduceDemo/orders

 

Supplement:

In order to support file replication, Hadoop provides a class distributedcache. The methods of using this class are as follows:

(1) The user uses the static method distributedcache. Addcachefile() to specify the file to be copied. Its parameter is the URI of the file (if it is a file on HDFS, it can be as follows: hdfs://namenode:9000/home/XXX/file Where 9000 is the namenode port number configured by yourself. Jobtracker will get the URI list before the job starts and copy the corresponding files to the local disk of each tasktracker.

(2) The user uses the distributedcache. Getlocalcachefiles() method to obtain the file directory, and uses the standard file read-write API to read the corresponding files.

Similar: [MapReduce] reports an error: java.io.filenotfoundexception: \ user \ Mr \ input \ information.txt (the system cannot find the specified path.)_ Riding the snail chase missiles ~-csdn blog

IDEA Run mapreduce error: PATH Set Error [How to Solve]

Error Message:
[root@node01 servers]# hadoop jar loginVisit.jar cn.itcast.loginVisit.step1.Step1Main
19/07/17 22:14:59 INFO client.RMProxy: Connecting to ResourceManager at node01/192.168.8.100:8032
19/07/17 22:14:59 WARN security.UserGroupInformation: PriviledgedActionException as:root (auth:SIMPLE) cause:java.net.ConnectException: Call From node01.hadoop.com/192.168.8.100 to node02:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
Exception in thread “main” java.net.ConnectException: Call From node01.hadoop.com/192.168.8.100 to node02:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

 

 

Solution:
Step1Main.java, path set error.
TextOutputFormat.setOutputPath(job,new Path(“hdfs://node02:8020/loginVisit/output3”));
Change node02 to node01.

Mapreduce:Split metadata size exceeded 10000000

Error message:

Failure Info:Job initialization failed: java.io.IOException: Split metadata size exceeded 10000000.   
Aborting job job_201205162059_1073852 at   
org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(SplitMetaInfoReader.java:48) at   
org.apache.hadoop.mapred.JobInProgress.createSplits(JobInProgress.java:817) at   
org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:711) at   
org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:4028) at   
org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:79) at   
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at   
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662)

 

Error reason: the size of the job.splitmetainfo file of this job exceeds the limit

 

1. Job.splitmetadata info, which records metadata information of split, job split —– > HDFS block && amp; slave node

  The storage path is located at ${Hadoop. TMP. Dir}/mapred/staging/${user. Name} /. Staging/jobid/

 

2. The parameter mapreduce.jobtracker.split.metainfo.maxsize controls the maximum size of the file. The default is 10000000 (10m)