Error background
After installing Tez,
Execute under hive-shell: select count(*) from student;
report an error.
error phenomenon
22 / 03 / 03 22 : 34 : 48 INFO client.DAGClientImpl: DAG: State: FAILED Progress: 0 % TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0 Killed: 0 22 / 03 / 03 22 : 34 : 48 INFO client.DAGClientImpl: DAG completed. FinalState= FAILED 22 / 03 / 03 22 : 34 : 48Examples.OrderedWordCount the INFO: the DAG Diagnostics: [Vertex failed, the Tokenizer vertexName =, = vertex_1646363809353_0002_1_00 vertexId, Diagnostics = [Vertex vertex_1646363809353_0002_1_00 [the Tokenizer] killed / failed Due to: ROOT_INPUT_INIT_FAILURE, Vertex the Input: initializer of the Input failed, Vertex = vertex_1646363809353_0002_1_00 [the Tokenizer], java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found. at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java: 135 ) at org.apache.hadoop.io.compress.CompressionCodecFactory. <init>(CompressionCodecFactory.java: 175 ) at org.apache.hadoop.mapreduce.lib.input.TextInputFormat.isSplitable(TextInputFormat.java: 58 ) at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits( FileInputFormat.java:399 ) at org.apache.hadoop.mapreduce.split.TezGroupedSplitsInputFormat.getSplits ( TezGroupedSplitsInputFormat.java : 97 ) at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateNewSplits(MRInputHelpers.java: 448 ) at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplitsToMem(MRInputHelpers.java: 329 ) at org.apache.tez.mapreduce.common.MRInputAMSplitGenerator.initialize(MRInputAMSplitGenerator.java: 122 ) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$ 1.run ( RootInputInitializerManager.java:278 ) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$ 1.run ( RootInputInitializerManager.java:269 ) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java: 422 ) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java: 1924 ) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java: 269 ) at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java: 253 ) at java.util.concurrent.FutureTask.run(FutureTask.java: 266 ) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: 1149 ) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java: 624 ) at java.lang.Thread.run(Thread.java: 748 ) Caused by: java.lang.ClassNotFoundException: Class com.hadoop.compression.lzo.LzoCodec not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java: 2255 ) at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java: 128 ) ... 18 more ], Vertex killed, vertexName =Summation, vertexId=vertex_1646363809353_0002_1_01, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1646363809353_0002_1_01 [Summation] killed/failed due to:OTHER_VERTEX_FAILURE], Vertex killed, vertexName=Sorter, vertex vertexId=vertex_1646363809353_0002_1_02, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1646363809353_0002_1_02 [Sorter] killed/failed due to:OTHER_VERTEX_FAILURE], DAG did not succeed due to VERTEX_FAILURE. failedVertices: 1 killedVertices: 2 ]
Reason for error
Lzo related dependencies not found.
Error solution
Lzo will put its dependencies into hadoop during installation, and only need to load hadoop’s dependencies into tez.
Modify tez-site.xml and add the following configuration:
< property > < name > tez.use.cluster.hadoop-libs </ name > < value > true </ value > </ property >
Similar Posts:
- [Solved] Hadoop Error: Input path does not exist: hdfs://Master:9000/user/hadoop/input
- Mapreduce:Split metadata size exceeded 10000000
- [Solved] Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses
- [Solved] Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses
- [Solved] Hadoop running jar package error: xception in thread “main” java.lang.ClassNotFoundException: Filter
- [Solved] Hadoop running jar package error: java.lang.exception: java.lang.arrayindexoutofboundsexception: 1
- The key technologies in hadoop-3.0.0 configuration yarn.nodemanager.aux -Services item
- [Solved] wordcount Error: org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist:
- An error is reported when sqoop imports data into MySQL database: error tool. Exporttool: error during export: export job failed!
- Hive Error: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask