Error background
After installing Tez,
Execute under hive-shell: select count(*) from student;
report an error.
error phenomenon
22 / 03 / 03 22 : 34 : 48 INFO client.DAGClientImpl: DAG: State: FAILED Progress: 0 % TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0 Killed: 0
22 / 03 / 03 22 : 34 : 48 INFO client.DAGClientImpl: DAG completed. FinalState= FAILED
22 / 03 / 03 22 : 34 : 48Examples.OrderedWordCount the INFO: the DAG Diagnostics: [Vertex failed, the Tokenizer vertexName =, = vertex_1646363809353_0002_1_00 vertexId, Diagnostics = [Vertex vertex_1646363809353_0002_1_00 [the Tokenizer] killed / failed Due to: ROOT_INPUT_INIT_FAILURE, Vertex the Input: initializer of the Input failed, Vertex = vertex_1646363809353_0002_1_00 [the Tokenizer], java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java: 135 )
at org.apache.hadoop.io.compress.CompressionCodecFactory. <init>(CompressionCodecFactory.java: 175 )
at org.apache.hadoop.mapreduce.lib.input.TextInputFormat.isSplitable(TextInputFormat.java: 58 )
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits( FileInputFormat.java:399 )
at org.apache.hadoop.mapreduce.split.TezGroupedSplitsInputFormat.getSplits ( TezGroupedSplitsInputFormat.java : 97 )
at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateNewSplits(MRInputHelpers.java: 448 )
at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplitsToMem(MRInputHelpers.java: 329 )
at org.apache.tez.mapreduce.common.MRInputAMSplitGenerator.initialize(MRInputAMSplitGenerator.java: 122 )
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$ 1.run ( RootInputInitializerManager.java:278 )
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$ 1.run ( RootInputInitializerManager.java:269 )
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java: 422 )
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java: 1924 )
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java: 269 )
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java: 253 )
at java.util.concurrent.FutureTask.run(FutureTask.java: 266 )
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: 1149 )
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java: 624 )
at java.lang.Thread.run(Thread.java: 748 )
Caused by: java.lang.ClassNotFoundException: Class com.hadoop.compression.lzo.LzoCodec not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java: 2255 )
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java: 128 )
... 18 more
], Vertex killed, vertexName =Summation, vertexId=vertex_1646363809353_0002_1_01, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1646363809353_0002_1_01 [Summation] killed/failed due to:OTHER_VERTEX_FAILURE], Vertex killed, vertexName=Sorter, vertex vertexId=vertex_1646363809353_0002_1_02, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1646363809353_0002_1_02 [Sorter] killed/failed due to:OTHER_VERTEX_FAILURE], DAG did not succeed due to VERTEX_FAILURE. failedVertices: 1 killedVertices: 2 ]
Reason for error
Lzo related dependencies not found.
Error solution
Lzo will put its dependencies into hadoop during installation, and only need to load hadoop’s dependencies into tez.
Modify tez-site.xml and add the following configuration:
< property >
< name > tez.use.cluster.hadoop-libs </ name >
< value > true </ value >
</ property >