[Solved] Tez Compression codec com.hadoop.compression.lzo.LzoCodec not found.

Error background

After installing Tez,

Execute under hive-shell: select count(*) from student;

report an error.

error phenomenon

22 / 03 / 03  22 : 34 : 48 INFO client.DAGClientImpl: DAG: State: FAILED Progress: 0 % TotalTasks: 1 Succeeded: 0 Running: 0 Failed: 0 Killed: 0 
22 / 03 / 03  22 : 34 : 48 INFO client.DAGClientImpl: DAG completed. FinalState= FAILED
 22 / 03 / 03  22 : 34 : 48Examples.OrderedWordCount the INFO: the DAG Diagnostics: [Vertex failed, the Tokenizer vertexName =, = vertex_1646363809353_0002_1_00 vertexId, Diagnostics = [Vertex vertex_1646363809353_0002_1_00 [the Tokenizer] killed / failed Due to: ROOT_INPUT_INIT_FAILURE, Vertex the Input: initializer of the Input failed, Vertex = vertex_1646363809353_0002_1_00 [the Tokenizer], java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
        at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java: 135 )
        at org.apache.hadoop.io.compress.CompressionCodecFactory. <init>(CompressionCodecFactory.java: 175 )
        at org.apache.hadoop.mapreduce.lib.input.TextInputFormat.isSplitable(TextInputFormat.java: 58 )
        at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits( FileInputFormat.java:399 )
        at org.apache.hadoop.mapreduce.split.TezGroupedSplitsInputFormat.getSplits ( TezGroupedSplitsInputFormat.java : 97 )
        at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateNewSplits(MRInputHelpers.java: 448 )
        at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplitsToMem(MRInputHelpers.java: 329 )
        at org.apache.tez.mapreduce.common.MRInputAMSplitGenerator.initialize(MRInputAMSplitGenerator.java: 122 )
        at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$ 1.run ( RootInputInitializerManager.java:278 )
        at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$ 1.run ( RootInputInitializerManager.java:269 )
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java: 422 )
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java: 1924 )
        at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java: 269 )
        at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java: 253 )
        at java.util.concurrent.FutureTask.run(FutureTask.java: 266 )
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: 1149 )
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java: 624 )
        at java.lang.Thread.run(Thread.java: 748 )
Caused by: java.lang.ClassNotFoundException: Class com.hadoop.compression.lzo.LzoCodec not found
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java: 2255 )
        at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java: 128 )
        ... 18  more 
], Vertex killed, vertexName =Summation, vertexId=vertex_1646363809353_0002_1_01, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1646363809353_0002_1_01 [Summation] killed/failed due to:OTHER_VERTEX_FAILURE], Vertex killed, vertexName=Sorter, vertex vertexId=vertex_1646363809353_0002_1_02, diagnostics=[Vertex received Kill in INITED state., Vertex vertex_1646363809353_0002_1_02 [Sorter] killed/failed due to:OTHER_VERTEX_FAILURE], DAG did not succeed due to VERTEX_FAILURE. failedVertices: 1 killedVertices: 2 ]

Reason for error

Lzo related dependencies not found.

Error solution

Lzo will put its dependencies into hadoop during installation, and only need to load hadoop’s dependencies into tez.

Modify tez-site.xml and add the following configuration:

    < property > 
         < name > tez.use.cluster.hadoop-libs </ name > 
         < value > true </ value > 
    </ property >

Similar Posts:

Leave a Reply

Your email address will not be published. Required fields are marked *