Tag Archives: Hive

When hive processes data, it cannot run directly and reports an error: execution error, return code 2 from org.apache.hadoop.hive.ql.exec.mr

terms of settlement:

set hive.exec.mode.local.auto=true;

 

Error reason:

The memory space of the namenode is not enough, and the remaining memory space of the JVM is not enough, which is caused by the operation of a new job

 

Reference link: https://www.it610.com/article/1297818114901221376.htm

 

Hive Start Error: Exception in thread “main” java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgumen

Error Messages:
Exception in thread “main” java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
at org.apache.hadoop.conf.Configuration.set(Configuration.java:1380)

at org.apache.hadoop.util.RunJar.main(RunJar.java:236)

Cause:
The two guava.jar versions of hadoop and hive are inconsistent
The two locations are located in the following two directories.
– /usr/local/hive/lib/
– /usr/local/hadoop/share/hadoop/common/lib/

Solution:
Delete the lower version and copy the higher version to the lower version directory

[Solved] Hive Error: java.sql.SQLException: No suitable driver found for jdbc:hive://localhost:10000/default

Error:

java.sql.SQLException: No suitable driver found for jdbc:hive://localhost:10000/default
    at java.sql.DriverManager.getConnection(DriverManager.java:596)
    at java.sql.DriverManager.getConnection(DriverManager.java:233)
    at demo.utils.JDBCUtils.getConnection(JDBCUtils.java:25)
    at demo.hive.HiveJDBCDemo.main(HiveJDBCDemo.java:16)
Exception in thread "main" java.lang.NullPointerException
    at demo.hive.HiveJDBCDemo.main(HiveJDBCDemo.java:18)

Solution:

url->change hive to hive2

before:

private static String driverString = "org.apache.hive.jdbc.HiveDriver";
private static String urlString = "jdbc:hive://localhost:10000/default";

after:

private static String driverString = "org.apache.hive.jdbc.HiveDriver";
private static String urlString = "jdbc:hive2://localhost:10000/default";

[Solved] Hive export MYSQL Error: Container [pid=3962,containerID=container_1632883011739_0002_01_000002] is running 270113280B beyond the ‘VIRTUAL’ memory limit.

Problem description

Container [pid=3962,containerID=container_1632883011739_0002_01_000002] is running 270113280B beyond the ‘VIRTUAL’ memory limit.

Current usage: 91.9 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used. Killing container.

Cause of problem

When yarn starts running, it checks the virtual memory and throws an exception.

Solution:

Modify Hadoop_Home/etc/Hadoop/yen-site.xml file

Add the following to save and exit:

1 <property>
2 <name>yarn.nodemanager.vmem-pmem-ratio</name>
3 <value>3.0</value>
4 </property>

Problem solved!

Apache Ranger integrates hive error records [How to Solve]

Version information is as follows:

hadoop2.9.2

hive 2.x

Ranger latest version 2.1.0

After deploying the Ranger plug-in on the hive side, an error is reported when using beeline to connect and query the database. The error information is as follows:

verbose: on
0: jdbc:hive2://192.168.0.9:10000> show databases;
Getting log thread is interrupted, since query is done!
Error: Error running query: java.lang.NoSuchFieldError: REPLLOAD (state=,code=0)
org.apache.hive.service.cli.HiveSQLException: Error running query: java.lang.NoSuchFieldError: REPLLOAD
    at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:264)
    at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:250)
    at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:309)
    at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:250)
    at org.apache.hive.beeline.Commands.executeInternal(Commands.java:977)
    at org.apache.hive.beeline.Commands.execute(Commands.java:1148)
    at org.apache.hive.beeline.Commands.sql(Commands.java:1063)
    at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1134)
    at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:965)
    at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:875)
    at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:499)
    at org.apache.hive.beeline.BeeLine.main(BeeLine.java:482)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.hive.service.cli.HiveSQLException: Error running query: java.lang.NoSuchFieldError: REPLLOAD
    at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:218)
    at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:269)
    at org.apache.hive.service.cli.operation.Operation.run(Operation.java:324)
    at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:460)
    at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:447)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
    at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
    at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)
    at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
    at com.sun.proxy.$Proxy52.executeStatementAsync(Unknown Source)
    at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:294)
    at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:497)
    at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1437)
    at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1422)
    at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
    at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
    at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoSuchFieldError: REPLLOAD
    at org.apache.ranger.authorization.hive.authorizer.RangerHiveAuthorizer.checkPrivileges(RangerHiveAuthorizer.java:700)
    at org.apache.hadoop.hive.ql.Driver.doAuthorizationV2(Driver.java:817)
    at org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:604)
    at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:472)
    at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:329)
    at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1158)
    at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1145)
    at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:184)
    ... 27 more

 

Most of these errors are caused by version conflicts or version mismatches

Go to the software package of Ranger admin and check the hive version supported by Ranger 2.1.0:

My ranger-admin
#cd /data/ranger/ranger-2.1.0-SNAPSHOT-admin/ews/webapp/WEB-INF/classes/ranger-plugins/hive
 Check hive package information:
#ll
total 41504
-rw-r--r-- 1 ranger ranger   492033 Nov  5 15:18 hive-common-3.1.0.jar
-rw-r--r-- 1 ranger ranger 40603464 Nov  5 15:20 hive-exec-3.1.0.jar
-rw-r--r-- 1 ranger ranger   125261 Nov  5 15:20 hive-jdbc-3.1.0.jar
-rw-r--r-- 1 ranger ranger    36852 Nov  5 15:19 hive-metastore-3.1.0.jar
-rw-r--r-- 1 ranger ranger   566570 Nov  5 15:19 hive-service-3.1.0.jar
-rw-r--r-- 1 ranger ranger   313702 Nov  5 15:19 libfb303-0.9.3.jar
-rw-r--r-- 1 ranger ranger   246445 Nov  5 15:57 libthrift-0.12.0.jar
-rw-r--r-- 1 ranger ranger    99652 Nov  5 17:18 ranger-hive-plugin-2.1.0-SNAPSHOT.jar

It is found that the supported hive version is 3. X, while the version I currently use is 2. X, resulting in an error using beeline

Solution 1: upgrade hive to 3. X

 

[[email protected] ~]$ beeline -u jdbc:hive2://192.168.0.230:10000 -n hduser
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data1/hadoop/hive_3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/data1/hadoop/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Connecting to jdbc:hive2://192.168.0.230:10000
Connected to: Apache Hive (version 3.1.2)
Driver: Hive JDBC (version 3.1.2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 3.1.2 by Apache Hive
0: jdbc:hive2://192.168.0.230:10000> show databases;
INFO  : Compiling command(queryId=hduser_20191108203220_9ae51d7b-38f0-4f3a-af96-c4ddb91ee9ad): show databases
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:database_name, type:string, comment:from deserializer)], properties:null)
INFO  : Completed compiling command(queryId=hduser_20191108203220_9ae51d7b-38f0-4f3a-af96-c4ddb91ee9ad); Time taken: 1.348 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hduser_20191108203220_9ae51d7b-38f0-4f3a-af96-c4ddb91ee9ad): show databases
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing command(queryId=hduser_20191108203220_9ae51d7b-38f0-4f3a-af96-c4ddb91ee9ad); Time taken: 0.039 seconds
INFO  : OK
INFO  : Concurrency mode is disabled, not creating a lock manager
+----------------+
| database_name  |
+----------------+
| default        |
+----------------+
1 row selected (1.878 seconds)
0: jdbc:hive2://192.168.0.230:10000> !q
Closing: 0: jdbc:hive2://192.168.0.230:10000
[[email protected] ~]$ hive --version
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data1/hadoop/hive_3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/data1/hadoop/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Hive 3.1.2
Git git://HW13934/Users/gates/tmp/hive-branch-3.1/hive -r 8190d2be7b7165effa62bd21b7d60ef81fb0e4af
Compiled by gates on Thu Aug 22 15:01:18 PDT 2019
From source with checksum 0492c08f784b188c349f6afb1d8d9847
[[email protected] ~]$

Solution 2: delete the package under Ranger admin hive, and then copy the corresponding package from hive client to this directory (I didn’t test it)

 

If this error is encountered during hive fetching

If you encounter such an error during hive fetching: parseexception line 1:78 cannot recognize input near ‘& lt; EOF>’ ‘& lt; EOF>’ ‘& lt; EOF>’ in expression specification
hive

The reason is: there is a problem with the last symbol. There are multiple ” or ”, and the computer does not know which symbol matches which symbol. When two quotation marks must be used, one uses’ ‘and the other uses”

Java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument

Website content quality is poor, distribution efficiency is too low how to do?Huawei engineers offer 5 unique skills>>>

Error message prompted:

SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
        at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)
        at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
        at org.apache.hadoop.mapred.JobConf.setJar(JobConf.java:536)
        at org.apache.hadoop.mapred.JobConf.setJarByClass(JobConf.java:554)
        at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:448)
        at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5141)
        at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5099)
        at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:97)
        at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:81)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:699)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:236)

The key point is: com.google.common.base.preconditions.checkargument. This is because the version of guava.jar in hive is inconsistent with that in Hadoop. Test method:

Check the guava.jar version in share/Hadoop/common/Lib in the Hadoop installation directory

Check the version of guava.jar in lib under hive installation directory. If the two versions are inconsistent, delete the lower version and copy the higher version to solve the problem

But I found out later that there were other problems

Other versions of guava can be downloaded from the MVN repository

Reference link: https://blog.csdn.net/GQB1226/article/details/102555820

Hive Error: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

Report errors during hive execution and pay attention to the key points (yellow)

2019-02-01 09:56:54,623 ERROR [pool-7-thread-4] dao.IHiveDaoImpl - java.sql.SQLException: org.apache.hive.service.cli.HiveSQLException: 
Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
    at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:380)
    at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:257)
    at org.apache.hive.service.cli.operation.SQLOperation.access$800(SQLOperation.java:91)
    at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:348)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1758)
    at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:362)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

It’s probably an error when executing MapReduce:

Check that MapReduce is actually implemented

Pull Mr error log:

2019-02-01 10:28:35,832 INFO [IPC Server handler 4 on 38091] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from attempt_1537175606568_162793_m_000000_3: Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"log":"5\u001aNEWEHIREWEB17.2019012911\u001a1\u001a3\u001a1548730807629\u001a43\u001a14\u001a2223123\u001a2577551\u001a8e56221be35a44f8845064b8cc8f21f9\u001a61.170.197.152\u001a","webname":"ehireLog","mon":"201901","dt":"20190129"}
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:169)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1758)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"log":"5\u001aNEWEHIREWEB17.2019012911\u001a1\u001a3\u001a1548730807629\u001a43\u001a14\u001a2223123\u001a2577551\u001a8e56221be35a44f8845064b8cc8f21f9\u001a61.170.197.152\u001a","webname":"ehireLog","mon":"201901","dt":"20190129"}
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:562)
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:160)
    ... 8 more
Caused by: com.tracker.common.db.simplehbase.exception.SimpleHBaseException: convert result exception. cells=[003\x111/data:id/1538028988105/Put/vlen=4/seqid=0, 003\x111/data:isInSelector/1538028988105/Put/vlen=4/seqid=0, 003\x111/data:isStats/1538028988105/Put/vlen=4/seqid=0, 003\x111/data:pageDesc/1538028988105/Put/vlen=6/seqid=0, 003\x111/data:pageType/1548918298621/Put/vlen=1/seqid=0, 003\x111/data:webId/1538028988105/Put/vlen=4/seqid=0] type=class com.tracker.common.data.model.dict.website.Page
    at com.tracker.common.db.simplehbase.HbaseClient.convertToHbaseObjectResult(HbaseClient.java:337)
    at com.tracker.common.db.simplehbase.HbaseClientImpl$6.handleData(HbaseClientImpl.java:177)
    at com.tracker.common.db.simplehbase.HbaseClientImpl.handData_internal(HbaseClientImpl.java:733)
    at com.tracker.common.db.simplehbase.HbaseClientImpl.handDataByRowPrefixList(HbaseClientImpl.java:651)
    at com.tracker.common.db.simplehbase.HbaseClientImpl.findObjectByRowPrefixList(HbaseClientImpl.java:174)
    at com.tracker.common.db.simplehbase.HbaseClientImpl.findObjectByRowPrefix(HbaseClientImpl.java:167)
    at com.tracker.common.data.dao.dict.WebDictDataDao$6.apply(WebDictDataDao.java:154)
    at com.tracker.common.data.dao.dict.WebDictDataDao$6.apply(WebDictDataDao.java:151)
    at com.tracker.common.cache.LocalMapCache.getOrElse(LocalMapCache.java:66)
    at com.tracker.common.data.dao.dict.WebDictDataDao.getPageList(WebDictDataDao.java:151)
    at com.tracker.common.data.dao.dict.WebDictDataDao.loadDictToCache(WebDictDataDao.java:36)
    at com.tracker.common.data.query.DictDataQuery.loadLogPaserDict(DictDataQuery.java:84)
    at com.tracker.hive.func.udf.parse.ParseLog.initialize(ParseLog.java:64)
    at org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:141)
    at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:146)
    at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:140)
    at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:140)
    at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:140)
    at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:140)
    at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluatorHead.initialize(ExprNodeEvaluatorHead.java:39)
    at org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:80)
    at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
    at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
    at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:148)
    at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:547)
    ... 9 more
Caused by: com.tracker.common.db.simplehbase.exception.SimpleHBaseException: java.lang.IllegalArgumentException: offset (0) + length (4) exceed the capacity of the array: 1
    at com.tracker.common.db.simplehbase.HbaseClient.convertBytesToPOJOField(HbaseClient.java:374)
    at com.tracker.common.db.simplehbase.HbaseClient.convertToHbaseObjectResult(HbaseClient.java:332)
    ... 33 more
Caused by: java.lang.IllegalArgumentException: offset (0) + length (4) exceed the capacity of the array: 1
    at org.apache.hadoop.hbase.util.Bytes.explainWrongLengthOrOffset(Bytes.java:632)
    at org.apache.hadoop.hbase.util.Bytes.toInt(Bytes.java:802)
    at org.apache.hadoop.hbase.util.Bytes.toInt(Bytes.java:778)
    at com.tracker.coprocessor.utils.TypeHandlerHolder$IntegerHandler.toObject(TypeHandlerHolder.java:311)
    at com.tracker.common.db.simplehbase.HbaseClient.convertBytesToPOJOField(HbaseClient.java:371)
    ... 34 more

If you look at the yellow part, you can see that it is the corresponding entity class error of HBase

Reason: the type – > in HBase data dictionary table has been modified; There is no jar package to update hive

“Execution error, return code 1 from org. Apache. Hadoop. Hive. QL. Exec. Movetask” error occurred when hive imported data locally

Phenomenon

Importing local files through load data local will cause an error that cannot be imported

hive> load data local inpath '/home/hadoop/out/mid_test.txt' overwrite into table my_mid.mid_test partition (etl_date=20190101);
Loading data to table my_mid.mid_test partition (etl_date=20190101)
Failed with exception Unable to move source file:/home/hadoop/out/mid_test.txt to destination hdfs://namenode01.my.com/my_etl/dw/mid/mid_test/etl_date=20190101/mid_test.txt
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask

Related operations

The first import was successful. Later, it was found that there was a problem with the number of lines in the source file. A new text file was copied to the local import directory. The above problem occurred during subsequent import

Solutions

Query the hive log for more detailed information. The common location of the log is/TMP/${user}/hive.log

2019-02-19 09:38:21,503 ERROR [main]: exec.Task (SessionState.java:printError(960)) - Failed with exception Unable to move source file:/home/hadoop/out/mid_test.txt to destination hdfs://namenode01.my.com/my_etl/dw/mid/mid_test/etl_date=20190101/mid_test.txt
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source file:/home/hadoop/out/mid_test.txt to destination hdfs://namenode01.my.com/my_etl/dw/mid/mid_test/etl_date=20190101/mid_test.txt
	at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2644)
	at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:2911)
	at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1398)
	at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1324)
	at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:438)
	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653)
	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
	at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.hadoop.fs.ChecksumException: Checksum error: file:/home/hadoop/out/mid_test.txt at 0 exp: -827044509 got: 624370567
	at org.apache.hadoop.fs.FSInputChecker.verifySums(FSInputChecker.java:323)
	at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:279)
	at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:228)
	at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:196)
	at java.io.DataInputStream.read(DataInputStream.java:100)
	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85)
	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:366)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338)
	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1965)
	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1933)
	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1898)
	at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2637)
	... 23 more
```shell

> Determine that it is a checksum failure and find the actual reason is that the checksum value of the old file crc file does not match with the new copy file.

## Check if there is a crc file

```shell
[[email protected] out]$ls -al
totally 112720
drwxrwxr-x   2 hadoop hadoop     4096 2月  19 09:31 .
drwx------. 39 hadoop hadoop     4096 2月  19 09:57 ..
-rw-r--r--   1 hadoop hadoop  1595242 2月  19 09:12 mid_test.txt
-rw-r--r--   1 hadoop hadoop     3128 2月  19 08:22 .mid_test.txt.crc

If there is such a file, just delete the CRC file

The subsequent import is successful