“Execution error, return code 1 from org. Apache. Hadoop. Hive. QL. Exec. Movetask” error occurred when hive imported data locally

Phenomenon

Importing local files through load data local will cause an error that cannot be imported

hive> load data local inpath '/home/hadoop/out/mid_test.txt' overwrite into table my_mid.mid_test partition (etl_date=20190101);
Loading data to table my_mid.mid_test partition (etl_date=20190101)
Failed with exception Unable to move source file:/home/hadoop/out/mid_test.txt to destination hdfs://namenode01.my.com/my_etl/dw/mid/mid_test/etl_date=20190101/mid_test.txt
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask

Related operations

The first import was successful. Later, it was found that there was a problem with the number of lines in the source file. A new text file was copied to the local import directory. The above problem occurred during subsequent import

Solutions

Query the hive log for more detailed information. The common location of the log is/TMP/${user}/hive.log

2019-02-19 09:38:21,503 ERROR [main]: exec.Task (SessionState.java:printError(960)) - Failed with exception Unable to move source file:/home/hadoop/out/mid_test.txt to destination hdfs://namenode01.my.com/my_etl/dw/mid/mid_test/etl_date=20190101/mid_test.txt
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move source file:/home/hadoop/out/mid_test.txt to destination hdfs://namenode01.my.com/my_etl/dw/mid/mid_test/etl_date=20190101/mid_test.txt
	at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2644)
	at org.apache.hadoop.hive.ql.metadata.Hive.replaceFiles(Hive.java:2911)
	at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1398)
	at org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1324)
	at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:438)
	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)
	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)
	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1653)
	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1412)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1195)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
	at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.hadoop.fs.ChecksumException: Checksum error: file:/home/hadoop/out/mid_test.txt at 0 exp: -827044509 got: 624370567
	at org.apache.hadoop.fs.FSInputChecker.verifySums(FSInputChecker.java:323)
	at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:279)
	at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:228)
	at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:196)
	at java.io.DataInputStream.read(DataInputStream.java:100)
	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85)
	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
	at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:366)
	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:338)
	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1965)
	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1933)
	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1898)
	at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2637)
	... 23 more
```shell

> Determine that it is a checksum failure and find the actual reason is that the checksum value of the old file crc file does not match with the new copy file.

## Check if there is a crc file

```shell
[hadoop@my-17 out]$ls -al
totally 112720
drwxrwxr-x   2 hadoop hadoop     4096 2月  19 09:31 .
drwx------. 39 hadoop hadoop     4096 2月  19 09:57 ..
-rw-r--r--   1 hadoop hadoop  1595242 2月  19 09:12 mid_test.txt
-rw-r--r--   1 hadoop hadoop     3128 2月  19 08:22 .mid_test.txt.crc

If there is such a file, just delete the CRC file

The subsequent import is successful

Similar Posts: