Geeks, please accept the hero post of 2021 Microsoft x Intel hacking contest>>>

Today, when the company uses LS scheduling system (Baidu’s internal tool) to execute remote scripts, it fails every time

The content of the script is relatively simple. In fact, it is to insert the data in HDFS (AFS) into Palo (doris) database. The script is as follows:

mysql -h xxx -P 9030 -uxxx -p'xxx' -e "LOAD LABEL baijiahao.bjh_spider_view_count_${day}_${label_time} (DATA INFILE('afs://xxx/user/feed-bjh/database/spider_view_count_showx/${day}/*') INTO TABLE bjh_spider_view_count) PROPERTIES('cluster'='feed_bjh','timeout'='86400','max_filter_ratio'='0.0001')"

At first, I thought it was my script problem. I have been testing this script on the remote development machine and found that it can be executed correctly, but it will fail through the scheduling system

At first, I didn’t pay attention to the error log. Suddenly, I saw that the error in the log was:

mysql command not found

There is no problem for me to test the MySQL instruction on the development machine

After searching for information, I found that the reason for this problem is that my MySQL is installed in the data partition I set, not in the default/usr/bin/MySQL. When SSH executes commands remotely, it reads from/usr/bin, so it can’t be found

Through which mysql, I found that my MySQL path is:

alias mysql='/home/work/opt/mysql/mysql-5.6.38/bin/mysql'
    ~/opt/mysql/mysql-5.6.38/bin/mysql

Now there are two solutions:

(1) Write the full name of MySQL in the script

/home/work/opt/mysql/mysql-5.6.38/bin/mysql -h xxx -P 9030 -uxxx -p'xxx' -e "LOAD LABEL baijiahao.bjh_spider_view_count_${day}_${label_time} (DATA INFILE('afs://xxx/user/feed-bjh/database/spider_view_count_showx/${day}/*') INTO TABLE bjh_spider_view_count) PROPERTIES('cluster'='feed_bjh','timeout'='86400','max_filter_ratio'='0.0001')"

(2) To establish a soft connection, you need root permission

ln -s /home/work/opt/mysql/mysql-5.6.38/bin/mysql /usr/bin/mysql

Reference

https://blog.csdn.net/qq1124794084/article/details/77529889