1. Modify global settings
Modify project -> Project Language Level-> Select a level lower than the current JDK version
2. Modify model settings
model-> Select model -> Language-> Select a level lower than the current JDK version
3. Modify setting
Compiler, java compiler version
1.File-> Invalidate caches/restart clear cache and restart idea
2. Check whether the dependencies in the POM file are correct
3.maven -> Reimport
4. Open the project structure to check whether there are sources
in the X:\XXX\XXXX\SRC\main\Java folder
It is said on the Internet that it is a jar package conflict, so I intend to open the dense Maven dependency graph through the dependencies of idea to find clues.
As a result, the direct persuasion was too complicated;
17:22:00.943 [main] ERROR org.springframework.boot.SpringApplication - Application run failed java.lang.NoSuchMethodError: org.yaml.snakeyaml.nodes.ScalarNode.getScalarStyle()Lorg/yaml/snakeyaml/DumperOptions$ScalarStyle; at org.springframework.boot.env.OriginTrackedYamlLoader$KeyScalarNode.<init>(OriginTrackedYamlLoader.java:124) at org.springframework.boot.env.OriginTrackedYamlLoader$KeyScalarNode.get(OriginTrackedYamlLoader.java:135) at org.springframework.boot.env.OriginTrackedYamlLoader$KeyScalarNode.get(OriginTrackedYamlLoader.java:130) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
Later, by clicking the code link in the console error prompt, you can directly choose to go to the jar package to see the code, and immediately find the two jar packages to do things again.
found a conflict between the selenium-server-standale-3.141.59.jar and the core package in Springboot.
This is still in vue’s node_modules
/Users/dairy/StudioProjects/mogu_blog_v2/vue_mogu_web/node_modules/selenium-server/lib/runner/selenium-server-standale-3.141.59.jar
delete it and this error will be solved.
When a new project is started in idea, it sometimes reports an error running ‘application’: command line is too long Short command line for application or aalso for spring boot default configuration, the error message is as follows:
Solution:
First find the idea/workspace.xml file inside the project, then find the <component name=”PropertiesComponent”></component > tag, which looks like this
Then add a line in the component tag <property name=”dynamic.classpath” value=”true” />, which would look like this
In this way, no error will be reported when starting the project.
I.Description
Using idea + Scala + spark, the running program code is as follows:
package cn.idcast.hello
import org.apache.spark.rdd.RDD
import org.apache.spark
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
/**
* Author itcast
* Desc Demo Spark Starter Case-WordCount
*/
object WordCount_bak {
def main(args: Array[String]): Unit = {
//TODO 1.env/preparesc/SparkContext/SparkContext execution environment
val conf: SparkConf = new SparkConf().setAppName("wc").setMaster("local[*]")
val sc: SparkContext = new SparkContext(conf)
sc.setLogLevel("WARN")
//TODO 2.source/read data
//RDD:A Resilient Distributed Dataset (RDD): Resilient Distributed Dataset, simply understood as a distributed collection! It is as simple to use as an ordinary collection!
// RDD [is a row of data]
val lines: RDD[String] = sc.textFile("data/input/words.txt")
//TODO 3.transformation/data manipulation/transformation
//cut:RDD[one word]
val words: RDD[String] = lines.flatMap(_.split(" "))
//record as 1:RDD[(word, 1)]
val wordAndOnes: RDD[(String, Int)] = words.map((_,1))
//group aggregation:groupBy + mapValues(_.map(_. _2).reduce(_+_)) ===>group+aggregate inside Spark in one step:reduceByKey
val result: RDD[(String, Int)] = wordAndOnes.reduceByKey(_+_)
//TODO 4.sink/output
//direct output
result.foreach(println)
//collect as a local collection and then output
println(result.collect().toBuffer)
//output to the specified path (can be a file/folder)
result.repartition(1).saveAsTextFile("data/output/result")
result.repartition(2).saveAsTextFile("data/output/result2")
result.saveAsTextFile("data/output/result3")
// For easy viewing of the Web-UI you can let the program sleep for a while
Thread.sleep(1000 * 60)
//TODO 5. Close the resource
sc.stop()
}
}
(forget the screenshot) an error is reported in the result: exception in thread “main” Java lang.NoSuchMethodError:com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;) V
It is said on the Internet that the jar package conflicts, but it does not solve the problem
II.Solution
Root cause of the problem: the scala version of windows is inconsistent with the scala version of spark, as shown in the figure:
This is spark’s own version, 2.12.10
I installed 2.12.11 on windows (forgot the screenshot), and later replaced it with 2.12.10 (reinstallation):
After that, it runs successfully without error
Error reported by idea running program:
Error:Abnormal build process termination:
"C:\Program Files\Java\jdk1.8.0_181\bin\java" -Xmx700m -Djava.awt.headless=true -Djava.endorsed.dirs=\"\" -Djdt.compiler.useSingleThread=true -Dcompile.parallel=false -Drebuild.on.dependency.change=true -Djava.net.preferIPv4Stack=true -Dio.netty.initialSeedUniquifier=3130135518625661539 -Dfile.encoding=GBK -Djps.file.types.component.name=CommunityFileTypes -Duser.language=zh -Duser.country=CN -Didea.paths.selector=IdeaIC2017.1 "-Didea.home.path=D:\intellij\IntelliJ IDEA Community Edition 2017.1.3" -Didea.config.path=C:\Users\Administrator\.IdeaIC2017.1\config -Didea.plugins.path=C:\Users\Administrator\.IdeaIC2017.1\config\plugins -Djps.log.dir=C:/Users/Administrator/.IdeaIC2017.1/system/log/build-log "-Djps.fallback.jdk.home=D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/jre64" -Djps.fallback.jdk.version=1.8.0_112-release -Dio.netty.noUnsafe=true -Djava.io.tmpdir=C:/Users/Administrator/.IdeaIC2017.1/system/compile-server/xmpos_version_1cf37ff8/_temp_ -Djps.backward.ref.index.builder=true -Dkotlin.incremental.compilation.experimental=true -Dkotlin.daemon.enabled -Dkotlin.daemon.client.alive.path=\"C:\Users\Administrator\AppData\Local\Temp\kotlin-idea-1543655368645972493-is-running\" -classpath "D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/lib/jps-launcher.jar;C:/Program Files/Java/jdk1.8.0_121/lib/tools.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/lib/optimizedFileManager.jar" org.jetbrains.jps.cmdline.Launcher "D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/lib/util.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/lib/snappy-in-java-0.5.1.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/lib/forms_rt.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/lib/openapi.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/lib/jps-builders-6.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/lib/protobuf-2.5.0.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/lib/idea_rt.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/lib/jdom.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/lib/resources_en.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/lib/log4j.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/lib/annotations.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/lib/asm-all.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/lib/jna-platform.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/lib/nanoxml-2.2.3.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/lib/oromatcher.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/lib/jgoodies-forms.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/lib/trove4j.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/lib/jps-builders.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/lib/netty-all-4.1.9.Final.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/lib/jps-model.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/lib/javac2.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/lib/jna.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/commons-io-2.2.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/commons-lang-2.6.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/gradle-base-services-3.3.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/gradle-base-services-groovy-3.3.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/gradle-cli-3.3.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/gradle-core-3.3.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/gradle-jps-plugin.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/gradle-logging-3.3.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/gradle-messaging-3.3.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/gradle-model-core-3.3.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/gradle-model-groovy-3.3.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/gradle-native-3.3.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/gradle-process-services-3.3.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/gradle-resources-3.3.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/gradle-tooling-api-3.3.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/gradle-tooling-extension-api.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/gradle-tooling-extension-impl.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/gradle-version-info-3.3.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/gradle-wrapper-3.3.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/gradle.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/guava-jdk5-17.0.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/jansi-1.14.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/kryo-2.22.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/minlog-1.2.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/native-platform-0.12.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/native-platform-freebsd-amd64-libcpp-0.12.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/native-platform-freebsd-amd64-libstdcpp-0.12.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/native-platform-freebsd-i386-libcpp-0.12.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/native-platform-freebsd-i386-libstdcpp-0.12.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/native-platform-linux-amd64-0.12.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/native-platform-linux-amd64-ncurses5-0.12.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/native-platform-linux-amd64-ncurses6-0.12.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/native-platform-linux-i386-0.12.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/native-platform-linux-i386-ncurses5-0.12.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/native-platform-linux-i386-ncurses6-0.12.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/native-platform-osx-amd64-0.12.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/native-platform-osx-i386-0.12.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/native-platform-windows-amd64-0.12.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/native-platform-windows-i386-0.12.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/objenesis-1.2.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/reflectasm-1.07.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/resources_en.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/lib/gson-2.5.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/android/lib/jarutils.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/ant/lib/ant-jps-plugin.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/IntelliLang/lib/intellilang-jps-plugin.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/Groovy/lib/groovy-jps-plugin.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/Groovy/lib/groovy-rt-constants.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/eclipse/lib/eclipse-jps-plugin.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/eclipse/lib/common-eclipse-util.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/gradle/lib/gradle-jps-plugin.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/android/lib/jps/android-jps-plugin.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/android/lib/jps/android-gradle-jps.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/android/lib/android-common.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/android/lib/android-rt.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/android/lib/sdk-common.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/android/lib/repository.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/android/lib/sdklib.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/android/lib/common.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/android/lib/jarutils.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/android/lib/layoutlib-api.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/android/lib/manifest-merger.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/Kotlin/lib/jps/kotlin-jps-plugin.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/Kotlin/lib/kotlin-runtime.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/Kotlin/lib/kotlin-reflect.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/Kotlin/lib/kotlin-plugin.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/Kotlin/lib/kotlin-android-extensions-plugin.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/Kotlin/lib/kotlin-android-extensions-compiler-plugin.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/javaFX/lib/javaFX-jps-plugin.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/plugins/javaFX/lib/common-javaFX-plugin.jar" org.jetbrains.jps.cmdline.BuildMain 127.0.0.1 49368 17dc4cd1-443b-4591-a8f4-08f506e32d55 C:/Users/Administrator/.IdeaIC2017.1/system/compile-server
Build process started. Classpath: D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/lib/jps-launcher.jar;C:/Program Files/Java/jdk1.8.0_121/lib/tools.jar;D:/intellij/IntelliJ IDEA Community Edition 2017.1.3/lib/optimizedFileManager.jarError connecting to 127.0.0.1:49368; reason: Connection refused: no further information: /127.0.0.1:49368io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: no further information: /127.0.0.1:49368 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:612) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:559) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:476) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:438) at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.jetbrains.jps.service.impl.SharedThreadPoolImpl.lambda$executeOnPooledThread$0(SharedThreadPoolImpl.java:42) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused: no further information
... 15 more
Exiting.
---------------------
Delete unused jdks:
Open the control panel, select the program or function, find the unused Java JDK and uninstall it.
SpringCloud Compile Error: java: JPS incremental annotation processing is disabled. Compilation results on partial recompilation may be inaccurate. Use build process “jps.track.ap.dependencies” VM flag to enable/disable incremental annotation processing environment.
Solution:
1.Add: -Djps.track.ap.dependencies=false
2.Clear the caches
Idea compile error report sun.misc.BASE64Decoder upgrade processing
warn:
17:01:15 /deploy/jenkins/workspace/auto-java-test/utils/ImageBase64Utils.java:67: warning: BASE64Encoder is internal proprietary API and may be removed in a future release
17:01:15 BASE64Encoder encoder = new BASE64Encoder();
import sun.misc.BASE64Decoder;
Alternative writing:
//Since JDK 1.8, the JDK public APIs of java.util.Base64.Decoder and java.util.Base64.Encoder have been provided, which can replace the JDK internal APIs of sun.misc.BASE64Decoder and sun.misc.BASE64Encoder.
//byte[] bytes = new BASE64Decoder().decodeBuffer(base64);
byte[] bytes = Base64.getDecoder().decode(base64);
//Or use method
import org.apache.commons.codec.binary.Base64;
return Base64.encodeBase64String(encrypted);
demo test class
@Test
public void test2() throws Exception{
//Decoder and java.util.Base64.Encoder are available from JDK 1.8 onwards, replacing the internal JDK APIs of sun.misc. .
//byte[] bytes = new BASE64Decoder().decodeBuffer(base64);
System.out.println("-----------------------Early Write ----------------------");
String text = "String text";
BASE64Encoder encoder = new BASE64Encoder();
BASE64Decoder decoder = new BASE64Decoder();
byte[] textByte = text.getBytes("UTF-8");
//Code
String encodedText = encoder.encode(textByte);
System.out.println(encodedText);
//Decoding
System.out.println(new String(decoder.decodeBuffer(encodedText), "UTF-8"));
/**
* Compared with the Base64 codecs provided by sun.mis c suite and Apache Commons Codec, the Base64 provided by Java 8 has better performance. In the actual test of encoding and decoding speed, Base64 provided by Java 8 is at least 11 times faster than that provided by sun.mis c suite and at least 3 times faster than that provided by Apache Commons Codec. So if you want to use Base64 in Java, the Base64 category provided by the java .util package under Java 8 is definitely the first choice!
* https://blog.csdn.net/zhou_kapenter/article/details/62890262
*/
System.out.println("-----------------------New writing style----------------------");
byte[] test11 = Base64.getEncoder().encode(textByte);
System.out.println("test11_string="+new String(test11, "UTF-8"));
byte[] bytes11 = Base64.getDecoder().decode(test11);
System.out.println("test11="+new String(bytes11, "UTF-8"));
System.out.println("-----------------------apache writing style----------------------");
String test22 = org.apache.tomcat.util.codec.binary.Base64.encodeBase64String(textByte);
System.out.println("test22_string="+test22);
byte[] bytes22 = org.apache.tomcat.util.codec.binary.Base64.decodeBase64(test22);
System.out.println("test22="+new String(bytes22, "UTF-8"));
/**
* Print results: Consistent results
* ----------------------- early write ----------------------
* 5a2X5Liy5paH5a2X
* String text
* ----------------------- new write ----------------------
* test11 string = 5a2X5Liy5paH5a2X
* test11=string literal
* -----------------------apache writeup ----------------------
* test22 string=5a2X5Liy5paH5a2X
* test22=word string literal
*/
}
Download online or ask colleagues to pass two jars, ojdbc6-11.2 0.7. 0. Jar and jconn3 0.jar
Put in the same folder:
CMD in this folder is as follows: press enter to enter the CMD console.
Enter the following two commands:
mvn install:install-file -DgroupId=com.oracle -DartifactId=ojdbc6 -Dversion=11.2.0.3 -Dpackaging=jar -Dfile=ojdbc6-11.2.0.3.jar mvn install:install-file -DgroupId=com.sybase -DartifactId=jconn -Dversion=3-6.0 -Dpackaging=jar -Dfile=jconn3-6.0.jar
Represents a successful addition. For build failure, please check whether the version number, groupid, file name, etc. in the statement are consistent with the actual situation, and then run it after modification.
Note: If you have installed multiple maven versions, please check which version the current maven_home address is. This command is to install the jar locally into our local maven warehouse. If the warehouse is wrong, it won’t work if you install it.
After installation, right-click the project and update project
Click OK, and then Maven install again. No error will be reported
DEA remote operation HDFS Hadoop
Idea remote operation HDFS Hadoop, error caused by: Java net. ConnectException: Connection refused: no further information
1.source code
package com.github.td.hdfs;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.junit.Test;
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
/**
* @Description: TODO
* @Author:
* @DateTime: 2021/12/20 17:52
**/
public class HdfsClient {
@Test
public void testMkdirs() throws IOException, URISyntaxException, InterruptedException{
// 1 Get the system of the files
Configuration configuration = new Configuration();
// master == 192.168.xx.xx
// the port of cdh version is 8020, the port of common version Hadoop is 9000.
FileSystem fs = FileSystem.get(new URI("hdfs://master:8020"), configuration, "root");
// 2 Create Directory
fs.mkdirs(new Path("/hadoop/hdfs-api"));
// 3 Close resouce
fs.close();
}
}
2. Error report log
java.net.ConnectException: Call From to master:8020 failed on connection exception: java.net.ConnectException: Connection refused: no further information; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:831)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:755)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1549)
at org.apache.hadoop.ipc.Client.call(Client.java:1491)
at org.apache.hadoop.ipc.Client.call(Client.java:1388)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
at com.sun.proxy.$Proxy12.mkdirs(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:657)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy13.mkdirs(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2420)
at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2396)
at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1319)
at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1316)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1333)
at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1308)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2275)
at com.github.td.hdfs.HdfsClient.testMkdirs(HdfsClient.java:29)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58)
Caused by: java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:715)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:700)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:804)
at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:421)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1606)
at org.apache.hadoop.ipc.Client.call(Client.java:1435)
... 46 more
3. cause analysis
Port problem
In the CDH environment, HDFS is port 8020, conf.set (“FS. Defaultfs”“ hdfs://192.168.0.4:8020 ”);
In a normal Hadoop environment, HDFS is a 9000 port, conf.set (“FS. Defaultfs”“ hdfs://192.168.0.121:9000 ”);
4. Solutions
Modify the port number to 9000