Idea remotely submits spark job Java io. IOException: Failed to connect to DESKTOP-H
1. Error report log
Exception in thread "main" java.lang.reflect.UndeclaredThrowableException
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1713)
at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:63)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.run(CoarseGrainedExecutorBackend.scala:188)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$.main(CoarseGrainedExecutorBackend.scala:293)
at org.apache.spark.executor.CoarseGrainedExecutorBackend.main(CoarseGrainedExecutorBackend.scala)
Caused by: org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:205)
at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)
at org.apache.spark.executor.CoarseGrainedExecutorBackend$$anonfun$run$1.apply$mcV$sp(CoarseGrainedExecutorBackend.scala:201)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:64)
at org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:63)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
... 4 more
Caused by: java.io.IOException: Failed to connect to DESKTOP-HSVM
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:245)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:187)
at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:198)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:194)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:190)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: 拒绝连接: DESKTOP-HSVM
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
... 1 more
Caused by: java.net.ConnectException: denied to connect
... 11 more
2. cause analysis
Idea cannot establish a connection with the local machine while submitting spark jobs to the remote cluster and returning the results to the local machine.
Caused by: java.io.IOException: Failed to connect to DESKTOP-HSVM
3. solution
Add the native name desktop-hsvm and IP to the/etc/hosts file of the remote cluster, as shown in the following figure
1. Complete exception information
[ERROR] error: java.lang.StackOverflowError [INFO] at scala.tools.nsc.transform.Erasure$Eraser.adaptMember(Erasure.scala:646) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:712) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401) [INFO] at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$102.apply(Typers.scala:4559) [INFO] at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$102.apply(Typers.scala:4559) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.silent(Typers.scala:680) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.normalTypedApply$1(Typers.scala:4558) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedApply$1(Typers.scala:4608) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5370) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5387) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:712) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5501) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5509) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.adaptMember(Erasure.scala:658) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:712) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401) [INFO] at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$102.apply(Typers.scala:4559) [INFO] at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$102.apply(Typers.scala:4559) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.silent(Typers.scala:680) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.normalTypedApply$1(Typers.scala:4558) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedApply$1(Typers.scala:4608) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5370) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5387) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:712) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5501) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5509) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.adaptMember(Erasure.scala:658) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:712) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401) [INFO] at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$102.apply(Typers.scala:4559) [INFO] at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$102.apply(Typers.scala:4559) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.silent(Typers.scala:680) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.normalTypedApply$1(Typers.scala:4558) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedApply$1(Typers.scala:4608) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5370) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5387) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:712) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5501) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5509) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.adaptMember(Erasure.scala:658) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:712) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401) [INFO] at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$102.apply(Typers.scala:4559) [INFO] at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$102.apply(Typers.scala:4559) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.silent(Typers.scala:680) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.normalTypedApply$1(Typers.scala:4558) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedApply$1(Typers.scala:4608) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5370) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5387) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:712) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5501) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5509) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.adaptMember(Erasure.scala:658) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:712) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401) [INFO] at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$102.apply(Typers.scala:4559) [INFO] at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$102.apply(Typers.scala:4559) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.silent(Typers.scala:680) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.normalTypedApply$1(Typers.scala:4558) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedApply$1(Typers.scala:4608) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5370) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5387) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:712) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5501) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5509) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.adaptMember(Erasure.scala:658) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:712) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401) [INFO] at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$102.apply(Typers.scala:4559) [INFO] at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$102.apply(Typers.scala:4559) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.silent(Typers.scala:680) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.normalTypedApply$1(Typers.scala:4558) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedApply$1(Typers.scala:4608) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5370) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5387) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:712) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5501) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5509) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.adaptMember(Erasure.scala:658) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:712) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401) [INFO] at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$102.apply(Typers.scala:4559) [INFO] at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$102.apply(Typers.scala:4559) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.silent(Typers.scala:680) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.normalTypedApply$1(Typers.scala:4558) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedApply$1(Typers.scala:4608) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5370) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5387) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:712) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5501) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5509) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.adaptMember(Erasure.scala:658) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:712) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401) [INFO] at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$102.apply(Typers.scala:4559) [INFO] at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$102.apply(Typers.scala:4559) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.silent(Typers.scala:680) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.normalTypedApply$1(Typers.scala:4558) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedApply$1(Typers.scala:4608) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5370) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5387) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:712) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5501) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5509) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.adaptMember(Erasure.scala:658) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:712) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401) [INFO] at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$102.apply(Typers.scala:4559) [INFO] at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$102.apply(Typers.scala:4559) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.silent(Typers.scala:680) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.normalTypedApply$1(Typers.scala:4558) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedApply$1(Typers.scala:4608) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5370) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5387) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:712) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5501) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5509) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.adaptMember(Erasure.scala:658) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:712) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401) [INFO] at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$102.apply(Typers.scala:4559) [INFO] at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$102.apply(Typers.scala:4559) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.silent(Typers.scala:680) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.normalTypedApply$1(Typers.scala:4558) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedApply$1(Typers.scala:4608) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5370) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5387) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:712) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5501) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5509) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.adaptMember(Erasure.scala:658) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:712) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401) [INFO] at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$102.apply(Typers.scala:4559) [INFO] at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$102.apply(Typers.scala:4559) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.silent(Typers.scala:680) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.normalTypedApply$1(Typers.scala:4558) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedApply$1(Typers.scala:4608) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5370) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5387) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:712) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5501) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5509) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.adaptMember(Erasure.scala:658) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:712) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401) [INFO] at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$102.apply(Typers.scala:4559) [INFO] at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$102.apply(Typers.scala:4559) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.silent(Typers.scala:680) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.normalTypedApply$1(Typers.scala:4558) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedApply$1(Typers.scala:4608) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5370) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5387) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:712) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5501) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5509) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.adaptMember(Erasure.scala:658) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:712) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401) [INFO] at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$102.apply(Typers.scala:4559) [INFO] at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$102.apply(Typers.scala:4559) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.silent(Typers.scala:680) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.normalTypedApply$1(Typers.scala:4558) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedApply$1(Typers.scala:4608) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5370) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5387) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:712) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5501) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5509) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.adaptMember(Erasure.scala:658) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:712) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401) [INFO] at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$102.apply(Typers.scala:4559) [INFO] at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$102.apply(Typers.scala:4559) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.silent(Typers.scala:680) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.normalTypedApply$1(Typers.scala:4558) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedApply$1(Typers.scala:4608) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedInAnyMode$1(Typers.scala:5370) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed1(Typers.scala:5387) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:712) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5501) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typedQualifier(Typers.scala:5509) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.adaptMember(Erasure.scala:658) [INFO] at scala.tools.nsc.transform.Erasure$Eraser.typed1(Erasure.scala:712) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.runTyper$1(Typers.scala:5423) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.scala$tools$nsc$typechecker$Typers$Typer$$typedInternal(Typers.scala:5450) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.body$2(Typers.scala:5397) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.typed(Typers.scala:5401) [INFO] at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$102.apply(Typers.scala:4559) [INFO] at scala.tools.nsc.typechecker.Typers$Typer$$anonfun$102.apply(Typers.scala:4559) [INFO] at scala.tools.nsc.typechecker.Typers$Typer.silent(Typers.scala:680) [INFO] at scala.t
2. reason: Scala code reports an error during construction
Plug in version used
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
<configuration>
<scalaVersion>${scala.version}</scalaVersion>
<args>
<arg>-target:jvm-1.5</arg>
</args>
</configuration>
</plugin>
3. solution
Delete the above plug-in and replace it with the following build plug-in to solve the problem. The corresponding version of Scala
<plugin>
<!-- This is a compiled scala code -->
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.2</version>
<executions>
<execution>
<id>scala-compile-first</id>
<phase>process-resources</phase>
<goals>
<goal>add-source</goal>
<goal>compile</goal>
</goals>
</execution>
</executions>
<configuration>
<scalaVersion>${scala.version}</scalaVersion>
<args>
<arg>-target:jvm-1.8</arg>
</args>
<jvmArgs>
<jvmArg>-Xss2048K</jvmArg>
</jvmArgs>
</configuration>
</plugin>
<plugin>
<groupId>org.scalastyle</groupId>
<artifactId>scalastyle-maven-plugin</artifactId>
<version>1.0.0</version>
<configuration>
<verbose>false</verbose>
<failOnViolation>true</failOnViolation>
<includeTestSourceDirectory>false</includeTestSourceDirectory>
<failOnWarning>false</failOnWarning>
<sourceDirectory>${basedir}/src/main/scala</sourceDirectory>
</configuration>
</plugin>
Scala corresponding version
<scala.version>2.11.12</scala.version>
<scala.binary.version>2.11</scala.binary.version>
1. First enter $JAVA_HOME in the shell. If it is not installed, you need to configure the Java environment variables with export first
2. Modify the JAVA_HOME in etc/hadoop/hadoop-env.sh
FileSystem fs = FileSystem.get(new URI("hdfs://hadoop000:8020"),new Configuration(),"hadoop");
#Error Messages:
Exception in thread "main" java.net.ConnectException: Call From DESKTOP-09ASUHR/192.168.56.1 to 192.168.137.101:8020 failed on connection exception: java.net.ConnectException: Connection refused: no further information; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
#Cause Analysis:
/root/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop/core-site.xml
The fs.defaultFS parameter is configured to use a domain name and the domain name is not mapped.
#Solution:
vim /root/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop/core-site.xml
Use IP address before #8020, not hostname
fs.defaultFS hdfs://192.168.154.129:8820
The reason for the error is the lack of slf4j NOP jar slf4j-simple.jar, slf4j-log4j12.jar, slf4j-jdk14.Jar or logback classic Just add one jar package in jar (only one, and there will be conflicts if there are more).
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-simple</artifactId>
<version>1.7.25</version>
</dependency>
Today, I started spark. I always reported an error when connecting hive. The error is as follows
21/12/12 19:55:26 ERROR Hive: Cannot initialize metastore due to autoCreate error javax.jdo.JDODataStoreException: Required table missing : "`DBS`" in Catalog "" Schema "". DataNucleus requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable "datanucleus.schema.autoCreateTables" Exception in thread "main" org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Hive metastore database is not initialized. Please use schematool (e.g. ./schematool -initSchema -dbType ...) to create the schema. If needed, don't forget to include the option to auto-create the underlying database in your JDBC connection string (e.g. ?createDatabaseIfNotExist=true for mysql));
Looking at the yellow font in the back, I first initialized the metadata. I wrote it in my last blog. Later, I found that it was not the metadata problem, but the need to open {datanucleus schema.autoCreateTables
<property>
<name>datanucleus.schema.autoCreateAll</name>
<value>true</value>
</property>
Complete hive site xm：
<configuration>
<!-- Location of metadata generated by Hive-->
<property>
<name>hive.metastore.warehouse.dir</name>
<value>/user/hive/warehouse</value>
</property>
<!--- Use local service to connect to Hive, default is true-->
<property>
<name>hive.metastore.local</name>
<value>true</value>
</property>
<!-- URL address of the database connection JDBC-->
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true&useSSL=false&serverTimezone=UTC</value>
</property>
<!-- Database connection driver, i.e. MySQL driver-->
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>
<!-- MySQL database user name-->
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
</property>
<!-- MySQL Database Password-->
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>123456</value>
</property>
<property>
<name>hive.metastore.schema.verification</name>
<value>false</value>
</property>
<property>
<name>datanucleus.schema.autoCreateAll</name>
<value>true</value>
</property>
</configuration>
* What went wrong:
An exception occurred applying plugin request [id: 'com.android.application']
> Failed to apply plugin 'com.android.internal.application'.
> Android Gradle plugin requires Java 11 to run. You are currently using Java 1.8.
You can try some of the following options:
- changing the IDE settings.
- changing the JAVA_HOME environment variable.
- changing `org.gradle.java.home` in `gradle.properties`.
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
Solution:
1.Add the follow in gradle.properties:
org.gradle.java.home=/Applications/Android Studio.app/Contents/jre/Contents/Home
2.Manually set the path to Java11
1 .Problem background
Split springboot into multi module projects (split in different forms)
Split by hierarchy: one item for controller, one item for service, and one item for Dao (mapper).
Split by business: a business module is split into one project (including the controller, service, Dao, etc. of the business)
After the split is completed, start the original application and prompt application failed to start
2 .Problem phenomenon
Description:
Field userMapper in com.fanzyx.crane.framework.security.service.JwtUserDetailsService required a bean of type ‘com. fanzyx.xx.mapper.XxxrMapper’ that could not be found.
The injection point has the following annotations:
– @org.springframework.beans.factory.annotation.Autowired(required=true)
Action:
Consider defining a bean of type ‘com.fanzyx.xx.mapper.XxxrMapper’ in your configuration.
Or prompt that a method (the method you called) in the basemapper of mybatis plus cannot be found
3. Cause of problem
Check whether the next package reference conflicts. If there are no conflicts, see the following
If the basic package path of mapper scanning is not configured in the application, it will only be found under the current module
e.g.
@SpringBootApplication()
public class XxxApplication {
public static void main(String[] args) {
SpringApplication.run(XxxApplication.class,args);
}
}
4. Solution
Add annotation on xxxapplication (startup class) and add scanning basic path
// Multi-module projects need to be configured to scan base packages, otherwise they will not be able to load classes from other modules
@SpringBootApplication(scanBasePackages = {"com.fanzyx.xx"})
// must be configured to a specific path, the middle ** can match multiple levels of directories, one * can only match a single level of directories
//@MapperScan(basePackages = {"com.fanzyx.xx.**.mapper"})
public class XxxApplication{
public static void main(String[] args) {
SpringApplication.run(XxxApplication.class,args);
}
}
Mybatis batch insert problem
Incorrect wording:
INSERT INTO t_csm_customer_product(id, customer_code, product_code) values
<foreach collection="lists" close=")" open="(" index="i" item="cstProduct" separator=",">
#{cstProduct.id},
#{cstProduct.customerCode},
#{cstProduct.productCode}
</foreach>
Error Messages: Error updating database…Cause: java.sql.SQLException: Column count doesn’t match value count at row 1
Print sql:
INSERT INTO table(id, name, value) values ( ?,?,?,?,?,?) The whole bracket is enclosed
The SQL we want:
INSERT INTO table(id, name, value) values ( ?,?,?),(?,?,?) Adjust XML to:
INSERT INTO t_csm_customer_product(id, customer_code, product_code) values
<foreach collection="lists" index="i" item="cstProduct" separator=",">
(#{cstProduct.id},
#{cstProduct.customerCode},
#{cstProduct.productCode})
</foreach>Missing parameter () in key foreach