Environmental description
During this period of time, I was sorting out flink. I just tested an application before today: submitting a task on the web ui reported an error:
As shown in the figure: The running main program class name and degree of parallelism are specified here, and then an error occurs when clicking submit
It is impossible to accurately locate the cause of the error simply from the error information on the page. At this time, you can view the cause of the error through the log file’
[hui@hadoop103 ~]$ cd /opt/module/flink-local/log/
[hui@hadoop103 log]$ ls - ltr
Total dosage 440
-rw-r--r-- 1 hui wd 43688 Jun 10 06:15 flink-hui-taskexecutor-0-hadoop103.log.1
-rw-r--r-- 1 hui wd 37268 Jun 10 06:15 flink-hui-standalonesession-0-hadoop103.log.1
-rw-r--r-- 1 hui wd 750 Jun 10 06:16 flink-hui-standalonesession-0- hadoop103.out
-rw-r--r-- 1 hui wd 750 Jun 10 06:16 flink -hui-taskexecutor-0- hadoop103.out
-rw-r--r-- 1 hui wd 36562 Jun 10 06:16 flink-hui-taskexecutor-0- hadoop103.log
-rw-r--r-- 1 hui wd 28430 Jun 10 06:22 flink-hui-standalonesession-1-hadoop103.log.1
-rw-r--r-- 1 hui wd 32769 Jun 10 06:22 flink-hui-standalonesession-1-hadoop103.log.2
-rw-r--r-- 1 hui wd 36750 Jun 10 06:26 flink-hui-standalonesession-0- hadoop103.log
-rw-r--r-- 1 hui wd 31715 Jun 10 08:28 flink -hui-client- hadoop103.log
-rw-r--r-- 1 hui wd 128992 Jun 10 09:42 flink-hui-standalonesession-1-hadoop103.log.3
-rw-r--r-- 1 hui wd 750 Jun 10 16:39 flink-hui-standalonesession-1- hadoop103.out
-rw-r--r-- 1 hui wd 47418 Jun 10 16:50 flink -hui-standalonesession-1- hadoop103.log
[hui@hadoop103 log]$ tail -100 flink-hui-standalonesession-1- hadoop103.log
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624 ) [?: 1.8.0_144]
at java.lang.Thread.run(Thread.java:748 ) [?: 1.8.0_144]
Caused by: java.lang.RuntimeException: No data for required key 'port'
at org.apache.flink.api.java.utils.AbstractParameterTool.getRequired(AbstractParameterTool.java: 79) ~[flink-dist_2.12-1.13. 0.jar:1.13.0 ]
at org.apache.flink.api.java.utils.AbstractParameterTool.getInt(AbstractParameterTool.java: 106) ~[flink-dist_2.12-1.13.0.jar:1.13.0 ]
at org.wdh01.wc.StreamWordCount.main(StreamWordCount.java: 20) ~[?:? ]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?: 1.8.0_144]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62 ) ~[?: 1.8.0_144]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43 ) ~[?: 1.8.0_144]
at java.lang.reflect.Method.invoke(Method.java:498 ) ~[?: 1.8.0_144]
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java: 355) ~[flink-dist_2.12-1.13.0.jar:1.13.0 ]
... 13 more
2022-06-10 16:49:21,738 ERROR org.apache.flink.runtime.webmonitor.handlers.JarRunHandler[] - Exception occurred in REST handler: Could not execute application.
2022-06-10 16: 49:30,964 WARN org.apache.flink.runtime.webmonitor.handlers.JarRunHandler[] - Configuring the job submission via query parameters is deprecated. Please migrate to submitting a JSON request instead.
2022-06-10 16:49:31,006 INFO org.apache.flink.client.ClientUtils[] - Starting program (detached: true )
2022-06-10 16:49:31,007 WARN org.apache.flink.client.deployment.application.DetachedApplicationRunner[] - Could not execute application :
org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: No data for required key 'port'
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java: 372) ~[ flink-dist_2.12-1.13.0.jar:1.13.0 ]
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java: 222) ~[flink-dist_2.12-1.13.0.jar:1.13.0 ]
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java: 114) ~[flink-dist_2.12-1.13.0.jar:1.13.0 ]
at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java: 84) ~[flink-dist_2.12-1.13.0.jar:1.13.0 ]
at org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java: 70) ~[flink-dist_2.12-1.13.0.jar:1.13.0 ]
at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$ 0(JarRunHandler.java:102) ~[flink-dist_2.12-1.13.0.jar:1.13.0 ]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java: 1590) [?: 1.8.0_144]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511 ) [?: 1.8.0_144]
at java.util.concurrent.FutureTask.run(FutureTask.java:266 ) [?: 1.8.0_144]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$ 201(ScheduledThreadPoolExecutor.java:180) [?: 1.8.0_144]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java: 293) [?: 1.8.0_144]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149 ) [?: 1.8.0_144]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624 ) [?: 1.8.0_144]
at java.lang.Thread.run(Thread.java:748 ) [?: 1.8.0_144]
Caused by: java.lang.RuntimeException: No data for required key 'port'
at org.apache.flink.api.java.utils.AbstractParameterTool.getRequired(AbstractParameterTool.java: 79) ~[flink-dist_2.12-1.13. 0.jar:1.13.0 ]
at org.apache.flink.api.java.utils.AbstractParameterTool.getInt(AbstractParameterTool.java: 106) ~[flink-dist_2.12-1.13.0.jar:1.13.0 ]
at org.wdh01.wc.StreamWordCount.main(StreamWordCount.java: 20) ~[?:? ]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?: 1.8.0_144]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62 ) ~[?: 1.8.0_144]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43 ) ~[?: 1.8.0_144]
at java.lang.reflect.Method.invoke(Method.java:498 ) ~[?: 1.8.0_144]
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java: 355) ~[flink-dist_2.12-1.13.0.jar:1.13.0 ]
... 13 more
2022-06-10 16:49:31,024 ERROR org.apache.flink.runtime.webmonitor.handlers.JarRunHandler[] - Exception occurred in REST handler: Could not execute application.
2022-06-10 16: 49:39,493 WARN org.apache.flink.runtime.webmonitor.handlers.JarRunHandler[] - Configuring the job submission via query parameters is deprecated. Please migrate to submitting a JSON request instead.
2022-06-10 16:49:39,501 INFO org.apache.flink.client.ClientUtils[] - Starting program (detached: true )
2022-06-10 16:49:39,501 WARN org.apache.flink.client.deployment.application.DetachedApplicationRunner[] - Could not execute application :
org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: No data for required key 'port'
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java: 372) ~[ flink-dist_2.12-1.13.0.jar:1.13.0 ]
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java: 222) ~[flink-dist_2.12-1.13.0.jar:1.13.0 ]
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java: 114) ~[flink-dist_2.12-1.13.0.jar:1.13.0 ]
at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java: 84) ~[flink-dist_2.12-1.13.0.jar:1.13.0 ]
at org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java: 70) ~[flink-dist_2.12-1.13.0.jar:1.13.0 ]
at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$ 0(JarRunHandler.java:102) ~[flink-dist_2.12-1.13.0.jar:1.13.0 ]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java: 1590) [?: 1.8.0_144]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511 ) [?: 1.8.0_144]
at java.util.concurrent.FutureTask.run(FutureTask.java:266 ) [?: 1.8.0_144]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$ 201(ScheduledThreadPoolExecutor.java:180) [?: 1.8.0_144]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java: 293) [?: 1.8.0_144]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149 ) [?: 1.8.0_144]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624 ) [?: 1.8.0_144]
at java.lang.Thread.run(Thread.java:748 ) [?: 1.8.0_144]
Caused by: java.lang.RuntimeException: No data for required key 'port'
at org.apache.flink.api.java.utils.AbstractParameterTool.getRequired(AbstractParameterTool.java: 79) ~[flink-dist_2.12-1.13. 0.jar:1.13.0 ]
at org.apache.flink.api.java.utils.AbstractParameterTool.getInt(AbstractParameterTool.java: 106) ~[flink-dist_2.12-1.13.0.jar:1.13.0 ]
at org.wdh01.wc.StreamWordCount.main(StreamWordCount.java: 20) ~[?:? ]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?: 1.8.0_144]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62 ) ~[?: 1.8.0_144]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43 ) ~[?: 1.8.0_144]
at java.lang.reflect.Method.invoke(Method.java:498 ) ~[?: 1.8.0_144]
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java: 355) ~[flink-dist_2.12-1.13.0.jar:1.13.0 ]
... 13 more
2022-06-10 16:49:39,507 ERROR org.apache.flink.runtime.webmonitor.handlers.JarRunHandler[] - Exception occurred in REST handler: Could not execute application.
2022-06-10 16: 50:18,223 WARN org.apache.flink.runtime.webmonitor.handlers.JarRunHandler[] - Configuring the job submission via query parameters is deprecated. Please migrate to submitting a JSON request instead.
2022-06-10 16:50:18,232 INFO org.apache.flink.client.ClientUtils[] - Starting program (detached: true )
2022-06-10 16:50:18,232 WARN org.apache.flink.client.deployment.application.DetachedApplicationRunner[] - Could not execute application :
org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: No data for required key 'port'
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java: 372) ~[ flink-dist_2.12-1.13.0.jar:1.13.0 ]
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java: 222) ~[flink-dist_2.12-1.13.0.jar:1.13.0 ]
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java: 114) ~[flink-dist_2.12-1.13.0.jar:1.13.0 ]
at org.apache.flink.client.deployment.application.DetachedApplicationRunner.tryExecuteJobs(DetachedApplicationRunner.java: 84) ~[flink-dist_2.12-1.13.0.jar:1.13.0 ]
at org.apache.flink.client.deployment.application.DetachedApplicationRunner.run(DetachedApplicationRunner.java: 70) ~[flink-dist_2.12-1.13.0.jar:1.13.0 ]
at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleRequest$ 0(JarRunHandler.java:102) ~[flink-dist_2.12-1.13.0.jar:1.13.0 ]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java: 1590) [?: 1.8.0_144]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511 ) [?: 1.8.0_144]
at java.util.concurrent.FutureTask.run(FutureTask.java:266 ) [?: 1.8.0_144]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$ 201(ScheduledThreadPoolExecutor.java:180) [?: 1.8.0_144]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java: 293) [?: 1.8.0_144]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149 ) [?: 1.8.0_144]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624 ) [?: 1.8.0_144]
at java.lang.Thread.run(Thread.java:748 ) [?: 1.8.0_144]
Caused by: java.lang.RuntimeException: No data for required key 'port'
at org.apache.flink.api.java.utils.AbstractParameterTool.getRequired(AbstractParameterTool.java: 79) ~[flink-dist_2.12-1.13. 0.jar:1.13.0 ]
at org.apache.flink.api.java.utils.AbstractParameterTool.getInt(AbstractParameterTool.java: 106) ~[flink-dist_2.12-1.13.0.jar:1.13.0 ]
at org.wdh01.wc.StreamWordCount.main(StreamWordCount.java: 20) ~[?:? ]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?: 1.8.0_144]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62 ) ~[?: 1.8.0_144]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43 ) ~[?: 1.8.0_144]
at java.lang.reflect.Method.invoke(Method.java:498 ) ~[?: 1.8.0_144]
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java: 355) ~[flink-dist_2.12-1.13.0.jar:1.13.0 ]
... 13 more
2022-06-10 16:50:18,247 ERROR org.apache.flink.runtime.webmonitor.handlers.JarRunHandler[] - Exception occurred in REST handler: Could not execute application.
Looking through the log, it turns out that the running java program requires two parameters, that is to say, the program needs to dynamically pass in parameters at runtime: its java program is like this
public static void main(String[] args) throws Exception {
// Read host & port from parameters
ParameterTool parameterTool = ParameterTool.fromArgs(args);
String host = parameterTool.get("host" );
int port = parameterTool.getInt("port" );
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
DataStreamSource <String> socketTextStream = env.socketTextStream(host, port);
socketTextStream.flatMap((String line, Collector <Tuple2<String, Long>> out) -> {
String[] s = line.split(" " );
for (String s1 : s) {
out.collect(Tuple2.of(s1, 1L ));
}
}).returns(Types.TUPLE(Types.STRING, Types.LONG))
.keyBy(data -> data.f0)
.sum( 1 )
.print();
env.execute();
}
So when submitting a job, you need to pass in two parameters’
This will submit the task. If there is no problem when running the program, don’t worry, in general, you can judge the problem through the log, so you can deal with it urgently.