Idea Run Scala Error: Exception in thread “main”;Ljava/lang/Object;)V


Using idea + Scala + spark, the running program code is as follows:

package cn.idcast.hello

import org.apache.spark.rdd.RDD
import org.apache.spark
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext

 * Author itcast
 * Desc Demo Spark Starter Case-WordCount
object WordCount_bak {
  def main(args: Array[String]): Unit = {
    //TODO 1.env/preparesc/SparkContext/SparkContext execution environment
    val conf: SparkConf = new SparkConf().setAppName("wc").setMaster("local[*]")
    val sc: SparkContext = new SparkContext(conf)

    //TODO 2.source/read data
    //RDD:A Resilient Distributed Dataset (RDD): Resilient Distributed Dataset, simply understood as a distributed collection! It is as simple to use as an ordinary collection!
    // RDD [is a row of data]
    val lines: RDD[String] = sc.textFile("data/input/words.txt")

    //TODO 3.transformation/data manipulation/transformation
    //cut:RDD[one word]
    val words: RDD[String] = lines.flatMap(_.split(" "))
    //record as 1:RDD[(word, 1)]
    val wordAndOnes: RDD[(String, Int)] =,1))
    //group aggregation:groupBy + mapValues( _2).reduce(_+_)) ===>group+aggregate inside Spark in one step:reduceByKey
    val result: RDD[(String, Int)] = wordAndOnes.reduceByKey(_+_)

    //TODO 4.sink/output
    //direct output
    //collect as a local collection and then output
    //output to the specified path (can be a file/folder)

    // For easy viewing of the Web-UI you can let the program sleep for a while
    Thread.sleep(1000 * 60)

    //TODO 5. Close the resource

(forget the screenshot) an error is reported in the result: exception in thread “main” Java;Ljava/lang/Object;) V

It is said on the Internet that the jar package conflicts, but it does not solve the problem


Root cause of the problem: the scala version of windows is inconsistent with the scala version of spark, as shown in the figure:

This is spark’s own version, 2.12.10

I installed 2.12.11 on windows (forgot the screenshot), and later replaced it with 2.12.10 (reinstallation):

After that, it runs successfully without error


Similar Posts: