Tag Archives: java

How to Solve JAVA Error: “well-known file is not secure” ?

Recently, JDK tools jmap and jstack are often used to dump JVM heap and JVM thread stack log to analyze problems. When executing these two commands, I was most puzzled when I encountered this problem: well-known file is not secure

eg:

Google the Internet and find the answer on stackoverflow http://stackoverflow.com/questions/9100149/jstack-well-known-file-is-not-secure?rq=1

It means:

We can find/TMP/hsperfdata on the machine_ $user/$PID is such a file. When we execute jmap or jstack, we first check whether the user executing the command is in contact with hsperfdata_ The users of the $user file are consistent. If not, switch to consistent and then execute.

Get current heap snapshot

jmap – dump:format=b ,file=./ heap.hprof 20344(pid)
20344: well-known file is not secure

sudo -u app-cfg-api jmap – dump:format=b ,file=./ heap.hprof 20344
Dumping heap to /tmp/hsperfdata_ app-cfg-api/ heap.hprof …
Heap dump file created

Org.yaml.snakeyaml.parser.ParserException: while parsing a block mapping

org.yaml.snakeyaml.parser.ParserException: while parsing a block mapping

Reason: yml file format error, this file requires strict requirements format

If the nodes are not aligned, align the Mybatis nodes with the Spring nodes, here is the yml configuration file for SpringBoot 2.0 + Mybatis + Mysql

server:
port: 8080
spring:
datasource:
name: daducha
type: com.alibaba.druid.pool.DruidDataSource
druid:
url: jdbc:mysql://127.0.0.1:3306/daducha
username: root
password: root
maxActive: 20
initialSize: 1
maxWait: 60000
minIdle: 1
timeBetweenEvictionRunsMillis: 60000
minEvictableIdleTimeMillis: 300000
testWhileIdle: true
testOnBorrow: false
testOnReturn: false
poolPreparedStatements: true
maxOpenPreparedStatements: 20
driver-class-name: com.mysql.jdbc.Driver
mybatis:
mapper-locations: classpath:mapping/*.xml
type-aliases-package: com.wuji.entity.po

Spring Custome schema Loading Error: White spaces are required between publicId and systemId.

Spring project startup error

The error log is as follows:

Caused by: org.springframework.beans.factory.xml.XmlBeanDefinitionStoreException: Line 1 in XML document from URL [jar:file:webapps/ROOT/WEB-INF/lib/xxx-biz-1.0.jar!/xxx/sponsor-group-pipeline.xml] is invalid; nested exception is org.xml.sax.SAXParseException; systemId: http://www.xxx.com/schema/hipac/pipeline.xsd; lineNumber: 1; columnNumber: 50; White spaces are required between publicId and systemId.
    at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.doLoadBeanDefinitions(XmlBeanDefinitionReader.java:399) ~[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
    at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:336) ~[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
    at org.springframework.beans.factory.xml.XmlBeanDefinitionReader.loadBeanDefinitions(XmlBeanDefinitionReader.java:304) ~[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
    at org.springframework.beans.factory.xml.DefaultBeanDefinitionDocumentReader.importBeanDefinitionResource(DefaultBeanDefinitionDocumentReader.java:245) ~[spring-beans-4.1.9.RELEASE.jar:4.1.9.RELEASE]
    ... 54 common frames omitted
Caused by: org.xml.sax.SAXParseException: White spaces are required between publicId and systemId.
    at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:203) ~[na:1.8.0_45]

Google has been saying for a long time that there should be a space between systemid and PublicID defined by XML header, which is the content of this lump

<?xml version="1.0" encoding="UTF-8"?&>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"  xmlns:hipac="http://www.xxx.com/schema/hipac"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
        http://www.xxx.com/schema/hipac http://www.xxx.com/schema/hipac/pipeline.xsd"&>

It’s the red part above. You can see that there is no problem with my configuration

What’s the problem?At the beginning, I always suspected the jar package conflict, but after a long time of investigation, compared with many jar versions of other normal applications, no problem was found

Since this scheme is defined by myself, there is a corresponding configuration resolution in spring, that is spring.handlers and spring.schemas Two documents. Let’s look at the contents of these two documents

spring.handlers

http\://www.xxx.com/schema/hipac=com.xxx.engine.choreography.pipeline.namespace.EngineNamespaceHandler

spring.schemas

http\://www.xxx.com/schema/hipac/pipeline.xsd=com/xxx/engine/choreography/pipeline/xsd/pipeline.xsd

One of these two files specifies the path of XSD, and the other is the parsing class of schemas. It is suspected that there is a problem with the loading of these two files

So I searched these two files in the access application code, and found that there are actually these two configuration files in the application, and the contents are similar (only the values are different, and the corresponding class and XSD can’t be found in the code). I’m very suspicious here, and the probability is the problem.

Next, delete these two files immediately and deploy them again. OK, it’s successful.

After this problem, let me remember that another application had this situation when it was just connected, but I forgot how to solve it at that time. In retrospect, it was actually caused by the wrong class path or XSD path in the configuration file at that time.

However, the prompt for spring startup is also a pitfall. It’s too misleading. I always think it’s a problem with the referenced configuration file, which wastes a lot of time. I’ll take a good look at spring’s parsing of sechemas later.

[leetcode] 280. Wiggle sort

Given an unsorted array nums, reorder it in-place such that nums[0] <= nums[1] &>= nums[2] <= nums[3]….

For example, given nums = [3, 5, 2, 1, 6, 4], one possible answer is [1, 6, 2, 5, 3, 4].

To give an array without sorting, reorder it to nums [0] & lt; = nums [1] &> = nums [2] & lt; = nums [3].

Solution: traverse the array, if it is an odd position and its value is larger than the next, then exchange its value, if it is an even position and its value is smaller than the next, then exchange its value. The time complexity is O (n). Note that the difference between the index and the actual position is 1, so the parity is opposite.

Java:

public class Solution {
    public void wiggleSort(int[] nums) {
        if (nums == null || nums.length < 2) return;
        for (int i = 1; i < nums.length; i++) {
            if ((i % 2 == 0 && nums[i] &> nums[i - 1]) || (i % 2 == 1 && nums[i] < nums[i - 1])) {
                int tmp = nums[i];
                nums[i] = nums[i - 1];
                nums[i - 1] = tmp;
            } 
        }
    }
}  

Java:

public class Solution {
    public void wiggleSort(int[] nums) {
        if (nums == null || nums.length == 0) {
            return;
        }
        for (int i = 1; i < nums.length; i++) {
            if (i % 2 == 1) {
                if (nums[i] < nums[i - 1]) {
                    swap(nums, i);
                } 
            } else {
                if (nums[i] &> nums[i - 1]) {
                    swap(nums, i);
                }
            }
        }
    }
    
    private void swap(int[] nums, int i) {
        int tmp = nums[i - 1];
        nums[i - 1] = nums[i];
        nums[i] = tmp;
    }
}  

Python:

# Time:  O(n)
# Space: O(1)
class Solution(object):
    def wiggleSort(self, nums):
        """
        :type nums: List[int]
        :rtype: void Do not return anything, modify nums in-place instead.
        """
        for i in xrange(1, len(nums)):
            if ((i % 2) and nums[i - 1] &> nums[i]) or \
                (not (i % 2) and nums[i - 1] < nums[i]):
                # Swap unordered elements.
                nums[i - 1], nums[i] = nums[i], nums[i - 1]

C++:

// Time:  O(n)
// Space: O(1)
class Solution {
public:
    void wiggleSort(vector<int&>& nums) {
        for (int i = 1; i < nums.size(); ++i) {
            if (((i % 2) && nums[i] < nums[i - 1]) ||
                (!(i % 2) && nums[i] &> nums[i - 1])) {
                // Swap unordered elements.
                swap(nums[i], nums[i - 1]);
            }
        }
    }
}; 

C++:

// Time Complexity O(nlgn)
class Solution {
public:
    void wiggleSort(vector<int&> &nums) {
        sort(nums.begin(), nums.end());
        if (nums.size() <= 2) return;
        for (int i = 2; i < nums.size(); i += 2) {
            swap(nums[i], nums[i - 1]);
        }
    }
};

C++:

// Time Complexity O(n)
class Solution {
public:
    void wiggleSort(vector<int&> &nums) {
        if (nums.size() <= 1) return;
        for (int i = 1; i < nums.size(); ++i) {
            if ((i % 2 == 1 && nums[i] < nums[i - 1]) || (i % 2 == 0 && nums[i] &> nums[i - 1])) {
                swap(nums[i], nums[i - 1]);
            }
        }
    }
};

>>

Similar theme:

[LeetCode] Wiggle Sort II free action II

All LeetCode Questions List Total

[leetcode] 140. Word break II word split II

Given anon-emptystringsand a dictionarywordDictcontaining a list ofnon-emptywords, add spaces insto construct a sentence where each word is a valid dictionary word.Return all such possible sentences.

Note:

The same word in the dictionary may be reused multiple times in the segmentation.

You may assume the dictionary does not contain duplicate words.

Example 1:

Input:
s = "catsanddog"
wordDict = ["cat", "cats", "and", "sand", "dog"]
Output:
[
 "cats and dog",
 "cat sand dog"
]

Example 2:

Input:
s = "pineapplepenapple"
wordDict = ["apple", "pen", "applepen", "pine", "pineapple"]
Output:
[
 "pine apple pen apple",
 "pineapple pen apple",
 "pine applepen apple"
]
Explanation: Note that you are allowed to reuse a dictionary word.

Example 3:

Input:
s = "catsandog"
wordDict = ["cats", "dog", "sand", "and", "cat"]
Output:
[]

139. Word break is an extension of word break. That question is just to judge whether it can be split, and this question requires the output of all possible split combinations.

Solution: DP + DFS, first use DP to calculate whether it can be split, and then use DFS to find the specific split combination.

Java:

public static List<String&> wordBreak(String s, Set<String&> dict) {
    //create an array of ArrayList<String&>
    List<String&> dp[] = new ArrayList[s.length()+1];
    dp[0] = new ArrayList<String&>();
 
    for(int i=0; i<s.length(); i++){
        if( dp[i] == null ) 
            continue; 
 
        for(String word:dict){
            int len = word.length();
            int end = i+len;
            if(end &> s.length()) 
                continue;
 
            if(s.substring(i,end).equals(word)){
                if(dp[end] == null){
                    dp[end] = new ArrayList<String&>();
                }
                dp[end].add(word);
            }
        }
    }
 
    List<String&> result = new LinkedList<String&>();
    if(dp[s.length()] == null) 
        return result; 
 
    ArrayList<String&> temp = new ArrayList<String&>();
    dfs(dp, s.length(), result, temp);
 
    return result;
}
 
public static void dfs(List<String&> dp[],int end,List<String&> result, ArrayList<String&> tmp){
    if(end <= 0){
        String path = tmp.get(tmp.size()-1);
        for(int i=tmp.size()-2; i&>=0; i--){
            path += " " + tmp.get(i) ;
        }
 
        result.add(path);
        return;
    }
 
    for(String str : dp[end]){
        tmp.add(str);
        dfs(dp, end-str.length(), result, tmp);
        tmp.remove(tmp.size()-1);
    }
}

Java:

public List<String&> wordBreak(String s, Set<String&> wordDict) {
    ArrayList<String&> [] pos = new ArrayList[s.length()+1];
    pos[0]=new ArrayList<String&>();
 
    for(int i=0; i<s.length(); i++){
        if(pos[i]!=null){
            for(int j=i+1; j<=s.length(); j++){
                String sub = s.substring(i,j);
                if(wordDict.contains(sub)){
                    if(pos[j]==null){
                        ArrayList<String&> list = new ArrayList<String&>();
                        list.add(sub);
                        pos[j]=list;
                    }else{
                        pos[j].add(sub);
                    }
 
                }
            }
        }
    }
 
    if(pos[s.length()]==null){
        return new ArrayList<String&>();
    }else{
        ArrayList<String&> result = new ArrayList<String&>();
        dfs(pos, result, "", s.length());
        return result;
    }
}
 
public void dfs(ArrayList<String&> [] pos, ArrayList<String&> result, String curr, int i){
    if(i==0){
        result.add(curr.trim());
        return;
    }
 
    for(String s: pos[i]){
        String combined = s + " "+ curr;
        dfs(pos, result, combined, i-s.length());
    }
} 

Java:

public class Solution {
    Map<String, List<String&>&> done = new HashMap<&>();
    Set<String&> dict;

    public List<String&> wordBreak(String s, Set<String&> dict) {
        this.dict = dict;
        done.put("", new ArrayList<&>());
        done.get("").add("");

        return dfs(s);
    }

    List<String&> dfs(String s) {
        if (done.containsKey(s)) {
            return done.get(s);
        }
        List<String&> ans = new ArrayList<&>();

        for (int len = 1; len <= s.length(); len++) { 
            String s1 = s.substring(0, len);
            String s2 = s.substring(len);

            if (dict.contains(s1)) {
                List<String&> s2_res = dfs(s2);
                for (String item : s2_res) {
                    if (item == "") {
                        ans.add(s1);
                    } else {
                        ans.add(s1 + " " + item);
                    }
                }
            }
        }
        done.put(s, ans);
        return ans;
    }
} 

Python:

class Solution(object):
    def wordBreak(self, s, wordDict):
        """
        :type s: str
        :type wordDict: Set[str]
        :rtype: List[str]
        """
        n = len(s)

        max_len = 0
        for string in wordDict:
            max_len = max(max_len, len(string))

        can_break = [False for _ in xrange(n + 1)]
        valid = [[False] * n for _ in xrange(n)]
        can_break[0] = True
        for i in xrange(1, n + 1):
            for l in xrange(1, min(i, max_len) + 1):
                if can_break[i-l] and s[i-l:i] in wordDict:
                    valid[i-l][i-1] = True
                    can_break[i] = True

        result = []
        if can_break[-1]:
            self.wordBreakHelper(s, valid, 0, [], result)
        return result

    def wordBreakHelper(self, s, valid, start, path, result):
        if start == len(s):
            result.append(" ".join(path))
            return
        for i in xrange(start, len(s)):
            if valid[start][i]:
                path += [s[start:i+1]]
                self.wordBreakHelper(s, valid, i + 1, path, result)
                path.pop() 

C++:

class Solution {
public:
    vector<string&> wordBreak(string s, unordered_set<string&>& wordDict) {
        vector<string&> res;
        string out;
        vector<bool&> possible(s.size() + 1, true);
        wordBreakDFS(s, wordDict, 0, possible, out, res);
        return res;
    }
    void wordBreakDFS(string &s, unordered_set<string&> &wordDict, int start, vector<bool&> &possible, string &out, vector<string&> &res) {
        if (start == s.size()) {
            res.push_back(out.substr(0, out.size() - 1));
            return;
        }
        for (int i = start; i < s.size(); ++i) {
            string word = s.substr(start, i - start + 1);
            if (wordDict.find(word) != wordDict.end() && possible[i + 1]) {
                out.append(word).append(" ");
                int oldSize = res.size();
                wordBreakDFS(s, wordDict, i + 1, possible, out, res);
                if (res.size() == oldSize) possible[i + 1] = false;
                out.resize(out.size() - word.size() - 1);
            }
        }
    }
};

>>

Similar theme:

[LeetCode] 139. Word Break units break

All LeetCode Questions List Total

The key technologies in hadoop-3.0.0 configuration yarn.nodemanager.aux -Services item

In the configuration of hadoop-3.0.0-alpha4, the default value of yarn.nodemanager.aux-services is “mapreduce.shuffle”, but if you continue to use this value in hadoop-2.2, the NodeManager will fail to start, and the following error will be reported in logs/yarn-biadmin-nodemanager-hostname.log.

java.lang.IllegalArgumentException: The ServiceName: mapreduce.shuffle set in yarn.nodemanager.aux-services is invalid.The valid service name should only contain a-zA-Z0-9_ and can not start with numbers

at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:115)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)

at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:303)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)

at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:406)

at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)

at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:748)

at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:809)

2017-07-21 03:21:42,946 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NodeManager at slave1/127.0.0.1

************************************************************/

Solution.

Change the value of “mapreduce.shuffle” to “mapreduce_shuffle” in yarn.nodemanager.aux-services.

Note: You need to change it in yarn-site.xml of each node at /usr/hadoop/hadoop-3.0.0-alpha4/etc/hadoop

The Difference Between Hadoop job-kill and Yarn application-kill

Hadoop job – kill calls CLI.java Inside job.killJob (); there are several situations. If you can find that the status is running, you can send a kill request directly to appmaster.
YARNRunner.java

@Override
  public void killJob(JobID arg0) throws IOException, InterruptedException {
    /* check if the status is not running, if not send kill to RM */
    JobStatus status = clientCache.getClient(arg0).getJobStatus(arg0);
    ApplicationId appId = TypeConverter.toYarn(arg0).getAppId();

    // get status from RM and return
    if (status == null) {
      killUnFinishedApplication(appId);
      return;
    }

    if (status.getState() != JobStatus.State.RUNNING) {
      killApplication(appId);
      return;
    }

    try {
      /* send a kill to the AM */
      clientCache.getClient(arg0).killJob(arg0);
      long currentTimeMillis = System.currentTimeMillis();
      long timeKillIssued = currentTimeMillis;
      while ((currentTimeMillis < timeKillIssued + 10000L)
          && !isJobInTerminalState(status)) {
        try {
          Thread.sleep(1000L);
        } catch (InterruptedException ie) {
          /** interrupted, just break */
          break;
        }
        currentTimeMillis = System.currentTimeMillis();
        status = clientCache.getClient(arg0).getJobStatus(arg0);
        if (status == null) {
          killUnFinishedApplication(appId);
          return;
        }
      }
    } catch(IOException io) {
      LOG.debug("Error when checking for application status", io);
    }
    if (status != null && !isJobInTerminalState(status)) {
      killApplication(appId);
    }
  }

MRClientService.java

@SuppressWarnings("unchecked")
    @Override
    public KillJobResponse killJob(KillJobRequest request) 
      throws IOException {
      JobId jobId = request.getJobId();
      UserGroupInformation callerUGI = UserGroupInformation.getCurrentUser();
      String message = "Kill job " + jobId + " received from " + callerUGI
          + " at " + Server.getRemoteAddress();
      LOG.info(message);
      verifyAndGetJob(jobId, JobACL.MODIFY_JOB);
      appContext.getEventHandler().handle(
          new JobDiagnosticsUpdateEvent(jobId, message));
      appContext.getEventHandler().handle(
          new JobEvent(jobId, JobEventType.JOB_KILL));
      KillJobResponse response = 
        recordFactory.newRecordInstance(KillJobResponse.class);
      return response;
    }

The yarn application -kill uses ApplicationCLI.java, which sends a kill request to RM

/**
   * Kills the application with the application id as appId
   * 
   * @param applicationId
   * @throws YarnException
   * @throws IOException
   */
  private void killApplication(String applicationId) throws YarnException,
      IOException {
    ApplicationId appId = ConverterUtils.toApplicationId(applicationId);
    ApplicationReport  appReport = null;
    try {
      appReport = client.getApplicationReport(appId);
    } catch (ApplicationNotFoundException e) {
      sysout.println("Application with id '" + applicationId +
          "' doesn't exist in RM.");
      throw e;
    }

    if (appReport.getYarnApplicationState() == YarnApplicationState.FINISHED
        || appReport.getYarnApplicationState() == YarnApplicationState.KILLED
        || appReport.getYarnApplicationState() == YarnApplicationState.FAILED) {
      sysout.println("Application " + applicationId + " has already finished ");
    } else {
      sysout.println("Killing application " + applicationId);
      client.killApplication(appId);
    }
  }

YarnClientImpl.java

@Override
  public void killApplication(ApplicationId applicationId)
      throws YarnException, IOException {
    KillApplicationRequest request =
        Records.newRecord(KillApplicationRequest.class);
    request.setApplicationId(applicationId);

    try {
      int pollCount = 0;
      long startTime = System.currentTimeMillis();

      while (true) {
        KillApplicationResponse response =
            rmClient.forceKillApplication(request);
        if (response.getIsKillCompleted()) {
          LOG.info("Killed application " + applicationId);
          break;
        }

        long elapsedMillis = System.currentTimeMillis() - startTime;
        if (enforceAsyncAPITimeout() &&
            elapsedMillis &>= this.asyncApiPollTimeoutMillis) {
          throw new YarnException("Timed out while waiting for application " +
            applicationId + " to be killed.");
        }

        if (++pollCount % 10 == 0) {
          LOG.info("Waiting for application " + applicationId + " to be killed.");
        }
        Thread.sleep(asyncApiPollIntervalMillis);
      }
    } catch (InterruptedException e) {
      LOG.error("Interrupted while waiting for application " + applicationId
          + " to be killed.");
    }
  }

Yet Another Java Service Wrapper

Artifact home page

http://sourceforge.net/projects/yajsw/

This thing can automatically wrap Java programs into windows services

Solve the boot problem of some Java programs that should be run after power recovery. Windows services similar to Tomcat can be booted automatically~

It’s easy to use

Run genconfig PID in bat directory

—–PID is the PID of the Java virtual machine process opened by the program, not the PID of the program you are running!!! (I’ve been inking this for a long time.)

It will automatically generate configuration files in the conf directory wrapper.conf

When you open a text document, you can see the parameters and configuration that are automatically read. The service name and service description of windows are also in it. Read the document for details~

Finally, run in bat installService.bat The Java program can be automatically registered as windows service~

The default is manual. Don’t forget to start it automatically~~

The log folder records the log of the program~~~