Outofmemoryerror: GC overhead limit exceeded error resolution

In short, garbage collection (GC) is the process of JVM recycling objects that are no longer used and releasing memory GC overhead limit exceeded error is a member of the java.lang.outofmemoryerror family, indicating that the JVM memory is exhausted. Next, let’s look at the causes of the java. Lang. outofmemoryerror: GC overhead limit exceeded error and how to solve it

Introduction to GC overhead limit exceeded error

outofmemoryerror is a subclass of java. Lang. virtualmachineerror , which is thrown when JVM resource utilization has problems. More specifically, this error is thrown when JVM takes too long to execute GC and can only recover very little heap memory. According to the Oracle official documents, by default, if the java process takes more than 98% time to execute GC , and less than 2% heap is recovered each time, then JVM throws this error. In other words, this means that our application runs out of almost all available memory, and the garbage collector takes too long to try to clean it up and fails many times

In this case, users will experience that the application response is very slow, and some operations that usually take only a few milliseconds to complete will take longer to complete. This is because all CPUs are garbage collecting, so they cannot perform other tasks

Error recurrence

The following code can reproduce the java. Lang. outofmemoryerror: GC overhead limit exceeded error, the code is as follows:

package com.galaxy.concurrency.jvm;

import java.util.HashMap;
import java.util.Map;
import java.util.Random;

public class OutOfMemoryGCLimitExceed {

    public static void addRandomDataToMap() {
        Map<Integer, String> dataMap = new HashMap<>();
        Random r = new Random();
        while (true) {
            dataMap.put(r.nextInt(), String.valueOf(r.nextInt()));
        }
    }

    public static void main(String[] args) {
        addRandomDataToMap();
    }
}

This code is very simple, that is to use a while dead loop to add random numbers to HashMap continuously. Before executing the main method, first set the JVM parameter to - xmx300m - XX: + useparallelgc ( JVM heap to 300MB , GC algorithm to parallelgc ), and then run the main method, you will encounter java.lang.outofmemoryerror: GC overhead limit exceeded error:

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.util.HashMap.newNode(HashMap.java:1747)
	at java.util.HashMap.putVal(HashMap.java:631)
	at java.util.HashMap.put(HashMap.java:612)
	at com.galaxy.concurrency.jvm.OutOfMemoryGCLimitExceed.addRandomDataToMap(OutOfMemoryGCLimitExceed.java:13)
	at com.galaxy.concurrency.jvm.OutOfMemoryGCLimitExceed.main(OutOfMemoryGCLimitExceed.java:18)

I use the MacOS operating system, 2-core 8g hardware configuration, JDK version is 1.8 . Due to different test environments, if you encounter java.lang.outofmemoryerror: Java heap space errors, you can adjust - Xmx appropriately to reproduce java.lang.outofmemoryerror: GC overhead limit exceeded errors. In order to better understand different garbage collection algorithms ( garbage collection algorithms ), you can refer to Oracle java garbage collection basics tutorial

Solutions

The ideal solution is to find the problems in the application by checking the code that may have memory leak

Which objects in the application occupy most of the heap spaceWhat are the objects in the application that occupy large portions of the heap?）

In which parts of the source code are these objects usedIn which parts of the source code are these objects being allocated?）

We can also use automated graphical tools, such as jvisualvm and jconsole, which can help detect performance problems in code, including java. Lang. outofmemoryerror

The last method is to increase the heap size by changing the JVM startup configuration, or to turn off GC overhead limit exceeded by adding the - XX: - usegcoverhead limit option in the JVM startup configuration. For example, the following JVM parameters provide 1GB heap space for java Applications:

java -Xmx1024m com.xyz.TheClassName

The following JVM parameters not only provide 1GB heap space for java applications, but also add the - XX: - usegcoverhead limit option to turn off GC overhead limit exceeded :

java -Xmx1024m -XX:-UseGCOverheadLimit com.xyz.TheClassName

However, adding the - XX: - usegcoverheadlimit option will cure the symptoms but not the root cause. JVM will eventually throw out the java.lang.outofmemoryerror: Java heap space error. If the above outofmemorygclimitexceed class test results are as follows:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at java.lang.Integer.toString(Integer.java:403)
	at java.lang.String.valueOf(String.java:3099)
	at com.galaxy.concurrency.jvm.OutOfMemoryGCLimitExceed.addRandomDataToMap(OutOfMemoryGCLimitExceed.java:16)
	at com.galaxy.concurrency.jvm.OutOfMemoryGCLimitExceed.main(OutOfMemoryGCLimitExceed.java:21)

In a word, if there is a memory leak in the actual application code, the methods listed above will not solve the problem. On the contrary, we will delay the error. Therefore, it is wiser to thoroughly reassess the memory usage of the application

Online accident solving process and summary

Exception log

This problem has recently occurred online. The following is the exception log of java. Lang. outofmemoryerror: GC overhead limit exceeded :

2019-04-03 10:48:21,253 [http-nio-8080-exec-121] ERROR c.u.n.s.c.c.GlobalExceptionHandler 44 - 服务器端异常！
org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: GC overhead limit exceeded
  at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1006)
  at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925)
  at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974)
  at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:877)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
  at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)
  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
  at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:496)
  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
  at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:803)
  at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
  at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:790)
  at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1468)
  at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
  at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
  at org.apache.phoenix.schema.types.PDataType.isBytesComparableWith(PDataType.java:92)
  at org.apache.phoenix.schema.types.PDataType.coerceBytes(PDataType.java:832)
  at org.apache.phoenix.schema.types.PDataType.coerceBytes(PDataType.java:822)
  at org.apache.phoenix.compile.UpsertCompiler$4.execute(UpsertCompiler.java:1011)
  at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:355)
  at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:338)
  at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
  at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:336)
  at org.apache.phoenix.jdbc.PhoenixPreparedStatement.executeUpdate(PhoenixPreparedStatement.java:199)
  at com.ucar.nosql.spacex.yarn.plugins.metric.timeline.PhoenixHBaseAccessor.commitMetrics(PhoenixHBaseAccessor.java:188)
  at com.ucar.nosql.spacex.yarn.plugins.metric.timeline.PhoenixHBaseAccessor.commitMetricsFromCache(PhoenixHBaseAccessor.java:138)
  at com.ucar.nosql.spacex.yarn.plugins.metric.timeline.PhoenixHBaseAccessor.insertMetricRecordsWithMetadata(PhoenixHBaseAccessor.java:348)
  at com.ucar.nosql.spacex.yarn.plugins.metric.timeline.HBaseTimelineMetricsService.putMetrics(HBaseTimelineMetricsService.java:299)
  at com.ucar.nosql.spacex.yarn.modules.monitor.controller.SinkConsumerController.postMetrics(SinkConsumerController.java:71)
  at sun.reflect.GeneratedMethodAccessor88.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209)
  at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
  at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)
  at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:877)
  at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:783)
  at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
  at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991)
  at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925)
  at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974)
  at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:877)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
  at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

Trouble shooting

oom occurs under linux , not necessarily because java service consumes memory, but also because other programs apply for a lot of memory. At this time, all applications need more memory than physical memory. Then java service consumes a lot of memory and is found by linux operating system, it will be kill , This is a memory protection mechanism adopted by linux to avoid system crash caused by physical memory overload. This mechanism is called oom killer . For the specific reasons, please refer to the oom killer under linux in the resources at the end of this paper. In my previous work, I encountered the situation that elasticsearch data storage service and fluent log collection service were deployed on the same server, and elasticsearch service was killed due to fluent memory leak. Later, Review created fluent , solved the memory leak problem, and deployed it separately from elasticsearch service. After confirmation, our service is deployed separately, so we will turn our attention to JVM memory configuration. This application has a small amount of visits, and the online server memory is 4G . First, we used the command tool of JDK to check the JVM configuration

#Find the process number of spacex.jar
sudo jps
# Check the jvm parameter, pid is the process number of spacex.jar
sudo jinfo -flags pid

JVM is configured as follows.

Attaching to process ID 16022, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.121-b13
Non-default VM flags: -XX:CICompilerCount=2 -XX:InitialHeapSize=62914560 -XX:+ManagementServer -XX:MaxHeapSize=1006632960 -XX:MaxNewSize=335544320 -XX:MinHeapDeltaBytes=524288 -XX:NewSize=20971520 -XX:OldSize=41943040 -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseParallelGC 
Command line:  -Dcom.sun.management.jmxremote -Dcom.sun.management.snmp.port=8044 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=8045 -Djava.rmi.server.hostname=10.212.27.54

Where - XX: maxheapsize = 1006632960kb , that is, 960mb , check the startup script and find java_ Opts is not added to the java startup command, java_ Opts we set the heap memory as - xms2048m - xmx2048m , but this variable is not referenced by java startup command, so the application uses the default memory configuration after startup. The comparison before and after the startup script modification is as follows. Pay attention to the difference of nohup Java line:

(1) Before revision:

JAVA_OPTS="-server -Xms2048m -Xmx2048m -Xmn512m -Xss256k -XX:PermSize=256m -XX:MaxPermSize=256m -XX:SurvivorRatio=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+UseCMSCompactAtFullCollection -XX:+CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=70 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${SPACEXLOG}/"
cd /usr/local/frame/spacex
nohup java -Dcom.sun.management.jmxremote -Dcom.sun.management.snmp.port=8044 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=8045 -Djava.rmi.server.hostname=$LOCAL_IP  -jar ${SPACEXJAR} > ${SPACEXLOG}/stdout.log 2>&1 &

(2) After modification:

JAVA_OPTS="-server -Xms2048m -Xmx2048m -Xmn512m -Xss256k -XX:PermSize=256m -XX:MaxPermSize=256m -XX:SurvivorRatio=8 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+UseCMSCompactAtFullCollection -XX:+CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=70 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=${SPACEXLOG}/"
cd /usr/local/frame/spacex
nohup java ${JAVA_OPTS} -Dcom.sun.management.jmxremote -Dcom.sun.management.snmp.port=8044 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=8045 -Djava.rmi.server.hostname=$LOCAL_IP  -jar ${SPACEXJAR} > ${SPACEXLOG}/stdout.log 2>&1 &

Measures and summary after the event

1. After modifying the script and starting the service in the production environment, use the JPS and Jinfo commands to check whether the JVM parameters are effective

2. The script to start the service must be tested completely and rigorously to make sure that each line of command and related variables take effect

References

OOM

1. The 1-3 section is compiled from outofmemoryerror: GC overhead limit exceeded

2. Oracle official summary of oom exception and handling method: understand the outofmemoryerror exception

Oom killer under Linux

1. Understanding and configuring oom killer under Linux

2、LinuxMM: OOM_ Killer

Similar Posts: