Re: QC server is mutated due to unrecoverable error

Hi Anuragjalan and Hristo,

Thank you for helping me in this regard.

I will put forward your points to my business team.

After analysing the Application Logs, I found that one particular User performed a task which called the below ALM Thread

http-0.0.0.0-8080-18 [ListRepositoryFiles] [login_id:294361] [connect_id:41279] [call_id:325] resulting in exhausting all of the JVM memory. PFA application log files.

There are 2 application servers, application on both these servers went to mutated state because of this reason.

In site admin logs since they are in debug mode, nothing got recorded.

In the server log attached heap size error message appeared, however there are no previous activities recorded.

I have approached the user and it seems user is running few  QTP scripts.

In which case the QTP scripts can cause memory leak exhausting all of the JVM heap size?

Interesting thing is after reboot of the servers user performed same action on both the servers, this time app did not go to mutated state but it gave the below error message.

org.apache.catalina.connector.ClientAbortException 
Messages:
Stack Trace:
java.net.SocketException: Connection reset by peer: socket write error
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:724)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:449)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:349)
at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:748)
at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:126)
at org.apache.coyote.http11.InternalOutputBuffer.doWrite(InternalOutputBuffer.java:557)
at org.apache.coyote.Response.doWrite(Response.java:567)
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:394)
wrapped in ClientAbortException: java.net.SocketException: Connection reset by peer: socket write error
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:399)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:325)
at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:422)
at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:411)
at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:89)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:263)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:106)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:116)
at java.io.OutputStreamWriter.write(OutputStreamWriter.java:203)
at java.io.Writer.write(Writer.java:140)
at com.mercury.td.web.CAbsServlet.sendResponse(CAbsServlet.java:585)
at com.mercury.td.web.CAbsServlet.returnResponseToClient(CAbsServlet.java:525)
at com.mercury.td.web.CAbsServlet.processRequest(CAbsServlet.java:476)
at com.mercury.td.web.CTdServlet.processRequest(CTdServlet.java:165)
at com.mercury.td.web.CAbsServlet.doPost(CAbsServlet.java:349)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.hp.qc.core.utils.gzipfilter.GZIPFilter.doFilter(GZIPFilter.java:24)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.mercury.optane.core.state.ServerStatusFilter.doFilter(ServerStatusFilter.java:33)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:235)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:190)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:92)
at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.process(SecurityContextEstablishmentValve.java:126)
at org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:70)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:330)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:829)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:598)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:619)

Why did the application behave two different ways before the restart and after the restrat for the same user action?

Is the root cause of this issue the User activity?

Any ideas guys.

Regards, Srihari