Category Archives: oracle management server

OMS Consuming High CPU due to ADF threads in OEM13c

In this Blog I have explained how I fixed one of the performance issues which was faced recently in my production OEM13c R2 environment.  From last couple of weeks we started facing performance issue, which resulted in degraded OEM13c Console performance.

Sudden increase in page response time was observed across all OMS, which was causing application to become unresponsive most of the time. Also the average time taken by any page to display data was over 30-40 seconds, which is not ideal at all.

Page processing time value across all OMS crossed the default warning threshold value and the maximum time hit the value of 96 seconds.

To troubleshoot further, I checked the EMGC_OMS logs and found below mentioned.

<Jan 15, 2019 2:41:53 PM EST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: ‘132’ for queue: ‘weblogic.kernel.Default (self-tuning)’ has been busy for “622” seconds working on the request “Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 622774 ms
“, which is more than the configured time (StuckThreadMaxTime) of “600” seconds in “server-failure-trigger”. Stack trace:
oracle.net.aso.o.a(Unknown Source)
oracle.net.aso.h.a(Unknown Source)
oracle.net.ano.CryptoDataPacket.receive(Unknown Source)
oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:306)
oracle.net.ns.NetInputStream.read(NetInputStream.java:250)
oracle.net.ns.NetInputStream.read(NetInputStream.java:172)
oracle.net.ns.NetInputStream.read(NetInputStream.java:90)

<Jan 15, 2019 2:41:57 PM EST> <Emergency> <oracle.dfw.incident> <BEA-000000> <incident 328 created with problem key “BEA-000337 [/empbs/upload]”>
<Jan 15, 2019 2:42:11 PM EST> <Warning> <org.apache.myfaces.trinidadinternal.agent.AgentFactoryImpl> <BEA-000000> <The User-Agent “EMCTL COMMAND: Oracle HTTPClient Version 10h” is unknown; creating an agent with “unknown” agent attributes.>
<Jan 15, 2019 2:42:53 PM EST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: ‘132’ for queue: ‘weblogic.kernel.Default (self-tuning)’ has been busy for “682” seconds working on the request “Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 682849 ms
“, which is more than the configured time (StuckThreadMaxTime) of “600” seconds in “server-failure-trigger”. Stack trace:
org.apache.log4j.Category.callAppenders(Category.java:204)
org.apache.log4j.Category.forcedLog(Category.java:391)
org.apache.log4j.Category.log(Category.java:826)

<Jan 15, 2019 2:47:11 PM EST> <Warning> <org.apache.myfaces.trinidadinternal.agent.AgentFactoryImpl> <BEA-000000> <The User-Agent “EMCTL COMMAND: Oracle HTTPClient Version 10h” is unknown; creating an agent with “unknown” agent attributes.>
<Jan 15, 2019 2:49:04 PM EST> <Warning> <org.apache.myfaces.trinidadinternal.agent.AgentFactoryImpl> <BEA-000000> <The User-Agent “EMCTL COMMAND: Oracle HTTPClient Version 10h” is unknown; creating an agent with “unknown” agent attributes.>
<Jan 15, 2019 2:49:47 PM EST> <Warning> <org.apache.myfaces.trinidadinternal.agent.AgentFactoryImpl> <BEA-000000> <The User-Agent “EMCTL COMMAND: Oracle HTTPClient Version 10h” is unknown; creating an agent with “unknown” agent attributes.>
<Jan 15, 2019 2:50:53 PM EST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: ’72’ for queue: ‘weblogic.kernel.Default (self-tuning)’ has been busy for “610” seconds working on the request “Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 611004 ms
“, which is more than the configured time (StuckThreadMaxTime) of “600” seconds in “server-failure-trigger”. Stack trace:
org.apache.log4j.Category.callAppenders(Category.java:204)
org.apache.log4j.Category.forcedLog(Category.java:391)
org.apache.log4j.Category.log(Category.java:826)

Referring the knowledge base at MOS, I came across this thread where it was found as a known BUG in EM 13c. Crosschecked EM 13c: OMS Java Process Consuming High CPU due to ADF threads ( Doc ID 2293299.1 )

And yes after checking the document, the log snippet indicated that we were hitting the similar issue. This issue is related to the OMS causing high cpu because of the internal ADF threads.

So to fix the issue I followed the steps as per the document Doc ID 2293299.1 and applied the recommended patch. The performance of OMS is back to normal now. I monitored the performance for next 3-4 days and the issue got fixed finally.

CAUSE :-  Issue is due to internal bugs on ADF framework

REFERENCES:-

BUG:25917466 – EM 13C: EMCTL DUMP OMS -INCLUDE_TOP DOES NOT COLLECT TOP CMD RESULTS CORRECTLY
BUG:27234155 – 13.2 OMS IS CONSUMING HIGH CPU DUE TO ADF THREADS

Hope this will help you as well in case you the similar issue.

Feel free to comment for any queries.