JAVA HEAP SIZE – Tune OMS Memory Settings using emctl in OEM13c

Thought of writing and sharing the scenario which I faced in my production environment. Recently I did migration of my production OEM13cR2 from RHEL6 to RHEL7 server. The migration was successful after couple of ups and downs and the application was working fine. It was last week we started facing performance Issue when there were more than 15 sessions running in parallel. All users started complaining that their pages have become unresponsive.

I checked the CPU utilization and found that the OMS process was consuming more than 1000% CPU.

[oracle@hanoemap2 scripts]$ top
top - 03:38:42 up 9 days, 4:41, 2 users, load average: 12.30, 11.83, 12.06
Tasks: 450 total, 1 running, 449 sleeping, 0 stopped, 0 zombie
%Cpu(s): 68.9 us, 1.3 sy, 0.0 ni, 29.1 id, 0.5 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem : 32942424 total, 525124 free, 8858128 used, 23559172 buff/cache
KiB Swap: 4095996 total, 1930836 free, 2165160 used. 18083860 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31247 oracle 20 0 13.8g 3.5g 32680 S 1112 11.2 58072:48 java
31914 oracle 20 0 8534208 2.0g 2.0g S 2.7 6.3 144:19.96 oracle_31914_oe
31912 oracle 20 0 8534248 2.0g 2.0g S 2.3 6.3 144:02.96 oracle_31912_oe
2826 oracle 20 0 13.3g 1.0g 31764 S 1.3 3.3 66:31.06 java
6788 oracle 20 0 9212528 572324 6612 S 1.3 1.7 121:51.87 java
[oracle@hanoemap2 product]$ ps -ef|grep java|grep EMGC_OMS
oracle 31247 31181 99 Nov22 ? 41-02:18:18 /app/oracle/product/middleware13C/oracle_common/jdk/bin/java -server -Xms256m -Xmx4096m -XX:MaxPermSize=768M -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:CompileThreshold=8000 -XX:PermSize=128m -XX:CompileThreshold=8000 -XX:-DoEscapeAnalysis -XX:+UseCodeCacheFlushing -XX:ReservedCodeCacheSize=100M -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -Dweblogic.Name=EMGC_OMS1 -Djava.security.policy=/app/oracle/product/middleware13C/wlserver/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.system.BootIdentityFile=/app/oracle/product/gc_inst01/user_projects/domains/GCDomain/servers EMGC_OMS1/data/nodemanager/boot.properties -Dweblogic.nodemanager.ServiceEnabled=true -Dweblogic.nmservice.RotationEnabled=true -Dweblogic.security.SSL.ignoreHostnameVerification=true -Dweblogic.ReverseDNSAllowed=false -DINSTANCE_HOME=/app/oracle/product/gc_inst01/em/EMGC_OMS1 -DORACLE_HOME=/app/oracle/product/middleware13C -Ddomain.home=/app/oracle/product/gc_inst01/user_projects/domains/GCDomain -Djava.awt.headless=true -Ddomain.name=GCDomain -Doracle.sysman.util.logging.mode=dual_mode -Djbo.doconnectionpooling=true -Djbo.txn.disconnect_level=1 -Docm.repeater.home=/app/oracle/product/middleware13C -Djbo.ampool.minavailablesize=1 -Djbo.ampool.timetolive=-1 -Djbo.load.components.lazily=true -Djbo.max.cursors=5 -Djbo.recyclethreshold=50 -Djbo.ampool.maxavailablesize=50 -Djavax.xml.bind.JAXBContext=com.sun.xml.bind.v2.ContextFactory -Djava.security.egd=file:///dev/./urandom -Dweblogic.debug.DebugWebAppSecurity=true -Dweblogic.SSL.LoginTimeoutMillis=300000 -Djps.auth.debug=true -Djps.authz=ACC -Djps.combiner.optimize.lazyeval=true -Djps.combiner.optimize=true -Djps.subject.cache.key=5 -Djps.subject.cache.ttl=600000 -Doracle.apm.home=/app/oracle/product/middleware13C/apm/ -DAPM_HELP_FILENAME=oesohwconfig.xml -Dweblogic.data.canTransferAnyFile=true -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -DHTTPClient.retryNonIdempotentRequest=false -Dweblogic.security.SSL.minimumProtocolVersion=TLSv1 -Djava.endorsed.dirs=/app/oracle/product/middleware13C/oracle_common/jdk/jre/lib/endorsed:/app/oracle/product/middleware13C/oracle_common/modules/endorsed -Djava.protocol.handler.pkgs=oracle.mds.net.protocol -Dopss.version=12.1.3 -Digf.arisidbeans.carmlloc=/app/oracle/product/gc_inst01/user_projects/domains/GCDomain/config/fmwconfig/carml -Digf.arisidstack.home=/app/oracle/product/gc_inst01/user_projects/domains/GCDomain/config/fmwconfig/arisidprovider -Doracle.security.jps.config=/app/oracle/product/gc_inst01/user_projects/domains/GCDomain/config/fmwconfig/jps-config.xml -Doracle.deployed.app.dir=/app/oracle/product/gc_inst01/user_projects/domains/GCDomain/servers/EMGC_OMS1/tmp/_WL_user -Doracle.deployed.app.ext=/- -Dweblogic.alternateTypesDirectory=/app/oracle/product/middleware13C/oracle_common/modules/oracle.ossoiap_12.1.3,/app/oracle/product/middleware13C/oracle_common/modules/oracle.oamprovider_12.1.3,/app/oracle/product/middleware13C/oracle_common/modules/oracle.jps_12.1.3 -Doracle.mds.filestore.preferred= -Dadf.version=12.1.3 -Dweblogic.jdbc.remoteEnabled=false -Dcommon.components.home=/app/oracle/product/middleware13C/oracle_common -Djrf.version=12.1.3 -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.Jdk14Logger -Ddomain.home=/app/oracle/product/gc_inst01/user_projects/domains/GCDomain -Doracle.server.config.dir=/app/oracle/product/gc_inst01/user_projects/domains/GCDomain/config/fmwconfig/servers/EMGC_OMS1 -Doracle.domain.config.dir=/app/oracle/product/gc_inst01/user_projects/domains/GCDomain/config/fmwconfig -Dohs.product.home=/app/oracle/product/middleware13C/ohs -da -Dwls.home=/app/oracle/product/middleware13C/wlserver/server -Dweblogic.home=/app/oracle/product/middleware13C/wlserver/server -Duser.timezone=America/New_York -Djavax.management.builder.initial=weblogic.management.jmx.mbeanserver.WLSMBeanServerBuilder -Dxdo.server.config.dir=/app/oracle/product/gc_inst01/user_projects/domains/GCDomain/config/bipublisher -DXDO_FONT_DIR=/app/oracle/product/middleware13C/bi/common/fonts -Dweblogic.management.server=https://hanoemap2.oii.oceaneering.com:7103 -Djava.util.logging.manager=oracle.core.ojdl.logging.ODLLogManager -Dweblogic.utils.cmm.lowertier.ServiceDisabled=true weblogic.Server
You have new mail in /var/spool/mail/oracle
[oracle@hanoemap2 product]$

The next thing came to my mind was checking the memory settings for the OMS and found that it was set to default ie. 1.5 Gb. However the recommended settings says that for medium installation it should be set to 4096 MB.

The JVM heap memory for both OMS and Agents plays an important role in the performance of these components.  The amount of memory used for the OMS and agents depends on multiple factors including the number of targets, number of concurrent sessions etc. The installer does provides you advanced options which helps you set size of your environment as Small, Medium, or Large (these options determine the tuning parameters that will be set during install).

You can later tune the memory parameters using below mentioned commands, just like I did to fix my performance issue.

“emctl” command to get the current value of property – Eg. To check current Settings of OMS_HEAP_MAX

[oracle@hanoemap2 ~]$ emctl get property -name OMS_HEAP_MAX
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
SYSMAN password:
Value for property OMS_HEAP_MAX for oms local_oms is 1740M
[oracle@hanoemap2 ~]

“emctl” command to set the new value of property 

For example, the Memory will be resized to 4096M.

[oracle@hanoemap2 ~]$ emctl set property -name OMS_HEAP_MAX -value 4096M
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
SYSMAN password:
Property OMS_HEAP_MAX for oms hanoemap2.oii.oceaneering.com:4890_Management_Service has been set to value 4096M
OMS restart is required to reflect the new property value
[oracle@hanoemap2 ~]$

Please note that whenever you will make a change it will require Restart of OMS. preferred – “emctl stop oms -all”

Restart your Oracle Management Server with emctl stop oms -all  as mentioned above.

[oracle@hanoemap2 ~]$ emctl stop oms -all
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
Stopping Oracle Management Server...
.WebTier Successfully Stopped
Oracle Management Server Successfully Stopped
Oracle Management Server is Down
JVMD Engine is Down
Stopping BI Publisher Server...
BI Publisher Server Successfully Stopped
AdminServer  Stopped successfully
BI Publisher Server is Down
You have new mail in /var/spool/mail/oracle
[oracle@hanoemap2 ~]$
[oracle@hanoemap2 ~]$ emctl start oms
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
Starting Oracle Management Server...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
JVMD Engine is Up
Starting BI Publisher Server ...
BI Publisher Server Successfully Started
BI Publisher Server is Up
[oracle@hanoemap2 ~]$

Now check if the new value has been applied or not.

[oracle@hanoemap2 ~]$ emctl get property -name OMS_HEAP_MAX
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
SYSMAN password:
Value for property OMS_HEAP_MAX for oms local_oms is 4096M
You have new mail in /var/spool/mail/oracle
[oracle@hanoemap2 ~]

The performance problem was fixed after increasing the JVM Heap Size to minimum requirement based on the documentation.  More details can be found at EM 13c: Enterprise Manager Configuration Best Practices (Doc ID 2139083.1)

Hope this information about modifying the JVM heap size is helpful and if you have any questions regarding this post, please add a comment below.

Thanks

Deepak Sharma

Please follow and like us:
RSS
Follow by Email
Facebook
Google+
http://emdeepaksharma.com/2018/11/java-heap-size-tune-oms-memory-settings-using-emctl-in-oem13c/
LinkedIn
Instagram

Leave a Reply

Your email address will not be published. Required fields are marked *