Monthly Archives: January 2019

Lock OEM SYSMAN Console Login – Security Tip 1

Keep “SYSMAN” user locked for general use in Oracle Enterprise Manager.

Sometimes, you may wish to prevent SYSMAN from logging into the OEM console. This is one of the good practices which I follow in my organization. I make sure that OEM administrators login using their own individual accounts to perform daily operations rather than using SYSMAN account. Also If required I make them Super Administrator than regular Administrator which gives them some extra permissions to perform admin operations.  However, Super Administrator privilege should be limited to users who truly need all the permissions that Super Administrator gives them.

Having Multiple Super Administrators accounts reduces the need for SYSMAN access. SYSMAN is the schema owner and is more privileged than Enterprise Manager Super Administrators.

By executing the following SQL statement on the Repository database as the SYSMAN user, you can Lock SYSMAN user login in OEM Console:

UPDATE MGMT_CREATED_USERS 
SET SYSTEM_USER='-1' 
WHERE user_name='SYSMAN'

Once you have disabled the account, you will still be able to login to “sysman” as repository user but this will restrict your access to OEM console and also using “emcli login -username=sysman“. So in both cases you will see errors like :-

SQL> UPDATE MGMT_CREATED_USERS
SET SYSTEM_USER='-1'
WHERE user_name='SYSMAN' 2 3 ;
1 row updated.

SQL> commit;
Commit complete.

SQL> conn / as sysdba
Connected.
SQL> show user
USER is "SYS"

SQL> conn sysman
Enter password:
Connected.
SQL>

SQL>
SQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
You have new mail in /var/spool/mail/oracle

[oracle@houoemap2 bin]$ pwd
/app/oracle/product/middleware/middleware13c/bin

[oracle@houoemap2 bin]$ ./emcli login -username=sysman
Enter password :

Error: Login failed. Retry with correct hostname, port or username / password else check the log files for further details.
Log file location is : /app/oracle/product/middleware/gc_inst1/em/EMGC_OMS1/sysman/emcli/setup/.emcli/.emcli.log
[oracle@houoemap2 bin]$

 

Also after disabling SYSMAN from logging into console, you can re-enable it by executing:

UPDATE MGMT_CREATED_USERS 
SET SYSTEM_USER='1' 
WHERE user_name='SYSMAN'

This is small and quick Tip using which you can Secure you SYSMAN login in Oracle Enterprise Manager.

 

OMS Consuming High CPU due to ADF threads in OEM13c

In this Blog I have explained how I fixed one of the performance issues which was faced recently in my production OEM13c R2 environment.  From last couple of weeks we started facing performance issue, which resulted in degraded OEM13c Console performance.

Sudden increase in page response time was observed across all OMS, which was causing application to become unresponsive most of the time. Also the average time taken by any page to display data was over 30-40 seconds, which is not ideal at all.

Page processing time value across all OMS crossed the default warning threshold value and the maximum time hit the value of 96 seconds.

To troubleshoot further, I checked the EMGC_OMS logs and found below mentioned.

<Jan 15, 2019 2:41:53 PM EST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: ‘132’ for queue: ‘weblogic.kernel.Default (self-tuning)’ has been busy for “622” seconds working on the request “Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 622774 ms
“, which is more than the configured time (StuckThreadMaxTime) of “600” seconds in “server-failure-trigger”. Stack trace:
oracle.net.aso.o.a(Unknown Source)
oracle.net.aso.h.a(Unknown Source)
oracle.net.ano.CryptoDataPacket.receive(Unknown Source)
oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:306)
oracle.net.ns.NetInputStream.read(NetInputStream.java:250)
oracle.net.ns.NetInputStream.read(NetInputStream.java:172)
oracle.net.ns.NetInputStream.read(NetInputStream.java:90)

<Jan 15, 2019 2:41:57 PM EST> <Emergency> <oracle.dfw.incident> <BEA-000000> <incident 328 created with problem key “BEA-000337 [/empbs/upload]”>
<Jan 15, 2019 2:42:11 PM EST> <Warning> <org.apache.myfaces.trinidadinternal.agent.AgentFactoryImpl> <BEA-000000> <The User-Agent “EMCTL COMMAND: Oracle HTTPClient Version 10h” is unknown; creating an agent with “unknown” agent attributes.>
<Jan 15, 2019 2:42:53 PM EST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: ‘132’ for queue: ‘weblogic.kernel.Default (self-tuning)’ has been busy for “682” seconds working on the request “Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 682849 ms
“, which is more than the configured time (StuckThreadMaxTime) of “600” seconds in “server-failure-trigger”. Stack trace:
org.apache.log4j.Category.callAppenders(Category.java:204)
org.apache.log4j.Category.forcedLog(Category.java:391)
org.apache.log4j.Category.log(Category.java:826)

<Jan 15, 2019 2:47:11 PM EST> <Warning> <org.apache.myfaces.trinidadinternal.agent.AgentFactoryImpl> <BEA-000000> <The User-Agent “EMCTL COMMAND: Oracle HTTPClient Version 10h” is unknown; creating an agent with “unknown” agent attributes.>
<Jan 15, 2019 2:49:04 PM EST> <Warning> <org.apache.myfaces.trinidadinternal.agent.AgentFactoryImpl> <BEA-000000> <The User-Agent “EMCTL COMMAND: Oracle HTTPClient Version 10h” is unknown; creating an agent with “unknown” agent attributes.>
<Jan 15, 2019 2:49:47 PM EST> <Warning> <org.apache.myfaces.trinidadinternal.agent.AgentFactoryImpl> <BEA-000000> <The User-Agent “EMCTL COMMAND: Oracle HTTPClient Version 10h” is unknown; creating an agent with “unknown” agent attributes.>
<Jan 15, 2019 2:50:53 PM EST> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: ’72’ for queue: ‘weblogic.kernel.Default (self-tuning)’ has been busy for “610” seconds working on the request “Workmanager: default, Version: 0, Scheduled=true, Started=true, Started time: 611004 ms
“, which is more than the configured time (StuckThreadMaxTime) of “600” seconds in “server-failure-trigger”. Stack trace:
org.apache.log4j.Category.callAppenders(Category.java:204)
org.apache.log4j.Category.forcedLog(Category.java:391)
org.apache.log4j.Category.log(Category.java:826)

Referring the knowledge base at MOS, I came across this thread where it was found as a known BUG in EM 13c. Crosschecked EM 13c: OMS Java Process Consuming High CPU due to ADF threads ( Doc ID 2293299.1 )

And yes after checking the document, the log snippet indicated that we were hitting the similar issue. This issue is related to the OMS causing high cpu because of the internal ADF threads.

So to fix the issue I followed the steps as per the document Doc ID 2293299.1 and applied the recommended patch. The performance of OMS is back to normal now. I monitored the performance for next 3-4 days and the issue got fixed finally.

CAUSE :-  Issue is due to internal bugs on ADF framework

REFERENCES:-

BUG:25917466 – EM 13C: EMCTL DUMP OMS -INCLUDE_TOP DOES NOT COLLECT TOP CMD RESULTS CORRECTLY
BUG:27234155 – 13.2 OMS IS CONSUMING HIGH CPU DUE TO ADF THREADS

Hope this will help you as well in case you the similar issue.

Feel free to comment for any queries.