ORACLE EM DBA

Problems, Solutions, Test Cases from my professional and personal experience in technologies like Oracle Enterprise Manager 13 c,Oracle Enterprise Manager 12c, Oracle Configuration Manager and some other stuff.

OMS decided to shutdown the agent : EM_PLUGIN_MISMATCH_AND_AGENT_NOT_YET_MANAGED

Yesterday I came across an issue when I was installing Oracle Agent 12cR3 on my TEST  environment,  the installation ended with a new error message for me this time. It was for the first time in 3 years while working on Oracle Enterprise Manager I faced this Issue.

This was the error message:- ” ERROR: Agent Configuration Failed SEVERE: emctl start agent command has failed with status1

I have added the complete installation logs to make picture more clear.

[oracle@testdb1 linux_x64]$ ./agentDeploy.sh OMS_HOST=testdb1.oii.test.com EM_UPLOAD_PORT=4903 AGENT_BASE_DIR=/app/oracle/product/agent12c AGENT_REGISTRATION_PASSWORD=formula123

Validating the OMS_HOST & EM_UPLOAD_PORT
Executing command : /app/oracle/product/agent12c/core/12.1.0.4.0/jdk/bin/java -classpath /app/oracle/product/agent12c/core/12.1.0.4.0/jlib/agentInstaller.jar:/app/oracle/product/agent12c/core/12.1.0.4.0/oui/jlib/OraInstaller.jar oracle.sysman.agent.installer.AgentInstaller /app/oracle/product/agent12c/core/12.1.0.4.0 /app/oracle/product/agent_sw/linux_x64 /app/oracle/product/agent12c -prereq
Validating oms host & port with url: http://testdb1.oii.test.com:4903/empbs/genwallet
Validating oms host & port with url: https://testdb1.oii.test.com:4903/empbs/genwallet
Return status:3-oms https port is passed
Unzipping the agentcoreimage.zip to /app/oracle/product/agent12c ....
12.1.0.4.0_PluginsOneoffs_226.zip
Executing command : /app/oracle/product/agent_sw/linux_x64/unzip -o /app/oracle/product/agent_sw/linux_x64/12.1.0.4.0_PluginsOneoffs_226.zip -d /app/oracle/product/agent12c
Executing command : /app/oracle/product/agent12c/core/12.1.0.4.0/jdk/bin/java -classpath /app/oracle/product/agent12c/core/12.1.0.4.0/oui/jlib/OraInstaller.jar:/app/oracle/product/agent12c/core/12.1.0.4.0/oui/jlib/xmlparserv2.jar:/app/oracle/product/agent12c/core/12.1.0.4.0/oui/jlib/srvm.jar:/app/oracle/product/agent12c/core/12.1.0.4.0/oui/jlib/emCfg.jar:/app/oracle/product/agent12c/core/12.1.0.4.0/jlib/agentInstaller.jar:/app/oracle/product/agent12c/core/12.1.0.4.0/oui/jlib/share.jar oracle.sysman.agent.installer.AgentInstaller /app/oracle/product/agent12c/core/12.1.0.4.0 /app/oracle/product/agent_sw/linux_x64 /app/oracle/product/agent12c /app/oracle/product/agent12c/agent_inst AGENT_BASE_DIR=/app/oracle/product/agent12c
Executing agent install prereqs...
Executing command: /app/oracle/product/agent12c/core/12.1.0.4.0/oui/bin/runInstaller -debug -ignoreSysPrereqs   -prereqchecker -silent -ignoreSysPrereqs -waitForCompletion  -prereqlogloc /app/oracle/product/agent12c/core/12.1.0.4.0/cfgtoollogs/agentDeploy -entryPoint oracle.sysman.top.agent_Complete -detailedExitCodes PREREQ_CONFIG_LOCATION=/app/oracle/product/agent12c/core/12.1.0.4.0/prereqs  -J-DAGENT_BASE_DIR=/app/oracle/product/agent12c
Agent install prereqs completed successfully
Cloning the agent home...
Executing command: /app/oracle/product/agent12c/core/12.1.0.4.0/oui/bin/runInstaller -debug -ignoreSysPrereqs   -clone -forceClone -silent -waitForCompletion -nowait ORACLE_HOME=/app/oracle/product/agent12c/core/12.1.0.4.0   AGENT_BASE_DIR=/app/oracle/product/agent12c OMS_HOST=testdb1.oii.test.com EM_UPLOAD_PORT=4903 AGENT_BASE_DIR=/app/oracle/product/agent12c AGENT_REGISTRATION_PASSWORD=formula123 -noconfig  ORACLE_HOME_NAME=agent12c3 -force b_noUpgrade=true AGENT_PORT=-1
Cloning of agent home completed successfully
Attaching sbin home...
Executing command: /app/oracle/product/agent12c/core/12.1.0.4.0/oui/bin/runInstaller -debug -ignoreSysPrereqs   -attachHome -waitForCompletion -nowait ORACLE_HOME=/app/oracle/product/agent12c/sbin ORACLE_HOME_NAME=sbin12c3 -force
Attach home for sbin home completed successfully.
Updating home dependencies...
Executing command: /app/oracle/product/agent12c/core/12.1.0.4.0/oui/bin/runInstaller -debug -ignoreSysPrereqs   -updateHomeDeps -waitForCompletion HOME_DEPENDENCY_LIST={/app/oracle/product/agent12c/sbin:/app/oracle/product/agent12c/core/12.1.0.4.0} -invPtrLoc /app/oracle/product/agent12c/core/12.1.0.4.0/oraInst.loc -force
Update home dependency completed successfully.
Executing command: /app/oracle/product/agent12c/core/12.1.0.4.0/oui/bin/runConfig.sh ORACLE_HOME=/app/oracle/product/agent12c/core/12.1.0.4.0 RESPONSE_FILE=/app/oracle/product/agent12c/core/12.1.0.4.0/agent.rsp ACTION=configure MODE=perform COMPONENT_XML={oracle.sysman.top.agent.11_1_0_1_0.xml} RERUN=true

ERROR: Agent Configuration Failed SEVERE: emctl start agent command has failed with status1
Agent Deploy Log Location:/app/oracle/product/agent12c/core/12.1.0.4.0/cfgtoollogs/agentDeploy/agentDeploy_2016-08-04_22-40-59-PM.log

[oracle@testdb1 linux_x64]$

I checked the log file as mentioned in the last line and this I what I came across.

——————————————————————————————————————————————

INFO: Executing command: /app/oracle/product/agent12c/core/12.1.0.4.0/oui/bin/runConfig.sh ORACLE_HOME=/app/oracle/product/agent12c/core/12.1.0.4.0 RESPONSE_FILE=/app/oracle/product/agent12c/core/12.1.0.4.0/agent.rsp ACTION=configure MODE=perform COMPONENT_XML={oracle.sysman.top.agent.11_1_0_1_0.xml} RERUN=true
INFO: Setting the invPtrLoc to /app/oracle/product/agent12c/core/12.1.0.4.0/oraInst.loc
INFO:
INFO: perform - mode is starting for action: configure
INFO:
INFO: ** Agent Port Check completed successfully.**
INFO: SEVERE: emctl start agent command has failed with status1
INFO: Follow the below steps to manually start the agent and add the host and agent targets:
INFO: 1. Start the agent by executing the command: /app/oracle/product/agent12c/agent_inst/bin/emctl start agent
INFO:  2. Add the targets by executing the command:/app/oracle/product/agent12c/agent_inst/bin/emctl config agent addinternaltargets.
INFO:
INFO: perform - mode finished for action: configure
INFO:
INFO: You can see the log file: /app/oracle/product/agent12c/core/12.1.0.4.0/cfgtoollogs/oui/configActions2016-08-04_10-41-54-PM.log
INFO: Plugin homes:
INFO: Plugin homes:
INFO: /app/oracle/product/agent12c/core/12.1.0.4.0/oui/bin/runConfig.sh ORACLE_HOME=/app/oracle/product/agent12c/core/12.1.0.4.0 RESPONSE_FILE=/app/oracle/product/agent12c/core/12.1.0.4.0/agent.rsp ACTION=configure MODE=perform COMPONENT_XML={oracle.sysman.top.agent.11_1_0_1_0.xml} RERUN=true completed with status=3
SEVERE:
ERROR: Agent Configuration Failed
SEVERE:
ERROR: Agent Deployment Failed

 

I followed the steps as mentioned in the log file and tried to start the agent. It failed again, to be honest I was expecting that.   😉

My Concern:-

Same setup is working fine for my production OEM why it is failing again and again for Test OEM environment. I used the same setup to install and configure agent on multiple servers which were being monitored by my PRODUCTION OEM12c and they were installed and configured successfully without any errors.

So why now it was failing for OMS in my TEST OEM12c environment.

Then last option left for me was to explore the trace files and see what exactly is causing issue. So I started digging deeper in the log and trace files until I reached common error message in all of them.


 

My findings.

— EMState agent

----------------------------------------

----- Thu Aug  4 23:01:15 2016::27430::Auto tuning the agent at time Thu Aug  4 23:01:15 2016 -----
----- Thu Aug  4 23:01:16 2016::27430::Finished auto tuning the agent at time Thu Aug  4 23:01:16 2016 -----
----- Thu Aug  4 23:01:16 2016::27430::Launching the JVM with following options: -Xmx128M -server -Djava.security.egd=file:///dev/./urandom -Dsun.lang.ClassLoader.allowArraySyntax=true -XX:+UseLinuxPosixThreadCPUClocks -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+UseCompressedOops -----
----- Thu Aug  4 23:01:16 2016::27430::Agent Launched with PID 27484 at time Thu Aug  4 23:01:16 2016 -----
----- Thu Aug  4 23:01:16 2016::27484::Time elapsed between Launch of Watchdog process and execing EMAgent is 1 secs -----
----- Thu Aug  4 23:01:16 2016::27430::Previous Thrash State(-1,-1) -----
2016-08-04 23:01:16,750 [1:main] WARN - Missing filename for log handler 'wsm'
2016-08-04 23:01:16,757 [1:main] WARN - Missing filename for log handler 'opss'
2016-08-04 23:01:16,759 [1:main] WARN - Missing filename for log handler 'opsscfg'
OMS decided to shutdown the agent because of the following reason sent from OMS:  EM_PLUGIN_MISMATCH_AND_AGENT_NOT_YET_MANAGED
----- Thu Aug  4 23:01:29 2016::27430::Checking status of EMAgent : 27484 -----
----- Thu Aug  4 23:01:29 2016::27430::EMAgent exited at Thu Aug  4 23:01:29 2016 with return value 0. -----
----- Thu Aug  4 23:01:29 2016::27430::writeAbnormalExitTimestampToAgntStmp: exitCause=NORMAL : restartRequired=0 -----
----- Thu Aug  4 23:01:29 2016::27430::EMAgent was shutdown normally. -----
----- Thu Aug  4 23:01:29 2016::27430::Exiting watchdog loop
-----

gcagent.log

----------------------
2016-08-04 23:01:19,441 [49:905DEBE9] WARN - improper ping interval (EM_PING_NOTIF_RESPONSE: SHUTDOWN EM_PLUGIN_MISMATCH_AND_AGENT_NOT_YET_MANAGED)
2016-08-04 23:01:19,443 [57:CB968A7D:GC.SysExecutor.2 (ShutdownFromPing)] INFO - OMS decided to shutdown the agent because of the following reason sent from OMS:  EM_PLUGIN_MISMATCH_AND_AGENT_NOT_YET_MANAGED
2016-08-04 23:01:19,443 [1:3305B9] INFO - Agent is shutting down
2016-08-04 23:01:19,444 [49:905DEBE9] WARN - improper ping interval (EM_PING_NOTIF_RESPONSE: SHUTDOWN EM_PLUGIN_MISMATCH_AND_AGENT_NOT_YET_MANAGED)
2016-08-04 23:01:19,444 [1:3305B9] INFO - Invoking SHUTDOWN (1) on Internal Metrics
2016-08-04 23:01:19,444 [1:3305B9] INFO - Done: SHUTDOWN (1) on Internal Metrics
2016-08-04 23:01:19,444 [1:3305B9] INFO - Invoking SHUTDOWN on Miscellaneous
2016-08-04 23:01:19,444 [1:3305B9] INFO - Done: SHUTDOWN on Miscellaneous


I gave one try more but this time with proper procedure. So I followed my own document (blog) for “Silent Installation of Agent” and this time it worked and agent was installed and configured successfully.

What was the issue ? as OMS mentioned in the Error logs.  “ OMS decided to shutdown the agent because of the following reason sent from OMS:  EM_PLUGIN_MISMATCH_AND_AGENT_NOT_YET_MANAGED

SOLUTION:- 

The setup which I used earlier was downloaded from the production environment this was the compatible with EM Plugin in the production not with test environment and this was the reason why it was giving error again and again.

When I downloaded agent from the test environment and deployed it got configured successfully.

 

Procedure is mentioned below:-

————————————————

[oracle@testdb1 bin]$ pwd
/app/oracle/product/middleware/OMS12CR4/oms/bin
[oracle@testdb1 bin]$ ./emcli login -username=sysman -password=formula1
Login successful

[oracle@testdb1 bin]$ ./emcli sync
Synchronized successfully

[oracle@testdb1 bin]$ ./emcli get_supported_platforms
-----------------------------------------------
Version = 12.1.0.3.0
Platform = Linux x86-64
-----------------------------------------------
Version = 12.1.0.4.0
Platform = Linux x86-64
-----------------------------------------------
Platforms list displayed successfully.

[oracle@testdb1 bin]$ ./emcli get_agentimage -destination=/tmp/deepak -platform="Linux x86-64" -version=12.1.0.3.0

=== Partition Detail ===
Space free : 3 GB
Space required : 1 GB
Check the logs at /app/oracle/product/middleware/gc_inst/em/EMGC_OMS1/sysman/emcli/setup/.emcli/get_agentimage_2016-08-05_00-57-38-AM.log
Downloading /tmp/deepak/12.1.0.3.0_AgentCore_226.zip
File saved as /tmp/deepak/12.1.0.3.0_AgentCore_226.zip
Downloading /tmp/deepak/12.1.0.3.0_PluginsOneoffs_226.zip
File saved as /tmp/deepak/12.1.0.3.0_PluginsOneoffs_226.zip
Downloading /tmp/deepak/unzip
File saved as /tmp/deepak/unzip
Agent Image Download completed successfully.
[oracle@testdb1 bin]$ pwd

[oracle@testdb1 agent_sw]$ ./agentDeploy.sh AGENT_BASE_DIR=/app/oracle/product/agent12cr3 OMS_HOST=testdb1.oii.test.com EM_UPLOAD_PORT=4903 AGENT_REGISTRATION_PASSWORD=formula123

Validating the OMS_HOST & EM_UPLOAD_PORT
Executing command : /app/oracle/product/agent12cr3/core/12.1.0.3.0/jdk/bin/java -classpath /app/oracle/product/agent12cr3/core/12.1.0.3.0/jlib/agentInstaller.jar:/app/oracle/product/agent12cr3/core/12.1.0.3.0/oui/jlib/OraInstaller.jar oracle.sysman.agent.installer.AgentInstaller /app/oracle/product/agent12cr3/core/12.1.0.3.0 /app/oracle/product/agent_sw /app/oracle/product/agent12cr3 -prereq
Validating oms host & port with url: http://testdb1.oii.test.com:4903/empbs/genwallet
Validating oms host & port with url: https://testdb1.oii.test.com:4903/empbs/genwallet
Return status:3-oms https port is passed
Unzipping the agentcoreimage.zip to /app/oracle/product/agent12cr3 ....
12.1.0.3.0_PluginsOneoffs_226.zip
Executing command : /app/oracle/product/agent_sw/unzip -o /app/oracle/product/agent_sw/12.1.0.3.0_PluginsOneoffs_226.zip -d /app/oracle/product/agent12cr3
Executing command : /app/oracle/product/agent12cr3/core/12.1.0.3.0/jdk/bin/java -classpath /app/oracle/product/agent12cr3/core/12.1.0.3.0/oui/jlib/OraInstaller.jar:/app/oracle/product/agent12cr3/core/12.1.0.3.0/oui/jlib/xmlparserv2.jar:/app/oracle/product/agent12cr3/core/12.1.0.3.0/oui/jlib/srvm.jar:/app/oracle/product/agent12cr3/core/12.1.0.3.0/oui/jlib/emCfg.jar:/app/oracle/product/agent12cr3/core/12.1.0.3.0/jlib/agentInstaller.jar:/app/oracle/product/agent12cr3/core/12.1.0.3.0/oui/jlib/share.jar oracle.sysman.agent.installer.AgentInstaller /app/oracle/product/agent12cr3/core/12.1.0.3.0 /app/oracle/product/agent_sw /app/oracle/product/agent12cr3 /app/oracle/product/agent12cr3/agent_inst AGENT_BASE_DIR=/app/oracle/product/agent12cr3

Executing agent install prereqs...
Executing command: /app/oracle/product/agent12cr3/core/12.1.0.3.0/oui/bin/runInstaller -debug   -ignoreSysPrereqs -prereqchecker -silent -ignoreSysPrereqs -waitForCompletion  -prereqlogloc /app/oracle/product/agent12cr3/core/12.1.0.3.0/cfgtoollogs/agentDeploy -entryPoint oracle.sysman.top.agent_Complete -detailedExitCodes PREREQ_CONFIG_LOCATION=/app/oracle/product/agent12cr3/core/12.1.0.3.0/prereqs  -J-DAGENT_BASE_DIR=/app/oracle/product/agent12cr3
Prereq Logs Location:/app/oracle/product/agent12cr3/core/12.1.0.3.0/cfgtoollogs/agentDeploy/prereq<timestamp>.log
Agent install prereqs completed successfully

Cloning the agent home...
Executing command: /app/oracle/product/agent12cr3/core/12.1.0.3.0/oui/bin/runInstaller -debug   -ignoreSysPrereqs -clone -forceClone -silent -waitForCompletion -nowait ORACLE_HOME=/app/oracle/product/agent12cr3/core/12.1.0.3.0   AGENT_BASE_DIR=/app/oracle/product/agent12cr3 AGENT_BASE_DIR=/app/oracle/product/agent12cr3 OMS_HOST=testdb1.oii.test.com EM_UPLOAD_PORT=4903 AGENT_REGISTRATION_PASSWORD=formula123 -noconfig  ORACLE_HOME_NAME=agent12c4 -force b_noUpgrade=true AGENT_PORT=-1
Clone Action Logs Location:/app/oracle/oraInventory//logs/cloneActions<timestamp>.log
Cloning of agent home completed successfully

Attaching sbin home...
Executing command: /app/oracle/product/agent12cr3/core/12.1.0.3.0/oui/bin/runInstaller -debug   -ignoreSysPrereqs -attachHome -waitForCompletion -nowait ORACLE_HOME=/app/oracle/product/agent12cr3/sbin ORACLE_HOME_NAME=sbin12c4 -force
Attach Home Logs Location:/app/oracle/product/agent12cr3/core/12.1.0.3.0/cfgtoollogs/agentDeploy/AttachHome<timestamp>.log
Attach home for sbin home completed successfully.

Updating home dependencies...
Executing command: /app/oracle/product/agent12cr3/core/12.1.0.3.0/oui/bin/runInstaller -debug   -ignoreSysPrereqs -updateHomeDeps -waitForCompletion HOME_DEPENDENCY_LIST={/app/oracle/product/agent12cr3/sbin:/app/oracle/product/agent12cr3/core/12.1.0.3.0} -invPtrLoc /app/oracle/product/agent12cr3/core/12.1.0.3.0/oraInst.loc -force
Update Home Dependencies Location:/app/oracle/product/agent12cr3/core/12.1.0.3.0/cfgtoollogs/agentDeploy/UpdateHomeDeps<timestamp>.log
Update home dependency completed successfully.

Performing the agent configuration...
Executing command: /app/oracle/product/agent12cr3/core/12.1.0.3.0/oui/bin/runConfig.sh ORACLE_HOME=/app/oracle/product/agent12cr3/core/12.1.0.3.0 RESPONSE_FILE=/app/oracle/product/agent12cr3/core/12.1.0.3.0/agent.rsp ACTION=configure MODE=perform COMPONENT_XML={oracle.sysman.top.agent.11_1_0_1_0.xml} RERUN=true
Configuration Log Location:/app/oracle/product/agent12cr3/core/12.1.0.3.0/cfgtoollogs/cfgfw/CfmLogger<timestamp>.log
Agent Configuration completed successfully

The following configuration scripts need to be executed as the "root" user.
#!/bin/sh
#Root script to run
/app/oracle/product/agent12cr3/core/12.1.0.3.0/root.sh
To execute the configuration scripts:
  1. Open a terminal window
  2. Log in as "root"
  3. Run the scripts
Agent Deployment Successful.
Agent deployment log location:
/app/oracle/product/agent12cr3/core/12.1.0.3.0/cfgtoollogs/agentDeploy/agentDeploy_2016-08-05_01-06-08-AM.log
Agent deployment completed successfully.

[oracle@testdb1 agent_sw]$ cd /app/oracle/product/agent12cr3/agent_inst/bin/
[oracle@testdb1 bin]$ ./emctl secure agent
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Agent successfully stopped...   Done.
Securing agent...   Started.
Enter Agent Registration Password :
Agent successfully restarted...   Done.
EMD gensudoprops completed successfully
Securing agent...   Successful.

[oracle@testdb1 bin]$ ./emctl config agent addinternaltargets
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.

[oracle@testdb1 bin]$ ./emctl upload
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully
[oracle@testdb1 bin]$

This time as well I learnt something new and I thought of sharing it with you so I did. Hope this will help you some time in future.

Thanks
ADDKS

1 Comment

  1. Ankush Sharma

    March 3, 2017 at 7:53 pm

    Thanks Deepak. Nice Info.Was facing the same issue.

Leave a Reply

Your email address will not be published.

*

© 2017 ORACLE EM DBA

Theme by Anders NorenUp ↑