Daily Archives: February 20, 2015

OMS failed to start “Connection to Repository Failed”

Hi,

Recently one issue blew my mind off, when all of a sudden my OEM test environment failed to startup. Since it was running fine earlier but this time when I start OMS using the command “>emctl start oms” it failed.

When I explored the logs I came across that some has changed the repository credentials for SYSMAN, which was done at the backend and was not synchronized with the OMS. This led the OMS not to connect with the repository. Something like this popped up in the logs.

“failureReason: DB Connection service is invalid or temporarily unavailable” and “ERROR commands.BaseCommand logAndPrint.623 – Oracle Management Server is not functioning because of the following reason:
Connection to the repository failed. Verify that the repository connection information provided is correct.”

Key Point to remember is that “SYSMAN” password should not be changed directly in repository database. The ONLY way to change SYSMAN password is using emctl command. So next thing I tried was reset SYSMAN password using “emctl” command.

[oracle@houoemap1 bin]$ ./emctl config oms -change_repos_pwd -use_sys_pwd -sys_pwd formula123 -new_pwd formula123
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
java.io.IOException
Error occurred. Check the log /app/oracle/product/middleware/gc_inst/em/EMGC_OMS1/sysman/log/secure.log

Which Failed again 🙁

I checked the logs and find that my “Admin Server” was not up and running but even after bringing Admin Server up I was not allowed to change password.

2015-02-10 07:03:32,915 [main] INFO oms.AdminCredsWalletUtil setInstanceHome.177 – Getting credentials from wallet
2015-02-10 07:03:33,218 [main] INFO oms.AdminCredsWalletUtil setInstanceHome.192 – Read the credentials from wallet
2015-02-10 07:03:33,221 [main] INFO util.EmctlUtil logp.251 – Connecting over t3s to: houoemap1.xxxxxx.com/7102 using id: weblogic
2015-02-10 07:03:33,851 [main] INFO util.EmctlUtil logp.251 – Unable to get mbean conn over t3s :null
2015-02-10 07:03:33,854 [main] ERROR oms.ChangeReposPwd logp.251 –

at weblogic.management.remote.common.ClientProviderBase.makeConnection(ClientProviderBase.java:178)
… 7 more
Caused by: java.net.ConnectException: t3s://houoemap1.xxxxxx.com:7102: Destination unreachable; nested exception is:
java.net.ConnectException: Connection refused; No available router to destination

I started Admin server using ” >emctl start oms -admin_only”

Followed by this I again tried to reset the sysman password using “EMCTL” but again no luck 🙁

[oracle@houoemap1 bin]$ ./emctl config oms -change_repos_pwd -use_sys_pwd -sys_pwd formula123 -new_pwd formula1
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
java.lang.NullPointerException
Error occurred. Check the log /app/oracle/product/middleware/gc_inst/em/EMGC_OMS1/sysman/log/secure.log
[oracle@houoemap1 bin]$

Now I had to open case with MOS and they guided me to reset the password of all four dependent users “sysman, sysman_apm, sysman_mds, sysman_opss” to common password and then try to start oms, so I did.

Reset Password—————

[oracle@houoemap1 db_1]$ ss
SQL*Plus: Release 11.2.0.2.0 Production on Fri Feb 13 02:41:33 2015
Copyright (c) 1982, 2010, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 – 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> conn sysman/formula1
Connected.
SQL> conn sysman_apm/formula1
Connected.
SQL> conn sysman_mds/formula1
Connected.
SQL> conn sysman_opss/formula1
Connected.

Start OMS —————-

[oracle@houoemap1 bin]$ ./emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Starting Oracle Management Server…
Starting WebTier…
WebTier Successfully Started
Oracle Management Server is not functioning because of the following reason:
Unexpected error occurred. Check error and log files.
Check the following log files:
EM log files: /app/oracle/product/middleware/gc_inst/em/EMGC_OMS1/sysman/log/emctl.log, emoms.trc, emoms_pbs.trc
[oracle@houoemap1 bin]$

Providing logs to Oracle Support we came to know that there is mismatch of the information in JDBC xml files.

Used “/app/oracle/product/middleware/gc_inst/user_projects/domains/GCDomain/config/jdbc/*” to find details.

Finally below action plan worked.

—————————————————————————-

1. Take a backup of the directory /app/oracle/product/middleware/gc_inst/user_projects/domains/GCDomain/config/jdbc
2. Access weblogic admin console. https://houoemap1.xxxxxx.com/7102/console
Enter ‘weblogic’ as the username and its password.
If the weblogic admin server is not up, then start it using:
OMS_HOME/bin>./emctl start oms -admin_only
3. From the admin console, Leftpanel > Services > Datasources
Here you will have list of all datasources. Click on “Lock and Edit” on top corner left side. Click on Datasources one by one.
For example: Click on emgc-sysman-pool > Connection Pool tab > Enter Password and Confirm Password as formula1
And Save each change.
Please repeat this for all the datasources.
4. Click on “Release Configuration” (Under Lock and Edit) after the changes are done.
5. Now restart the OMS with
$emctl stop oms -all -force
$emctl start oms

If you face this Issue ever, this Action plan will help you to get the Issue resolved.

Regards

Deepak