Wednesday 30 December 2015

Error while starting 12C OMS


Today i got the following error when i was trying to start 12C OMS, the error message was found in emctl log file location in <12c_install_home>/gc_inst/em/EMGC_OMS1/sysman/log/emctl.log, below was the error message :-

2015-12-30 14:41:24,136 [Thread-1] INFO wls.OMSController run.1744 - <OUT>Jython scans all the jar files it can find at first startup. Depending on the system, this process may take a few minutes to complete, and WLST may not return a prompt right away.
2015-12-30 14:41:24,136 [Thread-1] INFO wls.OMSController run.1744 - <OUT>
2015-12-30 14:41:24,771 [Thread-2] INFO wls.OMSController run.1744 - <ERR>*sys-package-mgr*: can't create package cache dir, '/tmpWLSTTemporacle/packages'
2015-12-30 14:41:45,764 [Thread-2] INFO wls.OMSController run.1744 - <ERR>java.io.IOException: Permission denied


# Interestingly the solution was to change the /tmp directory permission to 755 and make sure /tmp is readable and writeable. Sometimes what happens the permissions on tmp directory changes , and there is a 'Permission denied' message when trying to create the 'Packages' directory for the package cache directory.  In this case the directory is "/tmp/WLSTTemporacle' but could also be  /tmp/wlstTemp or /var/tmp/wlstTemp depending on Platform. Regardless of the platform the solution is the same i mentioned above.

Troubleshooting the 'Pending' status for a Listener Target in Enterprise Manager 12c and 13c Cloud Control


What Does the 'Pending' Status of a listener Mean?

# Sometimes what happens that the target has just been added and its status cannot be determined, not only this it can be some other issue as well, like for example due to some issue oms has not yet received the status of the target, so the Status 'pending' actually means that OMS has not yet received the 'availability' status of the target from the agent and the 'pending' status (clock icon) for a listener target should only be a temporary status (for example, when a new target is added or if a target has been under blackout and has just ended the blackout period). A listener target with a long status pending period should be investigated.  In 12.1.0.4 onwards additional status types were included for example 'Status pending (Target Addition in Progress)'

If a listener is showing as status 'pending' in Cloud Control, it means that the 'Response' metric has not been able to determine whether the listener is down or up. Sometimes even though a listener is shown as status 'pending' in the console, the agent has evaluated the status of the listener correctly but this status has not been received by the OMS.  Sometimes however, the agent is not able to evaluate the status correctly (eg. in the case of target being marked as 'broken', or target not existing at agent end).

In order to obtain the status of the listener target, the 'Response' metric is run every 5 minutes by the agent. The response metric runs an OS script called lsnrresp.pl which is found in <install home of agent>/plugins/oracle.sysman.db.agent.plugin_<pluginversion>/scripts. The lsnrresp.pl script performs a tnsping to the listener machine/port and confirms that the listener has been started with the 'correct' listener.ora file (the "correct" listener.ora file is that listener.ora file shown in "listener.ora location" (on the listener home page).  The listener target in Cloud Control is tied to a particular listener.ora file and directory (unlike previous versions of Enterprise Manager).  This means that for example, a listener in $TNS_ADMIN and a listener in $ORACLE_HOME/network/admin cannot be swapped between.


There is a very useful metalink i found and it is (Troubleshooting the 'Pending' status for a Listener Target in Enterprise Manager 12c and 13c Cloud Control (Doc ID 1546576.1))