Step-by-Step: Oracle Enterprise Manager(OEM) 13c Upgrade from 13.2 to 13.4
Updated: Apr 30, 2022
#oem13c #upgrade #oem13.2 #oem13.4 #oracle
“…Flash-based content will be blocked from running in Adobe Flash Player after the EOL Date.” from Adobe EOL page:
So Oracle support has released a MOS note on the same as described below .(Snippet taken from Oracle Support Note)
Adobe Flash End-of-Life Impact on Oracle Enterprise Manager Cloud Control (Doc ID 2709724.1)
Oracle Enterprise Manager’s database performance pages were initially written using Adobe Flash technology.
These database performance pages included the ‘Performance Home’, ‘Top Activity’, ‘ASH Analytics’, ‘Real-time SQL Monitoring’ among many others.
Oracle recommends that customers update their Oracle Enterprise Manager Cloud Control to version 13.4 before the Adobe Flash EOL.
Oracle also recommends that customers apply the latest release updates available to their Enterprise Manager
So here is a step by step guide to upgrade OEM from 13.2 to 13.4 .
Upgrade of OEM 13.2 to 13.4 involves below steps:
1- Software download
2- Prerequisite check
3- OEM 13.4 installation and upgrade
4- Agent Upgrade form OEM console
5- Removal of Old OMS homes
1- Software download :
As upgrade of OEM 13c to 13.4 is a new installation ,we need the binary of 13.4 version .
Use the link here and download the softwares .
2-Prerequisite check :
1- Check the current OEM details and status in the OMS server .
[oracle@oemnode1] echo $OMS_HOME
/u01/em/oms/middleware
[oracle@oemnode1]$ cd $OMS_HOME/bin
[oracle@oemnode1 bin]$ ./emctl status oms
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
WebTier is Up
Oracle Management Server is Up
JVMD Engine is Up
BI Publisher Server is Up
[oracle@oemnode1 bin]$ ./emctl getversion oms
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
Oracle Enterprise Manager Cloud Control OMS 13c Release 2
2- Check the Agent status :
[oracle@oemnode1 bin]$ echo $AGENT_HOME
/u01/em/oms/agent/agent_inst
[oracle@oemnode1 bin]$ cd $AGENT_HOME/bin
[oracle@pr-oemgcc1-ivl bin]$ ./emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 13.2.0.0.0
OMS Version : 13.2.0.0.0
Protocol Version : 12.1.0.1.0
Agent Home : /u01/em/oms/agent/agent_inst
Agent Log Directory : /u01/em/oms/agent/agent_inst/sysman/log
Agent Binaries : /u01/em/oms/agent/agent_13.2.0.0.0
Core JAR Location : /u01/em/oms/agent/agent_13.2.0.0.0/jlib
Agent Process ID : 2717
Parent Process ID : 1974
Agent URL : https://oemnode1:3872/emd/main/
Local Agent URL in NAT : https://oemnode1:3872/emd/main/
Repository URL : https://oemnode1:4903/empbs/upload
Started at : 2021-04-01 05:10:44
Started by user : oracle
Operating System : Linux version 3.10.0-693.el7.x86_64 (amd64)
Number of Targets : 34
Last Reload : (none)
Last successful upload : 2021-03-22 13:04:50
Last attempted upload : 2021-03-22 13:04:50
Total Megabytes of XML files uploaded so far : 72.89
Number of XML files pending upload : 0
Size of XML files pending upload(MB) : 0
Available disk space on upload filesystem : 34.15%
Collection Status : Collections enabled
Heartbeat Status : Ok
Last attempted heartbeat to OMS : 2021-03-22 13:08:44
Last successful heartbeat to OMS : 2021-03-22 13:08:44
Next scheduled heartbeat to OMS : 2021-03-22 13:09:44
---------------------------------------------------------------
Agent is Running and Ready
[oracle@oemnode1 bin]$ ./emctl getversion agent
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
Oracle Enterprise Manager Cloud Control Agent 13c Release 2
3- Check database version and PSU :
SQL> select version from sysman.mgmt_versions where component_name = 'CORE';
VERSION
--------------------------------
13.2.0.0.0
SQL> select * from v$version ;
BANNER CON_ID
-------------------------------------------------------------------------------- ----------
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production 0
PL/SQL Release 12.1.0.2.0 - Production 0
CORE 12.1.0.2.0 Production 0
TNS for Linux: Version 12.1.0.2.0 - Production 0
NLSRTL Version 12.1.0.2.0 - Production 0
4-Check the latest patch is applied in database home or not :
Oracle Database - Overview of Database Patch Delivery Methods - 12.1.0.2 and older (Doc ID 1962125.1)
Check if the latest patch has been applied on to the oracle home or not .If not apply the latest patch .
SQL> set lines 200
SQL> set pages 200
SQL> col DESCRIPTION for a70
SQL> SELECT patch_id,DESCRIPTION,version, status, bundle_id, bundle_series FROM dba_registry_sqlpatch;
PATCH_ID DESCRIPTION VERSION STATUS BUNDLE_ID BUNDLE_SERIES
31985579 DATABASE PATCH SET UPDATE 12.1.0.2.210119 12.1.0.2 SUCCESS 210119 PSU
5- EMKEY Copy Requirements :
export OMS_HOME=/u01/em/oms/middleware
$OMS_HOME/bin/emctl config emkey -copy_to_repos -sysman_pwd <>
$OMS_HOME/bin/emctl status emkey -sysman_pwd <>
[oracle@oemnode1 bin]$ ./emctl status emkey
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
The EMKey is configured properly.
[oracle@oemnode1 bin]$ ./emctl config emkey -copy_to_repos
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
The EMKey has been copied to the Management Repository. This operation will cause the EMKey to become unsecure.
After the required operation has been completed, secure the EMKey by running "emctl config emkey -remove_from_repos".
6- Optimizer Adaptive Features Disabling Requirements :
The OPTIMIZER_ADAPTIVE_FEATURES is not available in 12.2 DB onward and with DB 12.1.0.2 (if a patch is applied). For improved SQL performance all the adaptive features parameters should be unset prior to the upgrade.
NOTE: Before any parameter modification in database make sure to keep a backup of the parameter files .
Case A) If the Management Repository is using Oracle Database 12.1.0.2.0 and patch 22652097 has NOT been applied, ensure that you disable the optimizer adaptive features by connecting to the database as SYSDBA and running the following:
SQL> alter system set optimizer_adaptive_features=false scope=both
Case B) If the Management Repository is using Oracle Database 12.1.0.2 with patch 22652097 applied or the Management Repository is using Oracle Database 12.2, 18.x or 19.x, you need to do the following:
alter system set "_optimizer_nlj_hj_adaptive_join"= FALSE scope=both sid='*';
alter system set "_optimizer_strans_adaptive_pruning" = FALSE scope=both sid='*';
alter system set "_px_adaptive_dist_method" = OFF scope=both sid='*';
alter system set "_sql_plan_directive_mgmt_control" = 0 scope=both sid='*';
alter system set "_optimizer_dsdir_usage_control" = 0 scope=both sid='*';
alter system set "_optimizer_use_feedback" = FALSE scope=both sid='*';
alter system set "_optimizer_gather_feedback" = FALSE scope=both sid='*';
alter system set "_optimizer_performance_feedback" = OFF scope=both sid='*';
alter system set “ _allow_insert_with_update_check”=true scope=both;
alter system set job_queue_processes=0 scope=both;
alter system set optimizer_adaptive_features=false scope=both;Restart the repository database.
SQL> shu immediate ;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.
Total System Global Area 9663676416 bytes
Fixed Size 3720600 bytes
Variable Size 1107298920 bytes
Database Buffers 8522825728 bytes
Redo Buffers 29831168 bytes
Database mounted.
Database opened.
7-Check the deb level OMS plugins mismatch in OEM repository database .
Verify that all the OMS (including Standby) have the same version of Plugins deployed. Execute the following in the repository database, connected as SYSMAN user:
set serveroutput on
declare
oms_guids mgmt_medium_string_array := mgmt_medium_string_array();
oms_names mgmt_medium_string_array := mgmt_medium_string_array();
mismatches smp_emd_nvpair_array := smp_emd_nvpair_array();
begin
select host_url, em_plugin_util.get_guid_from_name(unique_oms_name)
bulk collect into oms_names, oms_guids
from gc_oms_info;
for j in 1..oms_guids.count
loop
dbms_output.put_line('Checking mismatch for OMS ' || oms_names(j));
mismatches := smp_emd_nvpair_array();
em_plugin_inventory.get_plugin_mismatch(oms_guids(j), mismatches);
if mismatches is null or mismatches.count = 0
then
dbms_output.put_line('No mismatch found for OMS ' || oms_names(j));
else
dbms_output.put_line('Mismatch found for OMS ' || oms_names(j));
for i in 1..mismatches.count
loop
dbms_output.put_line('Plug-in on repository: ' || nvl(mismatches(i).name, 'None') ||
', Plug-in on OMS: ' || nvl(mismatches(i).value, 'None'));
end loop;
end if;
end loop;
end;
/
8- Check if there are large number of jobs that are stuck:
If you have a large number of PropogateTarget jobs, then the Repository Upgrade can consume very long time or stuck while executing "upgrade/post_jobtype_upgrade.sql".
Run the below query to get the job count:
SELECT job_type, status, cnt
FROM (
SELECT job_type, status, count(*) as cnt
FROM MGMT_JOB_EXEC_SUMMARY join MGMT_JOB_TYPE_INFO using (job_type_id)
WHERE status NOT IN(3,4,5,8,18,19,23)
group by job_type, status
order by 3, 1, 2
)
where cnt > 10000;If jobs of type "PropagateTarget" are listed, clean-up the stuck jobs by running the below query till there are no more rows left for update:
UPDATE MGMT_JOB_EXEC_SUMMARY SET status = 23 WHERE status = 15 AND job_type_id IN (SELECT job_type_id FROM MGMT_JOB_TYPE_INFO WHERE job_type = 'PropagateTarget') AND rownum < 100*1000;
commit;9-Ensure that the tables in the Management Repository do not have any snapshots created:
select master , log_table from all_mview_logs where log_owner='SYSMAN';If there are snapshots, then drop them by running the following command as SYSMAN user:
Drop snapshot log on <master in above output>;10- Ensure that you do not have any logon or logoff triggers set in the Oracle Database that houses the Oracle Management Repository.
SELECT trigger_name FROM sys.dba_triggers WHERE TRIGGERING_EVENT LIKE 'LOGON%' AND status='ENABLED'
SELECT trigger_name FROM sys.dba_triggers WHERE TRIGGERING_EVENT LIKE 'LOGOFF%' AND status='ENABLED';If there are any logon or logoff triggers, disable them as below:
SQL> alter trigger <trigger_name> disable;NOTE :you may get the below triggered enabled .If you make it disabled .
SQL> SELECT trigger_name FROM sys.dba_triggers WHERE TRIGGERING_EVENT LIKE 'LOGOFF%' AND status='ENABLED';
TRIGGER_NAME
--------------------------------------------------------------------------------
GSMLOGOFF
SQL> alter trigger gsmadmin_internal.gsmlogoff disable;
Trigger altered.11-Disable the password_verify_function on the repository DB (if enabled).
alter profile default limit password_verify_function null;12- Once you stop all OMS before starting upgrade, make sure no active sysman sessions running in Repository database. Executing the below query as SYS user in Repository will help you to know the active sessions running in Repository database
select sid,serial#,username,status,action,machine from v$session;13- If the entity SUB_OracleGSM is present in Software Library, it could cause errors during Upgrade. Login to the repository as SYSMAN user and execute the below query to check this:
select * from EM_SWLIB_TS_ENT_SUBTYPES_E st where st.SUBTYPE_ID = '450' AND st.name = 'SUB_OracleGSM';14-If there are any Role Names with leading or trailing spaces, it can cause errors during OMS Configuration. Execute the following query in the repository as SYS user to identify any roles:
select * from sysman_opss.jps_dn where parentdn like '%cn=roles%' and rdn like 'cn=%\ %';15- Ensure that the ulimit is updated to the following values before Upgrade:
softnofiles : 30000
[root@oemnode1:~# ]cat /etc/security/limits.conf|grep soft
# - "soft" for enforcing the soft limits
#* soft core 0
#@faculty soft nproc 20
#oracle soft nofile 1024
oracle soft nofile 30000
oracle soft nproc 16384
oracle soft stack 10240
file-max : 65536
[oracle@oemnode1 bin]$ cat /proc/sys/fs/file-max
6815744
A lower value than recommended would cause the upgrade to move slowly or even fail
16-Stop the OMS and the agent.
$OMS_HOME/bin/emctl stop oms -all
$AGENT_HOME/bin/emctl stop agent
17- Take backup of OMS home,agent home :
OMS Backup Requirements :
1- Shutdown OMS
2- Shutdown Agent
3- Take tar backup of home
18 -Create directories for the new installation.
mkdir -p /u01/em/oms/agent2
mkdir -p /u01/em/oms/middleware2
19- Ensure that the CLASSPATH environment variable is unset prior to starting the upgrade
unset CLASSPATH
3- OEM 13.4 installation and upgrade :
1-Go to the directory where you have put the software and execute
[oracle@oemnode1 softwares]$ ./em13400_linux64.bin -J-Djava.io.tmpdir=/home/oracle/oemtmp/
NOTE: If you need at least 12G of temp space for installation .So to specify the alternate temp directory you can use the above command highlighted
Deselect the security updates option .
-skip the updates as we haven't provided any support id
-If any prerequisite failed resolve it and rerun .
-choose upgrade existing OEM home .make sure the correct home appears in the options .
-Give the new home of OMS which we had created earlier to hold the new installation
-Provide the SYS & SYSMAN password ,and make sure you have a healthy backup pf OMS ,AGENT homes .
-Note : While upgrade most of us faced the certificate issue ..The error message will be :
The upgrade process has detected MD5 certificate usage for some agent-OMS communication. These hosts and related targets will continue to be managed post-upgrade. However, it is strongly recommended to reconfigure them to use SHA-based certificates, since future versions of Enterprise Manager may not support MD5 certificates. Review the file /tmp/OraInstall<time_stamp>/md5Target.txt to find the agents that use MD5-based certificates. Follow the instructions outlined in My Oracle Support note 2179909.1 to reconfigure them with SHA-based certificates.Resolution :
There is a meta link doc available to resolve the issue .Follow the doc exactly step by step
and the error will be resolved .In a separate section will give the demo as well ,how to resolve this issue .
EM 13.2: SHA2 Certificate Pre-upgrade Verification for OMS & Agent (Doc ID 2179909.1)
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=350349323414694&parent=EXTERNAL_SEARCH&sourceId=REFERENCE&id=2179909.1&_afrWindowMode=0&_adf.ctrl-state=sw5st51px_4
For the above warning go to the repository database and run the command .
SQL> grant execute on DBMS_RANDOM to DBSNMP;
Grant succeeded.
Click OK and move ahead .
-Select all plugins
-deselect the shared location option if it is not there in your environment
-check the details before proceeding with upgrade
-click upgrade to start the upgrade process
-once the upgrade process is done ,it will ask to rn the root.sh as root user .In a separate session as root run the root.sh .
-This completes the upgrade process of OEM 13c from 13.2 to 13.4
POST UPGRADE:
1-Go to the new OMS home and check the status and version of OMS .It must point to the 13.4.0.0.0
[oracle@oemnode1 bin]$ ./emctl status oms -details
Oracle Enterprise Manager Cloud Control 13c Release 4
Copyright (c) 1996, 2020 Oracle Corporation. All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
2- Now as we had changed few parameters before upgrade was started ,will revert it back and bounce the database again .
alter system reset "_optimizer_nlj_hj_adaptive_join" scope=both sid='*';
alter system reset "_optimizer_strans_adaptive_pruning" scope=both sid='*';
alter system reset "_px_adaptive_dist_method" scope=both sid='*';
alter system reset "_sql_plan_directive_mgmt_control" scope=both sid='*';
alter system reset "_optimizer_dsdir_usage_control" scope=both sid='*';
alter system reset "_optimizer_use_feedback" scope=both sid='*';
alter system reset "_optimizer_gather_feedback" scope=both sid='*';
alter system reset "_optimizer_performance_feedback" scope=both sid='*';This completes the complete step by step details of OEM 13c upgrade from 13.2 to 13.4
In the Next Section we will see how to upgrade agent from OEM GUI after OMS was upgraded.