Step-by-Step: Oracle Enterprise Manager(OEM) 13c Upgrade from 13.2 to 13.4

Updated: Apr 30

#oem13c #upgrade #oem13.2 #oem13.4 #oracle



“…Flash-based content will be blocked from running in Adobe Flash Player after the EOL Date.” from Adobe EOL page:


So Oracle support has released a MOS note on the same as described below .(Snippet taken from Oracle Support Note)



Adobe Flash End-of-Life Impact on Oracle Enterprise Manager Cloud Control (Doc ID 2709724.1)

Oracle Enterprise Manager’s database performance pages were initially written using Adobe Flash technology.   
These  database performance pages included the ‘Performance Home’, ‘Top  Activity’,ASH Analytics’, ‘Real-time SQL Monitoring’ among many  others.

Oracle  recommends that customers update their Oracle Enterprise Manager Cloud  Control to version 13.4 before the Adobe Flash EOL.
Oracle also recommends that customers apply the latest release updates available to their Enterprise Manager


So here is a step by step guide to upgrade OEM from 13.2 to 13.4 .


Upgrade of OEM 13.2 to 13.4 involves below steps:

1- Software download

2- Prerequisite check

3- OEM 13.4 installation and upgrade

4- Agent Upgrade form OEM console

5- Removal of Old OMS homes


1- Software download :


As upgrade of OEM 13c to 13.4 is a new installation ,we need the binary of 13.4 version .

Use the link here and download the softwares .


2-Prerequisite check :


1- Check the current OEM details and status in the OMS server .


[oracle@oemnode1] echo $OMS_HOME

/u01/em/oms/middleware


[oracle@oemnode1]$ cd $OMS_HOME/bin

[oracle@oemnode1 bin]$ ./emctl status oms

Oracle Enterprise Manager Cloud Control 13c Release 2

Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.

WebTier is Up

Oracle Management Server is Up

JVMD Engine is Up

BI Publisher Server is Up


[oracle@oemnode1 bin]$ ./emctl getversion oms

Oracle Enterprise Manager Cloud Control 13c Release 2

Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.

Oracle Enterprise Manager Cloud Control OMS 13c Release 2


2- Check the Agent status :


[oracle@oemnode1 bin]$ echo $AGENT_HOME

/u01/em/oms/agent/agent_inst


[oracle@oemnode1 bin]$ cd $AGENT_HOME/bin


[oracle@pr-oemgcc1-ivl bin]$ ./emctl status agent

Oracle Enterprise Manager Cloud Control 13c Release 2

Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.

---------------------------------------------------------------


Agent Version : 13.2.0.0.0

OMS Version : 13.2.0.0.0

Protocol Version : 12.1.0.1.0

Agent Home : /u01/em/oms/agent/agent_inst

Agent Log Directory : /u01/em/oms/agent/agent_inst/sysman/log

Agent Binaries : /u01/em/oms/agent/agent_13.2.0.0.0

Core JAR Location : /u01/em/oms/agent/agent_13.2.0.0.0/jlib

Agent Process ID : 2717

Parent Process ID : 1974

Agent URL : https://oemnode1:3872/emd/main/

Local Agent URL in NAT : https://oemnode1:3872/emd/main/

Repository URL : https://oemnode1:4903/empbs/upload

Started at : 2021-04-01 05:10:44

Started by user : oracle

Operating System : Linux version 3.10.0-693.el7.x86_64 (amd64)

Number of Targets : 34

Last Reload : (none)

Last successful upload : 2021-03-22 13:04:50

Last attempted upload : 2021-03-22 13:04:50

Total Megabytes of XML files uploaded so far : 72.89

Number of XML files pending upload : 0

Size of XML files pending upload(MB) : 0

Available disk space on upload filesystem : 34.15%

Collection Status : Collections enabled

Heartbeat Status : Ok

Last attempted heartbeat to OMS : 2021-03-22 13:08:44

Last successful heartbeat to OMS : 2021-03-22 13:08:44

Next scheduled heartbeat to OMS : 2021-03-22 13:09:44


---------------------------------------------------------------

Agent is Running and Ready


[oracle@oemnode1 bin]$ ./emctl getversion agent

Oracle Enterprise Manager Cloud Control 13c Release 2

Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.

Oracle Enterprise Manager Cloud Control Agent 13c Release 2


3- Check database version and PSU :


SQL> select version from sysman.mgmt_versions where component_name = 'CORE';


VERSION

--------------------------------

13.2.0.0.0


SQL> select * from v$version ;


BANNER CON_ID

-------------------------------------------------------------------------------- ----------

Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production 0

PL/SQL Release 12.1.0.2.0 - Production 0

CORE 12.1.0.2.0 Production 0

TNS for Linux: Version 12.1.0.2.0 - Production 0

NLSRTL Version 12.1.0.2.0 - Production 0


4-Check the latest patch is applied in database home or not :


Oracle Database - Overview of Database Patch Delivery Methods - 12.1.0.2 and older (Doc ID 1962125.1)


Check if the latest patch has been applied on to the oracle home or not .If not apply the latest patch .


SQL> set lines 200

SQL> set pages 200

SQL> col DESCRIPTION for a70

SQL> SELECT patch_id,DESCRIPTION,version, status, bundle_id, bundle_series FROM dba_registry_sqlpatch;


PATCH_ID DESCRIPTION VERSION STATUS BUNDLE_ID BUNDLE_SERIES

31985579 DATABASE PATCH SET UPDATE 12.1.0.2.210119 12.1.0.2 SUCCESS 210119 PSU


5- EMKEY Copy Requirements :


export OMS_HOME=/u01/em/oms/middleware

$OMS_HOME/bin/emctl config emkey -copy_to_repos -sysman_pwd <>

$OMS_HOME/bin/emctl status emkey -sysman_pwd <>


[oracle@oemnode1 bin]$ ./emctl status emkey

Oracle Enterprise Manager Cloud Control 13c Release 2

Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.

Enter Enterprise Manager Root (SYSMAN) Password :


The EMKey is configured properly.


[oracle@oemnode1 bin]$ ./emctl config emkey -copy_to_repos

Oracle Enterprise Manager Cloud Control 13c Release 2

Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.

Enter Enterprise Manager Root (SYSMAN) Password :

The EMKey has been copied to the Management Repository. This operation will cause the EMKey to become unsecure.

After the required operation has been completed, secure the EMKey by running "emctl config emkey -remove_from_repos".



6- Optimizer Adaptive Features Disabling Requirements :

The OPTIMIZER_ADAPTIVE_FEATURES is not available in 12.2 DB onward and with DB 12.1.0.2 (if a patch is applied). For improved SQL performance all the adaptive features parameters should be unset prior to the upgrade.


NOTE: Before any parameter modification in database make sure to keep a backup of the parameter files .


Case A) If the Management Repository is using Oracle Database 12.1.0.2.0 and patch 22652097 has NOT been applied, ensure that you disable the optimizer adaptive features by connecting to the database as SYSDBA and running the following:


SQL> alter system set optimizer_adaptive_features=false scope=both


Case B) If the Management Repository is using Oracle Database 12.1.0.2 with patch 22652097 applied or the Management Repository is using Oracle Database 12.2, 18.x or 19.x, you need to do the following:


alter system set "_optimizer_nlj_hj_adaptive_join"= FALSE scope=both sid='*';
alter system set "_optimizer_strans_adaptive_pruning" = FALSE scope=both sid='*';
alter system set "_px_adaptive_dist_method" = OFF scope=both sid='*';
alter system set "_sql_plan_directive_mgmt_control" = 0 scope=both sid='*';
alter system set "_optimizer_dsdir_usage_control" = 0 scope=both sid='*';
alter system set "_optimizer_use_feedback" = FALSE scope=both sid='*';
alter system set "_optimizer_gather_feedback" = FALSE scope=both sid='*';
alter system set "_optimizer_performance_feedback" = OFF scope=both sid='*';
alter system set “ _allow_insert_with_update_check”=true scope=both;
alter system set job_queue_processes=0 scope=both;
alter system set optimizer_adaptive_features=false scope=both;


Restart the repository database.


SQL> shu immediate ;

Database closed.

Database dismounted.

ORACLE instance shut down.



SQL> startup

ORACLE instance started.


Total System Global Area 9663676416 bytes

Fixed Size 3720600 bytes

Variable Size 1107298920 bytes

Database Buffers 8522825728 bytes

Redo Buffers 29831168 bytes

Database mounted.

Database opened.


7-Check the deb level OMS plugins mismatch in OEM repository database .

Verify that all the OMS (including Standby) have the same version of Plugins deployed. Execute the following in the repository database, connected as SYSMAN user:



set serveroutput on
declare
oms_guids mgmt_medium_string_array := mgmt_medium_string_array();
oms_names mgmt_medium_string_array := mgmt_medium_string_array();
mismatches smp_emd_nvpair_array := smp_emd_nvpair_array();
begin
select host_url, em_plugin_util.get_guid_from_name(unique_oms_name)
bulk collect into oms_names, oms_guids
from gc_oms_info;
for j in 1..oms_guids.count
loop
dbms_output.put_line('Checking mismatch for OMS ' || oms_names(j));
mismatches := smp_emd_nvpair_array();
em_plugin_inventory.get_plugin_mismatch(oms_guids(j), mismatches);
if mismatches is null or mismatches.count = 0
then
dbms_output.put_line('No mismatch found for OMS ' || oms_names(j));
else
dbms_output.put_line('Mismatch found for OMS ' || oms_names(j));
for i in 1..mismatches.count
loop
dbms_output.put_line('Plug-in on repository: ' || nvl(mismatches(i).name, 'None') ||
', Plug-in on OMS: ' || nvl(mismatches(i).value, 'None'));
end loop;
end if;
end loop;
end;
/

8- Check if there are large number of jobs that are stuck:

If you have a large number of PropogateTarget jobs, then the Repository Upgrade can consume very long time or stuck while executing "upgrade/post_jobtype_upgrade.sql".


Run the below query to get the job count:


SELECT job_type, status, cnt
FROM (
SELECT job_type, status, count(*) as cnt
FROM MGMT_JOB_EXEC_SUMMARY join MGMT_JOB_TYPE_INFO using (job_type_id)
WHERE status NOT IN(3,4,5,8,18,19,23)
group by job_type, status
order by 3, 1, 2
)
where cnt > 10000;

If jobs of type "PropagateTarget" are listed, clean-up the stuck jobs by running the below query till there are no more rows left for update:


UPDATE MGMT_JOB_EXEC_SUMMARY SET status = 23 WHERE status = 15  AND job_type_id IN (SELECT job_type_id FROM MGMT_JOB_TYPE_INFO WHERE job_type = 'PropagateTarget') AND rownum < 100*1000;
commit;

9-Ensure that the tables in the Management Repository do not have any snapshots created:


select master , log_table from all_mview_logs where log_owner='SYSMAN';

If there are snapshots, then drop them by running the following command as SYSMAN user:


 Drop snapshot log on <master in above output>;


10- Ensure that you do not have any logon or logoff triggers set in the Oracle Database that houses the Oracle Management Repository.



SELECT trigger_name FROM sys.dba_triggers WHERE TRIGGERING_EVENT LIKE 'LOGON%' AND status='ENABLED'

SELECT trigger_name FROM sys.dba_triggers WHERE TRIGGERING_EVENT LIKE 'LOGOFF%' AND status='ENABLED';

If there are any logon or logoff triggers, disable them as below:


SQL> alter trigger <trigger_name> disable;

NOTE :you may get the below triggered enabled .If you make it disabled .


SQL> SELECT trigger_name FROM sys.dba_triggers WHERE TRIGGERING_EVENT LIKE 'LOGOFF%' AND status='ENABLED';
TRIGGER_NAME
--------------------------------------------------------------------------------
GSMLOGOFF

SQL> alter trigger gsmadmin_internal.gsmlogoff disable;

Trigger altered.

11-Disable the password_verify_function on the repository DB (if enabled).


alter profile default limit password_verify_function null;

12- Once you stop all OMS before starting upgrade, make sure no active sysman sessions running in Repository database. Executing the below query as SYS user in Repository will help you to know the active sessions running in Repository database


select sid,serial#,username,status,action,machine from v$session;

13- If the entity SUB_OracleGSM is present in Software Library, it could cause errors during Upgrade. Login to the repository as SYSMAN user and execute the below query to check this:



select * from EM_SWLIB_TS_ENT_SUBTYPES_E st where st.SUBTYPE_ID = '450' AND st.name = 'SUB_OracleGSM';

14-If there are any Role Names with leading or trailing spaces, it can cause errors during OMS Configuration. Execute the following query in the repository as SYS user to identify any roles:


select * from sysman_opss.jps_dn where parentdn like '%cn=roles%' and rdn like 'cn=%\ %';

15- Ensure that the ulimit is updated to the following values before Upgrade:

softnofiles : 30000


[root@oemnode1:~# ]cat /etc/security/limits.conf|grep soft

# - "soft" for enforcing the soft limits

#* soft core 0

#@faculty soft nproc 20

#oracle soft nofile 1024

oracle soft nofile 30000

oracle soft nproc 16384

oracle soft stack 10240

file-max : 65536

[oracle@oemnode1 bin]$ cat /proc/sys/fs/file-max

6815744

A lower value than recommended would cause the upgrade to move slowly or even fail


16-Stop the OMS and the agent.

$OMS_HOME/bin/emctl stop oms -all

$AGENT_HOME/bin/emctl stop agent


17- Take backup of OMS home,agent home :


OMS Backup Requirements :

1- Shutdown OMS

2- Shutdown Agent

3- Take tar backup of home


18 -Create directories for the new installation.

mkdir -p /u01/em/oms/agent2

mkdir -p /u01/em/oms/middleware2



19- Ensure that the CLASSPATH environment variable is unset prior to starting the upgrade

unset CLASSPATH



3- OEM 13.4 installation and upgrade :


1-Go to the directory where you have put the software and execute


[oracle@oemnode1 softwares]$ ./em13400_linux64.bin -J-Djava.io.tmpdir=/home/oracle/oemtmp/


NOTE: If you need at least 12G of temp space for installation .So to specify the alternate temp directory you can use the above command highlighted



Deselect the security updates option .


-skip the updates as we haven't provided any support id

-If any prerequisite failed resolve it and rerun .


-choose upgrade existing OEM home .make sure the correct home appears in the options .

-Give the new home of OMS which we had created earlier to hold the new installation

-Provide the SYS & SYSMAN password ,and make sure you have a healthy backup pf OMS ,AGENT homes .

-Note : While upgrade most of us faced the certificate issue ..The error message will be :


The upgrade process has detected MD5 certificate  usage for some agent-OMS communication. These hosts and related targets  will continue to be managed post-upgrade. However, it is strongly  recommended to reconfigure them to use SHA-based certificates, since  future versions of Enterprise Manager may not support MD5 certificates.  Review the file /tmp/OraInstall<time_stamp>/md5Target.txt  to find the agents that use MD5-based certificates. Follow the  instructions outlined in My Oracle Support note 2179909.1 to reconfigure  them with SHA-based certificates.

Resolution :

There is a meta link doc available to resolve the issue .Follow the doc exactly step by step

and the error will be resolved .In a separate section will give the demo as well ,how to resolve this issue .

EM 13.2: SHA2 Certificate Pre-upgrade Verification for OMS & Agent (Doc ID 2179909.1)
https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=350349323414694&parent=EXTERNAL_SEARCH&sourceId=REFERENCE&id=2179909.1&_afrWindowMode=0&_adf.ctrl-state=sw5st51px_4 


For the above warning go to the repository database and run the command .


SQL> grant execute on DBMS_RANDOM to DBSNMP;


Grant succeeded.

Click OK and move ahead .


-Select all plugins


-deselect the shared location option if it is not there in your environment



-check the details before proceeding with upgrade

-click upgrade to start the upgrade process



-once the upgrade process is done ,it will ask to rn the root.sh as root user .In a separate session as root run the root.sh .

-This completes the upgrade process of OEM 13c from 13.2 to 13.4




POST UPGRADE:


1-Go to the new OMS home and check the status and version of OMS .It must point to the 13.4.0.0.0


[oracle@oemnode1 bin]$ ./emctl status oms -details

Oracle Enterprise Manager Cloud Control 13c Release 4

Copyright (c) 1996, 2020 Oracle Corporation. All rights reserved.

Enter Enterprise Manager Root (SYSMAN) Password :


2- Now as we had changed few parameters before upgrade was started ,will revert it back and bounce the database again .



alter system reset "_optimizer_nlj_hj_adaptive_join" scope=both sid='*';
alter system reset "_optimizer_strans_adaptive_pruning" scope=both sid='*';
alter system reset "_px_adaptive_dist_method" scope=both sid='*';
alter system reset "_sql_plan_directive_mgmt_control" scope=both sid='*';
alter system reset "_optimizer_dsdir_usage_control" scope=both sid='*';
alter system reset "_optimizer_use_feedback" scope=both sid='*';
alter system reset "_optimizer_gather_feedback" scope=both sid='*';
alter system reset "_optimizer_performance_feedback" scope=both sid='*';


This completes the complete step by step details of OEM 13c upgrade from 13.2 to 13.4


In the Next Section we will see how to upgrade agent from OEM GUI after OMS was upgraded.



3,074 views2 comments

Recent Posts

See All

#oracle #ASM #controlfile #multiplexing #oracle19c It is a best practice for any database to have controlfile as well as redolog file to be multiplexed . This helps during any disk failure to keep one

In the previous post we saw how to upgrade OMS from 13.2 to 13.4 version .(Click Here) Once OMS is upgraded we have to upgrade the agents running in the servers which are monitored by OEM . This whole