High Availability Cloud

egyptiannorweiganInternet και Εφαρμογές Web

31 Οκτ 2013 (πριν από 3 χρόνια και 9 μήνες)

320 εμφανίσεις

Slide
1

High Availability Cloud
Control Recipes
-

EM12C

Session ID# 941



Mike Gangler

Ithaka
-

www.ithaka.org



Slide
2

Mike Gangler



Ann Arbor, Michigan


Database Specialist at Ithaka.org


aka

JSTOR



Multi
-
platform and Database DBA


Specializing in Performance, Tuning and management of databases
and environments


Working With Oracle since Versus 4 ….


Board of Directors for SEMOP (Southeast Michigan Oracle
Professionals), MOUS (Michigan Oracle User Summit, and charter
member (1993) of the Board of Directors of the International Oracle
Users Group(1OUG).



Slide
3

What this presentation will cover



EM12 Architecture High Availability options



Setting up an Level 3 MAA Environment



Installation of a Standby OMS


Silent Mode



If Time
---
Notifications Best Practices





Slide
4

Audience Experience



Grid Control Experience


Used the old 9i / Workgroup version ?



10G Grid Control



11G Grid Control



12C Cloud Control



Slide
5

Other Presentations



Enterprise Manager SIG


9:30


10:30am


Wednesday


MileHigh 4B ?




Slide
6

Presentation Goals



Goals



Learn the Different MAA architectures



How to run a installation of a Standby OMS
in silent Mode



Notifications Tips / Best Practices





Slide
7

Ithaka’s Mission

aka JSTOR


ITHAKA

is

a

not
-
for
-
profit

organization

dedicated

to

helping

the

academic

community

take

full

advantage

of

rapidly

advancing

information

and

networking

technologies
.

We

serve

scholars,

researchers,

and

students

by

providing

the

content,

tools,

and

services

needed

to

preserve

the

scholarly

record

and

to

advance

research

and

teaching

in

sustainable

ways
.

We

are

committed

to

working

in

collaboration

with

other

organizations

to

maximize

benefits

to

our

stakeholders
.


www
.
ithaka
.
org

-

www
.
jstor
.
org



Slide
8

EM12C Architecture

Options


Level 1


Single Site


No Failover



Level 2


Single Site


2 OMS


Active / Passive Mode


VIP Failover

Slide
9

EM12C Architecture

Options


Level 3


Single Site


Multiple OMS (Active/Active)


Load Balancer


RAC Data Guard Primary / Standby DB

Slide
10

EM12C Architecture

Options


Level 4


Multiple Sites


Primary OMS (Active/Active) w/ Load Balancer


Load Balancer


RAC Data Guard Primary / Standby DB


Multiple Standby OMS w/ Load Balancer


.

Slide
11

EM12C Architecture

Suggestions


Setup an additional / separate OMS for Dev/QA targets



Setup Level 3.5 Maximum Availability Architecture (MAA)


Multiple Sites


Multiple OMS (Primary / Standby)


Oracle Data Guard with Fast Start Failover setup.


Using VIP’s / DNS alias help with data Guard rollover.


Use Load Balancer (optional) for OMS.

Slide
12

EM12C Standby OMS
Architecture (level 3 MAA)

Slide
13

Monitoring at JSTOR



Production Oracle Databases


25


24x7


Development Oracle Databases


27


14x5


Microsoft SqlServer Databases
-

21


14x5


MySql Databases


9


14x5



Total 240 Production Targets
(Agents, hosts, db,
etc.)


Total 276 Dev/QA Targets
(Agents, hosts, db, etc.)

Slide
14

Ithaka OMS Layout

DATACENTER

1


DATA
CENTER 2

DATACENTER

3

DataCenter 1

Secondary

OMS

DataCenter 2

Primary
OMS

Datacenter1

Standby

DataGuard

DB

Primary

DB

DNS

DNS

Slide
15

Pre
-
requisites for High Availability


Setup VIP or DNS Alias for multiple
sites.


Create Data Guard database on other
site.


NFS Shared Library


Primary OMS Setup


Plugins Installed on Primary Site


Slide
16

Pre
-
requisites for High Availability


Optional setups:


Fast Start Failover Data guard
Setup(Requires additional hardware)


Load Balancer NOT REQUIRED


Although the documentation disagrees


RAC DB’s


Great if you can afford it.


Slide
17

Requirements Standby OMS


Machine Required for Standby OMS/DB
(Linux, Sparc, HPUX, Microsoft)


Plugins need to be the same version


Exact same directory structure


Load Balancer (optional)


NFS Drive ….


Slide
18

Configuration Steps:

1.
Install Primary OMS / DB

2.
Add targets to Primary

3.
Setup NFS Drive


Software Library

4.
Add All Plugins required to Primary OMS

5.
Add Dataguard
-

Standby Database

6.
Create Standby OMS

7.
Convert Standby Database to RAC (IF Available)

Slide
19

Installation Steps: (Silent)


Verify NFS Software Library on Standby Machine


Install a “Software
-
Only” Installation on Standby OMS
Machine


(Do NOT run the ConfigureGC.sh or ConfigureGC.bat)

Slide
20

Installation Steps: (Silent)



Ensure same Plugins as the source OMS

SOURCE_OMS_pgrid01
>l


1 SELECT epv.plugin_id, epv.version, epv.rev_version


2 FROM em_plugin_version epv, em_current_deployed_plugin ecp


3 WHERE epv.plugin_type NOT IN ('BUILT_IN_TARGET_TYPE','INSTALL_HOME')


4 AND ecp.dest_type='2'


5* AND epv.plugin_version_id =
ecp.plugin_version_id

SYSMAN_pgrid01>
/


PLUGIN_ID



VERSION REV_VERSION

-------------------
-

------------


-
----------

oracle.sysman.mos


12.1.0.2.0


0

oracle.em.satc



12.1.0.1.0


0

oracle.em.smss



12.1.0.2.0


0

oracle.sysman.emct


12.1.0.3.0


0

oracle.sysman.ssa


12.1.0.4.0


0

oracle.sysman.db


12.1.0.2.0

20120804

oracle.sysman.xa


12.1.0.3.0


0

oracle.sysman.emas


12.1.0.3.0


0

oracle.sysman.vt


12.1.0.3.0


0

pythian.mysql.prod


12.1.0.1.2


0


Slide
21

Installation Steps: (Silent)

Plug
-
in Definitions:


PLUGIN_ID





DEFINITION

-------------------
-

-----------------------

oracle.sysman.mos


My

Oracle Support

oracle.em.satc



Apache
Tomcat

oracle.em.smss



Microsoft SQL Server

oracle.sysman.emct


Charge Back

oracle.sysman.ssa


Cloud Application

oracle.sysman.db


Oracle
Database

oracle.sysman.xa


Exadata

oracle.sysman.emas


Fusion Middleware

oracle.sysman.vt


Virtualisation

pythian.mysql.prod


MySQL



Slide
22

Installation Steps: (Silent)

Plug
-
ins (Continued)

1.
To Install Additional Plug
-
ins


Manually download the
plug
-
ins from OTN to Accessible location (i.e., /tmp)

a.
Verify that they are the same version as primary OMS

2.
http://www.oracle.com/technetwork/oem/grid
-
control/downloads/index.html

3.
Run $OMS_HOME/sysman/install/PluginInstall.sh

pluginLocation /tmp
-

Choose the plugins you want to
install (screen print


next page)

Slide
23

PluginInstall.sh

Slide
24

Installation Steps: (Silent)

1.
Apply all patches that were applied to the first OMS

a.
To Check
-

$OMS_HOME/OPatch/opatch lsinventory

2.
Export the Configuration Details from the First OMS

a.
$OMS_HOME/bin/emctl exportconfig oms

dir <Path
Name>

b.
Creates bka file


i.e. opf_ADMIN_<DATE>_<TIME>.bka

c.
Copy bka file to standby OMS Machine

Slide
25

Installation Steps: (Silent)

oracle@pr2prodora01 ~]$ cd /u01/app/oracle/OracleHomes2/oms/bin/

[oracle@pr2prodora01 bin]$ ./emctl exportconfig oms
-
dir $HOME

Oracle Enterprise Manager Cloud Control 12c Release 2

Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.

Enter Enterprise Manager Root (SYSMAN) Password :

ExportConfig started...

Machine is Admin Server host. Performing Admin Server backup...

Exporting emoms properties...

Exporting secure properties...


Export has determined that the OMS is not fronted

by an SLB. The local hostname was NOT exported.

The exported data can be imported on any host but

resecure of all agents will be required. Please

see the EM Advanced Configuration Guide for more

details.


Exporting configuration for pluggable modules...

Preparing archive file...

Backup has been written to file: /home/oracle/opf_ADMIN_20130125_101524.bka


The export file contains sensitive data.

Please ensure that it is kept secure.


ExportConfig completed successfully!

[oracle@pr2prodora01 bin]$ cd $HOME

[oracle@pr2prodora01 ~]$ scp opf_ADMIN_20130125_101524.bka oracle@aa2prodora11.jstor.org:/home/oracle

oracle@aa2prodora11.jstor.org's password:

opf_ADMIN_20130125_101524.bka 100% 45MB 5.0MB/s 00:09

[oracle@pr2prodora01 ~]$

Slide
26

Installation Steps: (Silent)

OMSCA Breakdown on Standby database using the
backup file created



Uses the backup file just generated = bka file


Admin Server HTTPS Port = 7101


Managed Server Port=7202


Managed Server HTTPS Port=7301


EM Note Manager Port = 7403


EM Upload Port = 4889


EM Upload HTTPS Port=4900


EM Console Port = 7788


EM Console HTTPS Port= 7799


Config_home = /u01/oracle/product/OracleHomes2/gc_inst


EM_INSTANCE_HOST aa2prodora11.jstor.org (Standby machine)

Slide
27

Installation Steps: (Silent)

Of all these ports the following are relevant to the load
Balancer:



EM Upload HTTP Port: 4889


EM UPLOAD HTTPS Port: 4900


EM Central Console HTTP Port: 7788


EM Central Console HTTPS Port: 7799

Slide
28

Installation Steps: (Silent)


Run OMSCA on Standby database using the backup
file created

<OMS_HOME>/bin/omsca recover
-
ms
-
backup_file
/opt/oracle/product/backup/opf_ADMIN_20120504_031016.bka
-
AS_HTTPS_PORT 7101
-
MSPORT 7202
-
MS_HTTPS_PORT 7301
-
EM_NODEMGR_PORT 7403
-
EM_ UPLOAD_PORT 4889
-
EM_UPLOAD_HTTPS_PORT 4900
-
EM_CONSOLE_PORT 7788
-
EM_
CONSOLE_HTTPS_PORT 7799
-
config_home
/opt/oracle/product/omsmdw/gc_inst
-
EM_INSTANCE_HOST
example.com

!!! Many errors at this stage, due Failed attempts

Slide
29

Installation Issues:

MSEVERE: OMSCA
-
ERR:Post "Deploy and Repos
Setup" operations failed



Check the following items :


--

Check RAM on Standby Machine


needs 6GB


to Install


--

If using 32 Bit Linux




Modify


create_domain.py (under omsca/scripts/wls
directory)


Change
-

-
Xmx1524m to

Xmx1324m


Change

XX:MaxPermsize=612M to

X:MaxPermsize=412M


--

Check for Lock in repository


Slide
30

Installation Issues


Check for lock in the repository


Due to multiple tries t
here was a lock in the
repository during the plugin deployment stage.



select host_url, oms_id, status, is_first_oms,
em_release_version from gc_oms_info;


select mp_guid,file_id,file_name,file_type from
mgmt_mp_files;


select * from em_plugin_lock_track;


Slide
31

Installation Issues:


First Issue


wrong version of the database plugin



I upgraded the database plugin but didn

t include it
in the standby installation


Copied latest “DB” plugin and reran
pluginInstall.sh

Slide
32

Installation Issues
:


2
nd

Issue


Plugin Issue and multiple attempts
caused lock in EM database (SYSMAN)



Removed record from lock table
em_plugin_lock_track


Ran $OMS_HOME/bin
-

./pluginca

action deploy

isFirstOMS false

plugins


Started OMS

Slide
33

Final Installation Steps


If you haven

t 楮獴慬汥d th攠慧敮t on⁴h攠獴慮db礠
ho獴
:

$<AGENT_HOME>/sysman/install/agentDeploy.sh
AGENT_BASE_DIR=<middleware_ home>/agent
OMS_HOST=<second_oms_host_name> EM_UPLOAD_PORT=<second_oms_
port> AGENT_REGISTRATION_PASSWORD=<password>
-
configOnly

Note:
If you have Server Load Balancer (SLB) configured, then directly
enter the host name and port number of the SLB. If SLB is not
configured, then enter the secure upload port of the first OMS.


Slide
34

Final Installation Steps


After agent is installed and deployed



Deploy the required plug
-
ins on the management agent


Secure the agent
-

$AGENT_HOME/bin/emctl secure
add_trust_cert_to_jks (If applicable)


In the new OMS Grid control (standby)


discover the Oracle Weblogic
services


Targets, Middleware, click EMGC_DOMAIN


From EMGC_DOMAIN home page


Select Farm Menu, Refresh Weblogic Domain, click continue till
2
nd

management server is discovered on the standby OMS Host.


Slide
35

Installation logs / help


Plugin Logs:


$OMS_HOME/cfgtoollogs/pluginca/*


$OMS_HOME/cfgw/CfmLogger_<timestamp>.log


Repository Configuration logs


$OMS_HOME/cfgw/CfmLogger_<timestamp>.log


$OMS_HOME/sysman/log/schemamanager/m_<timestamp>/*



MDS Schema Configuration Assistant


$OMS_HOME/sysman/log/schemamanager/m_<timestamp>/*


$OMS_HOME/cfgtoollogs/cfgfw/emdcreate*.log



More Log file and Troubleshooting tips:

http://docs.oracle.com/cd/E24628_01/install.121/e24080/appdx_troubleshooting.htm#CEGFAHIG

Slide
50

What We Covered


High Available architecture options


Silent Standby OMS installations and
gotchas



References


Deploying a Highly Available EM12C Cloud Control


An Oracle
White Paper


May 2012
-

# 1631423.pdf

www.dbakevlar.com



Kellyn Pot’Vin’s blog contains many great
ideas about EM12c

e24089.pdf


Cloud Control Advanced Installation and
Configuration Guide (section 5
-

installing Add'l OMS in silent
mode)


References

https://forums.oracle.com/forums/category.jspa?categoryID=70


OTN Discussion on EM

www.oracle.com


Documentation is not bad for many of the
basic items with EM12c

Keeping your OMS Tidy


Ray Smith
-

IOUG Tips and Best
Practices


7
th

edition
-

www.ioug.org

http://oramanageability.wordpress.com

-

Ray Smiths Blog on
Cloud Control




References

www.pythian.com



MySQL plug
-
in and instructions

oemgc.wordpress.com



Rob Zoeteweij


Good information on
Incident management and ITL, including a blog

www.slideshare.net/Enkitec/em12c
-
monitoring
-
metric
-
extensions
-
and
-
performance
-
pages#btnPrevious



Em12C
Monitoring, Metric Extensions and Performance pages


By
Kellyn Pot’Vin


Enkitec

EM12C Log Cleanup



Script to clean up logs and information



Questions?
Comments?

Slide
55

Thank You for Attending


Please fill out your Evaluations

www.ioug.org
/
eval


Cloud Control Recipes for EM12C
Session # 941

Mike Gangler



Email:
michael.gangler@ithaka.org


Twitter: @mjgangler


Blog: mjgangler.wordpress.com