How to setup DGMGRL, broker with example

Before we start, remember that DG broker does not have the ability to create standby and is used for managing the dataguard configuration. Before proceeding with step-by-step instructions on how to set up DG broker, I would recommend you to get comfortable with the concepts first.
The following configuration was tested on RHEL4U2 64-bit with Oracle 10.1.0.5 database.
Pri db_unique_name = 'TESTPRI'DG db_unique_name = 'TESTDG'Configuration(any name) = 'DG_TEST'sys password = 'sys'Pri conn stg = 'TESTPRI'
1. Set up init parameters on primary to enable broker
Note: For RAC, ensure dg_broker_config_files are on shared storage and accessible to all the instances.
Note: Broker config files are named as dr1<>.dat and dr2<>.dat
SQL> alter system set dg_broker_start=false sid='*';System altered.SQL> alter system set dg_broker_config_file1='/n01/dg_broker_config_files/dr1TESTPRI.dat' sid='*';System altered.SQL> alter system set dg_broker_config_file2='/n01/dg_broker_config_files/dr2TESTPRI.dat' sid='*';System altered.SQL> alter system set dg_broker_start=true sid='*';System altered.
2. Verify if DMON process has started on all the instances of primary. Example:
$ ps -efgrep dmongrep -v greporacle 16190 1 0 08:53 ? 00:00:00 ora_dmon_TESTPRIR1$ ps -efgrep dmongrep -v greporacle 29723 1 0 08:53 ? 00:00:00 ora_dmon_TESTPRIR2
3. Set up init parameters on standby
SQL> alter system set dg_broker_start=false sid='*';System altered.SQL> alter system set dg_broker_config_file1='/export/crawlspace/dg_broker_config_files/dr1TESTDG.dat' sid='*';System altered.SQL> alter system set dg_broker_config_file2='/export/crawlspace/dg_broker_config_files/dr2TESTDG.dat' sid='*';System altered.SQL> alter system set dg_broker_start=true sid='*';System altered.
4. GLOBAL_DBNAME should be set to <>_DGMGRL.<> in listener.ora on all instances of both primary and standby.
This is important otherwise you'll have TNS-12154 error during switchover operation.
Example:
SID_LIST_LISTENER_TESTPRI = (SID_LIST = (SID_DESC = (SID_NAME = PLSExtProc) (ORACLE_HOME = /apps/oracle/product/10g/db) (PROGRAM = extproc) ) (SID_DESC = (SID_NAME = TESTPRIR1) (GLOBAL_DBNAME = TESTPRI_DGMGRL) (ORACLE_HOME = /apps/oracle/product/10g/db) ) )
5. DGMGRL Configuration5.1ConnectDGMGRL> CONNECT sys/sysConnected.5.2Create ConfigurationDGMGRL> CREATE CONFIGURATION 'DG_TEST' AS PRIMARY DATABASE IS 'TESTPRI' CONNECT IDENTIFIER IS TESTPRI;Configuration "DG_TEST" created with primary database "TESTPRI".5.3Verify configurationDGMGRL> SHOW CONFIGURATION;Configuration Name: DG_TEST Enabled: NO Protection Mode: MaxPerformance Databases: TESTPRI - Primary databaseCurrent status for "DG_TEST":DISABLED5.4Verify database; if RAC verify if all instances are validatedDGMGRL> show database 'TESTPRI';Database Name: TESTPRI Role: PRIMARY Enabled: NO Intended State: ONLINE Instance(s): TESTPRIR1 TESTPRIR2Current status for "TESTPRI":DISABLED5.5Add standby database to the configurationDGMGRL> ADD DATABASE 'TESTDG' AS CONNECT IDENTIFIER IS TESTDG MAINTAINED AS PHYSICAL;Database "TESTDG" added.5.6Enable the brokerDGMGRL> ENABLE CONFIGURATION;Enabled.5.7Verfying againDGMGRL> SHOW CONFIGURATION;Configuration Name: DG_TEST Enabled: YES Protection Mode: MaxPerformance Databases: TESTPRI - Primary database TESTDG - Physical standby databaseCurrent status for "DG_TEST":SUCCESS6. TroubleshootingLet us see some sample issues and their fixIssueDGMGRL> CONNECT sys/sysORA-16525: the Data Guard broker is not yet availableFixSet dg_broker_start=trueIssueAfter enabling the configuration, on issuing SHOW CONFIGURATION, this error comes Warning: ORA-16608: one or more sites have warningsFixTo know details of the error, you may check log which will be generated at bdump with naming as drc{DB_NAME}.log or there are various monitorable properties that can be used to query the database status and assist in further troubleshooting.Few Monitorable properties to troubleshootDGMGRL> SHOW DATABASE 'TESTPRI' 'StatusReport';DGMGRL> SHOW DATABASE 'TESTPRI' 'LogXptStatus';DGMGRL> SHOW DATABASE 'TESTPRI' 'InconsistentProperties';DGMGRL> SHOW DATABASE 'TESTPRI' 'InconsistentLogXptProps';DGMGRL> SHOW DATABASE 'TESTDG' 'StatusReport';DGMGRL> SHOW DATABASE 'TESTDG' 'LogXptStatus';DGMGRL> SHOW DATABASE 'TESTDG' 'InconsistentProperties';DGMGRL> SHOW DATABASE 'TESTDG' 'InconsistentLogXptProps';IssueDGMGRL> SHOW DATABASE 'TESTPRI' 'StatusReport';STATUS REPORT INSTANCE_NAME SEVERITY ERROR_TEXT TESTPRIR2 WARNING ORA-16714: The value of property ArchiveLagTarget is inconsistent with the database setting. TESTPRIR2 WARNING ORA-16714: The value of property LogArchiveMaxProcesses is inconsistent with the database setting.IssueDGMGRL> SHOW DATABASE 'TESTPRI' 'InconsistentProperties';INCONSISTENT PROPERTIES INSTANCE_NAME PROPERTY_NAME MEMORY_VALUE SPFILE_VALUE BROKER_VALUE TESTPRIR2 ArchiveLagTarget 0 0 TESTPRIR2 LogArchiveMaxProcesses 4 2 4 ExampleDGMGRL> SHOW DATABASE 'TESTPRI' 'LogArchiveMaxProcesses'; LogArchiveMaxProcesses = '4'FixDGMGRL> EDIT DATABASE 'TESTPRI' SET PROPERTY 'LogArchiveMaxProcesses'=2;orSQL> alter system set log_archive_max_processes=4 scope=spfile sid='*';System altered.DGMGRL> SHOW DATABASE 'TESTPRI' 'LogArchiveMaxProcesses'; LogArchiveMaxProcesses = '4'More commandsDGMGRL> SHOW DATABASE VERBOSE 'dbname';This will show all property values in detailDGMGRL> HELP;List of all broker commands with usage help
Equivalent Broker Commands to 'ALTER SYSTEM'SQL> alter database recover managed standby database cancel;DGMGRL> edit database 'stby_dbname' set state='LOG-APPLY-OFF';SQL> alter database recover managed standby database disconnect;DGMGRL> edit database 'stby_dbname' set state='ONLINE';SQL> alter system set log_archive_max_processes=4;DGMGRL> edit database 'dbname' set property 'LogArchiveMaxProcesses'=4;SQL> alter system set log_archive_dest_state_2='enable' scope=both;DGMGRL> edit database 'stby_dbname' set property 'LogShipping'='ON';SQL> alter system set log_archive_dest_state_2='defer' scope=both;DGMGRL> edit database 'stby_dbname' set property 'LogShipping'='OFF';DGMGRL> edit database 'pri_dbname' set state='LOG-TRANSPORT-OFF';This will defer all standby databases
That's it my friends. You are all set to use broker now for managing your dataguard configuration. In my next blog, I'll talk about switchover using DG broker

Windows Service Not Starting Your Database?

If you are on a Windows Server you need to know about the Windows Services and the oradim utility.

On all versions of Windows the Oracle Database (and the Application Server) are started as Windows Services. This allows them to start in the background, normally owned by SYSTEM.

In an earlier Tip I discussed starting and stopping the Application Server 10g on Windows using a script to start and stop the component's service. The AS10g (and 9iAS) components can be started from the command line and will start the windows service however, the database (both the back end and the Metadata Repository) can not start without the database service running.

The Oracle Database must have a running service.

The Oracle database must attach itself to a running process. It is the Win Service that provides this process. Most DBAs think that the service is the database but that is not true. The service can start/stop the database but it can also be started without starting the database.

Most Problem with the Win Service for the Database involve the service starting but the database not starting.

If the Win service is set to automatically start when the server boots but the database does not start, you may have a improper registry setting or you may have a bad service.

1. Check Task Manager for the ORACLE.EXE process. If it is present, then the service started.

2. Check the Alert Log for the database. If the problem is not with the database, there will be no indication in the log that the database even tried to start.

3. Check the oradim.log in the $ORACLE_HOME/database directory for errors. Check the date on the log file as versions before 9i did not date/time stamp the entries.

If there are no errors in the logs then try and start the database.

C:> sqlplus "/ as sysdba"
connected to an idle instance
SQL> starup

If the database starts great, the problem is in the service.

To check the Win service:

Open the registry with regedit. Always back up the registry before making changes. Navigate to the key:

HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE\oracle_home_name.

There will be a key called:

ORA_SID_AUTOSTART (SID is your database SID)

This key should be set to TRUE. If not the server starts but does not start the database. There is also an ORA_SID_SHUTDOWN which you want to be TRUE so that if the server is shut down the service will shutdown the database.

If you want to manually start the database set ORA_SID_AUTOSTART to FALSE. The service will start but not the database.

Test the service.

If the ORA_SID_AUTOSTART setting was the problem, change it to TRUE and then test the service by stopping and then restarting the service to see it the database automatically starts. If it does, then that fixed your problem......or maybe is didn't. Reboot the server to verify that the database will start automatically. Sometimes the service will work, only to fail again after a reboot. If the service fail after rebooting you need to recreate it. This is where the oradim utility comes in.
Deleting a Service
First delete or rename the oradim.log file.
Next delete the current service.
C:>oradim -delete -sid SID
SID is of course you database SID. This may take a while so check the services to insure that the OracleServiceSID is removed before proceeding.
Creating a new Service.
Again we use oradim to recreate the service. This entire command is on one line.
c:> oradim -new -sid SID -intpwd password -startmode AUTO -pfile c:\oracle\admin\SID\pfile\initSID.ora
This command does a lot and will take some time to complete (if startmode is set to AUTO it will start the database). It recreate the database password and sets the internal password to password. NOTE: oradim changes the internal password.
The startmode determines if the service starts the database when it starts. If set to AUTO then the key ORA_SID_AUTOSTART is set to TRUE. If startmode is MANUAL then the key is set to FALSE.
Check the oradim.log for errors. Finally, verify the service works as needed by starting and stopping it. Then test with a reboot. If the service fails try recreating it again.
Scripting the Database Start in Windows
The dbstart and dbstop shell scripts do not exist on Windows platforms. Consequently Oracle database startup and shutdown is implemented completely differently. The oradim utility is used on the Windows platform to perform these tasks.
C:\oracle9i\bin\oradim -startup -sid ORCL92 –usrpwd manager
-starttype SRVC,INST -pfile
C:\oracle9i\admin\ORCL92\pfile\init.ora
• startup – Indicates that the specified instance should be started.
• sid – The SID of the database to start.
• usrpwd – The password for the database user.
• starttype – Specifies whether to start the instance, the service, or both (SRVC, INST).

The following command can be used to shutdown the instance with oradim:

C:\oracle9i\bin\oradim -shutdown -sid ORCL92 -shutttype SRVC,INST
–shutmode A

Notice that no password is needed to perform this task.

If you can't get the service to function properly you are left with scripting the starting of the database. Recreate the service with the -startmode set to MANUAL. Then create a batch file as follows:
REM Wait for the server to start.
sleep 60
REM Start the database
%ORACLE_HOME%\bin\sqlplus -s "/ as sysdba" @startup.sql
exit
The startup.sql file
-- start the database
startup
exit
-Now schedule the batch file in the Windows Scheduler to run at startup.
-If this is part of you AS10g, you can implement the startup in the script used to start the application server components.

Configuring ASM on Windows environment

1) With asmtool

Follow below steps and create a ASM diskgroup on your local m/c

1) Creating a disks with asmtool

D:\>mkdir asmdisks
D:\>cd asmdisks
D:\asmdisks>asmtool -create D:\asmdisks\disk1 1024
D:\asmdisks>asmtool -create D:\asmdisks\disk2 1024
D:\asmdisks>asmtool -create D:\asmdisks\disk3 1024

Note: - size should be in MB’s

Now you have 3 disks of 1024mb each which can be used to create a ASM disk group.

2) Create ASM instance: -

a) Configure Cluster Synchronization Service: -

C:\oracle\product\10.2.0\db_1\BIN>localconfig add

Step 1: stopping local CSS stack
Step 2: deleting OCR repository
Step 3: creating new OCR repository
successfully accumulated necessary OCR keys.
Creating OCR keys for user 'hostname', privgrp ' ' . .
Operation successful.
Step 4: creating new CSS service
successfully created local CSS service
successfully reset location of CSS setup

b) Create init file: -

Open notepad edit the following parameters and save file as "C:\oracle\product\ 10.2.0\db_ 1\database\ init+ASM. ora"
INSTANCE_TYPE= ASM
DB_UNIQUE_NAME= +ASM
LARGE_POOL_SIZE= 8M
ASM_DISKSTRING= 'D:\asmdisks\ *'
_ASM_ALLOW_ONLY_ RAW_DISKS= FALSE

c) Create service and password file
oradim will create an ASM instance and start it automatically.
D:\> orapwd file=C:\oracle\ product\10. 2.0\db_1\ database\ PWD+ASM.ora password=asm
D:\> oradim -NEW -ASMSID +ASM -STARTMODE auto

3) Starting ASM instance: -

a) Change PFILE to SPFILE, Add ASM Diskgroup parameter and your all set to go and use ASM.
SQL> create spfile from pfile;
SQL> startup nomount;

Create asm disk group: -

SQL> select path, mount_status from v$asm_disk;
PATH MOUNT_STATUS
----------------------- -------------------
D:\ASMDISKS\ DISK1 CLOSED
D:\ASMDISKS\ DISK3 CLOSED
D:\ASMDISKS\ DISK2 CLOSED

SQL> create diskgroup data external redundancy disk
2 'D:\ASMDISKS\DISK1',
3 'D:\ASMDISKS\DISK2',
4 'D:\ASMDISKS\DISK3'
5 /
Diskgroup created.

SQL> alter system set asm_diskgroups= data scope=spfile;
SQL> shutdown immediate
SQL> startup

ASM instance started
Total System Global Area 83886080 bytes
Fixed Size 1247420 bytes
Variable Size 57472836 bytes
ASM Cache 25165824 bytes
ASM diskgroups mounted

Now you can go ahead and use your DBCA and create a database and on step 6 of 13, you can use Automatic Storage management as your Filesystem.

SYSRESV Utility

This utility was the result of enhancement request 566223 [BUG: 566223]:

"Currently, many Oracle applications determine whether a particular instance is up by checking for the presence of the sgadef file. We should provide a separate utility for this purpose instead, based on the contents of the SGA."

The sysresv utility included with Oracle 8.1.5 and above provides instance status (and OS resources used) for specified ORACLE_SIDs. This utility is especially useful when multiple instances are running. OS resources can be removed using this utility if the specified instance is detected to be dead.

This utility may be useful when an instance has crashed or was aborted and memory and semaphores related to this instance were not cleaned up automatically. This utility is also helpful in determining which instance is running. The sysresv utility, located in $ORACLE_HOME/bin, can be used from locations other than $ORACLE_HOME/bin.

Point your environment to the instance of interest before using sysresv.

NOTE: (9i 64-bit):

Must set up as follows to get sysresv to work.

Oracle 9.X.X (64Bit) on Solaris (64Bit) OS
- Set LD_LIBRARY_PATH=$ORACLE_HOME/lib32
- Set LD_LIBRARY_PATH_64=$ORACLE_HOME/lib

Usage:

sysresv:
usage :
sysresv [-if] [-d (on/off)] [-l sid1 (sid2) ...]
-i : Prompt before removing ipc resources for each sid
-f : Remove ipc resources silently, over rides -i option
-d (on/off) : List ipc resources for each sid if on
-l sid1 (sid2) .. : apply sysresv to each sid

Default: sysresv -d on -l $ORACLE_SID
Note: ipc resources are attempted to be deleted for a sid only if there is no currently running instance with that sid.


Examples:

Instance is not running:

/u02/app/oracle/product/8.1.7 $ sysresv
IPC Resources for ORACLE_SID “R817”:
Shared Memory
ID KEY
No shared memory segments used
Semaphores:
ID KEY
No semaphore resources used
Oracle Instance not alive for sid "R817"

Instance is running:

/u03/app/oracle/product/8.1.6 $ sysresv

IPC Resources for ORACLE_SID "X816" :
Shared Memory:
ID KEY
16437 0xe4efa8dc
Semaphores:
ID KEY
12320802 0x09d48346
Oracle Instance alive for sid "X816"

Attempting to remove memory and semaphores using sysresv when Oracle detects an instance is running:

/u03/app/oracle/product/8.1.6 $ sysresv -f
IPC Resources for ORACLE_SID "X816" :
Shared Memory:
ID KEY
16437 0xe4efa8dc
Semaphores:
ID KEY
12320802 0x09d48346
Oracle Instance alive for sid "X816"
SYSRESV-005: Warning
Instance maybe alive - aborting remove for sid "X816"

Removing IPC resources:

[Sysresv shows memory and semaphores exist but Oracle determines the instance is not alive. Cleanup is needed.]

/u03/app/oracle/product/8.1.6 $ sysresv
IPC Resources for ORACLE_SID "X816" :
Shared Memory:
ID KEY
16837 0xe4efa8dc
Semaphores:
ID KEY
12714018 0x09d48346
Oracle Instance not alive for sid "X816"

Removing IPC resources using sysresv:

/u03/app/oracle/product/8.1.6 $ sysresv -i

IPC Resources for ORACLE_SID "X816" :
Shared Memory
ID KEY
No shared memory segments used
Semaphores:
ID KEY
No semaphore resources used

Oracle Instance not alive for sid "X816"
Remove ipc resources for sid "X816" (y/n)?y
Done removing ipc resources for sid "X816"
/u03/app/oracle/product/8.1.6

Verify the resources were removed:

/u03/app/oracle/product/8.1.6 $ sysresv

IPC Resources for ORACLE_SID "X816" :
Shared Memory
ID KEY
No shared memory segments used
Semaphores:
ID KEY
No semaphore resources used
Oracle Instance not alive for sid "X816"

If you need to remove memory segments, and Oracle detects the instance is alive through sysresv:

$ ipcrm -m (memid)

Where (memid) is the memory id shown in the sysresv output.

Example:
$ ipcrm -m 16437

If you need to remove semaphores, and Oracle detects the instance is alive through sysresv:

$ ipcrm -s (semid)

where (semid) is the semaphore id shown in the sysresv output.

Example:
$ ipcrm -s 12320802

115235.1
Resolving ORA-7279 or ORA-27146 errors when starting instance

Problem Description
-------------------

When attempting to start an instance, you receive the following error:

ORA-7279: spcre: semget error, unable to get first semaphore set

This error may possibly be accompanied by one or more of the following
additional errors:

ORA-600: internal error code, arguments [SKGMBUSY]
SVR4 error: 28: no space left on device

If you are using version 8.1.5 or 8.1.6, you receive the following error instead of the ora-7279 when attempting to start an instance:

ORA-27146: post/wait initialization failed

With 8.1.7, instance startup fails with ora-3113.

These errors happen whenever attempting to start an instance, which includes, but is not limited to, creating a new database during an install, creating a new database with an existing install, starting an existing database which had the initSID.ora parameter PROCESSES changed, or starting an existing database which was either shutdown incorrectly or crashed.


Solution Description

The problem is that there are not enough semaphores to start the instance. This can be resolved based on one of the two scenarios below:

Scenario 1
If no new instances have been created and you are trying to start an existing database which was shutdown incorrectly or crashed, the semaphores from the crashed instance may still be allocated and will need to be removed.

Step 1: -

Before removing semaphores, you should be certain that there are no running instances using them. First check for running instances using the following command:
$ ps -ef | grep -v grep | grep pmon
If there are no pmon processes running, then there should be no semaphores owned by the oracle user, and any semaphores owned by the oracle user can safely be removed.

See steps 3 and 4 for the commands to list and remove semaphores.

Step 2: -

If you have multiple instances running, you will need to either shut them down or determine which semaphores are being used by each instance. It is recommended that you shut down all of the instances, but if this is not possible, then follow the instructions below.

A) If the running instances are version 8.1.x or higher, set your environment to that of the running instance, then run the following command:
% $ORACLE_HOME/bin/sysresv
This will show you the semaphore ids being used by this instance. Make a note of them and repeat the sysresv command for each running instance.
B) If you are not running version 8.1.x or higher, use the following commands to get a listing of semaphores:
$ svrmgrl
svrmgr) connect internal
svrmgr) oradebug ipc

Repeat these commands for each running instance making note of the "Semaphore identifiers" for each.

Step 3)

Now get a list of all of the allocated semaphores on the system using the following command:

% ipcs -sb | grep -v grep | grep (oracle)
where (oracle) is the name of the oracle user.

Compare the output of the ipcs command with the notes you made from the sysresv command or oradebug ipc command , and remove any semaphores that do not belong to a running instance.

Step 4)
You can remove semaphores using the following command:
% ipcrm -s (semid)
where (semid) is the semaphore id shown in the second column of the ipcs output titled ID.

Step 5)
Now try to start your instance again.


Scenario 2
----------
You are trying to start a new instance, or an existing instance that had the initSID.ora parameter PROCESSES changed. You will need to raise the kernel parameter SEMMNS.

Set the kernel parameter SEMMNS to double the sum of all of the initSID.ora PROCESSES parameters on the system.

For example, if you have 5 instances with the initSID.ora PROCESSES parameter set to 25, 50, 100, 200, and 250, then set the kernel parameter SEMMNS equal to 1250.

((25 + 50 + 100 + 200 + 250) * 2) = 1250

After making changes to the kernel parameter SEMMNS, you will be required to reboot the system. Depending on your operating system, you may also be required to rebuild the kernel. For more information on changing kernel parameters, please consult your OS documentation and/or your OS vendor.

Oracle Merge Statements

Syntax: -
MERGE (hint) INTO (table_name)
USING (table_view_or_query)
ON ((condition))
WHEN MATCHED THEN (update_clause)
DELETE (where_clause)
WHEN NOT MATCHED THEN (insert_clause)
[LOG ERRORS (log_errors_clause) (reject limit (integer unlimited)];

Ex: -

CREATE TABLE employee (employee_id NUMBER(5),first_name VARCHAR2(20),last_name VARCHAR2(20),dept_no NUMBER(2),salary NUMBER(10));
INSERT INTO employee VALUES (1, 'Dan', 'Morgan', 10, 100000);INSERT INTO employee VALUES (2, 'Helen', 'Lofstrom', 20, 100000);INSERT INTO employee VALUES (3, 'Akiko', 'Toyota', 20, 50000);INSERT INTO employee VALUES (4, 'Jackie', 'Stough', 20, 40000);INSERT INTO employee VALUES (5, 'Richard', 'Foote', 20, 70000);INSERT INTO employee VALUES (6, 'Joe', 'Johnson', 20, 30000);INSERT INTO employee VALUES (7, 'Clark', 'Urling', 20, 90000);

CREATE TABLE bonuses (employee_id NUMBER, bonus NUMBER DEFAULT 100);
INSERT INTO bonuses (employee_id) VALUES (1);INSERT INTO bonuses (employee_id) VALUES (2);INSERT INTO bonuses (employee_id) VALUES (4);INSERT INTO bonuses (employee_id) VALUES (6);INSERT INTO bonuses (employee_id) VALUES (7);COMMIT;

SELECT * FROM employee;
SELECT * FROM bonuses;

MERGE INTO bonuses bUSING ( SELECT employee_id, salary, dept_no FROM employee WHERE dept_no =20) eON (b.employee_id = e.employee_id)WHEN MATCHED THEN UPDATE SET b.bonus = e.salary * 0.1 DELETE WHERE (e.salary <> 40000);

SELECT * FROM bonuses;