ORA-00752 on standby and DB_ULTRA_SAFE parameter

Alert file from standby database got following messages:

ORA-00752: recovery detected a lost write of a data block
ORA-10567: Redo is inconsistent with data block (file# 8, block# 351475)
ORA-10564: tablespace TD_DATA
ORA-01110: data file 8: ‘+TD_DATA/test/datafile/test01.dbf’
ORA-10561: block type ‘TRANSACTION MANAGED DATA BLOCK’, data object# 105684

and second part

ORA-10877: error signaled in parallel recovery slave PR00
ORA-00752: recovery detected a lost write of a data block
MRP0: Background Media Recovery process shutdown (test1)

The alert file  shows that an ORA-00752 error is raised on the standby database
and the managed recovery is cancelled:

The recommended procedure to recover from such errors is a failover to the physical standby!!!

If you read Oracle Documentation

http://download.oracle.com/docs/cd/B28359_01/server.111/b28294.pdf

Chapter 13.6

During media recovery in a Data Guard configuration, a physical standby database
can be used to detect lost-write data corruption errors on the primary database. This is
done by comparing SCNs of blocks stored in the redo log on the primary database to
SCNs of blocks on the physical standby database. If the SCN of the block on the
primary database is lower than the SCN on the standby database, then there was a
lost-write error on the primary database. Read more of this post

On Database Trigger and current schema issue

I run into this issue few months ago:

Login as sysdba

bash-3.2$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Tue Jan 5 15:13:49 2010

Copyright (c) 1982, 2009, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 – 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL>  create table test as select * from dba_tables;
create table test as select * from dba_tables
*
ERROR at line 1:
ORA-01950: no privileges on tablespace ‘USERS’

or try to compile invalid objects:

SQL>  @?/rdbms/admin/utlrp.sql
SELECT dbms_registry_sys.time_stamp(‘utlrp_bgn’) as timestamp from dual
*
ERROR at line 1:
ORA-00904: “DBMS_REGISTRY_SYS”.”TIME_STAMP”: invalid identifier

——–

PL/SQL procedure successfully completed.

I got a bunch of errors.It seems that SYSDBA  does not have enough privileges to run this code.

SQL> show user
USER is “SYS”
Read more of this post

Tracing Oracle Data Guard

Data Guard Broker (DGMGRL) can connect to database using following options:

-echo
-silent
-xml
-debug
-logfile

So if I need to run dgmgrl command in debug mode I would execute following :

dgmgrl -debug / or

-bash-3.2$ dgmgrl -debug sys/secret@test

DGMGRL for Linux: Version 11.2.0.1.0 - 64bit Production
Copyright (c) 2000, 2009, Oracle. All rights reserved.
Welcome to DGMGRL, type "help" for information.
[W000 01/05 12:27:03.81] Connecting to database using test.
[W000 01/05 12:27:03.85] Executing query [select dbms_drs.dg_broker_info('VERSION') from dual].
[W000 01/05 12:27:03.86] Query result is '11.2.0.1.0'
Connected.

or dgmgrl  -debug / shutdown command  will shutdown instance ,generate trace file and then exit.

If Data Guard Broker is used then tracing can be enabled using DGMGRL command
and setting property LogArchiveTrace to value other than 0.

/** Tracing primary site **/

DGMGRL> edit database ‘primary_db_name’ set property LogArchiveTrace=’1′;

/** Tracing standby site **/

DGMGRL> edit database ‘standby_db_name’ set property LogArchiveTrace=’1′;

Also this parameter can be set by issueing a SQL statement :

SQL> ALTER SYSTEM SET LOG_ARCHIVE_TRACE=1;

Values are described in table bellow:

Level Meaning
0 Disables archived redo log tracing (default setting)
1 Tracks archiving of log files
2 Tracks archive status by archive log file destination
4 Tracks archive operational phase
8 Tracks archive log destination activity
16 Tracks detailed archive log destination activity
32 Tracks archive log destination parameter modifications
64 Tracks ARCn process state activity
128 Tracks FAL server process activity
256 Tracks RFS Logical Client
512 Tracks LGWR redo shipping network activity
1024 Tracks RFS physical client
2048 Tracks RFS/ARCn ping heartbeat
4096 Tracks real-time apply activity
8192 Tracks Redo Apply activity (media recovery or physical standby)

Open database with delay

The event db_open_begin delay (opening database with delay)  is undocumented event ( read unsupported ) and it will do exactly what is says e.g. open database with delay .Although alter database open command is executed it will wait for specified interval before opening database.

So here is scenario:

SQL> startup mount;
ORACLE instance started.

Total System Global Area 2973003776 bytes
Fixed Size                  2217064 bytes
Variable Size            2080377752 bytes
Database Buffers          872415232 bytes
Redo Buffers               17993728 bytes
Database mounted.
SQL> oradebug setmypid
Statement processed.
SQL> oradebug event db_open_begin delay=10
Statement processed.
SQL> set timing on
SQL> alter database open;

Database altered.

Elapsed: 00:00:11.51

Database was opened after approx 12 sec ( 10 sec delay and 1.51sec to run all oracle code related to opening database  starting with  bootstrap$  object. )

alter system command will work too:

SQL> alter system set events ‘db_open_begin delay =50′;

High Availability – Oracle GoldenGate

The GoldenGate ( now owned by Oracle Corporation ) for Active-Active solution provides  real-time bidirectional data movement between heterogeneous dual-active or multi-master databases. GoldenGate’s Active-Active solution also offers advanced features for data conflict detection and resolution.

KEY FEATURES:

  • Allows application transactions to be distributed across multiple databases
  • Provides full bi-directional data movement without distance constraints
  • Allows conflict detection and resolution
  • Enables sub-second latency even at high data volumes
  • Preserves transaction integrity for confidence in data across multiple active environments
  • Decouples source and target environments to support a wide variety of heterogeneous databases and platforms

 

Main page:

http://www.oracle.com/technology/products/goldengate/index.html

Documentation:

http://download.oracle.com/docs/cd/E15881_01/index.htm

Additional downloads from  Oracle E delivery site:

http://edelivery.oracle.com/EPD/Download/get_form?egroup_aru_number=11897207

Red Hat Enterprise Virtualization (RHEV)

If you haven’t heard Red Hat released Red Hat Enterprise Virtualization (RHEV).  It has a lot of advantages over proprietary virtualization solutions.

http://www.redhat.com/virtualization/rhev/

http://www.redhat.com/v/swf/rhev/demo.html

Find waiter with Oradebug 11gR2

There is small addition to oradebug unit_test command in 11g R2 .

oradebug unit_test per_session_find_one_waiter

This command has four mandatory arguments and one optional.

First parameter is find_waiters_for which can have two different values : current_sess or all_local_sess .Self descriptive.
Second parameter is wait_event.This is name of the wait event which search is done.Third and fourth parameters are related to time and rusults.

For my simple test I will test this command on event enq TX -row lock contention for my own session.

Session 1:

create table test(a int primary key);

insert into test select rownum from dba_tables where rownum <9;
commit;
update test set a=9 where a=1;
update test set a=10 where a=2;

Session 2:

update test set a=11 where a=3;
update test set a=12 where a=4;

SQL> /

PID SPID                            SID USERNAME                          SERIAL# SQL_HASH_VALUE
———- ———————— ———- —————————— ———- ————–
22 1030                            190 SYS                                  1115     3889092034

Session 1 again:

SQL> /

PID SPID                    SID USERNAME                          SERIAL# SQL_HASH_VALUE
———- ———————— ———- —————————— ———- ————–
21 31360                   156 SYS                                 59123     3889092034

update test set a=25 where a=8;
update test set a=13 where a=3;

Now I got  event to test oradebug command:

oradebug unit_test per_session_find_one_waiter find_waiters_for=current_sess wait_event=”enq: TX – row lock contention” waiter_min_secs_blkd=300 min_results=1

SQL> oradebug unit_test per_session_find_one_waiter find_waiters_for=current_sess wait_event=”enq: TX – row lock contention” waiter_min_secs_blkd=1 min_results=1

Here is output:

WAITERS_FOUND_BEGIN
(inst=1, sid=190, osid=1030) is blocking (inst=1, sid=156) for at least 74 secs
WAITERS_FOUND_END

if I run same command again

SQL> oradebug unit_test per_session_find_one_waiter find_waiters_for=current_sess wait_event=”enq: TX – row lock contention” waiter_min_secs_blkd=1 min_results=1

WAITERS_FOUND_BEGIN
(inst=1, sid=190, osid=1030) is blocking (inst=1, sid=156) for at least 221 secs
WAITERS_FOUND_END

wait time will increase.

Let’s increase waiter_min_secs_blkd=300 and run command:

SQL> oradebug unit_test per_session_find_one_waiter find_waiters_for=current_sess wait_event=”enq: TX – row lock contention” waiter_min_secs_blkd=300 min_results=1
WAITERS_FOUND_BEGIN
(inst=1, sid=190, osid=1030) is blocking (inst=1, sid=156) for at least 301 secs
WAITERS_FOUND_END

and I got output after aprox 301 secs..

ORION (Oracle I/O Calibration Tool) included in 11g R2

Apparently Oracle  included this tool into 11g release 2 database.Orion binary is located under  $ORACLE_HOME/bin .

Quick run with help command will give very detail explanation how to use this tool.Also

Kevin Closson’s Oracle blog got post about Orion:

http://kevinclosson.wordpress.com/2006/12/11/a-tip-about-the-orion-io-generator-tool/

-bash-3.2$ orion -help
ORION: ORacle IO Numbers — Version 11.2.0.1.0

ORION runs IO performance tests that model Oracle RDBMS IO workloads.
It measures the performance of small (2-32K) IOs and large (128K+) IOs
at various load levels.
Read more of this post

Uncommitted transactions

Let say someone have an update,insert or delete statement that runs but has not been commited yet.So question is how to show uncommitted transactions ?

First some background about transaction management in Oracle.

A transaction is a logical unit of work that contains one or more SQL statements.

A transaction ends when any of the following occurs:

A user issues a COMMIT or ROLLBACK statement without a SAVEPOINT clause.
A user runs a DDL statement such as CREATE, DROP, RENAME, or ALTER. If the current transaction contains any DML statements,
Oracle first commits the transaction, and then runs and commits the DDL statement as a new, single statement transaction.
A user disconnects from Oracle. The current transaction is committed.
A user process terminates abnormally. The current transaction is rolled back.

When a transaction is committed, the following occurs:

1  The internal transaction table for the associated undo tablespace that the transaction has committed,
and the corresponding unique system change number (SCN) of the transaction is assigned and recorded in the table.

This can be seen using v$transaction view.To get more  info  I normally join v$transaction and v$session views.

e.g.

select t.start_time,s.sid,s.serial#,s.username,s.status
from v$transaction t, v$session s
where s.saddr = t.ses_addr

2. The log writer process (LGWR) writes redo log entries in the SGA’s redo log buffers to the redo log file.
It also writes the transaction’s SCN to the redo log file. Read more of this post

Blocking locks history

In my previous blogs I have already wrote script for   monitoring blocking locks in real time.

http://oraclue.com/2009/05/18/procedure-to-kill-blocking-session-in-rac-11g/

But what if I am accessing database next day or week later and trying to find out who caused blocking lock? I know there are trace files etc but this time I will use v$views.

Let me start with familiar  scenario to produce blocking lock (  I have use same code for deadlock scenario in previous blogs):

Session 1:

create table test(a int primary key);

insert into test select rownum from dba_tables where rownum <9;
commit;
update test set a=9 where a=1;
update test set a=10 where a=2;

Table created.
8 rows created.
Commit complete.
1 row updated.
1 row updated.

Session 2:

update test set a=11 where a=3;

update test set a=12 where a=4;
1 row updated.
1 row updated.

again session 1:

update test set a=25 where a=8;
update test set a=13 where a=3;

So to detect blocking lock that occurred in the past  I   use active session history or to be precise column blocking_session from view V$ACTIVE_SESSION_HISTORY. Note that  blocking_session column is populated only when the session was waiting for enqueues or a “buffer busy” wait.    

For mutex related waits, v$session.blocking_session is not populated in 10.2.

Use Metalink Note 786507.1

   

v$session.blocking_session is populated in 11g .

My  simple script  looks like: Read more of this post