ora-01115 solution

karson

Problem Description:
====================

An ORA-01115 is issued whenever Oracle is unable to read from an open
datafile because of an I/O error:

ORA-01115: "IO error reading block from file %s (block # %s)"
     Cause: Device on which the file resides is probably offline
    Action: Restore access to the device

ORA-01115 errors are usually followed by:

- an ORA-01110 error
- an operating system level Oracle error message such as ORA-0737X
- an operating system error (e.g., error# 5 in Unix)


Solution Description:
=====================

Because most ORA-01115s are caused by hardware problems, the solution
consists in first isolating those, and then addressing the problem at
the database level, if necessary.

PERFORMING HARDWARE CHECKS IS ESSENTIAL.  If hardware problems are not
fixed, trying to solve the problem at the database level will be useless.
Run operating system level utilities and diagnostic tools that check
for the sanity of disks, controllers, and the I/O subsystem.  Pay special
attention to the disk where the datafile referenced in the ORA-01115 resides.
Your system administrator should be able to assist you in this task.
Such diagnostics should be done in parallel with the steps
recommended here, if feasible, or as soon as possible thereafter.

Determining the exact cause of an ORA-01115 is not always trivial.
Approaches differ according to whether you know the cause of the problem
or not.


I. STEPS FOR SOLVING THE PROBLEM WHEN THE CAUSE IS NOT KNOWN
------------------------------------------------------------

1. Try to assess the cause and extent of the problem.

        Examine the alert.log file for this instance.  Scan the last few
        days' entries for other occurrences of ORA-01115. If you find them,

        A) Do they reference files in different disks?
                If so, it is likely that there you have controller problems.
                Move on to Scenario II.A below.
        B) Do they reference different files in the same disk?
                If so, it is likely that there are problems with that disk.
                Move on to Scenario II.B below.
        C) Do they always reference the same datafile?
                If so, it is likely that the datafile contains bad blocks.
                Move on to Scenario II.C below.  If the file is bigger
                than 2GB and you are running 7.1.4 or lower on a Solaris
                platform, see Scenario II.D below.
        D) If none of the above applies, move on to Step 2.

2. If the datafile is in the SYSTEM tablespace, or the database is in
   NOARCHIVELOG mode, shut the database down. Move on to Step 4.

        If shutdown immediate fails, do a shutdown abort.

3. If the database is in ARCHIVELOG mode, you should still shut the
   database down.  If the database cannot be shut down, offline the
   datafile.

        ALTER DATABASE DATAFILE '<full_path_file_name>' OFFLINE;

4. Try to copy the datafile to another disk (managed by a different
   controller, if possible).

5. If the copy fails, even after you retry, the datafile must be considered
   lost at this point.  The next action depends on the tablespace to
   which the lost file belongs. See the following Solution References to
   PR entries, according to the different types of tablespaces,
   for instructions on how to proceed.

   IMPORTANT: While going through the references below, keep in mind that
   if you restore the datafile from backup, you need to place it in another
   disk, preferably under a different controller, and rename it inside Oracle
   (see [NOTE:115424.1] for details).  If you recreate any tablespace, make
   sure its datafiles are created in another disk, preferably under a
   different controller.

        TABLESPACE      Reference
        ----------      ---------
        system          [NOTE:1013182.6]
        rollback        [NOTE:1013221.6]
        user            [NOTE:1013173.6]
        index           [NOTE:1013115.6]
        temporary       [NOTE:1013104.6]
        read-only       [NOTE:1013129.6]

6. If the database is down, mount it.

7. Rename the datafile that you succeeded in copying inside Oracle.

        ALTER DATABASE RENAME FILE '<old_full_path_file_name>'
        TO '<new_full_path_file_name>';

8. If the database is mounted, open it.  If you offlined the datafile,
   perform media recovery on it, and then bring it online.

        RECOVER DATAFILE '<full_path_file_name>';

        ALTER DATABASE DATAFILE '<full_path_file_name>' ONLINE;


II. STEPS FOR SOLVING THE PROBLEM WHEN THE CAUSE IS KNOWN
---------------------------------------------------------

II.A  CONTROLLER PROBLEMS
-------------------------

These are typically intermittent.  Usually, there is no damage to the
datafiles.   Unless you can quickly fix the controller and restore
access to the datafile, follow these steps:

1. Find out which datafiles are under the bad controller.

        Query V$DATAFILE to obtain the names of all datafiles in the
        database. You may need the help of the system administrator
        to determine which datafiles reside in disks managed by this
        controller.

2. If any of the datafiles under the bad controller belongs to the SYSTEM
   tablespace, or if the database is in NOARCHIVELOG mode, shut the database
   down.  Move on to step 4.

        If shutdown immediate fails, do a shutdown abort.

3. If the database is in ARCHIVELOG mode and none of the datafiles under
   the bad controller are in the SYSTEM tablespace, you should shut the
   database down.  If the database cannot be shut down,
   offline all the datafiles under the bad controller.

        ALTER DATABASE DATAFILE '<full_path_file_name>' OFFLINE;

4. Try to copy all the datafiles under the bad controller to disks
   managed by different controllers.

5. If the database is down, mount it.

6. Rename all the files that you succeeded in copying inside Oracle.

        ALTER DATABASE RENAME FILE '<old_full_path_file_name>'
        TO '<new_full_path_file_name>';

7. If the copy fails for one or more of the datafiles, even after you retry
   copying them, those datafiles have to be considered lost at this point.
   See the following Solution References to PR entries, according to the
   tablespaces to which the lost datafiles belong, for instructions on
   how to proceed.

   IMPORTANT: While going through the references below, keep in mind that if
   you restore datafiles from backup, you need to place them in disks under
   other controllers and rename them inside Oracle (see [NOTE:115424.1] for
   details). If you recreate any tablespace, make sure its datafiles are
   created under other controllers.

        TABLESPACE      REFERENCE
        ----------      ---------
        system          [NOTE:1013182.6]
        rollback        [NOTE:1013221.6]
        user            [NOTE:1013173.6]
        index           [NOTE:1013115.6]
        temporary       [NOTE:1013104.6]
        read-only       [NOTE:1013129.6]

8. If the database is mounted, open it.  If any of the moved datafiles
   is offline, apply media recovery to it, and then online it:

        RECOVER DATAFILE '<full_path_file_name>';
         
        ALTER DATABASE DATAFILE '<full_path_file_name>' ONLINE;

karson

II.B  DISK PROBLEMS
-------------------

If you know for a fact that a disk has bad blocks or is malfunctioning,
you should focus on moving the datafiles in it to a different disk, if at
all possible.  If not, you must consider the files lost and address the
issue according to the tablespaces to which they belong, while working
in parallel on fixing the disk.  The steps to follow in this scenario
are analogous to those in Scenario II.A above.


II.C  DATA BLOCK CORRUPTION
---------------------------

If you are sure that the datafile has bad blocks, it should be considered LOST
if it belongs to the SYSTEM tablespace or to a ROLLBACK or READ-ONLY tablespace.
See the following References, depending on the tablespace to which the datafile
belongs.

   IMPORTANT: While going through the references below, keep in mind that if
   you restore datafiles from backup, you need to place them in different disks
   (preferably under other controllers) and rename them inside Oracle (see the
   [NOTE:115424.1] for details). If you recreate any tablespace, make sure its
   datafiles are created on different disks (preferably under other
   controllers).

        TABLESPACE      REFERENCE
        ----------      ---------
        system          [NOTE:1013182.6]
        rollback        [NOTE:1013221.6]
        user            [NOTE:1013173.6]
        index           [NOTE:1013115.6]
        temporary       [NOTE:1013104.6]
        read-only       [NOTE:1013129.6]

If the datafile belongs to a user or index tablespace,  you may also
address the problem as an object recreation issue if the ORA-01115
occurs consistently against the same objects (tables, indexes, etc.).
The following query returns the object in which the bad block is:

        SELECT SEGMENT_NAME, SEGMENT_TYPE FROM DBA_EXTENTS
        WHERE FILE_ID = <file_number> and <block_number> BETWEEN BLOCK_ID
        AND BLOCK_ID + BLOCKS - 1;

where <file_number> and <block_number> are those listed in the ORA-01115.
If this query consistently points to a table or index, you may try
recreating them, if possible in a different tablespace.


II.D  VERY LARGE DATAFILE PROBLEMS ON SOLARIS
---------------------------------------------

If you are running Oracle 7.1.4 or lower on a Solaris platform, and
you get an ORA-07371 with the ORA-01115, and the file is
bigger than 2GB, you are very likely running into [BUG:233569].
This bug is fixed in 7.1.6 and patches are available for 7.1.3
([BUG:233569]) and 7.1.4 ([BUG:281904]).


Explanation:
============

What causes ORA-01115 error?
----------------------------

Oracle hands over read from file requests to the underlying operating system
(except if raw devices are being used).  A read request specifies a
datafile and a block number to be accessed.  If a low-level I/O error
prevents the read from completing successfully, Oracle signals an
ORA-01115.

The main causes for an ORA-01115 are:

1. HARDWARE PROBLEMS

- Disk controller problems: the most common, and usually intermittent.

- Disk problems: these include bad blocks, disk malfunctioning, etc.

2. DATA BLOCK CORRUPTION (AT THE PHYSICAL LEVEL)

Usually caused by previous hardware problems.

3. PROBLEMS HANDLING VERY LARGE DATAFILES

In Oracle 7.1.4 and lower on Sun Solaris, [BUG:233569] causes ORA-01115
and ORA-07371 when handling datafiles bigger than 2GB.

Typical scenarios where ORA-01115 can happen include:
- On execution of DML statements
- During exports or imports
- At startup or shutdown