Tag Archives: ORA03113

Oracle startup error ora-03113 solution

Environment: rhel6.4 + Oracle 11.2.0.4

Step summary:
1. Start the error ora-03113
2. Check the alert log to find the cause
3. Take reasonable measures according to the actual situation. Here, we first increase the size of the flash back area and start the library
4. Formulate the corresponding Archive log deletion strategy

1. Startup error ora-03113

$ sqlplus/as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on Stauday Feb 28 13:56:44 2015

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup mount
ORACLE instance started.

Total System Global Area 3206836224 bytes
Fixed Size                  2257520 bytes
Variable Size            2365590928 bytes
Database Buffers          822083584 bytes
Redo Buffers               16904192 bytes
Database mounted.
SQL> alter database open; 
alter database open
*
ERROR at line 1:
ORA-03113: End of file for communication channel
Process ID: 13501
Session ID: 853 Sequence number: 5


SQL>

2. Check the alert log to find the reason

Sat Feb 28 13:56:52 2015
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Initial number of CPU is 8
CELL communication is configured to use 0 interface(s):
CELL IP affinity details:
    NUMA status: non-NUMA system
    cellaffinity.ora status: N/A
CELL communication will use 1 IP group(s):
    Grp 0: 
Picked latch-free SCN scheme 3
Using LOG_ARCHIVE_DEST_1 parameter default value as USE_DB_RECOVERY_FILE_DEST
Autotune of undo retention is turned on. 
IMODE=BR
ILAT =249
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options.
ORACLE_HOME = /opt/app/oracle/product/11.2.0.4/dbhome_1
System name:    Linux
Node name:      JJFAB01
Release:        2.6.32-358.el6.x86_64
Version:        #1 SMP Tue Jan 29 11:47:41 EST 2013
Machine:        x86_64
VM name:        Xen Version: 4.1 (PVM)
Using parameter settings in server-side spfile /opt/app/oracle/product/11.2.0.4/dbhome_1/dbs/spfileJJFAB.ora
System parameters with non-default values:
  processes                = 1500
  sessions                 = 2272
  memory_target            = 3G
  control_files            = "/opt/oradata/gxwj/control01.ctl"
  control_files            = "/opt/app/oracle/fast_recovery_area/gxwj/control02.ctl"
  db_block_size            = 8192
  compatible               = "11.2.0.0.0"
  db_recovery_file_dest    = "/opt/app/oracle/fast_recovery_area"
  db_recovery_file_dest_size= 4122M
  undo_tablespace          = "UNDOTBS1"
  remote_login_passwordfile= "EXCLUSIVE"
  db_domain                = ""
  dispatchers              = "(PROTOCOL=TCP) (SERVICE=gxwjXDB)"
  audit_file_dest          = "/opt/app/oracle/admin/gxwj/adump"
  audit_trail              = "DB"
  db_name                  = "JJFAB"
  open_cursors             = 300
  diagnostic_dest          = "/opt/app/oracle"
Sat Feb 28 13:56:54 2015
PMON started with pid=2, OS id=13461 
Sat Feb 28 13:56:54 2015
PSP0 started with pid=3, OS id=13463 
Sat Feb 28 13:56:55 2015
VKTM started with pid=4, OS id=13465 at elevated priority
VKTM running at (1)millisec precision with DBRM quantum (100)ms
Sat Feb 28 13:56:55 2015
GEN0 started with pid=5, OS id=13469 
Sat Feb 28 13:56:55 2015
DIAG started with pid=6, OS id=13471 
Sat Feb 28 13:56:55 2015
DBRM started with pid=7, OS id=13473 
Sat Feb 28 13:56:55 2015
DIA0 started with pid=8, OS id=13475 
Sat Feb 28 13:56:55 2015
MMAN started with pid=9, OS id=13477 
Sat Feb 28 13:56:55 2015
DBW0 started with pid=10, OS id=13479 
Sat Feb 28 13:56:55 2015
LGWR started with pid=11, OS id=13481 
Sat Feb 28 13:56:55 2015
CKPT started with pid=12, OS id=13483 
Sat Feb 28 13:56:55 2015
SMON started with pid=13, OS id=13485 
Sat Feb 28 13:56:55 2015
RECO started with pid=14, OS id=13487 
Sat Feb 28 13:56:55 2015
MMON started with pid=15, OS id=13489 
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
Sat Feb 28 13:56:55 2015
MMNL started with pid=16, OS id=13491 
starting up 1 shared server(s) ...
ORACLE_BASE from environment = /opt/app/oracle
Sat Feb 28 13:56:56 2015
ALTER DATABASE   MOUNT
Successful mount of redo thread 1, with mount id 3498004520
Database mounted in Exclusive Mode
Lost write protection disabled
Completed: ALTER DATABASE   MOUNT
Sat Feb 28 13:57:48 2015
alter database open
Sat Feb 28 13:57:48 2015
LGWR: STARTING ARCH PROCESSES
Sat Feb 28 13:57:48 2015
ARC0 started with pid=20, OS id=13553 
ARC0: Archival started
LGWR: STARTING ARCH PROCESSES COMPLETE
ARC0: STARTING ARCH PROCESSES
Sat Feb 28 13:57:49 2015
ARC1 started with pid=21, OS id=13555 
Sat Feb 28 13:57:49 2015
ARC2 started with pid=22, OS id=13557 
Errors in file /opt/app/oracle/diag/rdbms/jjfab/JJFAB/trace/JJFAB_ora_13501.trc:
ORA-19815: 警告: db_recovery_file_dest_size 字节 (共 4322230272 字节) 已使用 100.00%, 尚有 0 字节可用。
************************************************************************
You have following choices to free up space from recovery area:
1. Consider changing RMAN RETENTION POLICY. If you are using Data Guard,
   then consider changing RMAN ARCHIVELOG DELETION POLICY.
2. Back up files to tertiary device such as tape using RMAN
   BACKUP RECOVERY AREA command.
3. Add disk space and increase db_recovery_file_dest_size parameter to
   reflect the new space.
4. Delete unnecessary files using RMAN DELETE command. If an operating
   system command was used to delete files, then use RMAN CROSSCHECK and
   DELETE EXPIRED commands.
************************************************************************
ARCH: Error 19809 Creating archive log file to '/opt/app/oracle/fast_recovery_area/JJFAB/archivelog/2015_02_28/o1_mf_1_130_%u_.arc'
Errors in file /opt/app/oracle/diag/rdbms/jjfab/JJFAB/trace/JJFAB_ora_13501.trc:
ORA-16038: Log 1 sequence# 130 could not be archived
ORA-19809: Exceeded recovery file count limit
ORA-00312: Online log 1 thread 1: '/opt/oradata/gxwj/redo01.log'
Sat Feb 28 13:57:49 2015
ARC3 started with pid=23, OS id=13559 
USER (ospid: 13501): terminating the instance due to error 16038
Sat Feb 28 13:57:50 2015
System state dump requested by (instance=1, osid=13501), summary=[abnormal instance termination].
System State dumped to trace file /opt/app/oracle/diag/rdbms/jjfab/JJFAB/trace/JJFAB_diag_13471_20150228135750.trc
Dumping diagnostic data in directory=[cdmp_20150228135750], requested by (instance=1, osid=13501), summary=[abnormal instance termination].
Instance terminated by USER, pid = 13501

 

It is found that the reasons and suggestions have been clearly given in the alarm log

3. Take reasonable measures according to the actual situation. Here, we first increase the size of the flashback area and start the library

SQL> startup mount
ORA-24324: Service handle not initialized
ORA-01041: Internal error, hostdef extension does not exist
SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
$ sqlplus/as sysdba

SQL*Plus: Release 11.2.0.4.0 Production on 星期六 2月 28 14:05:59 2015

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup mount
ORACLE instance started.

Total System Global Area 3206836224 bytes
Fixed Size                  2257520 bytes
Variable Size            2365590928 bytes
Database Buffers          822083584 bytes
Redo Buffers               16904192 bytes
Database mounted.
SQL> show parameter db_recover

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest                string      /opt/app/oracle/fast_recovery_
                                                 area
db_recovery_file_dest_size           big integer 4122M
SQL> !
$ df -h /opt/
File System Capacity Used Available Used %% Mount Point
/dev/mapper/vg_extend-lv_opt
                      493G  423G   45G  91% /opt
$ exit
exit

SQL> alter system set db_recovery_file_dest_size=10G;

System altered.

SQL> alter database open;

Database altered.

SQL> select status from v$instance;

STATUS
------------
OPEN

SQL>

4. Formulate corresponding archive log deletion strategy

 

SQL> set linesize 120
SQL> select * from v$flash_recovery_area_usage;

FILE_TYPE            PERCENT_SPACE_USED PERCENT_SPACE_RECLAIMABLE NUMBER_OF_FILES
-------------------- ------------------ ------------------------- ---------------
CONTROL FILE                          0                         0               0
REDO LOG                              0                         0               0
ARCHIVED LOG                      41.54                         0             133
BACKUP PIECE                          0                         0               0
IMAGE COPY                            0                         0               0
FLASHBACK LOG                         0                         0               0
FOREIGN ARCHIVED LOG                  0                         0               0

7 rows selected.
You can rman directly delete archives that are no longer needed, for example, delete archives that are 1 day old
RMAN> delete noprompt archivelog until time 'sysdate-1';
The output of the deleted archive is omitted here...
RMAN> exit

Recovery Manager complete.

SQL> select * from v$flash_recovery_area_usage;

FILE_TYPE            PERCENT_SPACE_USED PERCENT_SPACE_RECLAIMABLE NUMBER_OF_FILES
-------------------- ------------------ ------------------------- ---------------
CONTROL FILE                          0                         0               0
REDO LOG                              0                         0               0
ARCHIVED LOG                        .35                         0               1
BACKUP PIECE                          0                         0               0
IMAGE COPY                            0                         0               0
FLASHBACK LOG                         0                         0               0
FOREIGN ARCHIVED LOG                  0                         0               0

7 rows selected.

Of course, you can also set crontab to delete archives 1 day ago every day:

$ crontab -l
00 04 * * * /opt/shell/del_arch.sh
$ more /opt/shell/del_arch.sh
#!/bin/bash
export ORACLE_BASE=/opt/app/oracle
export ORACLE_HOME=/opt/app/oracle/product/11.2.0.4/dbhome_1 
export ORACLE_SID=JJFAB
export NLS_LANG="simplified chinese_china.ZHS16GBK"
export NLS_DATE_FORMAT="YYYY -MM-DD HH24:Mi:SS"
export PATH=.:/opt/app/oracle/product/11.2.0.4/dbhome_1:/usr/sbin:$ORACLE_HOME/bin:$PATH
export LD_LIBRARY_PATH=$ORACLE_HOME/lib

rman target/log=/opt/backup/del_arch.log <<EOF
list archivelog all;
delete noprompt archivelog until time 'sysdate-1';
show all;
list archivelog all;
EOF