Oracle 11gr2 RAC database cannot be connected (ora-12537: TNS: connection closed)

Oracle 11gR2 RAC database can not connect (ORA-12537: TNS:connection closed) solution

 

[oracle@rac01 ~]$ sqlplus /nolog

SQL*Plus: Release 11.2.0.1.0 Production on Wed Aug 29 21:06:10 2012

Copyright (c) 1982, 2009, Oracle. All rights reserved.

SQL> connect system/***@astt
ERROR:
ORA-12537: TNS:connection closed

[root@rac01 app]# ls -ltr
drwxr-xr-x 3 root oinstall 4096 08-28 19:05 grid
drwxrwx— 6 grid oinstall 4096 08-29 00:03 oraInventory
drwxrwxr-x 7 oracle oinstall 4096 08-29 00:20 oracle
[root@rac01 app]# chmod 755 oracle
[root@rac01 app]# chmod 755 grid
[root@rac01 app]# ls -ltr
drwxr-xr-x 3 root oinstall 4096 08-28 19:05 grid
drwxrwx— 6 grid oinstall 4096 08-29 00:03 oraInventory
drwxr-xr-x 7 oracle oinstall 4096 08-29 00:20 oracle

[oracle@rac01 ~]$ sqlplus /nolog

SQL*Plus: Release 11.2.0.1.0 Production on Wed Aug 29 21:28:23 2012

Copyright (c) 1982, 2009, Oracle. All rights reserved.

SQL> connect system/***@astt;
????
SQL>

SQL> select instance_name from gv$instance;

INSTANCE_NAME
—————-
astt1
astt2

SQL> select instance_name from v$instance;

INSTANCE_NAME
—————-
astt1

SQL>

metalink [ID 1069517.1]:

Symptoms

In environment where listener home (including SCAN listener which resides in Grid Infrastructure/ASM home) and database home are owned by different OS user, ORA-12537 could happen when connecting through listener, when creating database through DBCA, or when installing database software and creating a database in runInstaller. Job Role Separation is a typical example as SCAN and local grid home listener is owned differently than database.

Error detail

12537, 00000, “TNS:connection closed”

// *Cause: “End of file” condition has been reached; partner has disconnected.

// *Action: None needed; this is an information message.

Before the error is reported, connection could hang for a while, and a core file may also be generated.

Screen output:

SQL> conn
system/xxx@OSES

ERROR:

ORA-12537: TNS:connection closed

OR

PRCR-1079 : Failed to start resource ora.db11.db

CRS-5011: Check of resource db11 failed: details at ..

CRS-2674: Start of ora.db11.db on racnode1 failed

ORA-12546: TNS:permission denied

listener_scann.log or listener.log

TNS-12518: TNS:listener could not hand off client connection

TNS-12546: TNS:permission denied

TNS-12560: TNS:protocol adapter error

TNS-00516: Permission denied

Linux Error: 13: Permission denied

OR

TNS-12518: TNS:listener could not hand off client connection

TNS-12547: TNS:lost contact

TNS-12560: TNS:protocol adapter error

TNS-00517: Lost contact

Linux Error: 32: Broken pipe

runInstaller error if installing

INFO: Starting Output Reader Threads for process /ocw/grid/bin/kfod

INFO: Parsing KFOD-00300: OCI error [-1] [OCI error] [ORA-12547: TNS:lost contact

INFO: Parsing ] [12547]

INFO: Parsing

INFO: The process /ocw/grid/bin/kfod exited with code 1

..

SEVERE: [FATAL] [INS-30502] No ASM disk group found.

CAUSE: There were no disk groups managed by the ASM instance +ASM1.

Cause

1. Listener owner (including SCAN listener) can not access oracle binary in database home:

As listener owner:

$ ls -l $RDBMS_HOME/bin/oracle

ls: /home/oracle/app/oracle/product/11.2/db/bin/oracle: Permission denied

2. Oracle binary in database home has wrong permission:

ls -l $RDBMS_HOME/bin/oracle

-rwxr-x–x 1 oracle asmadmin 184286251 Aug 9 16:25 /home/oracle/app/oracle/product/11.2/db/bin/oracle

The permission “-rwxr-x–x” is wrong as it’s missing suid bit, oracle binary should have permission of 6751:

-rwsr-s–x 1 oracle asmadmin 184286251 Aug 9 16:25 /home/oracle/app/oracle/product/11.2/db/bin/oracle

Note: If Job Role Separation is in place, the group will be , otherwise it will be

3. File System for database home does not support setuid/suid or has nosetuid/nosuid set:

mount| grep

/home/oracle on /dev/dsk/diskoracle read/write/nosuid..

Solution

Solution is to make sure file system for database home has setuid/suid set, database binary($RDBMS_HOME/bin/oracle) has correct ownership and permission, and listener owner is able to access database oracle binary (as listener owner, “ls -l $RDBMS_HOME/bin/oracle” will tell)

If its Job Role Separation environment, please refer to the following document:

Oracle® Grid Infrastructure

Installation Guide

11g Release 2 (11.2)

Oracle ASM Job Role Separation Option with SYSASM

The official explanation is as follows.

Applies to:
Oracle Net Services – Version: 8.1.7.0 to 11.2.0.1 – Release: 8.1.7 to 11.2
Generic UNIX
Checked for relevance on 30-Apr-2010
Symptoms
Connection to the database server fails with ORA-12546 or ORA-12537

Local connections working fine but the connection via listener fails with ORA-12537 or ORA-12547

[oracle@myhost]$ sqlplusmyuser/mypass@mytns

SQL*Plus: Release 11.1.0.7.0 – Production on Fri Apr 30 21:32:18 2010

Copyright (c) 1982, 2008, Oracle. All rights reserved.

ERROR:
ORA-12537: TNS:connection closed

The listener log for the failing connection shows following error message –

TNS-12518: TNS:listener could not hand off client connection
TNS-12546: TNS:permission denied
TNS-12560: TNS:protocol adapter error
TNS-00516: Permission denied
Linux Error: 13: Permission denied

Or

TNS-12518: TNS:listener could not hand off client connection
TNS-12547: TNS:lost contact
TNS-12560: TNS:protocol adapter error
TNS-00517: Lost contact
IBM/AIX RISC System/6000 Error: 32: Broken pipe

Cause
Permissions on oracle binary ( $ORACLE_HOME/bin/oracle ) are not set correctly.

Solution
The OS level error message in the listener log “Linux Error: 13: Permission denied” indicates that the listener was not able to hand off the connection to oracle processes or not able to spawn a new dedicated user process because of lack of permission at the OS level. To resolve the error –

1. Check and correct the permission on /var/tmp/.oracle directory. Permissions on this directory should be 777 with sticky bit set.

# chmod 01777 /var/tmp/.oracle

2. Check and correct permissions on $ORACLE_HOME/bin/oracle binary and make sure that it is set to 6751.

$ chmod 6751 $ORACLE_HOME/bin/oracle

Check your own environment and indeed find that it is not the correct permission

[root@ract1 bin]# pwd
/oracle/product/11.2.0/bin
[root@ract1 bin]# ls -la oracle
-rwxr-s–x 1 oracle asmadmin 228886426 Jan 7 15:46 oracle
[root@ract1 bin]# chmod 6751 oracle- -Note that the root user is used to give permissions
[root@ract1 bin]# ls -la oracle
-rwsr-s–x 1 oracle asmadmin 228886426 Jan 7 15:46 oracle

After the modification, the login is normal.

[oracle@ract2 ~]$ sqlplussys/[email protected]:1521/racdbas sysdba

SQL*Plus: Release 11.2.0.2.0 Production on Thu Jan 17 14:48:25 2013

Copyright (c) 1982, 2010, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 – 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

Similar Posts: