Oracle 11gR2 RAC database can not connect (ORA-12537: TNS:connection closed) solution
[oracle@rac01 ~]$ sqlplus /nolog
SQL*Plus: Release 11.2.0.1.0 Production on Wed Aug 29 21:06:10 2012
Copyright (c) 1982, 2009, Oracle. All rights reserved.
SQL> connect system/***@astt
ERROR:
ORA-12537: TNS:connection closed
[root@rac01 app]# ls -ltr
drwxr-xr-x 3 root oinstall 4096 08-28 19:05 grid
drwxrwx— 6 grid oinstall 4096 08-29 00:03 oraInventory
drwxrwxr-x 7 oracle oinstall 4096 08-29 00:20 oracle
[root@rac01 app]# chmod 755 oracle
[root@rac01 app]# chmod 755 grid
[root@rac01 app]# ls -ltr
drwxr-xr-x 3 root oinstall 4096 08-28 19:05 grid
drwxrwx— 6 grid oinstall 4096 08-29 00:03 oraInventory
drwxr-xr-x 7 oracle oinstall 4096 08-29 00:20 oracle
[oracle@rac01 ~]$ sqlplus /nolog
SQL*Plus: Release 11.2.0.1.0 Production on Wed Aug 29 21:28:23 2012
Copyright (c) 1982, 2009, Oracle. All rights reserved.
SQL> connect system/***@astt;
????
SQL>
SQL> select instance_name from gv$instance;
INSTANCE_NAME
—————-
astt1
astt2
SQL> select instance_name from v$instance;
INSTANCE_NAME
—————-
astt1
SQL>
metalink [ID 1069517.1]:
Symptoms
In environment where listener home (including SCAN listener which resides in Grid Infrastructure/ASM home) and database home are owned by different OS user, ORA-12537 could happen when connecting through listener, when creating database through DBCA, or when installing database software and creating a database in runInstaller. Job Role Separation is a typical example as SCAN and local grid home listener is owned differently than database.
Error detail
12537, 00000, “TNS:connection closed”
// *Cause: “End of file” condition has been reached; partner has disconnected.
// *Action: None needed; this is an information message.
Before the error is reported, connection could hang for a while, and a core file may also be generated.
Screen output:
SQL> conn
system/xxx@OSES
ERROR:
ORA-12537: TNS:connection closed
OR
PRCR-1079 : Failed to start resource ora.db11.db
CRS-5011: Check of resource db11 failed: details at ..
CRS-2674: Start of ora.db11.db on racnode1 failed
ORA-12546: TNS:permission denied
listener_scann.log or listener.log
TNS-12518: TNS:listener could not hand off client connection
TNS-12546: TNS:permission denied
TNS-12560: TNS:protocol adapter error
TNS-00516: Permission denied
Linux Error: 13: Permission denied
OR
TNS-12518: TNS:listener could not hand off client connection
TNS-12547: TNS:lost contact
TNS-12560: TNS:protocol adapter error
TNS-00517: Lost contact
Linux Error: 32: Broken pipe
runInstaller error if installing
INFO: Starting Output Reader Threads for process /ocw/grid/bin/kfod
INFO: Parsing KFOD-00300: OCI error [-1] [OCI error] [ORA-12547: TNS:lost contact
INFO: Parsing ] [12547]
INFO: Parsing
INFO: The process /ocw/grid/bin/kfod exited with code 1
..
SEVERE: [FATAL] [INS-30502] No ASM disk group found.
CAUSE: There were no disk groups managed by the ASM instance +ASM1.
Cause
1. Listener owner (including SCAN listener) can not access oracle binary in database home:
As listener owner:
$ ls -l $RDBMS_HOME/bin/oracle
ls: /home/oracle/app/oracle/product/11.2/db/bin/oracle: Permission denied
2. Oracle binary in database home has wrong permission:
ls -l $RDBMS_HOME/bin/oracle
-rwxr-x–x 1 oracle asmadmin 184286251 Aug 9 16:25 /home/oracle/app/oracle/product/11.2/db/bin/oracle
The permission “-rwxr-x–x” is wrong as it’s missing suid bit, oracle binary should have permission of 6751:
-rwsr-s–x 1 oracle asmadmin 184286251 Aug 9 16:25 /home/oracle/app/oracle/product/11.2/db/bin/oracle
Note: If Job Role Separation is in place, the group will be , otherwise it will be
3. File System for database home does not support setuid/suid or has nosetuid/nosuid set:
mount| grep
/home/oracle on /dev/dsk/diskoracle read/write/nosuid..
Solution
Solution is to make sure file system for database home has setuid/suid set, database binary($RDBMS_HOME/bin/oracle) has correct ownership and permission, and listener owner is able to access database oracle binary (as listener owner, “ls -l $RDBMS_HOME/bin/oracle” will tell)
If its Job Role Separation environment, please refer to the following document:
Oracle® Grid Infrastructure
Installation Guide
11g Release 2 (11.2)
Oracle ASM Job Role Separation Option with SYSASM
The official explanation is as follows.
Applies to:
Oracle Net Services – Version: 8.1.7.0 to 11.2.0.1 – Release: 8.1.7 to 11.2
Generic UNIX
Checked for relevance on 30-Apr-2010
Symptoms
Connection to the database server fails with ORA-12546 or ORA-12537
Local connections working fine but the connection via listener fails with ORA-12537 or ORA-12547
[oracle@myhost]$ sqlplusmyuser/mypass@mytns
SQL*Plus: Release 11.1.0.7.0 – Production on Fri Apr 30 21:32:18 2010
Copyright (c) 1982, 2008, Oracle. All rights reserved.
ERROR:
ORA-12537: TNS:connection closed
The listener log for the failing connection shows following error message –
TNS-12518: TNS:listener could not hand off client connection
TNS-12546: TNS:permission denied
TNS-12560: TNS:protocol adapter error
TNS-00516: Permission denied
Linux Error: 13: Permission denied
Or
TNS-12518: TNS:listener could not hand off client connection
TNS-12547: TNS:lost contact
TNS-12560: TNS:protocol adapter error
TNS-00517: Lost contact
IBM/AIX RISC System/6000 Error: 32: Broken pipe
Cause
Permissions on oracle binary ( $ORACLE_HOME/bin/oracle ) are not set correctly.
Solution
The OS level error message in the listener log “Linux Error: 13: Permission denied” indicates that the listener was not able to hand off the connection to oracle processes or not able to spawn a new dedicated user process because of lack of permission at the OS level. To resolve the error –
1. Check and correct the permission on /var/tmp/.oracle directory. Permissions on this directory should be 777 with sticky bit set.
# chmod 01777 /var/tmp/.oracle
2. Check and correct permissions on $ORACLE_HOME/bin/oracle binary and make sure that it is set to 6751.
$ chmod 6751 $ORACLE_HOME/bin/oracle
Check your own environment and indeed find that it is not the correct permission
[root@ract1 bin]# pwd
/oracle/product/11.2.0/bin
[root@ract1 bin]# ls -la oracle
-rwxr-s–x 1 oracle asmadmin 228886426 Jan 7 15:46 oracle
[root@ract1 bin]# chmod 6751 oracle- -Note that the root user is used to give permissions
[root@ract1 bin]# ls -la oracle
-rwsr-s–x 1 oracle asmadmin 228886426 Jan 7 15:46 oracle
After the modification, the login is normal.
[oracle@ract2 ~]$ sqlplussys/[email protected]:1521/racdbas sysdba
SQL*Plus: Release 11.2.0.2.0 Production on Thu Jan 17 14:48:25 2013
Copyright (c) 1982, 2010, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 – 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options