SR Detail

https://support.oracle.com/epmos/faces/SrDetailP ...

SR 3-8635585771 : ORA-27140: attach to post/wait facility failed Severity Escalation Status Last Updated Bug Reference Related Articles Support Identifier Account Name Primary Contact System Product 1-Critical Never Escalated Mar 2, 2014 11:23 AM (Sunday) No Related Bugs No Related Articles 3161100170 Seguros Generales Suramericana SA Cristian Gmez Jimnez Oracle Database - Enterprise Edition Product Version 11.2.0.3 OS Version 6.1 Project Milestone Status Customer Working Opened Mar 2, 2014 9:05 AM (Sunday) Attachments alert_logs.zip Related SRs No Related SRs

Alternate Contact Host No Related Hosts

Operating System IBM AIX on POWER Systems (64-bit) Project all instance in the cluster appear with this errors

Errors in file /oracle/app/oracle/diag/rdbms/pdnsbk/PDNSBK2/trace/PDNSBK2_j000_27788006.trc: ORA-27140: attach to post/wait facility failed ORA-27300: OS system dependent operation:invalid_egid failed with status: 1 ORA-27301: OS failure message: Not owner Problem Description ORA-27302: failure occurred at: skgpwinit6 ORA-27303: additional information: startup egid = 10000 (oinstall), current egid = 10001 (asmadmin) Sun Mar 02 02:19:59 2014 Process J000 died, see its trace file kkjcre1p: unable to spawn jobq slave process Errors in file /oracle/app/oracle/diag/rdbms/pdnsbk/PDNSBK2/trace/PDNSBK2_cjq0_48234732.trc: Sun Mar 02 02:19:59 2014 History Notes Hi! Oracle Support- Mar 2, 2014 11:23 AM (Sunday)

Not, these patches are not related with this error. The problem happend because the files owner were change with instance up. All the cases i had found is caused by manual changes It is more likely these files are oinstall. It is extrange to see them part of the group asmadmin. Maybe somebody were implementing Roles separation.

Thanks Miguel Lopez Update from Customer CMGOMEZJ@SURA.COM.CO- Mar 2, 2014 11:04 AM (Sunday) i apply the patches 14393542 and 16619892 in this server with the instances down, there are any relation with the permissions and this patches ??? Notes Hi! Oracle Support- Mar 2, 2014 10:30 AM (Sunday)

The problem occurred because the group of the files with errors were change after the instance were started. The message is a warning that indicated the files had change of group. there are two ways to solve it: a) Restore the previous group owner to those files (oinstall), This does not required to restart the instance Or b) Stop the instances and start it again.

Thanks Miguel Lopez Update from Customer CMGOMEZJ@SURA.COM.CO- Mar 2, 2014 10:11 AM (Sunday) hi, the databases in this moment are open, after the problem appear i restart the databases and the problem in this moment not appear, i want to know why the problem appear and, if i don't take any action with the permission of the oracle file why with a restart in the instance the problem as solved ODM Action Plan Hi! Oracle Support- Mar 2, 2014 9:52 AM (Sunday)

1.- It seems that the files had been change on group since the instance were started. According to the error those trace file belong to "oinstall" group when the instance started and currently belong to "asmadmin". This change was done when the instance running and it had not been

1 of 4

03/03/2014 02:09 PM

SR Detail
restarted. The solutions are:

https://support.oracle.com/epmos/faces/SrDetailP ...

a.- Bounce all the instances, so they start using the current group b.- Restore the files to the "oistall" group c.- Make sure the oracle binary file have 6751 ("-rwsr-s--x") permissions, If it is not please set it as follow ========================== Binary on the grid home 1-. Stop the CRS. 2-. Change the file permission of GI_HOME/bin/oracle to "-rwsr-s--x": .- $su - grid .- $cd GI_HOME/bin .- $chmod 6751 oracle .- $ls -l oracle 3-. Start the /CRS. ===============================Binary on oracle Home 1.- Stop Databases, listener and other processs running from this Oracle Home 2-. Change the file permission of ORACLE_HOME/bin/oracle to "-rwsr-s--x": .- $su - oracle .- $cd ORACLE_HOME/bin .- $chmod 6751 oracle .- $ls -l oracle 3.- Start database, listener and other process that run from this home ============================ Thanks Miguel Lopez ODM Issue Verification Verified the issue in the log file as noted below: Oracle Support- Mar 2, 2014 9:41 AM (Sunday)

LOG FILE ----------------------Filename =https://orioniss.us.oracle.com/iss/faces/issviewer/checkout?context=SR&contextNum=3-8635585771&file=alert_logs.zip See the following error: Sun Mar 02 08:09:27 2014 Process PZ99 died, see its trace file Sun Mar 02 08:09:27 2014 Errors in file /oracle/app/oracle/diag/rdbms/pdnbanca/PDNBANCA2/trace/PDNBANCA2_j000_48693560.trc: ORA-27140: attach to post/wait facility failed ORA-27300: OS system dependent operation:invalid_egid failed with status: 1 ORA-27301: OS failure message: Not owner ORA-27302: failure occurred at: skgpwinit6 ORA-27303: additional information: startup egid = 10000 (oinstall), current egid = 10001 (asmadmin) Process J000 died, see its trace file kkjcre1p: unable to spawn jobq slave process Errors in file /oracle/app/oracle/diag/rdbms/pdnbanca/PDNBANCA2/trace/PDNBANCA2_cjq0_10421026.trc: Sun Mar 02 08:09:28 2014 License high water mark = 51 USER (ospid: 20447576): terminating the instance ODM Issue Clarification Customer is getting the following messages on the alert .log Oracle Support- Mar 2, 2014 9:39 AM (Sunday)

Errors in file /oracle/app/oracle/diag/rdbms/pdnbanca/PDNBANCA2/trace/PDNBANCA2_j000_64881046.trc: ORA-27140: attach to post/wait facility failed ORA-27300: OS system dependent operation:invalid_egid failed with status: 1 ORA-27301: OS failure message: Not owner ORA-27302: failure occurred at: skgpwinit6 ORA-27303: additional information: startup egid = 10000 (oinstall), current egid = 10001 (asmadmin) Process J000 died, see its trace file kkjcre1p: unable to spawn jobq slave process Errors in file /oracle/app/oracle/diag/rdbms/pdnbanca/PDNBANCA2/trace/PDNBANCA2_cjq0_10421026.trc: First Contact First Phone Contact Comments -------------------------------------------------------** Primary Customer Contact Method: Web Oracle Support- Mar 2, 2014 9:20 AM (Sunday)

2 of 4

03/03/2014 02:09 PM

SR Detail
Hi!

https://support.oracle.com/epmos/faces/SrDetailP ...

My name is Miguel Lopez and I would be working on this SR. I am reviewing and searching information about this error. Thanks Miguel Lopez Update from Customer Upload to TDS successful for the file alert_logs.zip. Update from Customer the intance startup but then all instance in this cluster show this error CMGOMEZJ@SURA.COM.CO- Mar 2, 2014 9:15 AM (Sunday) CMGOMEZJ@SURA.COM.CO- Mar 2, 2014 9:05 AM (Sunday)

Sun Mar 02 02:00:00 2014 Closing scheduler window Closing Resource Manager plan via scheduler window Clearing Resource Manager plan via parameter Sun Mar 02 02:09:16 2014 Thread 2 advanced to log sequence 50873 (LGWR switch) Current log# 17 seq# 50873 mem# 0: +DGDATA_PDNSBK/pdnsbk/onlinelog/group_17.535.758567501 Current log# 17 seq# 50873 mem# 1: +DGFLASH_PDNSBK/pdnsbk/onlinelog/group_17.260.758567503 Sun Mar 02 02:09:17 2014 Archived Log entry 83856 added for thread 2 sequence 50872 ID 0x22d7e13f dest 1: Sun Mar 02 02:18:20 2014 Reconfiguration started (old inc 28, new inc 30) List of instances: 1 2 (myinst: 2) Global Resource Directory frozen Communication channels reestablished Sun Mar 02 02:18:20 2014 * domain 0 valid = 1 according to instance 1 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Sun Mar 02 02:18:20 2014 Sun Mar 02 02:18:20 2014 LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete Sun Mar 02 02:19:58 2014 Errors in file /oracle/app/oracle/diag/rdbms/pdnsbk/PDNSBK2/trace/PDNSBK2_j000_27788006.trc: ORA-27140: attach to post/wait facility failed ORA-27300: OS system dependent operation:invalid_egid failed with status: 1 ORA-27301: OS failure message: Not owner ORA-27302: failure occurred at: skgpwinit6 ORA-27303: additional information: startup egid = 10000 (oinstall), current egid = 10001 (asmadmin) Sun Mar 02 02:19:59 2014 Process J000 died, see its trace file kkjcre1p: unable to spawn jobq slave process Errors in file /oracle/app/oracle/diag/rdbms/pdnsbk/PDNSBK2/trace/PDNSBK2_cjq0_48234732.trc: Sun Mar 02 02:19:59 2014 Customer Problem Description Severity 1 Contact Information --------------------------------------------------** Customer's Management 24x7 contact name: cristianmatiasgomez jimenez ** Customer's Management 24x7 contact number: (57) 318 221 01 96 ** Customer's Management E-mail Address: cmgomezj@sura.com.co ** ** ** ** ** ** ** ** Primary Primary Primary Primary Customer Customer Customer Customer 24x7 contact name: Cristian Gmez Jimnez 24x7 contact number: +574.4355792 E-mail Address: cmgomezj@sura.com.co Contact Method: Web CMGOMEZJ@SURA.COM.CO- Mar 2, 2014 9:05 AM (Sunday)

Secondary Customer 24x7 contact name: Gustavo Adolfo Orrego Toro Secondary Customer 24x7 contact number: +574-435-5616 Alternate 24x7 Contact Number: (57) 318 221 08 51 Secondary Customer E-mail Address: gaorrego@sura.com.co

3 of 4

03/03/2014 02:09 PM

SR Detail
Customer Problem Description --------------------------------------------------Problem Summary --------------------------------------------------ORA-27140: attach to post/wait facility failed Problem Description --------------------------------------------------all instance in the cluster appear with this errors

https://support.oracle.com/epmos/faces/SrDetailP ...

Errors in file /oracle/app/oracle/diag/rdbms/pdnsbk/PDNSBK2/trace/PDNSBK2_j000_27788006.trc: ORA-27140: attach to post/wait facility failed ORA-27300: OS system dependent operation:invalid_egid failed with status: 1 ORA-27301: OS failure message: Not owner ORA-27302: failure occurred at: skgpwinit6 ORA-27303: additional information: startup egid = 10000 (oinstall), current egid = 10001 (asmadmin) Sun Mar 02 02:19:59 2014 Process J000 died, see its trace file kkjcre1p: unable to spawn jobq slave process Errors in file /oracle/app/oracle/diag/rdbms/pdnsbk/PDNSBK2/trace/PDNSBK2_cjq0_48234732.trc: Sun Mar 02 02:19:59 2014

Error Codes ---------------------------------------------------

Problem Category/Subcategory --------------------------------------------------Database RAC (Real Application Cluster)/Database/Instance Startup (hang, slow, etc) Uploaded Files ---------------------------------------------------

Template Question Responses --------------------------------------------------1) ### NUMBER OF INSTANCES/NODES ###

2) ### O/S DETAILS ###

3) ### CLUSTERWARE ###

4) ### CLUSTER INTERCONNECT ###

5) ### DATABASE FILE LOCATION ###

6) Describe how this problem is impacting your business. Include relevant information such as critical events (i.e., upgrade or project milestones), dates (i.e., go live dates), number of users affected, financial impact, etc.

4 of 4

03/03/2014 02:09 PM