crs_stat -t offline问题

只看该作者 · 发表于 2009-6-9 12:25

今天检查两个节点都 offline：

Name          Type          Target State    Host
------------------------------------------------------------
ora.hp01.gsd application ONLINE ONLINE hp01
ora.hp01.ons application ONLINE OFFLINE
ora.hp01.vip application ONLINE ONLINE hp01
ora.hp02.gsd application ONLINE ONLINE hp02
ora.hp02.ons application ONLINE OFFLINE
ora.hp02.vip application ONLINE ONLINE hp02
ora.metro.db application ONLINE ONLINE hp02
ora....o1.inst application ONLINE ONLINE hp01
ora....o2.inst application ONLINE ONLINE hp02

只看该作者 · 发表于 2009-6-9 12:26

其中 hp01.log日志内容如下：

2009-06-01 08:10:04.799: Attempting to start `ora.hp02.ons` on member `hp02`
2009-06-01 08:10:06.224: Start of `ora.hp02.ons` on member `hp02` succeeded.
`ora.hp01.ons` on `hp01` went OFFLINE unexpectedly
2009-06-02 06:39:46.645: Attempting to stop `ora.hp01.ons` on member `hp01`
2009-06-02 06:39:47.353: Stop of `ora.hp01.ons` on member `hp01` succeeded.
Restarting `ora.hp01.ons` on `hp01`
2009-06-02 06:39:47.395: Attempting to start `ora.hp01.ons` on member `hp01`
2009-06-02 06:39:47.971: Start of `ora.hp01.ons` on member `hp01` succeeded.
Successfully restarted `ora.hp01.ons` on `hp01`
`ora.hp01.ons` on `hp01` went OFFLINE unexpectedly
2009-06-02 06:49:48.569: Attempting to stop `ora.hp01.ons` on member `hp01`
2009-06-02 06:49:49.098: Stop of `ora.hp01.ons` on member `hp01` succeeded.
Restarting `ora.hp01.ons` on `hp01`
2009-06-02 06:49:49.128: Attempting to start `ora.hp01.ons` on member `hp01`
2009-06-02 06:49:49.691: Start of `ora.hp01.ons` on member `hp01` succeeded.
Successfully restarted `ora.hp01.ons` on `hp01`
`ora.hp01.ons` on `hp01` went OFFLINE unexpectedly
2009-06-02 06:59:50.303: Attempting to stop `ora.hp01.ons` on member `hp01`
2009-06-02 06:59:50.884: Stop of `ora.hp01.ons` on member `hp01` succeeded.
Restarting `ora.hp01.ons` on `hp01`
2009-06-02 06:59:50.925: Attempting to start `ora.hp01.ons` on member `hp01`
2009-06-02 06:59:51.464: Start of `ora.hp01.ons` on member `hp01` succeeded.
Successfully restarted `ora.hp01.ons` on `hp01`
`ora.hp01.ons` on `hp01` went OFFLINE unexpectedly
2009-06-02 07:09:52.090: Attempting to stop `ora.hp01.ons` on member `hp01`
2009-06-02 07:09:52.631: Stop of `ora.hp01.ons` on member `hp01` succeeded.

hp02.log日志内容：

`ora.hp02.ons` on `hp02` went OFFLINE unexpectedly
2009-06-02 06:41:20.324: Attempting to stop `ora.hp02.ons` on member `hp02`
2009-06-02 06:41:21.071: Stop of `ora.hp02.ons` on member `hp02` succeeded.
Restarting `ora.hp02.ons` on `hp02`
2009-06-02 06:41:21.131: Attempting to start `ora.hp02.ons` on member `hp02`
2009-06-02 06:41:21.753: Start of `ora.hp02.ons` on member `hp02` succeeded.
Successfully restarted `ora.hp02.ons` on `hp02`
`ora.hp02.ons` on `hp02` went OFFLINE unexpectedly
2009-06-02 06:51:22.359: Attempting to stop `ora.hp02.ons` on member `hp02`
2009-06-02 06:51:22.937: Stop of `ora.hp02.ons` on member `hp02` succeeded.
Restarting `ora.hp02.ons` on `hp02`
2009-06-02 06:51:22.989: Attempting to start `ora.hp02.ons` on member `hp02`
2009-06-02 06:51:23.520: Start of `ora.hp02.ons` on member `hp02` succeeded.
Successfully restarted `ora.hp02.ons` on `hp02`
`ora.hp02.ons` on `hp02` went OFFLINE unexpectedly
2009-06-02 07:01:24.107: Attempting to stop `ora.hp02.ons` on member `hp02`
2009-06-02 07:01:24.634: Stop of `ora.hp02.ons` on member `hp02` succeeded.
Restarting `ora.hp02.ons` on `hp02`
2009-06-02 07:01:24.678: Attempting to start `ora.hp02.ons` on member `hp02`
2009-06-02 07:01:25.210: Start of `ora.hp02.ons` on member `hp02` succeeded.
Successfully restarted `ora.hp02.ons` on `hp02`
`ora.hp02.ons` on `hp02` went OFFLINE unexpectedly
2009-06-02 07:11:25.846: Attempting to stop `ora.hp02.ons` on member `hp02`
2009-06-02 07:11:26.368: Stop of `ora.hp02.ons` on member `hp02` succeeded.

[ 本帖最后由 fuger2 于 2009-6-9 13:30 编辑 ]

cc59 · 发表于 2009-6-10 00:43

....
crs_start -f ....

只看该作者 · 发表于 2009-6-14 17:29

启动后相隔10分钟还是offline

2009-06-09 15:07:03.864: Attempting to start `ora.hp01.ons` on member `hp01`
2009-06-09 15:07:04.394: Start of `ora.hp01.ons` on member `hp4601` succeeded.
Successfully restarted `ora.hp01.ons` on `hp01`
`ora.hp01.ons` on `hp01` went OFFLINE unexpectedly
2009-06-09 15:17:05.029: Attempting to stop `ora.hp01.ons` on member `hp01`
2009-06-09 15:17:05.567: Stop of `ora.hp01.ons` on member `hp01` succeeded.

只看该作者 · 发表于 2009-6-14 17:31

由于ONS的offline状态对数据库和应用没有影响，这个问题PASS了。谢谢大家。

cc59 · 发表于 2009-6-15 23:35

ons.log中会没有日志吗?

空间足够吧?

cc59 · 发表于 2009-6-15 23:36

如果可以的话,最好是把日志级别调为5级.

crsctl debug log res ora.hp02.ons:5'

oktongyan · 发表于 2009-6-20 16:22

oracle rac的服务 crsd.bin每天都产生core dump 具体情况为:
lwyxora01:/oracle/product/crs/10.2/log/lwyxora01/crsd$ ps -eaf | grep crsd.bin
root 1585172    1 0 11:15:34    -  0:11 /oracle/product/crs/10.2/bin/crsd.bin restart
  oracle 2547768 2588724 0 13:26:39  pts/0  0:00 grep crsd.bin

errpt -as 0620000009
------------------------------------------
PCSS/SPI2 FLDS/crsd.bin SIG/6 FLDS/pthread_k VALU/80
---------------------------------------------------------------------------
LABEL:       CORE_DUMP
IDENTIFIER:    A63BEB70

Date/Time:    Sat Jun 20 03:13:08 BEIST 2009
Sequence Number: 4531
Machine Id:    00CEFEFF4C00
Node Id:       lwyxora01
Class:          S
Type:          PERM
Resource Name: SYSPROC

Description
SOFTWARE PROGRAM ABNORMALLY TERMINATED

Probable Causes
SOFTWARE PROGRAM

User Causes
USER GENERATED SIGNAL

      Recommended Actions
      CORRECT THEN RETRY

Failure Causes
SOFTWARE PROGRAM

      Recommended Actions
      RERUN THE APPLICATION PROGRAM
      IF PROBLEM PERSISTS THEN DO THE FOLLOWING
      CONTACT APPROPRIATE SERVICE REPRESENTATIVE

Detail Data
SIGNAL NUMBER
         6
USER'S PROCESS ID:
            2601034
FILE SYSTEM SERIAL NUMBER
      12
INODE NUMBER
   22437
PROCESSOR ID
      -1
CORE FILE NAME
/oracle/product/crs/10.2/log/lwyxora01/crsd/core
PROGRAM NAME
crsd.bin
STACK EXECUTION DISABLED
         0
ADDITIONAL INFORMATION
pthread_k 80
??

Symptom Data
REPORTABLE
1
INTERNAL ERROR
0
SYMPTOM CODE
PCSS/SPI2 FLDS/crsd.bin SIG/6 FLDS/pthread_k VALU/80
------------------------------------------------------------------------------------------
tail -100 crsd.log
--------------------------------------------------------------------
2009-06-20 07:15:11.969: [ default][1][ENTER]32
Oracle Database 10g CRS Release 10.2.0.2.0 Production Copyright 1996, 2004, Oracle.  All rights reserved
2009-06-20 07:15:11.969: [ default][1]32CRS Daemon Starting
2009-06-20 07:15:11.969: [ CRSMAIN][1]32Checking the OCR device
2009-06-20 07:15:11.973: [ CRSMAIN][1]32Connecting to the CSS Daemon
2009-06-20 07:15:12.407: [ CRSD][1]32Daemon Version: 10.2.0.2.0 Active Version: 10.2.0.2.0
2009-06-20 07:15:12.407: [ CRSD][1]32Active Version and Software Version are same
2009-06-20 07:15:12.407: [ CRSMAIN][1]32Initializing OCR
2009-06-20 07:15:12.412: [  OCRRAW][1]proprioo: for disk 0 (/dev/rocrdisk), id match (1), my id set (1935938242,1028247821) total id sets (1), 1st set (1935938242,1028247821), 2nd set (0,0) my votes (2), total votes (2)
2009-06-20 07:15:12.586: [ CRSD][1]32ENV Logging level for Module: allcomp  0
2009-06-20 07:15:12.588: [ CRSD][1]32ENV Logging level for Module: default  0
2009-06-20 07:15:12.590: [ CRSD][1]32ENV Logging level for Module: COMMCRS  0
2009-06-20 07:15:12.591: [ CRSD][1]32ENV Logging level for Module: COMMNS  0
2009-06-20 07:15:12.592: [ CRSD][1]32ENV Logging level for Module: CRSUI  0
2009-06-20 07:15:12.594: [ CRSD][1]32ENV Logging level for Module: CRSCOMM  0
2009-06-20 07:15:12.595: [ CRSD][1]32ENV Logging level for Module: CRSRTI  0
2009-06-20 07:15:12.596: [ CRSD][1]32ENV Logging level for Module: CRSMAIN  0
2009-06-20 07:15:12.598: [ CRSD][1]32ENV Logging level for Module: CRSPLACE  0
2009-06-20 07:15:12.599: [ CRSD][1]32ENV Logging level for Module: CRSAPP  0
2009-06-20 07:15:12.600: [ CRSD][1]32ENV Logging level for Module: CRSRES  0
2009-06-20 07:15:12.602: [ CRSD][1]32ENV Logging level for Module: CRSOCR  0
2009-06-20 07:15:12.603: [ CRSD][1]32ENV Logging level for Module: CRSTIMER  0
2009-06-20 07:15:12.605: [ CRSD][1]32ENV Logging level for Module: CRSEVT  0
2009-06-20 07:15:12.606: [ CRSD][1]32ENV Logging level for Module: CRSD  0
2009-06-20 07:15:12.607: [ CRSD][1]32ENV Logging level for Module: CLUCLS  0
2009-06-20 07:15:12.609: [ CRSD][1]32ENV Logging level for Module: OCRRAW  0
2009-06-20 07:15:12.610: [ CRSD][1]32ENV Logging level for Module: OCROSD  0
2009-06-20 07:15:12.611: [ CRSD][1]32ENV Logging level for Module: CSSCLNT  0
2009-06-20 07:15:12.613: [ CRSD][1]32ENV Logging level for Module: OCRAPI  0
2009-06-20 07:15:12.614: [ CRSD][1]32ENV Logging level for Module: OCRUTL  0
2009-06-20 07:15:12.615: [ CRSD][1]32ENV Logging level for Module: OCRMSG  0
2009-06-20 07:15:12.617: [ CRSD][1]32ENV Logging level for Module: OCRCLI  0
2009-06-20 07:15:12.618: [ CRSD][1]32ENV Logging level for Module: OCRCAC  0
2009-06-20 07:15:12.619: [ CRSD][1]32ENV Logging level for Module: OCRSRV  0
2009-06-20 07:15:12.621: [ CRSD][1]32ENV Logging level for Module: OCRMAS  0
2009-06-20 07:15:12.629: [ CRSMAIN][1]32Filename is /oracle/product/crs/10.2/crs/init/lwyxora02.pid
[  clsdmt][7970]Listening to (ADDRESS=(PROTOCOL=ipc)(KEY=lwyxora02DBG_CRSD))
2009-06-20 07:15:12.758: [ CRSMAIN][1]32Using Authorizer location: /oracle/product/crs/10.2/crs/auth/
2009-06-20 07:15:12.778: [ CRSMAIN][1]32Initializing RTI
2009-06-20 07:15:12.845: [CRSTIMER][8484]32Timer Thread Starting.
2009-06-20 07:15:12.846: [  CRSRES][1]32Parameter SECURITY = 1, running in USER Mode
2009-06-20 07:15:12.852: [ CRSMAIN][1]32Initializing EVMMgr
2009-06-20 07:15:13.209: [ CRSMAIN][1]32CRSD locked during state recovery, please wait.
2009-06-20 07:15:14.078: [  CRSRES][1]32ora.lwyxora02.vip check shows ONLINE
2009-06-20 07:15:23.944: [  CRSRES][1]32ora.lwyxora02.gsd check shows ONLINE
2009-06-20 07:15:24.793: [  CRSRES][1]32ora.lwyxora02.ons check shows ONLINE
2009-06-20 07:15:25.652: [  CRSRES][1]32ora.lwyxora02.LISTENER_LWYXORA02.lsnr check shows ONLINE
2009-06-20 07:15:26.805: [  CRSRES][1]32ora.lwdlyx.lwdlyx2.inst check shows ONLINE
2009-06-20 07:15:34.977: [  CRSRES][1]32ora.lwdlyx.lwdlyx.lwdlyx2.srv check shows ONLINE
2009-06-20 07:15:35.206: [ CRSMAIN][1]32CRSD recovered, unlocked.
2009-06-20 07:15:35.274: [ CRSMAIN][1]32QS socket on: (ADDRESS=(PROTOCOL=ipc)(KEY=ora_crsqs))
2009-06-20 07:15:35.331: [ CRSMAIN][1]32CRSD UI socket on: (ADDRESS=(PROTOCOL=ipc)(KEY=CRSD_UI_SOCKET))
2009-06-20 07:15:35.359: [ CRSMAIN][1]32E2E socket on: (ADDRESS=(PROTOCOL=tcp)(HOST=lwyxora02_priv)(PORT=49896))
2009-06-20 07:15:35.359: [ CRSMAIN][1]32Starting Threads
2009-06-20 07:15:35.360: [ CRSMAIN][1]32CRS Daemon Started.

oinstall · 发表于 2009-7-2 18:37

crs_stat -t 结果是:

Name             Type          Target    State    Host
------------------------------------------------------------------
ora.hp01.gsd application ONLINE ONLINE hp01
ora.hp01.ons application ONLINE ONLINE hp01
ora.hp01.vip application ONLINE ONLINE    hp01
ora.hp02.gsd application ONLINE ONLINE hp02
ora.hp02.ons application OFFLINE OFFLINE
ora.hp02.vip application ONLINE ONLINE    hp02
ora.metro.db application ONLINE ONLINE hp02
ora....o1.inst application ONLINE ONLINE    hp01
ora....o2.inst application ONLINE ONLINE    hp02

这里面的ora.metro.db应该指的是两个实例共享的库吧，我发现有的时候这个app会在hp01，有的时候会在hp02上，当在hp02上的时候，我把hp02上的instance shutdown的时候，这个ora.metro.db用crs_stat看还是在hp02上，谁帮忙解释下这个nodeapp具体作用及故障切换或启动规律什么的，多谢。

inthirties2 · 发表于 2010-7-31 16:13

你这里是ora.hp02.ons application OFFLINE OFFLINE
两个都是offline是不是自己停了，

ons可以先查看一下ons的日志，在racg下，
看看ons的进程是否在。onsctl debug看看状况。
ping localhost有没有问题。

[笔记] crs_stat -t offline问题

crsd.bin 每天都core dump 求救!