10g RAC 实例突然宕掉

owlstudio · 发表于 2009-1-5 20:54

crs_stat -t 看了吗？

ifconfig -a 看看IP情况

ricky_zhu · 发表于 2009-1-5 21:38

instance在节点上，尝试失败了多次，就放弃了：

0Start of `ora.superdb.db` on member `myhost_2` failed.
2009-01-05 10:44:42.881: [  CRSRES][579038128]0ora.superdb.db exceeded it's failure threshold.  Stopping it and its dependents!

从前面可以看出，是vip启动失败，导致其他resource包括vip和listener都要进行relocate

0CheckResource error for ora.myhost_2.vip error code = 1
2009-01-05 10:44:26.342: [  CRSRES][537078704]0In stateChanged, ora.myhost_2.vip target is ONLINE
2009-01-05 10:44:26.342: [  CRSRES][537078704]0ora.myhost_2.vip on myhost_2 went OFFLINE unexpectedly

看看vip的log，跟crs在一个目录

ricky_zhu · 发表于 2009-1-5 21:42

具体版本是多少？

10.2.0.2以及以前，the instance relys on vip, this changed from 10.2.0.3

so vip failed cause the db instance failed to start

jlttt · 发表于 2009-1-5 22:34

vip log:
2009-01-05 11:23:37.770: [ RACG][3076396224] [30487][3076396224][ora.myhost_2.vip]: Interface eth0 checked failed (host=myhost_2)
Invalid parameters, or failed to bring up VIP (host=myhost_2)

2009-01-05 11:23:37.771: [ RACG][3076396224] [30487][3076396224][ora.myhost_2.vip]: clsrcexecut: env ORACLE_CONFIG_HOME=/opt/oracle/product/10.2.0/crs

2009-01-05 11:23:37.771: [ RACG][3076396224] [30487][3076396224][ora.myhost_2.vip]: clsrcexecut: cmd = /opt/oracle/product/10.2.0/crs/bin/racgeut -e _USR_ORA_DEBUG=0 54 /opt/o
racle/product/10.2.0/crs/bin/racgvip check myhost_2

2009-01-05 11:23:37.771: [ RACG][3076396224] [30487][3076396224][ora.myhost_2.vip]: clsrcexecut: rc = 1, time = 6.610s

2009-01-05 11:23:37.771: [ RACG][3076396224] [30487][3076396224][ora.myhost_2.vip]: end for resource = ora.myhost_2.vip, action = check, status = 1, time = 6.670s

SQL> select * from v$version;

BANNER
----------------------------------------------------------------
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Prod
PL/SQL Release 10.2.0.1.0 - Production
CORE 10.2.0.1.0 Production
TNS for Linux: Version 10.2.0.1.0 - Production
NLSRTL Version 10.2.0.1.0 - Production

jlttt · 发表于 2009-1-6 10:51

哎呀妈呀，终于发现一点规律了。

库正常的时候：
[root@myhost_2 root]# ifconfig
eth0    Link encap:Ethernet  HWaddr 00:0C:29:5A:84:40
      inet addr:192.168.0.8  Bcast:192.168.0.255  Mask:255.255.255.0
      UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
      RX packets:442 errors:0 dropped:0 overruns:0 frame:0
      TX packets:64 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1000
      RX bytes:49158 (48.0 Kb)  TX bytes:2688 (2.6 Kb)
      Interrupt:5 Base address:0x1824

eth0:1 Link encap:Ethernet  HWaddr 00:0C:29:5A:84:40
      inet addr:192.168.0.18  Bcast:192.168.0.255  Mask:255.255.255.0
      UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
      Interrupt:5 Base address:0x1824

eth0:2 Link encap:Ethernet  HWaddr 00:0C:29:5A:84:40
      inet addr:192.168.0.16  Bcast:192.168.0.255  Mask:255.255.255.0
      UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
      Interrupt:5 Base address:0x1824

eth1    Link encap:Ethernet  HWaddr 00:0C:29:5A:84:4A
      inet addr:192.168.46.8  Bcast:192.168.46.255  Mask:255.255.255.0
      UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
      RX packets:50 errors:0 dropped:0 overruns:0 frame:0
      TX packets:57 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1000
      RX bytes:6808 (6.6 Kb)  TX bytes:6791 (6.6 Kb)
      Interrupt:9 Base address:0x18a4

lo       Link encap:Local Loopback
      inet addr:127.0.0.1  Mask:255.0.0.0
      UP LOOPBACK RUNNING  MTU:16436  Metric:1
      RX packets:2749 errors:0 dropped:0 overruns:0 frame:0
      TX packets:2749 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:0
      RX bytes:1308906 (1.2 Mb)  TX bytes:1308906 (1.2 Mb)

库宕掉后：
[oracle@myhost_2 admin]$ ifconfig
eth0    Link encap:Ethernet  HWaddr 00:0C:29:5A:84:40
      inet addr:192.168.0.8  Bcast:192.168.0.255  Mask:255.255.255.0
      UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
      RX packets:3841 errors:0 dropped:0 overruns:0 frame:0
      TX packets:829 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1000
      RX bytes:463783 (452.9 Kb)  TX bytes:34818 (34.0 Kb)
      Interrupt:5 Base address:0x1824

eth1    Link encap:Ethernet  HWaddr 00:0C:29:5A:84:4A
      inet addr:192.168.46.8  Bcast:192.168.46.255  Mask:255.255.255.0
      UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
      RX packets:1992 errors:0 dropped:0 overruns:0 frame:0
      TX packets:1824 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:1000
      RX bytes:168405 (164.4 Kb)  TX bytes:251797 (245.8 Kb)
      Interrupt:9 Base address:0x18a4

lo       Link encap:Local Loopback
      inet addr:127.0.0.1  Mask:255.0.0.0
      UP LOOPBACK RUNNING  MTU:16436  Metric:1
      RX packets:16553 errors:0 dropped:0 overruns:0 frame:0
      TX packets:16553 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:0
      RX bytes:3947461 (3.7 Mb)  TX bytes:3947461 (3.7 Mb)

/etc/hosts内容：
192.168.0.6    myhost_1
192.168.0.8    myhost_2
192.168.46.6 myhost_1-priv
192.168.0.16 myhost_1-vip
192.168.46.8 myhost_2-priv
192.168.0.18 myhost_2-vip