RAC一節點DOWN機﹐附alert.log

biti_rainy · 发表于 2005-1-29 01:40

OER 7451 in Load Indicator : Error Code = Additional information: 1 !

[oracle@ocn2 oracle]$  oerr  ora  7451
07451, 00000, "slskstat: unable to obtain load information."
// *Cause:  kstat library returned an error. Possible OS failure
// *Action: Check result code in sercose[0] for more information.
[oracle@ocn2 oracle]$

首先检查网络是不是出问题了，根据提示看来是和负载均衡有关
（注意remote  listener 的设置）remote_listener       = LISTENERS_RAC

先看网络是否存在问题

dob_zhu7781 · 发表于 2005-1-29 08:10

SQL> show parameter remote_listener

NAME                               TYPE       VALUE
------------------------------------ ----------- ------------------------------
remote_listener                   string    LISTENERS_RAC

tnsnames.ora文件如下﹕
  LISTENERS_RAC =
  (DESCRIPTION =
   (ADDRESS_LIST =
         (ADDRESS = (PROTOCOL = TCP)(HOST = IALDB01)(PORT = 1521))
         (ADDRESS = (PROTOCOL = TCP)(HOST = IALDB02)(PORT = 1521))
   )
  )

LISTENER_ialdb1 =
  (DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = IALDB01)(PORT = 1521))
  )

ialdb1 =
  (DESCRIPTION =
   (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = IALDB01)(PORT = 1521))
   )
   (CONNECT_DATA =
      (SERVICE_NAME = ialdb)
      (INSTANCE_NAME = ialdb1)
   )
  )

ialdb2 =
  (DESCRIPTION =
   (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = IALDB02)(PORT = 1521))
   )
   (CONNECT_DATA =
      (SERVICE_NAME = ialdb)
      (INSTANCE_NAME = ialdb2)
   )
  )

ialdb =
  (DESCRIPTION =
   (LOAD_BALANCE = yes)
      (ADDRESS = (PROTOCOL = TCP)(HOST = IALDB01)(PORT = 1521))
      (ADDRESS = (PROTOCOL = TCP)(HOST = IALDB02)(PORT = 1521))
      (CONNECT_DATA =
      (SERVICE_NAME = ialdb)
      )
   )

RAC =
  (DESCRIPTION =
(LOAD_BALANCE = yes)
(FAILOVER = yes)
(ADDRESS = (PROTOCOL = TCP)(HOST = IALDB01)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = IALDB02)(PORT = 1521))
(CONNECT_DATA =
   (SERVICE_NAME = ialdb)
)
  )

failover =
  (DESCRIPTION =
(enable=broken)
(LOAD_BALANCE = yes)
(ADDRESS = (PROTOCOL = TCP)(HOST = IALDB01)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = IALDB02)(PORT = 1521))
(CONNECT_DATA =
   (SERVICE_NAME = ialdb)
   (failover_mode=(type=select)(method=basic))
)
  )

心跳線對連﹐但綱絡正常﹐重起后正﹐如何防止﹖

dob_zhu7781 · 发表于 2005-1-29 08:41

檢查第二節點LOG﹐在2小時之后有error,但沒有造成down機
以前一年半沒有出這種錯誤﹐log 如下
Tue Jun  1 17:54:20 2004
ARC0: Completed archiving  log 4 thread 2 sequence 779
Tue Jun  1 17:59:35 2004
SMON: Parallel transaction recovery tried
:200000
ARC1: Completed archiving  log 6 thread 2 sequence 2674
Fri Jan 28 16:58:12 2005
SMON offlining US=10
SMON offlining US=57
SMON offlining US=59
SMON offlining US=67
SMON offlining US=68
SMON offlining US=69
SMON offlining US=70
SMON offlining US=71
Fri Jan 28 17:09:22 2005
Undo Segment 10 Onlined
Fri Jan 28 17:09:22 2005
Undo Segment 57 Onlined
Fri Jan 28 17:09:22 2005
Undo Segment 59 Onlined
Fri Jan 28 17:09:24 2005
Undo Segment 67 Onlined
Fri Jan 28 17:09:27 2005
Undo Segment 68 Onlined
Fri Jan 28 17:09:27 2005
Undo Segment 69 Onlined
Fri Jan 28 17:09:28 2005
Undo Segment 70 Onlined
Fri Jan 28 17:09:28 2005
Undo Segment 71 Onlined
Fri Jan 28 17:09:28 2005
Created Undo Segment _SYSSMU72$
Undo Segment 72 Onlined
Fri Jan 28 17:09:29 2005
Created Undo Segment _SYSSMU73$
Undo Segment 73 Onlined
Fri Jan 28 17:09:29 2005
Created Undo Segment _SYSSMU74$
Undo Segment 74 Onlined
Fri Jan 28 17:09:29 2005
Created Undo Segment _SYSSMU75$
Undo Segment 75 Onlined
Fri Jan 28 17:09:30 2005
Created Undo Segment _SYSSMU76$
Undo Segment 76 Onlined
Fri Jan 28 17:09:33 2005
Created Undo Segment _SYSSMU77$
Undo Segment 77 Onlined
Fri Jan 28 17:09:35 2005
Created Undo Segment _SYSSMU78$
Undo Segment 78 Onlined
Fri Jan 28 17:09:35 2005
Created Undo Segment _SYSSMU79$
Undo Segment 79 Onlined
Fri Jan 28 17:11:48 2005
SSKGXP_IP: Primary interface down
operation gethostbyname
location clusterip2
error No such file or directory
SSKGXP_IP: Primary interface down
operation gethostbyname
location clusterip2
error No such file or directory
SSKGXP_IP: Primary interface down
operation gethostbyname
location clusterip2
error No such file or directory
SSKGXP_IP: Primary interface down
operation gethostbyname
location clusterip2
error No such file or directory
SSKGXP_IP: Primary interface down
operation gethostbyname
location clusterip2
error No such file or directory
SSKGXP_IP: Primary interface down
operation gethostbyname
location clusterip2
error No such file or directory
SSKGXP_IP: Primary interface down
operation gethostbyname
location clusterip2
error No such file or directory
SSKGXP_IP: Primary interface down
operation gethostbyname
location clusterip2
error No such file or directory
SSKGXP_IP: Primary interface down
operation gethostbyname
location clusterip2
error No such file or directory
SSKGXP_IP: Primary interface down
operation gethostbyname
location clusterip2
error No such file or directory
SSKGXP_IP: Primary interface down
operation gethostbyname
location clusterip2
error No such file or directory
Fri Jan 28 17:13:14 2005
OER 7451 in Load Indicator : Error Code = Additional information: 1 !
Fri Jan 28 17:13:46 2005
OER 7451 in Load Indicator : Error Code = Additional information: 1 !
Fri Jan 28 17:14:17 2005
OER 7451 in Load Indicator : Error Code = Additional information: 1 !
Fri Jan 28 17:14:48 2005
OER 7451 in Load Indicator : Error Code = Additional information: 1 !
Fri Jan 28 17:15:24 2005
SSKGXP_IP: Primary interface down
operation gethostbyname
location clusterip2
error No such file or directory
SSKGXP_IP: Primary interface down
operation gethostbyname
location clusterip2
error No such file or directory
Fri Jan 28 17:19:02 2005
Thread 2 advanced to log sequence 2676
  Current log# 3 seq# 2676 mem# 0: /dev/raw/raw9
Fri Jan 28 17:19:02 2005
ARC0: Evaluating archive log 4 thread 2 sequence 2675
ARC0: Beginning to archive log 4 thread 2 sequence 2675
Creating archive destination LOG_ARCHIVE_DEST_1: '/u01/app/oracle/admin/ialdb/arch/T0002S0000002675.ARC'
Fri Jan 28 17:19:18 2005
ARC0: Completed archiving  log 4 thread 2 sequence 2675
Fri Jan 28 20:40:41 2005
Thread 2 advanced to log sequence 2677
  Current log# 6 seq# 2677 mem# 0: /dev/raw/raw11
Fri Jan 28 20:40:41 2005
ARC0: Evaluating archive log 3 thread 2 sequence 2676
ARC0: Beginning to archive log 3 thread 2 sequence 2676

biti_rainy · 发表于 2005-1-29 12:58

LISTENERS_RAC =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = IALDB01)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = IALDB02)(PORT = 1521))
)
)

通常我们设置remote_listener 不是使用上面这样的使者，而是使用下面这样的

LISTENER_ialdb1 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = IALDB01)(PORT = 1521))
)

remote listener 只是指向远程的 listener 而不用和本地的相关

要检查网络是否有问题，是检查 IALDB01 对应的IP 地址，这应该不是心跳线对应的IP，而是对外的IP和网卡。如果排除掉公网和私网都不存在问题，能否尝试把 remote listener 去掉取消负载均衡试一下？

另：该系统以前有这样的问题吗？确保在rac的安装和配置不存在问题。

xyz3000 · 发表于 2005-1-29 13:22

书签转到末尾

文档 ID:  注释:280452.1
主题:  LINUX: OER-7451 in the Alert.log
类型:  PROBLEM
状态:  PUBLISHED
内容类型:  TEXT/X-HTML
创建日期:  11-AUG-2004
上次修订日期:  07-DEC-2004

The information in this article applies to:
Oracle Server - Enterprise Edition - Version: 9.2.0.4
Red Hat Advanced Server

Errors
OER 7451

Symptoms
In the Alert.log , Following Messages can be seen:

OER 7451 in Load Indicator : Error Code = Additional information: 1 !
Cause
Also in /var/log/messages , following message will be noticed :

"hendrix kernel: VFS: file-max limit 8192 reached jui 13 16:12:03 hendrixsu(pam_unix)".

Cause:

+Maximum of open files reached set in kernel by file-max
+OER-7451 reported when we try to access /proc/loadavg. We can not open
this device because mamixmum of files are already open
Fix
Change file-max for running system:

Increase file-max for running system and permanent:

sysctl -w fs.file-max=65536
References

--------------------------------------------------------------------------------

Help us improve our service. Please email us your comments for this document. .

--------------------------------------------------------------------------------

Copyright (c) 1995,2000 Oracle Corporation. All Rights Reserved. 法律声明和使用条款。

Angel.John · 发表于 2005-1-30 12:15

网卡停止工作了。

husthxd · 发表于 2005-1-30 18:43

Too many open files in system
在hp－ux 11.11 9.0.1.0.0下碰到过这个问题，升级后解决。

ghost_zy1979 · 发表于 2005-1-30 19:07

关注。。。。。。。。

biti_rainy · 发表于 2005-1-30 21:01

这个问题，假如不是bug的原因，那么我们可以这样来分析这个问题：

本问题的产生的根源是因为负载均衡的缘故，设置了 remote  listener （或者同时使用了负载均衡），使得连接数据库的时候需要去比较两边节点的load的高低，而比较节点load（包括实现负载均衡）是通过remote  listener 去比较实现的。而判别节点的load，需要访问os来比较load，对于linux而言，内核往往是通过内存文件  的方式展示给应用的。比如load的获取，是输出到内存文件系统的 /proc/loadavg  这个  *文件* 。而所有应用，包括 top  / sar  等等工具都是通过读这个文件来获得当前os的load状况的。

由于linux里面内核大量地使用了内存文件系统，外界大量地通过文件的访问方式来访问系统数据，对于应用来说，就是 open  file (only  read )。这样可能造成文件打开数量限制的现象,oracle中在os中可能会存在不少类似的状况，所以适当地加大这个os限制有是必要的。

所以，我猜测，如果楼主确认网卡没有问题的话：  关闭负载均衡（取消remote  listener  and  tnsnames.ora 设置），或者增大 os打开文件数量都是可能解决问题的（ sysctl -w fs.file-max=65536 ，楼主的 1024 （1k）可能确实太小了，一般推荐都是 64k个），但是关闭负载均衡可能是暂时缓解问题，还是修改os 限制最合适。

当然，session_max_open_files integer 这个参数应该和本问题无关,：
http://download-west.oracle.com/ ... 1188.htm#REFRN10196

SESSION_MAX_OPEN_FILES specifies the maximum number of BFILEs that can be opened in any session. Once this number is reached, subsequent attempts to open more files in the session by using DBMS_LOB.FILEOPEN() or OCILobFileOpen() will fail. The maximum value for this parameter depends on the equivalent parameter defined for the underlying operating system.

biti_rainy · 发表于 2005-1-30 21:43

题外话：

就linux的这个把内核内存通过文件系统的方式提供读写接口给用户，让用户非常方便。并且，我们经常用到linux的一个便捷功能，那就是在 linux 系统上为了安装oracle 通常我们需要修改一些内核参数，比如
[root@member-crm-db kernel]# more  /proc/sys/kernel/sem
250    32000 32    128
[root@member-crm-db kernel]# more  /proc/sys/kernel/shmmax
3221225472
[root@member-crm-db kernel]#

在linux上，我们只要把这些参数当一些文件中的值一样处理，要修改的时候，echo  xxx  > more  /proc/sys/kernel/shmmax 这样就修改了内核参数配置并且立即生效，这是因为这样不仅修改了配置文件，而且是真正地修改了内核的内存中的值，适时地反应到了os中。

而比如 sun  solaris 这类系统，修改内核参数配置之后需要重新启动os才可以，这就是因为我们修改的仅仅是配置文件而内核内存中的值并没有变化，需要os重新启动的时候从配置文件读入内存。

当然，对于linux 而言，我们在 /proc  下将可以看见大量的文件和目录，这些都是内核中的内存的适时状态。