求教：SMON: enabling cache recovery

biti_rainy · 发表于 2005-1-26 20:05

我想的是，你将故障节点 shutdown 之后，正常情况下在另外一个节点 alert log 里面会看到它会接管恢复的

写出所有dirty buffer 并且切换一圈 logfile ，看alert log 里面是否会提示恢复完毕……如果提示恢复完毕，然后再起这个节点会如何？

或者不知道是不是跟 cluster manager 有关系,对方还没有检测到节点失败？

我也是不得已的情况下说这样试一下，注意观察两个节点的 alert log 提示

kuz1211 · 发表于 2005-1-26 20:20

系统里面节点2的alert日志显示了Ending instance recovery of 1 threads，这个应该表明实例2已经将实例1恢复完毕了，然后开始mount实例1，mount成功后，实例2的alter日志会显示：
Reconfiguration complete
Wed Jan 26 20:09:48 2005
Instance recovery: looking for dead threads
Instance recovery: lock domain invalid but no dead threads
然后开始open实例1，就会发现SMON: enabling cache recovery ，从而实例1一直无法open。

kuz1211 · 发表于 2005-1-26 20:27

现在实例1还在等待OPEN状态，我此时查看v$instance等一些视图，都是正常的，什么情况会导致现在数据库不能被OPEN呢？难道它真的在等待recovery吗？其实现在也没什么好恢复的啊。

biti_rainy · 发表于 2005-1-26 20:40

可否在 node 1 上做一个 hanganalyze 的 oradebug ，看看在等什么

oradebug hanganalyze level

level : 3/4/5/10

可以尝试从3开始，看是否看的出名堂

kuz1211 · 发表于 2005-1-26 20:52

用的level1，trace文件的内容如下：
JServer Release 9.2.0.5.0 - Production
ORACLE_HOME = /home/oracle/app/oracle/product/9.2
System name: AIX
Node name:icdnode1
Release:1
Version:5
Machine:00562BFD4C00
Instance name: ora921
Redo thread mounted by this instance: 1
Oracle process number: 16
Unix process pid: 4628648, image: oracle@icdnode1 (TNS V1-V3)

*** SESSION ID

18.7) 2005-01-26 20:41:42.782
*** 2005-01-26 20:41:42.782
==============
HANG ANALYSIS:
==============
Open chains found:
Other chains found:
Chain 1 : <cnode/sid/sess_srno/ospid/wait_event> :
<0/2/1/4186240/No Wait>
Chain 2 : <cnode/sid/sess_srno/ospid/wait_event> :
<0/15/1/4194330/DFS lock handle>
Chain 3 : <cnode/sid/sess_srno/ospid/wait_event> :
<0/18/7/4628648/No Wait>
Extra information that will be dumped at higher levels:
[level 5] : 3 node dumps -- [SINGLE_NODE] [SINGLE_NODE_NW] [IGN_DMP]
[level 10] : 12 node dumps -- [IGN]

State of nodes
([nodenum]/cnode/sid/sess_srno/session/ospid/state/start/finish/[adjlist]/predec
essor):
[0]/0/1/1/0xf93b6348/4571170/IGN/1/2//none
[1]/0/2/1/0xf846b5a8/4186240/SINGLE_NODE_NW/3/4//none
[2]/0/3/1/0xf93b6dd8/2998520/IGN/5/6//none
[3]/0/4/1/0xf846c038/2244856/IGN/7/8//none
[4]/0/5/1/0xf93b7868/4562998/IGN/9/10//none
[6]/0/7/1/0xf93b82f8/3997842/IGN/11/12//none
[8]/0/9/1/0xf93b8d88/4497480/IGN/13/14//none
[9]/0/10/1/0xf846dfe8/3653716/IGN/15/16//none
[10]/0/11/1/0xf93b9818/3768414/IGN/17/18//none
[11]/0/12/1/0xf846ea78/3866796/IGN/19/20//none
[12]/0/13/1/0xf93ba2a8/1327152/IGN/21/22//none
[13]/0/14/1/0xf846f508/4014128/IGN/23/24//none
[14]/0/15/1/0xf93bad38/4194330/SINGLE_NODE/25/26//none
[16]/0/17/1/0xf93bb7c8/4341886/IGN/27/28//none
[17]/0/18/7/0xf8470a28/4628648/SINGLE_NODE_NW/29/30//none
====================
END OF HANG ANALYSIS
====================

biti_rainy · 发表于 2005-1-26 21:24

DFS Lock Handles
Pointers to global resources. To perform operations on global enqueue service resources, the process first needs to acquire a DFS handle.

DFS lock handle
The session waits for the lock handle of a global lock request. The lock handle identifies a global lock. With this lock handle, other operations can be performed on this global lock (to identify the global lock in future operations such as conversions or release). The global lock is maintained by the DLM.

Wait Time: The session waits in a loop until it has obtained the lock handle from the DLM. Inside the loop there is a wait of 0.5 seconds.

Parameters:

name
See "name and type"

mode
See "mode"

id1
See "id1"

id2
See "id2"

The session needs to get the lock handle.

有些怀疑是不是你的 cluster manager 出了问题？能否把 cluster manager 重新启动？

biti_rainy · 发表于 2005-1-26 21:30

或者，做个 level 3 ,或者查询看看 v$lock ，v$session_wait ,希望看能否找出更详细的在等待什么资源

btw:另一个节点是不是非常的繁忙？两个节点是否有负载均衡（如果有的话能否关掉，可以手立即修改 remote_listener）?

以前有 sequence 没有 cache 导致出现 DFS lock handle 的global resource 争用

kuz1211 · 发表于 2005-1-26 21:43

只重启过实例1的双机，结果还是不行。
另一个节点确实很繁忙，两个节点应该没有负载均衡，因为我的客户端配置文件当中load_balance=off。level3的oradebug消息等下奉上。

kuz1211 · 发表于 2005-1-26 21:47

刚看了实例1的等待事件，确实有DFL LOCK的字样，该如何解决啊？
另外ORADEBUG LEVEL3的消息如下（似乎和LEVEL1的没什么区别）：
ORACLE_HOME = /home/oracle/app/oracle/product/9.2
System name: AIX
Node name:    icdnode1
Release:       1
Version:       5
Machine:       00562BFD4C00
Instance name: ora921
Redo thread mounted by this instance: 1
Oracle process number: 17
Unix process pid: 4227320, image: oracle@icdnode1 (TNS V1-V3)

*** SESSION ID

18.19) 2005-01-26 21:37:22.179
*** 2005-01-26 21:37:22.179
==============
HANG ANALYSIS:
==============
Open chains found:
Other chains found:
Chain 1 : <cnode/sid/sess_srno/proc_ptr/ospid/wait_event> :
<0/2/1/0xf83c5d58/4186240/No Wait>
Chain 2 : <cnode/sid/sess_srno/proc_ptr/ospid/wait_event> :
<0/15/1/0xf83c9458/4194330/DFS lock handle>
Chain 3 : <cnode/sid/sess_srno/proc_ptr/ospid/wait_event> :
<0/18/19/0xf83ca358/4227320/No Wait>
Extra information that will be dumped at higher levels:
[level 5] : 3 node dumps -- [SINGLE_NODE] [SINGLE_NODE_NW] [IGN_DMP]
[level 10] : 12 node dumps -- [IGN]

State of nodes
([nodenum]/cnode/sid/sess_srno/session/ospid/state/start/finish/[adjlist]/predec
essor):
[0]/0/1/1/0xf93b6348/4571170/IGN/1/2//none
[1]/0/2/1/0xf846b5a8/4186240/SINGLE_NODE_NW/3/4//none
[2]/0/3/1/0xf93b6dd8/2998520/IGN/5/6//none
[3]/0/4/1/0xf846c038/2244856/IGN/7/8//none
[4]/0/5/1/0xf93b7868/4562998/IGN/9/10//none
[6]/0/7/1/0xf93b82f8/3997842/IGN/11/12//none
[8]/0/9/1/0xf93b8d88/4497480/IGN/13/14//none
[9]/0/10/1/0xf846dfe8/3653716/IGN/15/16//none
[10]/0/11/1/0xf93b9818/3768414/IGN/17/18//none
[11]/0/12/1/0xf846ea78/3866796/IGN/19/20//none
[12]/0/13/1/0xf93ba2a8/1327152/IGN/21/22//none
[13]/0/14/1/0xf846f508/4014128/IGN/23/24//none
[14]/0/15/1/0xf93bad38/4194330/SINGLE_NODE/25/26//none
[16]/0/17/1/0xf93bb7c8/4341886/IGN/27/28//none
[17]/0/18/19/0xf8470a28/4227320/SINGLE_NODE_NW/29/30//none
====================
END OF HANG ANALYSIS
====================

kuz1211 · 发表于 2005-1-26 21:54

另外我又看了实例2的等待事件，没有发现有什么异常，也没有DFL LOCK HANDLER这个事件，只是在实例1上有该等待事件。