enq: KO - fast object checkpoint Fatal NI connect error 12170

pengpengfly

系统登录不了，因为是互联网项目，很紧急，所以只能把nginx切换到备机上去，然后恢复正常，备机也是连的同一数据库
事后检查应用程序日志，发现连不上数据库，然后检查数据库日志：

**********************************************************************
Fatal NI connect error 12170.
Fatal NI connect error 12170.
Fatal NI connect error 12170.
Tue Nov 25 10:40:25 2014

***********************************************************************
Tue Nov 25 10:40:25 2014

***********************************************************************
Fatal NI connect error 12170.
Fatal NI connect error 12170.
  VERSION INFORMATION:
TNS for Linux: Version 11.2.0.4.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production
  VERSION INFORMATION:
TNS for Linux: Version 11.2.0.4.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production
  VERSION INFORMATION:
TNS for Linux: Version 11.2.0.4.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production
  VERSION INFORMATION:
TNS for Linux: Version 11.2.0.4.0 - Production
Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.4.0 - Production
  Time: 25-11月-2014 10:40:25
  Time: 25-11月-2014 10:40:25
  Time: 25-11月-2014 10:40:25
  Time: 25-11月-2014 10:40:25
  Tracing not turned on.
  Tracing not turned on.
  Tracing not turned on.
  Tns error struct:
  Tns error struct:
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12535
    ns main err code: 12535
  Tns error struct:
    ns main err code: 12535
        
   
    ns main err code: 12535
TNS-12535: TNS: 操作超时
TNS-12535: TNS: 操作超时
   
TNS-12535: TNS: 操作超时
    ns secondary err code: 12606
    ns secondary err code: 12606
    ns secondary err code: 12606
    nt main err code: 0
    nt main err code: 0
    nt main err code: 0
TNS-12535: TNS: 操作超时
    nt secondary err code: 0
    nt secondary err code: 0
    nt secondary err code: 0
    ns secondary err code: 12606
    nt OS err code: 0
    nt OS err code: 0
    nt OS err code: 0
    nt main err code: 0
  Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=192.168.19.6*)(PORT=47725))
  Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=192.168.19.6*)(PORT=47722))
  Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=192.168.19.**)(PORT=47727))
    nt secondary err code: 0
WARNING: inbound connection timed out (ORA-3136)WARNING: inbound connection timed out (ORA-3136)
    nt OS err code: 0
  Client address: (ADDRESS=(PROTOCOL=tcp)(HOST=192.168.19.6*)(PORT=47726))
  VERSION INFORMATION:

从日志显示是应用程序所在服务器连不上数据库，可以肯定的是网络是没问题的（有监控，如果有问题会告警），网上说数据库负载高也会出现这种情况
登录控制台，察看此时间段的情况：发现此时间段的average active session很高：


运行addm如下:
addm_12129.rar (6.71 KB, 下载次数: 5)

2014-11-28 12:01 上传

点击文件名下载附件

请大侠一起分析下

Yong Huang

Is "enq: KO - fast object checkpoint" your top wait event during the time? Your ADDM report doesn't show that wait and does not give a clear indication of the cause of your problem. But generally, Oracle Net (SQL*Net) timeout could be due to network problems and/or server high load. If you're sure your network at the lower layer is fine, check CPU load on the server. If it's Linux or UNIX, you can find the historical data with sar command.

The ADDM does tell you that the SGA may be under-sized (recommend you increase it from 16G to 26G). How much memory do you have on the box? Some SQLs may need to be tuned to reduce CPU usage.

If the KO enqueue wait is indeed high for an extended period of time, check your I/O. Is it on ASM? Do you truncate or drop lots of tables?