难道真是程序导致Out Of Memory??

shuayang · 发表于 2009-5-4 17:02

软件环境： aix 5.3+weblogic9.2+oracle1023
系统每天在线人数2500左右，数据库的数据量有4张频繁操作的表数据记录在1千万，2千万，3千万以上，其它表至少有20个数据量在百万级别以上。
系统前段时间运行一直正常(已经运行了1年多，以前没有出现过OOM)，只是从4月底出现OOM,最近差不多一个星期出现一次。
对weblogic产生的phd文件和javacore文件分析的不太清楚。。。
以下是weblogic的日志：
.
.
JAVA Memory arguments: -Xms768m -Xmx1536m
.
WLS Start Mode=Development
.
CLASSPATH=/usr/java5_64/lib/:/usr/java5_64/jre/lib/:/usr/java5_64/lib/dt.jar:/usr/java5_64/lib/tools.jar::/bea/patch_weblogic922/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/usr/java5_64/lib/tools.jar:/bea/weblogic92/server/lib/weblogic_sp.jar:/bea/weblogic92/server/lib/weblogic.jar:/bea/weblogic92/server/lib/webservices.jar::/bea/weblogic92/common/eval/pointbase/lib/pbclient51.jar:/bea/weblogic92/server/lib/xqrl.jar::
.
PATH=/bea/weblogic92/server/bin:/usr/java5_64/jre/bin:/usr/java5_64/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java5_64/jre/bin:/usr/java5_64/bin
.
***************************************************
*  To start WebLogic Server, use a username and *
*  password assigned to an admin-level user.  For *
*  server administration, use the WebLogic Server *
*  console at http://hostname

ort/console       *
***************************************************
starting weblogic with Java version:
java version "1.5.0"
Java(TM) 2 Runtime Environment, Standard Edition (build pap64dev-20080315 (SR7))
IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 AIX ppc64-64 j9vmap6423-20080315 (JIT enabled)
J9VM - 20080314_17962_BHdSMr
JIT  - 20080130_0718ifx2_r8
GC - 200802_08)
JCL  - 20080314
Starting WLS with line:
/usr/java5_64/bin/java -Xms768m -Xmx1536m -Dcom.sun.xml.namespace.QName.useCompatibleSerialVersionUID=1.0  -da -Dplatform.home=/bea/weblogic92 -Dwls.home=/bea/weblogic92/server -Dwli.home=/bea/weblogic92/integration  -Dweblogic.management.discover=false -Dweblogic.management.server=http://xxx.xxx.xxx.xxx  -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole= -Dfile.encoding=GB18030 -Dweblogic.ext.dirs=/bea/patch_weblogic922/profiles/default/sysext_manifest_classpath -Dweblogic.management.username=weblogic -Dweblogic.management.password=weblogic -Dweblogic.Name=OASvrA03 -Djava.security.policy=/bea/weblogic92/server/lib/weblogic.policy weblogic.Server
<2009-4-30 下午04时03分51秒 GMT+08:00> <Notice> <WebLogicServer> <BEA-000395> <Following extensions directory contents added to the end of the classpath:
/bea/weblogic92/platform/lib/p13n/p13n-schemas.jar:/bea/weblogic92/platform/lib/p13n/p13n_common.jar:/bea/weblogic92/platform/lib/p13n/p13n_system.jar:/bea/weblogic92/platform/lib/wlp/netuix_common.jar:/bea/weblogic92/platform/lib/wlp/netuix_schemas.jar:/bea/weblogic92/platform/lib/wlp/netuix_system.jar:/bea/weblogic92/platform/lib/wlp/wsrp-common.jar>
<2009-4-30 下午04时03分51秒 GMT+08:00> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with IBM J9 VM Version 2.3 from IBM Corporation>
<2009-4-30 下午04时03分53秒 GMT+08:00> <Info> <Management> <BEA-141107> <Version: WebLogic Server 9.2 MP2  Mon Jun 25 01:32:01 EDT 2007 952826 >
<2009-4-30 下午04时03分56秒 GMT+08:00> <Info> <WebLogicServer> <BEA-000215> <Loaded License : /bea/license.bea>
<2009-4-30 下午04时03分56秒 GMT+08:00> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING>
<2009-4-30 下午04时03分56秒 GMT+08:00> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool>
<2009-4-30 下午04时03分57秒 GMT+08:00> <Notice> <Log Management> <BEA-170019> <The server log file /bea/user_projects/domains/oa_domain/servers/OASvrA03/logs/OASvrA03.log is opened. All server side log events will be written to this file.>
<2009-5-4 上午10时06分19秒 GMT+08:00> <Critical> <Health> <BEA-310003> <Free memory in the server is 838,960 bytes. There is danger of OutOfMemoryError>
JVMDUMP006I Processing Dump Event "systhrow", detail "java/lang/OutOfMemoryError" - Please Wait.
JVMDUMP007I JVM Requesting Snap Dump using '/bea/user_projects/domains/oa_domain/Snap.20090504.100639.139302.0001.trc'
JVMDUMP010I Snap Dump written to /bea/user_projects/domains/oa_domain/Snap.20090504.100639.139302.0001.trc
JVMDUMP006I Processing Dump Event "systhrow", detail "java/lang/OutOfMemoryError" - Please Wait.
JVMDUMP006I Processing Dump Event "systhrow", detail "java/lang/OutOfMemoryError" - Please Wait.
JVMDUMP006I Processing Dump Event "systhrow", detail "java/lang/OutOfMemoryError" - Please Wait.
JVMDUMP007I JVM Requesting Snap Dump using '/bea/user_projects/domains/oa_domain/Snap.20090504.100641.139302.0002.trc'
JVMDUMP010I Snap Dump written to /bea/user_projects/domains/oa_domain/Snap.20090504.100641.139302.0002.trc
JVMDUMP007I JVM Requesting Snap Dump using '/bea/user_projects/domains/oa_domain/Snap.20090504.100641.139302.0003.trc'
JVMDUMP010I Snap Dump written to /bea/user_projects/domains/oa_domain/Snap.20090504.100641.139302.0003.trc
JVMDUMP007I JVM Requesting Snap Dump using '/bea/user_projects/domains/oa_domain/Snap.20090504.100641.139302.0004.trc'
JVMDUMP010I Snap Dump written to /bea/user_projects/domains/oa_domain/Snap.20090504.100641.139302.0004.trc
JVMDUMP007I JVM Requesting Snap Dump using '/bea/user_projects/domains/oa_domain/Snap.20090504.100641.139302.0005.trc'
JVMDUMP010I Snap Dump written to /bea/user_projects/domains/oa_domain/Snap.20090504.100641.139302.0005.trc
JVMDUMP006I Processing Dump Event "systhrow", detail "java/lang/OutOfMemoryError" - Please Wait.
JVMDUMP007I JVM Requesting Heap Dump using '/bea/user_projects/domains/oa_domain/heapdump.20090504.100641.139302.0006.phd'
JVMDUMP010I Heap Dump written to /bea/user_projects/domains/oa_domain/heapdump.20090504.100641.139302.0006.phd
JVMDUMP013I Processed Dump Event "systhrow", detail "java/lang/OutOfMemoryError".
JVMDUMP007I JVM Requesting Heap Dump using '/bea/user_projects/domains/oa_domain/heapdump.20090504.100641.139302.0007.phd'
JVMDUMP010I Heap Dump written to /bea/user_projects/domains/oa_domain/heapdump.20090504.100641.139302.0007.phd
JVMDUMP013I Processed Dump Event "systhrow", detail "java/lang/OutOfMemoryError".
Exception in thread "Timer-1" java.lang.OutOfMemoryError
JVMDUMP007I JVM Requesting Java Dump using '/bea/user_projects/domains/oa_domain/javacore.20090504.100641.139302.0008.txt'
JVMDUMP012E Error in Java Dump: /bea/user_projects/domains/oa_domain/javacore.20090504.100641.139302.0008.txt
JVMDUMP013I Processed Dump Event "systhrow", detail "java/lang/OutOfMemoryError".
at weblogic.utils.collections.Stack.<init>(Stack.java:18)
at weblogic.utils.collections.Stack.<init>(Stack.java:13)
at weblogic.kernel.ThreadLocalStack$StackInitialValue.initialValue(ThreadLocalStack.java:159)
at weblogic.kernel.FinalThreadLocal$FinalThreadStorage.<init>(FinalThreadLocal.java:183)
at weblogic.kernel.AuditableThread.<init>(AuditableThread.java:13)
at weblogic.work.ExecuteThread.<init>(ExecuteThread.java:63)
at weblogic.work.RequestManager.create(RequestManager.java:304)JVMDUMP007I JVM Requesting Java Dump using '/bea/user_projects/domains/oa_domain/javacore.20090504.100641.139302.0009.txt'
JVMDUMP012E Error in Java Dump: /bea/user_projects/domains/oa_domain/javacore.20090504.100641.139302.0009.txt
JVMDUMP013I Processed Dump Event "systhrow", detail "java/lang/OutOfMemoryError".

********************************

[ 本帖最后由 shuayang 于 2009-5-6 10:06 编辑 ]