|
|
本帖最后由 skywalker418 于 2011-11-21 17:08 编辑
jinguanding 发表于 2011-11-16 18:38 ![]()
InnoDB: Failing assertion: page_get_n_recs(page) > 1
InnoDB: We intentionally generate a memory tra ...
今天另一台从库又挂了。错误信息相同。
再补充一些信息：
1、这台从库自那天之后已调整了内存参数，可使用内存之和没有超过主机物理内存
2、今天早晨每次启动这个从库，会在Crash recovery finished后，依然出现Assertion failure。这个过程会不断重复，Crash recovery、Assertion failure，重复一定次数后宕掉。
3、在做了一些操作(如启动数据库、用gdb启动、用--core-file参数）后，Crash recovery 进行到46%就会出现Assertion failure（之前可以顺利完成Crash recovery）
4、下面附上Crash Covery finished、 Crash Covery 46%和gdb获取的信息
完全没头绪，求大拿解救
Crash Covery finished
111121 12:41:46 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
111121 12:41:46 [Note] Failed to execute mysql_file_stat on file './mysql-bin.000143'
111121 12:41:46 [Note] Plugin 'FEDERATED' is disabled.
111121 12:41:46 InnoDB: The InnoDB memory heap is disabled
111121 12:41:46 InnoDB: Mutexes and rw_locks use GCC atomic builtins
111121 12:41:46 InnoDB: Compressed tables use zlib 1.2.3
111121 12:41:46 InnoDB: Using Linux native AIO
111121 12:41:46 InnoDB: Initializing buffer pool, size = 8.0G
111121 12:41:47 InnoDB: Completed initialization of buffer pool
111121 12:41:47 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 537728331465
111121 12:41:47 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 537728615077
111121 12:41:47 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 3
7 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 8
1 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: Last MySQL binlog file position 0 455970077, file name ./mysql-bin.000021
111121 12:41:48 InnoDB: Waiting for the background threads to start
111121 12:41:49 InnoDB: 1.1.8 started; log sequence number 537728615077
111121 12:41:49 [Note] Recovering after a crash using mysql-bin
111121 12:41:49 [Note] Starting crash recovery...
111121 12:41:49 [Note] Crash recovery finished.
111121 12:41:49 [Warning] 'proxies_priv' entry '@ root@db01' ignored in --skip-name-resolve mode.
111121 12:41:49 [Warning] Neither --relay-log nor --relay-log-index were used; so replication may break when this MySQL server acts
as a slave and has his hostname changed!! Please use '--relay-log=db04-relay-bin' to avoid this problem.
111121 12:41:49 [Note] Event Scheduler: Loaded 0 events
111121 12:41:49 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.14-log' socket: '/var/lib/mysql/mysql.sock' MySQL Community Server (GPL)
111121 12:41:49 InnoDB: Assertion failure in thread 1190238528 in file ibuf0ibuf.c line 4185
InnoDB: Failing assertion: page_get_n_recs(page) > 1
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/ ... nnodb-recovery.html
InnoDB: about forcing recovery.
111121 12:41:49 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=33554432
read_buffer_size=2097152
max_used_connections=0
max_threads=500
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2086459 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = (nil) thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x39)[0x7ccc79]
/usr/sbin/mysqld(handle_segfault+0x379)[0x4fe7b9]
/lib64/libpthread.so.0[0x364900eb10]
/lib64/libc.so.6(gsignal+0x35)[0x3648430265]
/lib64/libc.so.6(abort+0x110)[0x3648431d10]
/usr/sbin/mysqld[0x8f2bbf]
/usr/sbin/mysqld[0x8ac94e]
/usr/sbin/mysqld[0x8dbd78]
/usr/sbin/mysqld[0x863440]
/lib64/libpthread.so.0[0x364900673d]
/lib64/libc.so.6(clone+0x6d)[0x36484d3f6d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file
111121 12:41:49 mysqld_safe Number of processes running now: 0
111121 12:41:49 mysqld_safe mysqld restarted
111121 12:41:50 [Note] Plugin 'FEDERATED' is disabled.
111121 12:41:50 InnoDB: The InnoDB memory heap is disabled
111121 12:41:50 InnoDB: Mutexes and rw_locks use GCC atomic builtins
111121 12:41:50 InnoDB: Compressed tables use zlib 1.2.3
111121 12:41:50 InnoDB: Using Linux native AIO
111121 12:41:50 InnoDB: Initializing buffer pool, size = 8.0G
111121 12:41:50 InnoDB: Completed initialization of buffer pool
111121 12:41:50 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 537728331465
111121 12:41:50 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
Crash recovery 46%
111121 16:40:18 [Note] Plugin 'FEDERATED' is disabled.
111121 16:40:18 InnoDB: The InnoDB memory heap is disabled
111121 16:40:18 InnoDB: Mutexes and rw_locks use GCC atomic builtins
111121 16:40:18 InnoDB: Compressed tables use zlib 1.2.3
111121 16:40:18 InnoDB: Using Linux native AIO
111121 16:40:18 InnoDB: Initializing buffer pool, size = 8.0G
111121 16:40:18 InnoDB: Completed initialization of buffer pool
111121 16:40:18 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 537728331465
111121 16:40:19 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 537728651913
111121 16:40:19 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 111121 16:40:19 InnoDB: Assertion failure in thread 1157658944 in file ibuf0ibuf.c line 4185
InnoDB: Failing assertion: page_get_n_recs(page) > 1
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/ ... nnodb-recovery.html
InnoDB: about forcing recovery.
111121 16:41:33 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
111121 16:41:33 [Note] Plugin 'FEDERATED' is disabled.
111121 16:41:33 InnoDB: The InnoDB memory heap is disabled
111121 16:41:33 InnoDB: Mutexes and rw_locks use GCC atomic builtins
111121 16:41:33 InnoDB: Compressed tables use zlib 1.2.3
111121 16:41:33 InnoDB: Using Linux native AIO
111121 16:41:33 InnoDB: Initializing buffer pool, size = 8.0G
111121 16:41:34 InnoDB: Completed initialization of buffer pool
111121 16:41:34 InnoDB: highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 537728331465
111121 16:41:34 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 537728651913
111121 16:41:34 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 111121 16:41:34 InnoDB: Assertion failure in thread 1169197376 in file ibuf0ibuf.c line 4185
InnoDB: Failing assertion: page_get_n_recs(page) > 1
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/ ... nnodb-recovery.html
InnoDB: about forcing recovery.
47 111121 16:41:34 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=33554432
read_buffer_size=2097152
max_used_connections=0
max_threads=100
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 443506 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = (nil) thread_stack 0x30000
/usr/sbin/mysqld(my_print_stacktrace+0x39)[0x7ccc79]
/usr/sbin/mysqld(handle_segfault+0x379)[0x4fe7b9]
/lib64/libpthread.so.0[0x364900eb10]
/lib64/libc.so.6(gsignal+0x35)[0x3648430265]
/lib64/libc.so.6(abort+0x110)[0x3648431d10]
/usr/sbin/mysqld[0x8f2bbf]
/usr/sbin/mysqld[0x8ac94e]
/usr/sbin/mysqld[0x8dbd78]
/usr/sbin/mysqld[0x863440]
/lib64/libpthread.so.0[0x364900673d]
/lib64/libc.so.6(clone+0x6d)[0x36484d3f6d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
111121 16:41:34 mysqld_safe mysqld from pid file /var/lib/mysql/db04.pid ended
gdb
-bash-3.2$ gdb --args /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/lib/mysql/db04.err --open-files-limit=8192 --pid-file=/var/lib/mysql/db04.pid --socket=/var/lib/mysql/mysql.sock
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-23.el5)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/mysqld...(no debugging symbols found)...done.
(gdb) run
Starting program: /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/lib/mysql/db04.err --open-files-limit=8192 --pid-file=/var/lib/mysql/db04.pid --socket=/var/lib/mysql/mysql.sock [Thread debugging using libthread_db enabled]
[New Thread 0x40a00940 (LWP 5369)]
[Thread 0x40a00940 (LWP 5369) exited]
[New Thread 0x40a00940 (LWP 5370)]
[New Thread 0x41401940 (LWP 5371)]
[New Thread 0x41e02940 (LWP 5372)]
[New Thread 0x42803940 (LWP 5373)]
[New Thread 0x43204940 (LWP 5374)]
[New Thread 0x43c05940 (LWP 5375)]
[New Thread 0x44606940 (LWP 5376)]
[New Thread 0x45007940 (LWP 5377)]
[New Thread 0x45a08940 (LWP 5378)]
[New Thread 0x46409940 (LWP 5379)]
[New Thread 0x46e0a940 (LWP 5380)]
[New Thread 0x4780b940 (LWP 5381)]
[New Thread 0x4820c940 (LWP 5382)]
[New Thread 0x48c0d940 (LWP 5383)]
[New Thread 0x4960e940 (LWP 5384)]
[New Thread 0x4a00f940 (LWP 5385)]
[New Thread 0x4aa10940 (LWP 5386)]
[New Thread 0x4b411940 (LWP 5387)]
Program received signal SIGABRT, Aborted.
[Switching to Thread 0x45007940 (LWP 5377)]
0x0000003648430265 in raise () from /lib64/libc.so.6
(gdb) backtrace full
#0 0x0000003648430265 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x0000003648431d10 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x00000000008f2bbf in ibuf_merge_or_delete_for_page (block=0x2aaab1768300, space=110, page_no=690544, zip_size=0,
update_ibuf_bitmap=1)
at /export/home/pb2/build/sb_0-3516584-1308676814.36/rpm/BUILD/mysql-5.5.14/mysql-5.5.14/storage/innobase/ibuf/ibuf0ibuf.c:4185
success = <value optimized out>
entry = <value optimized out>
max_trx_id = <value optimized out>
dummy_index = 0x2aacd8005648
op = IBUF_OP_DELETE
rec = <value optimized out>
heap = 0x10080240
pcur = {btr_cur = {index = 0x1000cab8, page_cur = {rec = 0x2aaabefed472 "", block = 0x2aaab1768900},
purge_node = 0x8b10ec, left_block = 0x450069b0, thr = 0x866fa2, flag = BTR_CUR_BINARY, tree_height = 2, up_match = 3,
up_bytes = 0, low_match = 2, low_bytes = 2, n_fields = 10922504, n_bytes = 1, fold = 1, ibuf_cnt = 3, path_arr = 0x0},
latch_mode = 2, old_stored = 122766467, old_rec = 0x0, old_n_fields = 1157656512, rel_pos = 9496870,
block_when_stored = 0x0, modify_clock = 1157655264, pos_state = 1997660512, search_mode = 2, trx_if_known = 0x0,
old_rec_buf = 0x0, buf_size = 0}
search_tuple = 0x100802b8
page_zip = 0x0
tablespace_being_deleted = 0
corruption_noticed = 0
mtr = {memo = {heap = 0x0, used = 80,
data = "\002\000\000\000\000\000\000\000\000\203v\261\252*\000\000\067", '\000' <repeats 15 times>, "\003\000\000\000\000\000\000\000\000\260m\261\252*\000\000\003\000\000\000\000\000\000\000\000\211v\261\252*\000\000\002\000\000\000\000\000\000\000\000\211v\261\252*", '\000' <repeats 433 times>, base = {count = 0, start = 0x0, end = 0x0}, list = {prev = 0x0, next = 0x0}}, log = {
heap = 0x0, used = 0,
data = '\000' <repeats 472 times>"\200, o\000E\000\000\000\000\365\222\220", '\000' <repeats 28 times>, base = {
count = 0, start = 0x0, end = 0x0}, list = {prev = 0x0, next = 0x27c37e0}}, inside_ibuf = 1, modifications = 0,
n_log_recs = 0, log_mode = 21, start_lsn = 0, end_lsn = 690544}
mops = {0, 0, 0}
dops = {0, 0, 0}
#3 0x00000000008ac94e in buf_page_io_complete (bpage=0x2aaab1768300)
at /export/home/pb2/build/sb_0-3516584-1308676814.36/rpm/BUILD/mysql-5.5.14/mysql-5.5.14/storage/innobase/buf/buf0buf.c:4077
read_page_no = 690544
read_space_id = 110
frame = 0x2aaabefdc000 "\307`\r{"
buf_pool = 0x27fe5d8
uncompressed = 1
#4 0x00000000008dbd78 in fil_aio_wait (segment=7)
at /export/home/pb2/build/sb_0-3516584-1308676814.36/rpm/BUILD/mysql-5.5.14/mysql-5.5.14/storage/innobase/fil/fil0fil.c:4565
ret = <value optimized out>
fil_node = 0x10034638
message = 0x2aaab1768300
type = 10
#5 0x0000000000863440 in io_handler_thread (arg=<value optimized out>)
at /export/home/pb2/build/sb_0-3516584-1308676814.36/rpm/BUILD/mysql-5.5.14/mysql-5.5.14/storage/innobase/srv/srv0start.c:482
---Type <return> to continue, or q <return> to quit---
segment = 7
#6 0x000000364900673d in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#7 0x00000036484d3f6d in clone () from /lib64/libc.so.6
No symbol table info available.
|
|
楼主: skywalker418
IP卡
狗仔卡
发表于 2011-11-17 14:40
显身卡
楼主