修复MongoDB数据库，解决因Unclean Shutdown导致服务不能启动的问题

吴_飚

今天原本想研究点mongodb的东西，发现服务启不来了：

1. [root@biao ~]# mongod -f /etc/mongod.conf
   
2. forked process: 19465
   
3. all output going to: /mongodb/log/mongodb.log
   
4. [root@biao ~]# service mongod status
   
5. mongod is stopped

复制代码

打开log看了下，日志太多了，密密麻麻的，用tail查看又怕看得不全，索性重新指定个log文件，让日志记录到新的日志文件中:

1. [root@biao ~]# mongod -f /etc/mongod.conf -logpath /mongodb/log/mongodb_new.log
   
2. forked process: 20310
   
3. all output going to: /mongodb/log/mongodb_new.log
   
4. [root@biao ~]# service mongod status
   
5. mongod is stopped

复制代码

查看log：

1. root@biao ~]# more /mongodb/log/mongodb_new.log
   
2. Wed Apr  3 23:02:40
   
3. Wed Apr  3 23:02:40 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durabi
   
4. lity.
   
5. Wed Apr  3 23:02:40
   
6. Wed Apr  3 23:02:40 [initandlisten] MongoDB starting : pid=20310 port=27017 dbpath=/mongodb/data 32-bit host=biao.oracle.com
   
7. Wed Apr  3 23:02:40 [initandlisten]
   
8. Wed Apr  3 23:02:40 [initandlisten] ** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
   
9. Wed Apr  3 23:02:40 [initandlisten] **       see [url]http://blog.mongodb.org/post/137788967/32-bit-limitations[/url]
   
10. Wed Apr  3 23:02:40 [initandlisten] **       with --journal, the limit is lower
    
11. Wed Apr  3 23:02:40 [initandlisten]
    
12. Wed Apr  3 23:02:40 [initandlisten] db version v2.2.3, pdfile version 4.5
    
13. Wed Apr  3 23:02:40 [initandlisten] git version: f570771a5d8a3846eb7586eaffcf4c2f4a96bf08
    
14. Wed Apr  3 23:02:40 [initandlisten] build info: Linux bs-linux32.10gen.cc 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 20
    
15. 08 i686 BOOST_LIB_VERSION=1_49
    
16. Wed Apr  3 23:02:40 [initandlisten] options: { config: "/etc/mongod.conf", dbpath: "/mongodb/data", fork: "true", logappend:
    
17. "true", logpath: "/mongodb/log/mongodb_new.log", pidfilepath: "/var/run/mongodb/mongod.pid" }
    
18. Wed Apr  3 23:02:40 [initandlisten] Unable to check for journal files due to: boost::filesystem::basic_directory_iterator con
    
19. structor: No such file or directory: "/mongodb/data/journal"
    
20. **************
    
21. Unclean shutdown detected.
    
22. Please visit [url]http://dochub.mongodb.org/core/repair[/url] for recovery instructions.
    
23. *************
    
24. Wed Apr  3 23:02:40 [initandlisten] exception in initAndListen: 12596 old lock file, terminating
    
25. Wed Apr  3 23:02:40 dbexit:
    
26. Wed Apr  3 23:02:40 [initandlisten] shutdown: going to close listening sockets...
    
27. Wed Apr  3 23:02:40 [initandlisten] shutdown: going to flush diaglog...
    
28. Wed Apr  3 23:02:40 [initandlisten] shutdown: going to close sockets...
    
29. Wed Apr  3 23:02:40 [initandlisten] shutdown: waiting for fs preallocator...
    
30. Wed Apr  3 23:02:40 [initandlisten] shutdown: closing all files...
    
31. Wed Apr  3 23:02:40 [initandlisten] closeAllFiles() finished
    
32. Wed Apr  3 23:02:40 dbexit: really exiting now
    
33. [root@biao ~]#

复制代码

发现其中有一段，也是最中要的一段日志：

**************

Unclean shutdown detected.

Please visit http://dochub.mongodb.org/core/repairfor recovery instructions.

*************

日志中说发现有unclean shutdown，还给了个指导链接，真人性化！上去研究了下。

服务器断电、异常关闭以及直接killall命令导致服务终止的情况都可能会被mondodb认为是unclean shutdown，

因为unclean shutdown可能会导致数据不一致性或者数据损坏，所以必须要手动修复后才能继续提供服务。

mongodb修复unclean shutdown的方式有：

1、从journal修复，journal就是类似于Oracle数据库中的redo log，工作机制也是类似的，都是write-ahead的。

2、从集群中其他节点的副本中修复。

3、如果服务不是运行在集群环境中，也没有启用journal，那必须在再次启动服务时使用--repair或者--repair加上--repairpath修复。

带与不带--repairpath的区别在于，来一段官方的解释，英文不好，翻译得费劲：

Use the --repair optionin conjunction with the --repairpath option. mongod will read the existing datafiles, and write the existing data to new data files. This does not modify oralter the existing data files.

You do not need to removethe mongod.lock file before using this procedure.

Use the --repair option.mongod will read the existing data files, write the existing data to new filesand replace the existing, possibly corrupt, files with new files.

You must remove themongod.lock file before using this procedure.

很明显我的mongodb跑的是单节点，也没有启用journal，journal配置据说在2.0以下的版本以及32位系统上默认是不启用的，据说是因为耗内存。。。

本机的mongod版本为2.2.3：

1. [root@biao ~]# mongod --version
   
2. db version v2.2.3, pdfile version 4.5
   
3. Thu Apr  4 01:14:14 git version: f570771a5d8a3846eb7586eaffcf4c2f4a96bf08

复制代码

本机是32位系统：

1. [root@biao ~]# getconf LONG_BIT
   
2. 32
   
3. [root@biao ~]# getconf WORD_BIT
   
4. 32
   
5. [root@biao ~]# uname -a
   
6. Linux biao.oracle.com 2.6.18-348.3.1.el5 #1 SMP Mon Mar 11 19:40:14 EDT 2013 i686 i686 i386 GNU/Linux

复制代码

在mongod --help发现有启用journal的参数，但是在默认配置文件中的没有发现有启用journal的参数，只有禁用的参数，

但是想想的话，我这个情况应该是没有启用的，否则就自动修复了.

1. # Disables write-ahead journaling
   
2. # nojournal = true

复制代码

这里我使用--repair修复，因为本来也就没啥有用的数据，替不替换都无所谓。

查看dbpath下的mogod.lock文件：

1. [root@biao ~]# ll /mongodb/data/
   
2. total 4
   
3. -rwxr-xr-x 1 root root 6 Jan 25 02:11 mongod.lock

复制代码

官方指导说可以直接删了，我还是先移走吧，免得回头需要：

1. [root@biao ~]# mv /mongodb/data/mongod.lock  ~/.

复制代码

执行修复，必须要指定dbpath：

1. [root@biao ~]# mongod --dbpath /mongodb/data --repair
   
2. Thu Apr  4 00:08:07
   
3. Thu Apr  4 00:08:07 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability.
   
4. Thu Apr  4 00:08:07
   
5. Thu Apr  4 00:08:07 [initandlisten] MongoDB starting : pid=23196 port=27017 dbpath=/mongodb/data 32-bit host=biao.oracle.com
   
6. Thu Apr  4 00:08:07 [initandlisten]
   
7. Thu Apr  4 00:08:07 [initandlisten] ** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
   
8. Thu Apr  4 00:08:07 [initandlisten] **       see [url]http://blog.mongodb.org/post/137788967/32-bit-limitations[/url]
   
9. Thu Apr  4 00:08:07 [initandlisten] **       with --journal, the limit is lower
   
10. Thu Apr  4 00:08:07 [initandlisten]
    
11. Thu Apr  4 00:08:07 [initandlisten] db version v2.2.3, pdfile version 4.5
    
12. Thu Apr  4 00:08:07 [initandlisten] git version: f570771a5d8a3846eb7586eaffcf4c2f4a96bf08
    
13. Thu Apr  4 00:08:07 [initandlisten] build info: Linux bs-linux32.10gen.cc 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_49
    
14. Thu Apr  4 00:08:07 [initandlisten] options: { dbpath: "/mongodb/data", repair: true }
    
15. Thu Apr  4 00:08:07 [initandlisten] Unable to check for journal files due to: boost::filesystem::basic_directory_iterator constructor: No such file or directory: "/mongodb/data/journal"
    
16. Thu Apr  4 00:08:07 [initandlisten] finished checking dbs
    
17. Thu Apr  4 00:08:07 dbexit:
    
18. Thu Apr  4 00:08:07 [initandlisten] shutdown: going to close listening sockets...
    
19. Thu Apr  4 00:08:07 [initandlisten] shutdown: going to flush diaglog...
    
20. Thu Apr  4 00:08:07 [initandlisten] shutdown: going to close sockets...
    
21. Thu Apr  4 00:08:07 [initandlisten] shutdown: waiting for fs preallocator...
    
22. Thu Apr  4 00:08:07 [initandlisten] shutdown: closing all files...
    
23. Thu Apr  4 00:08:07 [initandlisten] closeAllFiles() finished
    
24. Thu Apr  4 00:08:07 [initandlisten] shutdown: removing fs lock...
    
25. Thu Apr  4 00:08:07 dbexit: really exiting now

复制代码

也没看到有是否修复成功的提示

直接启动下看看：

1. [root@biao ~]# mongod --dbpath /mongodb/data/
   
2. Thu Apr  4 00:12:06
   
3. Thu Apr  4 00:12:06 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability.
   
4. Thu Apr  4 00:12:06
   
5. Thu Apr  4 00:12:06 [initandlisten] MongoDB starting : pid=23395 port=27017 dbpath=/mongodb/data/ 32-bit host=biao.oracle.com
   
6. Thu Apr  4 00:12:06 [initandlisten]
   
7. Thu Apr  4 00:12:06 [initandlisten] ** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
   
8. Thu Apr  4 00:12:06 [initandlisten] **       see [url]http://blog.mongodb.org/post/137788967/32-bit-limitations[/url]
   
9. Thu Apr  4 00:12:06 [initandlisten] **       with --journal, the limit is lower
   
10. Thu Apr  4 00:12:06 [initandlisten]
    
11. Thu Apr  4 00:12:06 [initandlisten] db version v2.2.3, pdfile version 4.5
    
12. Thu Apr  4 00:12:06 [initandlisten] git version: f570771a5d8a3846eb7586eaffcf4c2f4a96bf08
    
13. Thu Apr  4 00:12:06 [initandlisten] build info: Linux bs-linux32.10gen.cc 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_49
    
14. Thu Apr  4 00:12:06 [initandlisten] options: { dbpath: "/mongodb/data/" }
    
15. Thu Apr  4 00:12:06 [initandlisten] Unable to check for journal files due to: boost::filesystem::basic_directory_iterator constructor: No such file or directory: "/mongodb/data/journal"
    
16. Thu Apr  4 00:12:06 [initandlisten] waiting for connections on port 27017
    
17. Thu Apr  4 00:12:06 [websvr] admin web console waiting for connections on port 28017

复制代码

如上启动貌似是交互模式的，直接就定格在那了。。。

重开个窗口查看下状态：

1. [root@biao ~]# service mongod status
   
2. mongod (pid 23395) is running...
   
3. [root@biao ~]# ps aux|grep mongod
   
4. root     23395  0.1  0.7 119404 22800 pts/9    Sl+  00:12   0:01 mongod --dbpath /mongodb/data/
   
5. root     24417  0.0  0.0   4032   724 pts/11   S+   00:33   0:00 grep mongod
   
6. [root@biao ~]# netstat -aux|grep mongod
   
7. unix  2      [ ACC ]     STREAM     LISTENING     402385 /tmp/mongodb-27017.sock

复制代码

可以看到mongod服务启动成功，监听端口为默认的27017。

重启一下再看看，记得用--shutdown关闭：

1. [root@biao ~]# mongod --shutdown
   
2. There doesn't seem to be a server running with dbpath: /data/db/
   
3. [root@biao ~]# mongod -f /etc/mongod.conf --shutdown
   
4. killing process with pid: 23395
   
5. [root@biao ~]# service mongod status
   
6. mongod is stopped
   
7. [root@biao ~]# mongod -f /etc/mongod.conf
   
8. forked process: 24592
   
9. all output going to: /mongodb/log/mongodb.log
   
10. child process started successfully, parent exiting

复制代码

再检查下：

1. [root@biao ~]# service mongod status
   
2. mongod (pid 24592) is running...
   
3. [root@biao ~]# ps aux|grep mongod
   
4. root     24592  0.6  0.6 118388 21060 ?        Sl   00:36   0:00 mongod -f /etc/mongod.conf
   
5. root     24631  0.0  0.0    292    52 pts/11   R+   00:36   0:00 grep mongod
   
6. [root@biao ~]# netstat -aux|grep mongod
   
7. unix  2      [ ACC ]     STREAM     LISTENING     404328 /tmp/mongodb-27017.sock
   

复制代码

可以看到mongod服务依然坚挺的启动成功了，监听端口依然坚挺的为默认的27017。

至此，修复成功。

心得：关闭服务时，一定要注意，最好就是用--shutdown命令关闭，既方便又安全。

可以的话还是要启用journal功能，至于journal有多耗内存，还需后续进一步研究。

吴_飚

yinhuazhe

沒人支持就幫一下

jieforest

写得不错。

jieforest

如果是在Windows操作系统下安装MongoDB，不妨把它安装为系统服务，这样在操作系统关闭时，会自动关闭MongoDB服务，不会出现这种数据损坏的异常。

吴_飚

jieforest 发表于 2013-4-5 22:23
写得不错。

更多原创在dataguru的nosql版块，不过我也准备开始混pub了，大师多关照！

吴_飚

jieforest 发表于 2013-4-5 22:25
如果是在Windows操作系统下安装MongoDB，不妨把它安装为系统服务，这样在操作系统关闭时，会自动关闭MongoD ...

嗯，确实。。不过我现在学习的这些个nosql都是安装在linux上。。。

jieforest

吴_飚 发表于 2013-4-5 22:25
更多原创在dataguru的nosql版块，不过我也准备开始混pub了，大师多关照！

jieforest

吴_飚 发表于 2013-4-5 22:28
嗯，确实。。不过我现在学习的这些个nosql都是安装在linux上。。。

把MongoDB安装为Linux系统服务，应该也具备这个功能——关机时系统服务会随之关闭。当然，掉电又另当别论。