MongoDB规模化的设计模式

jieforest · 发表于 2012-9-30 01:29

We've actually improvised performance in two ways using this approach:

1. Pre-allocation means that our documents never grow, so they never get moved

2. By pre-allocating throughout the day, we don't have a "midnight problem" where our upserts all end up inserting a new document and increasing load on the server.

We do, however, have a curious downward trend in performance throughout the day (though much less drastic than before). Where did that come from?

jieforest · 发表于 2012-10-1 13:06

MongoDB's storage format

To figure out the downward performance through the day, we need to take a brief detour into the actual format that MongoDB uses to store data on disk (and memory), BSON. Normally, we don't need to worry about it, since the pymongo driver converts everything so nicely into native Python types, but in this case BSON presents us a performance problem.

Although MongoDB documents, such as our minute embedded document, are represented in Python as a dict (which is a constant-speed lookup hash table), BSON actually stores documents as an association list. So rather than having a nice hash table for minute, we actually have something that looks more like the following:

1.minute = [
2.[ "0000", 3612 ],
3.[ "0001", 3241 ],
4.# ...
5.[ "1439", 2819 ] ]

复制代码

jieforest · 发表于 2012-10-1 13:06

Now to actually update a particular minute, the MongoDB server performs the something like the following operations (psuedocode, with lots of special cases ignored):

1.inc_value(minute, "1439", 1)
2.
3.def inc_value(document, key, value)
4.for entry in document:
5.if entry[0] == key:
6.entry[1] += value
7.break

复制代码

The performance of this algorithm, far from our nice O(1) hash table, is actually O(N) in the number of entries in the document. In the case of the minute document, MongoDB has to actually perform 1439 comparisons before it finds the appropriate slot to update.

jieforest · 发表于 2012-10-1 13:07

Fixing the downward trend with hierarchy

To fix the problem, then, we need to reduce the number of comparisons MongoDB needs to do to find the right minute to increment. The way we can do this is by splitting up the minutes into hours. Our daily stats document now looks like the following:

01.{ _id: "20101010/metric-1",
02.metadata: {
03.date: ISODate("2000-10-10T00:00:00Z"),
04.metric: "metric-1" },
05.daily: 5468426,
06.hourly: {
07."0": 227850,
08."1": 210231,
09....
10."23": 20457 },
11.minute: {
12."00": {
13."0000": 3612,
14."0100": 3241,
15....
16.}, ...,
17."23": { ..., "1439": 2819 }
18.}

复制代码

jieforest · 发表于 2012-10-1 13:07

Our record_hit and preallocate routines have to change a bit as well:

01.def record_hit_hier(coll, dt, measure):
02.if PREALLOC and random.random() < (1.0/1500.0):
03.preallocate_hier(coll, dt + timedelta(days=1), measure)
04.sdate = dt.strftime('%Y%m%d')
05.metadata = dict(
06.date=datetime.combine(
07.dt.date(),
08.time.min),
09.measure=measure)
10.id='%s/%s' % (sdate, measure)
11.coll.update(
12.{ '_id': id, 'metadata': metadata },
13.{ '$inc': {
14.'daily': 1,
15.'hourly.%.2d' % dt.hour: 1,
16.('minute.%.2d.%.2d' % (dt.hour, dt.minute)): 1 } },
17.upsert=True)
18.
19.def preallocate(coll, dt, measure):
20.'''Once again, simplified for explanatory purposes'''
21.metadata, id = # compute metadata and ID
22.coll.update(
23.{ '_id': id },
24.{ '$set': { 'metadata': metadata },
25.'$inc': {
26.'daily': 0,
27.'hourly.00': 0,
28.# ...
29.'hourly.23': 0,
30.'minute.00.00': 0,
31.# ...
32.'minute.00.59': 0,
33.'minute.01.00': 0,
34.# ...
35.'minute.23.59': 0 } },
36.upsert=True)

复制代码

jieforest · 发表于 2012-10-1 13:07

Once we've added the hierarchy and re-run our experiment, we get the nice, level performance we'd like to see:

jieforest · 发表于 2012-10-2 14:32

Conclusion

It's always nice to see "tips and tricks" borne out through actual, quantitative results, so this was probably the most enjoyable talk I've ever put together. The things I got out of it were the following:

1. Growing documents is a very bad thing for performance. Avoid it if at all possible.

2. Awareness of the BSON specification and data representation can actually be quite useful when diagnosing performance problems.

3. To get the best performance out of your system, you need to actually run it (or a highly representative stand-in). Actually seeing the results of performance tweaking in graphical form is incredibly helpful in targeting your efforts.

jieforest · 发表于 2012-10-2 14:32

The source code for all these updates is available in my mongodb-sdas Github repo, and I welcome any feedback either there, or here in the comments. In particular, I'd love to hear of any performance problems you've run into and how you got around them. And of course, if you've got a really perplexing problem, I'm always available for consulting by emailing me at Arborian.com.

jieforest · 发表于 2012-10-2 14:32

over.

wang1352083 · 发表于 2012-10-22 15:51

求git的url.一段一段看不容易看懂