把数据导入CouchDB－java、Ruby、Erlang三种方法

jieforest

Saving the records

Once we have the erlang structure ready to be saved into our CouchDB instance things get much easier. For this purpose I have used some code snippets from the ‘Hovercraft‘ library which looks like its not developed since 2010 but the code is easy to follow and I was able to reuse some functionality I needed.

An easy direct Erlang CouchDB library.

Use this to abstract CouchDB behind a simple Erlang function call. Currently supports the database and document APIs, with views on the way.

The ‘lightning’ test works with the latest CouchDB version and thats more than enough for my needs, if it wouldn’t work, it was a good starting point to play with the CouchDB from Erlang anyway.

jieforest

Saving our erlang structure is as easy as a single method call:

1. %
   
2. %% Closing tags and resetting counters..
   
3. %
   
4. processEndTag(<<"artist">>,Acc) ->
   
5.   [_,_,_,_,ArtistData] = Acc,
   
6.   save_doc(<<"erlang_music">>,ArtistData),
   
7.   [];
   

复制代码

jieforest

Whole part for interacting with the CouchDB from erlang looks like the following: (taken mostly from Hovercraft sources):

1. open_db(DbName) ->
   
2.   couch_db:open(DbName, [?ADMIN_USER_CTX]).
   
3. 
   
4. create_db(DbName) ->
   
5.   create_db(DbName, []).
   
6. 
   
7. create_db(DbName, Options) ->
   
8.   case couch_server:create(DbName, Options) of
   
9.     {ok, Db} ->
   
10.       couch_db:close(Db),
    
11.       {ok, created};
    
12.     Error ->
    
13.       {error, Error}
    
14.   end.
    
15. 
    
16. delete_db(DbName) ->
    
17.   delete_db(DbName,  [?ADMIN_USER_CTX]).
    
18. 
    
19. delete_db(DbName, Options) ->
    
20.   case couch_server:delete(DbName, Options) of
    
21.     ok ->
    
22.       {ok, deleted};
    
23.     Error ->
    
24.       {error, Error}
    
25.   end.
    
26. 
    
27. save_doc(#db{}=Db, Doc) ->
    
28.   CouchDoc = ejson_to_couch_doc(Doc),
    
29.   {ok, Rev} = couch_db:update_doc(Db, CouchDoc, []),
    
30.   {ok, {[{id, CouchDoc#doc.id}, {rev, couch_doc:rev_to_str(Rev)}]}};
    
31. 
    
32. save_doc(DbName, Docs) ->
    
33.   {ok, Db} = open_db(DbName),
    
34.   save_doc(Db, Docs).
    
35. 
    
36. ejson_to_couch_doc({DocProps}) ->
    
37.   Doc = case proplists:get_value(<<"_id">>, DocProps) of
    
38.     undefined ->
    
39.       DocId = couch_uuids:new(),
    
40.       {[{<<"_id">>, DocId}|DocProps]};
    
41.     _DocId ->
    
42.       {DocProps}
    
43.   end,
    
44.   couch_doc:from_json_obj(Doc).
    

复制代码

Full source code for the erlang script as well as ruby is available from GitHub, for Java version you can get it from ‘woodstoxex‘ example under the ‘couchdb’ branch.

jieforest

The Test

Assuming you have downloaded the test data from Jamendo website, and modified the sources appropriately you do the following:

Java – this is really straightforward, just execute the main method from the ‘Main’ class to get the job done.

Ruby –  run the script feeding it with the xml stream using cat command:

1. cat /path_to_your_download_folder/dbdump_artistalbumtrack.0.290905586176.xml | ruby jamendo.rb

复制代码

jieforest

Our erlang script will be installed and used the same way a Hovercraft library was used, inside the CouchDB trunk, create a folder called ‘xmlimporter’ and place our erlang script in there (modifying the xml file location earlier).

Once you have a CouchDB build you can start it up in interactive mode with the following command:

1. erlc xmlimporter/*erl && ERL_LIBS="xmlimporter" utils/run -i

复制代码

jieforest

Hit Enter when the CouchDB finishes launching and you should be inside the erlang console.

To start our script execute the following command:

1. 1> xmlimporter:start().

复制代码

where (1>) is the erlang shell prompt.

jieforest

The Results

Putting the database inserts on the side, it has to be highlighted that every script described here takes a different amount of time to parase xml (without doing anything usefull with the records themselves). To compare the results between those 3 scripts I have to take that into account and first find out the time of parsing and then the time of parsing and inserting the records.

jieforest

The results of parsing and inserting for each language/library of choice used is as follows:

jieforest

Final Notes

Process of importing data could be done quicker if we would insert that data in batches instead of each record one by one. This would eat a bit more memory but will increase the speed significantly I believe. Another thing to note is that the scripts described here are by no means perfect (especially the erlang one) and could be improved for speed and efficiency. It was obvious from the beginning that the Java and Ruby versions would be most probably slower than the erlang version (no matter how bad the code is) as they are communicating with the database through the HTTP protocol where the Erlang script is using the CouchDB code directly.

Surprisingly the results differs just a bit. Where parsing in Java using Woodstox was the fastest one it had the slowest insertion times, probably because of the Ektorp performance. Ruby and Erlang parsing times look similar but Erlang was quicker while inserting the data as it didn’t had to go through the loopback interface.

jieforest

over.