嵌入式NoSQL数据库RaptorDB 3.1.0发布

jieforest

How the LINQ parser works in RaptorDB

In RaptorDB we want to be able to extract and query the index for each clause in the filter expression based on the order and logic of the expression. Because the indexes are built on the WAHBitArray the result will be a WAHBitArray. All this is done in the following very small code (compared to writing a language parser) :

1.    delegate WAHBitArray QueryExpression(string colname, RDBExpression exp, object from);
   
2. 
   
3.     internal class QueryVisitor : ExpressionVisitor
   
4.     {
   
5.         public QueryVisitor(QueryExpression express)
   
6.         {
   
7.             qexpression = express;
   
8.         }
   
9.         public Stack<object> _stack = new Stack<object>();
   
10.         public Stack<object> _bitmap = new Stack<object>();
    
11.         QueryExpression qexpression;
    
12. 
    
13.         protected override Expression VisitBinary(BinaryExpression b)
    
14.         {
    
15.             this.Visit(b.Left);
    
16.             ExpressionType t = b.NodeType;
    
17. 
    
18.             if (t == ExpressionType.Equal || t == ExpressionType.LessThan || t == ExpressionType.LessThanOrEqual ||
    
19.                 t == ExpressionType.GreaterThan || t == ExpressionType.GreaterThanOrEqual)
    
20.                 _stack.Push(b.NodeType);
    
21.            
    
22.             this.Visit(b.Right);
    
23.             t = b.NodeType;
    
24.             if (t == ExpressionType.Equal || t == ExpressionType.NotEqual ||
    
25.                 t == ExpressionType.LessThanOrEqual || t == ExpressionType.LessThan ||
    
26.                 t == ExpressionType.GreaterThanOrEqual || t == ExpressionType.GreaterThan)
    
27.             {
    
28.                 // binary expression
    
29.                 object lv = _stack.Pop();
    
30.                 ExpressionType lo = (ExpressionType)_stack.Pop();
    
31.                 object ln = _stack.Pop();
    
32.                 RDBExpression exp = RDBExpression.Equal;
    
33.                
    
34.                 if (lo == ExpressionType.LessThan)
    
35.                     exp = RDBExpression.Less;
    
36.                 else if (lo == ExpressionType.LessThanOrEqual)
    
37.                     exp = RDBExpression.LessEqual;
    
38.                 else if (lo == ExpressionType.GreaterThan)
    
39.                     exp = RDBExpression.Greater;
    
40.                 else if (lo == ExpressionType.GreaterThanOrEqual)
    
41.                     exp = RDBExpression.GreaterEqual;
    
42. 
    
43.                 _bitmap.Push(qexpression("" + ln, exp, lv));
    
44.             }
    
45. 
    
46.             if (t == ExpressionType.And || t == ExpressionType.AndAlso ||
    
47.                 t == ExpressionType.Or || t == ExpressionType.OrElse)
    
48.             {
    
49.                 // do bitmap operations
    
50.                 WAHBitArray r = (WAHBitArray)_bitmap.Pop();
    
51.                 WAHBitArray l = (WAHBitArray)_bitmap.Pop();
    
52. 
    
53.                 if (t == ExpressionType.And || t == ExpressionType.AndAlso)
    
54.                     _bitmap.Push(r.And(l));
    
55.                 if (t == ExpressionType.Or || t == ExpressionType.OrElse)
    
56.                     _bitmap.Push(r.Or(l));
    
57.             }
    
58.             return b;
    
59.         }
    
60. 
    
61.         protected override Expression VisitMethodCall(MethodCallExpression m)
    
62.         {
    
63.             string s = m.ToString();
    
64.             _stack.Push(s.Substring(s.IndexOf('.') + 1));
    
65.             return m;
    
66.         }
    
67. 
    
68.         protected override Expression VisitMember(MemberExpression m)
    
69.         {
    
70.             var e = base.VisitMember(m);
    
71.             var c = m.Expression as ConstantExpression;
    
72.             if (c != null)
    
73.             {
    
74.                 Type t = c.Value.GetType();
    
75.                 var x = t.InvokeMember(m.Member.Name, BindingFlags.GetField, null, c.Value, null);
    
76.                 _stack.Push(x);
    
77.             }
    
78.             if (m.Expression != null && m.Expression.NodeType == ExpressionType.Parameter)
    
79.             {
    
80.                 _stack.Push(m.Member.Name);
    
81.                 return e;
    
82.             }
    
83.             return e;
    
84.         }
    
85. 
    
86.         protected override Expression VisitConstant(ConstantExpression c)
    
87.         {
    
88.             IQueryable q = c.Value as IQueryable;
    
89.             if (q != null)
    
90.                 _stack.Push(q.ElementType.Name);
    
91.             else if (c.Value == null)
    
92.                 _stack.Push(null);
    
93.             else
    
94.             {
    
95.                 _stack.Push(c.Value);
    
96.                 if (Type.GetTypeCode(c.Value.GetType()) == TypeCode.Object)
    
97.                     _stack.Pop();
    
98.             }
    
99.             return c;
    
100.         }
     
101.     }

复制代码

jieforest

Most of the work is done in the VisitBinary method (for evaluating logical [&& || ] operations and clauses [b>3] ) so to distinguish the two a stack is used store the clause values for further processing. VisitBinary will be called recursively for left and right sides of expressions so a stack of bitmap is also required for aggregating the results of the expression.

The constructor to the class takes two delegates which are supplied by the caller for handles to the underlying indexes which this class calls when a binary clause is completely parsed. The results are push onto the bitmap stack.

The VisitMember method is responsible for replacing the compiler generated code for constant values with the appropriate value ( the j in the above example).

The rest of the code is generally for extracting the "column names" without the prefixes (s.Serial -> Serial etc.).

jieforest

A sample application

To work with RaptorDB all you need to do is follow these steps :

1）Define your Entities (plain c# objects) as you would doing domain driven development.

2）Create a Primary View for your base Entities.

3）Register your views with RaptorDB.

4）Save and query your data.

5）Add new views based on your requirements.

As you will see below this is so easy and simple that it just happens and you don't need to learn anything new or worry about configurations or breaking things at runtime as the compiler will catch your error at compile time.

A great feature is the total absence of anything SQL related, the associated schema pains and having to switch to a database management product to define and check tables and columns as everything is in your source file.

jieforest

1. Creating Entities

The first thing you should do is define your entities or data classes (referred to as domain driven development), these are plain c# (vb.net) classes or POCO's like the following :.

1. public class LineItem
   
2.     {
   
3.         public decimal QTY { get; set; }
   
4.         public string Product { get; set; }
   
5.         public decimal Price { get; set; }
   
6.         public decimal Discount { get; set; }
   
7.     }
   
8. 
   
9.     public class SalesInvoice
   
10.     {
    
11.         public SalesInvoice()
    
12.         {
    
13.             ID = Guid.NewGuid();
    
14.         }
    
15.         public Guid ID { get; set; }
    
16.         public string CustomerName { get; set; }
    
17.         public string Address { get; set; }
    
18.         public List<LineItem> Items { get; set; }
    
19.         public DateTime Date { get; set; }
    
20.         public int Serial { get; set; }
    
21.         public byte Status { get; set; }
    
22.     }

复制代码

There is nothing special about the above other than the lack of anything extra you need to do like adding attributes etc. (even Serializable) as they are not needed.

jieforest

2. Creating Views

Next you create your primary view for your entities as follows :

1.    public class SalesInvoiceView : View<SalesInvoice> // create a view for the SalesInvoice type
   
2.     {
   
3.         public class RowSchema  // define the schema for this view
   
4.         {
   
5.             public NormalString CustomerName; // CustomerName is a normal string index
   
6.             public DateTime InvoiceDate;
   
7.             public string Address;
   
8.             public int Serial;
   
9.             public byte Status;
   
10.         }
    
11. 
    
12.         public SalesInvoiceView()
    
13.         {
    
14.             this.Name = "SalesInvoice";
    
15.             this.Description = "A primary view for SalesInvoices";
    
16.             this.isPrimaryList = true;
    
17.             this.isActive = true;
    
18.             this.BackgroundIndexing = true;
    
19. 
    
20.             this.Schema = typeof(SalesInvoiceView.RowSchema);
    
21. 
    
22.             this.AddFireOnTypes(typeof(SalesInvoice));
    
23. 
    
24.             this.Mapper = (api, docid, doc) =>
    
25.             {
    
26.                 api.Emit(docid, doc.CustomerName, doc.Date, doc.Address, doc.Serial, doc.Status);
    
27.             };
    
28.         }
    
29.     }

复制代码

jieforest

This is pretty straight forward also.

1）This view is for a SalesInvoice object type.

2）RowSchema is a class which defines the columns for this view
You can name this class anything as long as you register it with the Schema property
All value types are supported (int, string, decimal etc.)
NormalString is a special type which instructs the indexer to index this column as a string so you will have to specify all the string when querying.
If you specify a string property then RaptorDB will do full text indexing on that column, so you can search for words within that column when querying.

3）BackgroundIndexing controls how the indexer does it's work on this view (i.e. block saves until each document is indexed when false).

4）AddFireOnTypes controls when this view is called based on the input document type

5）Mapper is the map function which will populate this view (i.e. extract information from the input document), you can add logic here if you need it. The order of the items must be the same as the schema you defined.

6）With api you can Fetch a document, log debug information and Query another view.

jieforest

3. Registering Views

Registering a view is as simple as :

1. if (rap.RegisterView(new SalesInvoiceView()).OK == false)
   
2.         {
   
3.                 Console.WriteLine("Error registering view");
   
4.                 return;
   
5.         }

复制代码

RaptorDB will do some checks on your view and if everything is fine it will return true, which means you are good to go.

jieforest

4. Saving and querying data

Now you can use RaptorDB and save documents as follows :

1. var inv = new SalesInvoice()
   
2.    {
   
3.        Date = FastDateTime.Now,
   
4.        Serial = i % 10000,
   
5.        CustomerName = "me " + i % 10,
   
6.        Status = (byte)(i % 4),
   
7.        Address = "df asd sdf asdf asdf"
   
8.    };
   
9.    inv.Items = new List<LineItem>();
   
10.    for (int k = 0; k < 5; k++)
    
11.        inv.Items.Add(new LineItem()
    
12.             { Product = "prod " + k, Discount = 0, Price = 10 + k, QTY = 1 + k });
    
13. 
    
14.    rap.Save(inv.ID, inv); // save to RaptorDB

复制代码

jieforest

Querying is as simple as writing LINQ predicates like the following:

1. var q = rap.Query(typeof(SalesInvoice), // call by the view type or the primary document type
   
2.                 (SalesInvoice s) => (s.Serial < j) && (s.Status == 1 || s.Status == 3));
   
3. 
   
4. q = rap.Query("SalesItemRows",  // call by the view name
   
5.         (LineItem l) => (l.Product == "prod 1" || l.Product == "prod 3"));

复制代码

As you can see you can call the query in 2 ways by specifying the type of the view (or type of the document type for primary views) or by calling the string name of the view.

jieforest

Screen Shots

From the below image you can see the test application doing its work. RaptorDB was configured to do background indexing, so 100,000 documents were inserted in 12 secs and the primary view was populated (the query results for 500 items) and the background indexer is working on populating the other view defined, which after a couple of queries shows the final results of 50,000 items.