用Neo4j来管理微服务

jieforest

At Lending Club, we use blue-green deployments. In a blue-green deployment, we have a live pool and a dark pool at any given time, so half the servers are live and half of them are dark within a service group.

The live pool takes all the traffic, while the dark pool is inactive. During a deployment, we deploy new code to the dark pool, and then QA and Release run testing on it.

jieforest

Once they say okay, the dark pool is good to go, we change the state on the dark pool so that it becomes active with a high priority, and we also lower the priority of the pool that’s on the left.

After that switch, all new connections are sent to the newly live pool on the right and we wait for all of the old connections on the previously live pool to drain out on the left. Once those old connections reach zero, we cut that pool over.

jieforest

Essentially, we’ve just done a pool flip. Now the pool on the right is live, and the pool on the left is dark.

Although we had this concept of pools, the load balancer could associate servers with each service group, but it couldn’t tell which servers were in what pool.

Also, the app instances that were being reported to MacGyver knew their revision and app ID, but they didn’t know their state. They didn’t know if they were live or dark.

To solve that problem, we were able to use Neo4j to map these servers into pools and automate our deployments.

This involves what we just talked about where app instances are saved into Neo4j. We took the info from the app instances – such as app ID, revisions, environment – and we combined that information. We then polled our load balancer for information on servers – such as state; whether it’s active, inactive, draining; how many connections does that server have – and we combine that with our app instance nodes, and we were able to create virtual server nodes.

jieforest

By collecting virtual servers and aggregating them according to their app ID and their state, we were able to create pools, which is where it gets a little more interesting.

These (above) are application pools in Neo4j, and you’ll see the purple dot is a pool. Each pool contains many servers and each of those servers has a one-to-one mapping to our app instances.

jieforest

Because it’s pulling info from its virtual servers, each pool knows whether it’s live, dark or draining. Each service has a live pool and a dark pool; pool A and pool B. So by mapping the pools into a virtual service, for every service we have a green dot and there are two pools (below).

As before, within a pool there are many servers which are then mapped to app instances.

Because of this data model, we gained a lot of visibility into apps that we didn’t have before, and questions that had once been difficult to answer became easy with Neo4j.

jieforest

For example, let’s answer the question, “What pool should I deploy to?”

Usually, we would have to bounce into the A10 GUI and look up which servers were active or inactive, but that’s all solved with a simple Cypher query (below).

Right away, it’s easy to tell what pool we want to deploy to.

jieforest

Another example: “Are live pool revisions in sync in different environments?”

This is important because, say, our main load balancer fails over to our backup load balancer. In this case, we want to make sure there’s not old code running and that it’s the same revision.

This also becomes a simple Cypher query (below) which before would have required manual look-up.

Here’s an important one: “Do multiple revisions exist within a single pool?”

jieforest

We don’t want old code and new code running at the same time, and so we just query: “Show me all the servers within a pool” and if there’s more than one revision within a pool then we know that’s a problem.

If you think about the Knight Capital meltdown in 2012, they lost $440 million because they had old and new code deployed at the same time.

jieforest

Rob: That was the end of that company, I believe, or they got sold off for nothing.

Ashley: Yeah, this happened in like 45 minutes. So if you do the math, it’s $163,000 that they were losing per second, which is crazy. They should have been using Neo4j.

Rob: It’s an easy mistake to make, so if it were to happen, we get paged.

Ashley: A lot of these queries we periodically run throughout the day, and they’ll page us. They’ll tell us if something is wrong.

So, a high-level overview of what we just covered: The app instances report in to MacGyver, they get saved to Neo4j, we take some info from the load balancer and combine that with our app instance info and from that we are able to create virtual servers. From there, we group those servers into pools and then we connect those pools into virtual services.

jieforest

The reason we were able to do all this for deployment and release automation was because of these pools that we were able to create with Neo4j.

Because Neo4j is so good at mapping relationships, it was really perfect for this use case. Not only can we monitor all the virtual services, but at the same time, we can also send commands to the load balancer and this is how we’re able to automate our deployments.

At any time you can say, “I want to deploy this revision of this app into this environment.” MacGyver can then respond, “This pool is dark. We’re going to deploy to this pool.”

So you can deploy MacGyver, and then you can start a drain in MacGyver. Then, because MacGyver knows the number of connections to all of its servers, it can automatically just cut the pool over.