用Neo4j来管理微服务

jieforest

Infrastructure Mapping

The last use case: infrastructure mapping.

Similar to the problem with our services, we didn’t have a lot of visibility into what pools were live, what pools were dark, what servers were in a virtual service or even what app instances were out there. This problem extended to our entire infrastructure.

Once we got started with Neo4j, we played around with virtual servers and services and realized, “Hey, we can map out our entire infrastructure with this.”

So here you’ll see what we just talked about:

jieforest

Every virtual service contains two pools, which contain a number of virtual servers, which maps to an app instance. Exposing this data allowed us to ask, “Are any servers in the live pool degraded?”

jieforest

We don’t want an unhealthy server in our live pool, so basically the state and the priority is what determines if the server is live and active, if it’s healthy. If those conditions aren’t met, then it returns that server and we get a ping about it.

From there, we decided to add stuff from vCenter, so all the app instances map to compute instances which are hosted by compute hosts and so we added a bunch more nodes from vCenter into Neo4j (below).

jieforest

We were able to extend this so now we can ask the question, “Do we have a single point of failure among our services?”

If at any given time within a pool all those servers are hosted on one host in vCenter, if that host is to go down, then we definitely have a problem. Because we are mapping this in Neo4j, we are able to expose this data in a way that we weren’t able to before.

This query is kind of blurry, but it shows the traversal pretty well from pool to virtual server to app instance to compute instance to compute host.

jieforest

This query is kind of blurry, but it shows the traversal pretty well from pool to virtual server to app instance to compute instance to compute host.

jieforest

Rob: I love this because vCenter has a habit with vMotion of deciding to move things around. If you don’t have affinity set up in vCenter, it’s real easy to set yourself up for these single points of failure.

Before Neo4j, we were constantly scanning the infrastructure, and I’d be asking staff, “Is it okay? Can you look again?”

But we want the infrastructure to tell us. We want to be reacting to problems, not having to go constantly look.

Ashley: This problem is also really hard if you’re just looking in vCenter at a host. It’s hard to know if there is a single point of failure.
So really, we weren’t just taking data but making sense of it within Neo4j and exposing it in a way that was useful to us.

So to continue with our infrastructure diagram:

jieforest

The nimbles at the bottom are our storage arrays and our storage volumes. If we wanted, we can traverse this graph all the way from a virtual service to, for example, a storage array or a storage volume.

Now we want to ask a question: “If this storage volume goes down, what services are going to be impacted?”

jieforest

Again, it’s a really simple query, it runs really quickly and it tells us things we never had visibility into before.

Other Use Cases

I think the cool thing here, as Rob was saying, is that it started out where we were using Neo4j as a hobby. We thought, “Oh, it would be cool if we put our app instances into Neo4j.” From there, we were like, “Oh, app instances, we can wrap those pretty easily into virtual services.”

Then, because that information was there, we were able to automate our deployments and from there we kept building and building on our dataset that we already had. That is one of the things I think is cool about Neo4j: that you can develop incrementally.

When we first started, we didn’t know we were going to end up with this full deployment. And we’re still progressing.

We’re adding firewall rules; we could add databases; as we move into Amazon, we can get EC2 instances and security groups. It’s pretty cool. It’s easy to build on your dataset, and make it more complex.

Also, our information security group has recently taken an interest in MacGyver for service onboarding. We now have a service registration in MacGyver, so when we get new services, we can register with MacGyver.

We can determine if a service is allowed to talk to another one, and we have a graph of relationships of services that depend on one another and talk to each other. We also use that graph for rezoning. Is this server in the correct security zone?

jieforest

Using Neo4j and Microservices for Greater Agility

Rob: We want to keep this agile culture going in the company, and we don’t want to have meetings and change review boards and all that kind of stuff that nobody really likes.

The information security use cases are a great one for the continuation of DevOps. And when we’re thinking about how we are going to move from five services to 139 to 400 – at the same time as the company is getting bigger – we can’t have meetings and change review boards to get there.

Our information security group is on board with this because they want to hook into it, and they want to react to chaining workflows around new services created by developers.

Then, they want to dig into it, asking, “Now maybe I need to start running app scans on the new service. Maybe I need to have a conversation with that scrum team to understand what it is. Maybe I’m going to ask them to provide attributes about it so that we know how to do data classification.”

The opportunities for continuing this DevOps mindset with Neo4j, it’s limitless.

We’re really excited, particularly with moving into AWS and the API control plane that has such a rich variety of information, even though it’s kind of hard to query. As we’ve started doing that automation, it’s slow – something that will take 15 seconds to make 20 different queries out to AWS – but we want it to come back in a snap so we can have fast, responsive APIs that allow us to deliver the services that we want.

jieforest

Ashley: In the end, everything is awesome when you use Neo4j.

We have a lot of individual microservices (or “Lego blocks”) and we can switch them out or move them around. As Rob was saying, sometimes things can get messy, but using Neo4j to manage it has made it a lot easier, and we’ll continue to use Neo4j.

Inspired by Ashley and Rob’s talk? Register for GraphConnect Europe on April 26, 2016 at for more industry-leading presentations and workshops on the evolving world of graph database technology.