Microservice architecture diagram composition

A low code approach to composing microservice architecture diagrams from per service context diagrams.

Background

On a recent assignment I was one of multiple new engineers joining a start-up transitioning into a scale-up. The past two years had been spent rapidly developing to prove the concept, and while at the outset some diagrams were drawn up, at the time I joined these were outdated, lacking or even misleading. To help get ourselves and other new developers up to speed, Andrew Morgan and I set out to create architecture diagrams for the system as a whole.

Approach

In typical fashion these days, the system consists of microservices communicating with one-another in a mix of direct HTTP invocation and by publishing and subscribing to Kafka topics. In the absence of tracing information or tools to deduce the diagrams, we settled on a simple approach:

  1. Create context diagrams per service
    Using PlantUML we created diagrams for each service, in the root of the project repository. The diagrams contain the topics published/subscribed to, and direct HTTP invocations going out from the service, as well as any data stores and interactions with external services.
  2. Compose overall diagrams
    Using the GitLab API we gather all projects with a particular tag, and for those projects extract the PlantUML diagrams from the repository. These are then combined into a single overall diagram by stripping out the @start/enduml tags and simply appending them one after the other. By using distinct tags we’re able to create diagrams for services in active development, in production or zoom in on a particular aspect such as data processing. New projects are automatically picked up, and warnings are sent out when diagrams are missing.
  3. Expose diagrams as images
    A nightly pipeline creates images for each service individually, as well as a number of composite diagram images. The individual service diagrams are visualized in the GitLab repository README of each project, giving passing developers a quick glance of the context of a single service. The composed service diagrams are visualized in the README of the composition project, for easy reference elsewhere.

Compose individual diagrams

Implementation

The code consists of a Gradle task, a Gradle build file and a GitLab CI file. We place the task in the buildSrc directory, so we can easily use it in our project’s build.gradle file.

Gradle Task

buildSrc build.gradle

Project build.gradle

.gitlab-ci.yml

Strengths

We chose this approach as a simple way to document the overall architecture, in a landscape that’s still undergoing changes and needs to be kept up to date. Having the individual diagrams close to the code ensures the individual diagrams are easily maintained, while the overall diagrams always reflect the latest changes. These qualities are not as easily achieved when the diagram is further removed from the code. The low-tech approach allows anyone to quickly contribute diagrams, with minimal tooling.

Weaknesses

As the overall diagrams are composed from individual diagrams, a common set of guidelines should be enforced for a consistent overall view. Inconsistent use of arrow directionality for instance, can have big consequences for the overall diagram layout. Also, layout of the overall diagram is not stable with regards to previous versions in light of recent changes. Introduction of a new inter-service-dependency for example, can rearrange large parts of the diagram, which can confuse those familiar with previous versions.

Future extensions

GitLab PlantUML integration

Above we use separate files for the diagrams for simplicity, and render the diagrams using the PlantUML jar. However, GitLab can integrate with PlantUML directly to visualize diagrams present in any AsciiDoc and Markdown documents in a repository. This would allow one to make the diagrams part of the README for instance, with only slight changes to the diagram composition.

Other platforms

While the code above uses the GitLab API the same approach can just as easily be applied to different platforms such as GitHub.

Alternative diagrams

As briefly mentioned above, there are alternatives to generate overall system architecture diagrams. One such approach is Structurizer, which inspects the code to generate diagrams at multiple levels. Another, more deductive approach is to use the operational platform and available tracing information to compose diagrams based on runtime service dependencies. These and more are explored to extend and supplement the current approach.

Groovy Goodness: Download Grab Dependencies In IntelliJ IDEA

In our Groovy scripts we can use the @Grab annotation. With this annotation we define dependencies for our script and they will be automatically downloaded and added to the class path when we run our script. When we use IntelliJ IDEA we can use a nice intention that is part of the IntelliJ IDEA Groovy support to download the dependencies from our IDE editor. IDEA downloads the dependencies and add it to the project module dependencies. This is useful, because this will also adds code completion for the classes in the dependency to our editor.

Continue reading

Groovy Goodness: Make Sure Closeable Objects Are Closed Using withCloseable Method

If a class implements the Closeable interface Groovy adds the withCloseable method to the class. The withCloseable method has a closure as argument. The code in the closure is executed and then the implementation of the close method of the Closeable interface is invoked. The Closeable object is passed as argument to the closure, so we can refer to it inside the closure.

Continue reading

Spocklight: Reuse Variables In Data Providers

Writing a parameterized specification in Spock is easy. We need to add the where: block and use data providers to specify different values. For each set of values from the data providers our specifications is run, so we can test for example very effectively multiple input arguments for a method and the expected outcome. A data provider can be anything that implements the Iterable interface. Spock also adds support for a data table. In the data table we define columns for each variable and in the rows values for each variable. Since Spock 1.1 we can reuse the value of the variables inside the data provider or data table. The value of the variable can only be reused in variables that are defined after the variable we want to reuse is defined.

Continue reading

Spocklight: Group Assertions With verifyAll

We all know writing tests or specifications with Spock is fun. We can run our specifications and when one of our assertions in a feature method invalid, the feature method is marked as failed. If we have more than one assertion in our feature method, only the first assertion that fails is returned as an error. Any other assertion after a failing assertion are not checked. To let Spock execute all assertions and return all failing assertions at once we must use the verifyAll method. We pass a closure with all our assertions to the method. All assertions will be checked when use the verifyAll and if one or more of the assertions is invalid the feature method will fail.

Continue reading

Ratpacked: Assert No Exceptions Are Thrown With RequestFixture

Writing unit tests for our handlers in Ratpack is easy with RequestFixture. We invoke the handle method and use a Handler or Chain we want to test as argument. We can provide extra details on the fixture instance with a second argument, for example adding objects to the registry or setting the request method. The handle method returns a HandlingResult object. This object has the method exception that we can use to see if an exception occurred in our code under test. The method throws a HandlerExceptionNotThrownException if the expected exception doesn’t occurr.

Continue reading

Jenkins Joy: Shared library for Jenkins Declarative Pipeline

Since the introduction of the Jenkins Declarative Pipeline syntax (as opposed to the Scripted Pipeline syntax) the concept of a shared library has become more important as we are otherwise restricted to the sections of the pipeline model. So we’ll make a basic set-up with Continue reading

Groovy Goodness: Redirecting Print Methods In Scripts

To run external Groovy scripts in our Java or Groovy application is easy to do. For example we can use GroovyShell to evaluate Groovy code in our applications. If our script contains print methods like println we can redirect the output of these methods. The Script class, which is a base class to run script code, has an implementation for the print, printf and println methods. The implementation of the method is to look for a property out, either as part of a Script subclass or in the binding added to a Script class. If the property out is available than all calls to print, printf and println methods are delegated to the object assigned to the out property. When we use a PrintWriter instance we have such an object, but we could also write our own class with an implementation for the print methods. Without an assignment to the out property the fallback is to print on System.out.

Continue reading

Groovy Goodness: Using The Call Operator ()

In Groovy we can add a method named call to a class and then invoke the method without using the name call. We would simply just type the parentheses and optional arguments on an object instance. Groovy calls this the call operator: (). This can be especially useful in for example a DSL written with Groovy. We can add multiple call methods to our class each with different arguments. The correct method is invoked at runtime based on the arguments.

Continue reading

Groovy Goodness: Creating Root JSON Array With JsonBuilder

To create JSON output with Groovy is easy using JsonBuilder and StreamingJsonBuilder. In the samples mentioned in the links we create a JSON object with a key and values. But what if we want to create JSON with a root JSON array using JsonBuilder or StreamingJsonBuilder? It turns out to be very simple by passing a list of values using the constructor or using the implicit method call.

Continue reading