Microservice architecture diagram composition

A low code approach to composing microservice architecture diagrams from per service context diagrams.

Background

On a recent assignment I was one of multiple new engineers joining a start-up transitioning into a scale-up. The past two years had been spent rapidly developing to prove the concept, and while at the outset some diagrams were drawn up, at the time I joined these were outdated, lacking or even misleading. To help get ourselves and other new developers up to speed, Andrew Morgan and I set out to create architecture diagrams for the system as a whole.

Approach

In typical fashion these days, the system consists of microservices communicating with one-another in a mix of direct HTTP invocation and by publishing and subscribing to Kafka topics. In the absence of tracing information or tools to deduce the diagrams, we settled on a simple approach:

  1. Create context diagrams per service
    Using PlantUML we created diagrams for each service, in the root of the project repository. The diagrams contain the topics published/subscribed to, and direct HTTP invocations going out from the service, as well as any data stores and interactions with external services.
  2. Compose overall diagrams
    Using the GitLab API we gather all projects with a particular tag, and for those projects extract the PlantUML diagrams from the repository. These are then combined into a single overall diagram by stripping out the @start/enduml tags and simply appending them one after the other. By using distinct tags we’re able to create diagrams for services in active development, in production or zoom in on a particular aspect such as data processing. New projects are automatically picked up, and warnings are sent out when diagrams are missing.
  3. Expose diagrams as images
    A nightly pipeline creates images for each service individually, as well as a number of composite diagram images. The individual service diagrams are visualized in the GitLab repository README of each project, giving passing developers a quick glance of the context of a single service. The composed service diagrams are visualized in the README of the composition project, for easy reference elsewhere.

Compose individual diagrams

Implementation

The code consists of a Gradle task, a Gradle build file and a GitLab CI file. We place the task in the buildSrc directory, so we can easily use it in our project’s build.gradle file.

Gradle Task

buildSrc build.gradle

Project build.gradle

.gitlab-ci.yml

Strengths

We chose this approach as a simple way to document the overall architecture, in a landscape that’s still undergoing changes and needs to be kept up to date. Having the individual diagrams close to the code ensures the individual diagrams are easily maintained, while the overall diagrams always reflect the latest changes. These qualities are not as easily achieved when the diagram is further removed from the code. The low-tech approach allows anyone to quickly contribute diagrams, with minimal tooling.

Weaknesses

As the overall diagrams are composed from individual diagrams, a common set of guidelines should be enforced for a consistent overall view. Inconsistent use of arrow directionality for instance, can have big consequences for the overall diagram layout. Also, layout of the overall diagram is not stable with regards to previous versions in light of recent changes. Introduction of a new inter-service-dependency for example, can rearrange large parts of the diagram, which can confuse those familiar with previous versions.

Future extensions

GitLab PlantUML integration

Above we use separate files for the diagrams for simplicity, and render the diagrams using the PlantUML jar. However, GitLab can integrate with PlantUML directly to visualize diagrams present in any AsciiDoc and Markdown documents in a repository. This would allow one to make the diagrams part of the README for instance, with only slight changes to the diagram composition.

Other platforms

While the code above uses the GitLab API the same approach can just as easily be applied to different platforms such as GitHub.

Alternative diagrams

As briefly mentioned above, there are alternatives to generate overall system architecture diagrams. One such approach is Structurizer, which inspects the code to generate diagrams at multiple levels. Another, more deductive approach is to use the operational platform and available tracing information to compose diagrams based on runtime service dependencies. These and more are explored to extend and supplement the current approach.