Microservice architecture diagram composition

A low code approach to composing microservice architecture diagrams from per service context diagrams.

Background

On a recent assignment I was one of multiple new engineers joining a start-up transitioning into a scale-up. The past two years had been spent rapidly developing to prove the concept, and while at the outset some diagrams were drawn up, at the time I joined these were outdated, lacking or even misleading. To help get ourselves and other new developers up to speed, Andrew Morgan and I set out to create architecture diagrams for the system as a whole.

Approach

In typical fashion these days, the system consists of microservices communicating with one-another in a mix of direct HTTP invocation and by publishing and subscribing to Kafka topics. In the absence of tracing information or tools to deduce the diagrams, we settled on a simple approach:

  1. Create context diagrams per service
    Using PlantUML we created diagrams for each service, in the root of the project repository. The diagrams contain the topics published/subscribed to, and direct HTTP invocations going out from the service, as well as any data stores and interactions with external services.
  2. Compose overall diagrams
    Using the GitLab API we gather all projects with a particular tag, and for those projects extract the PlantUML diagrams from the repository. These are then combined into a single overall diagram by stripping out the @start/enduml tags and simply appending them one after the other. By using distinct tags we’re able to create diagrams for services in active development, in production or zoom in on a particular aspect such as data processing. New projects are automatically picked up, and warnings are sent out when diagrams are missing.
  3. Expose diagrams as images
    A nightly pipeline creates images for each service individually, as well as a number of composite diagram images. The individual service diagrams are visualized in the GitLab repository README of each project, giving passing developers a quick glance of the context of a single service. The composed service diagrams are visualized in the README of the composition project, for easy reference elsewhere.

Compose individual diagrams

Implementation

The code consists of a Gradle task, a Gradle build file and a GitLab CI file. We place the task in the buildSrc directory, so we can easily use it in our project’s build.gradle file.

Gradle Task

buildSrc build.gradle

Project build.gradle

.gitlab-ci.yml

Strengths

We chose this approach as a simple way to document the overall architecture, in a landscape that’s still undergoing changes and needs to be kept up to date. Having the individual diagrams close to the code ensures the individual diagrams are easily maintained, while the overall diagrams always reflect the latest changes. These qualities are not as easily achieved when the diagram is further removed from the code. The low-tech approach allows anyone to quickly contribute diagrams, with minimal tooling.

Weaknesses

As the overall diagrams are composed from individual diagrams, a common set of guidelines should be enforced for a consistent overall view. Inconsistent use of arrow directionality for instance, can have big consequences for the overall diagram layout. Also, layout of the overall diagram is not stable with regards to previous versions in light of recent changes. Introduction of a new inter-service-dependency for example, can rearrange large parts of the diagram, which can confuse those familiar with previous versions.

Future extensions

GitLab PlantUML integration

Above we use separate files for the diagrams for simplicity, and render the diagrams using the PlantUML jar. However, GitLab can integrate with PlantUML directly to visualize diagrams present in any AsciiDoc and Markdown documents in a repository. This would allow one to make the diagrams part of the README for instance, with only slight changes to the diagram composition.

Other platforms

While the code above uses the GitLab API the same approach can just as easily be applied to different platforms such as GitHub.

Alternative diagrams

As briefly mentioned above, there are alternatives to generate overall system architecture diagrams. One such approach is Structurizer, which inspects the code to generate diagrams at multiple levels. Another, more deductive approach is to use the operational platform and available tracing information to compose diagrams based on runtime service dependencies. These and more are explored to extend and supplement the current approach.

Starting as a new technical lead – Introducing change

Congratulations! Someone has made the wise decision to hire you as the new technical lead. It is an exciting time. You start in a new environment, will be working with a new team and maybe even have to learn new technologies along the way. This can be quite challenging. In this two-part article I want to share my personal views regarding Introducing change and shaping teams as a technical lead.

Introducing change

When starting in this new environment you probably bring lots of energy and want to leverage your experience to change things for the better. In my opinion introducing changes in a new environment requires some consideration.

Judging

As technical lead you are (hopefully) up to date with the latest principles and technology trends. As technical lead your life purpose is to “build the thing right” because “building the right thing” is someone else’s concern ;-) You quickly determine that the project you inherited has some some design/architectural issues that should be addressed promptly. So the next day you climb on a chair and give a passionate speech about how things need to change. Wait right there! Even if you are right, you need to consider some things.

First remember that you are judging the work of the current development team. They probably put their heart and soul into it. Secondly there are often influences from outside the team that drive technical decisions. The most obvious are deadline but there are others. Don’t be judgemental about the current state of development, because it  can quickly create a gap between you and the rest of the team.

Change velocity

Changing something requires some time for the team to adjust. This means you can’t pile change on change, even if they are beneficial. If you make to many changes you effectively try to change the team’s entire way of working.

Consider the change velocity of the team. This doesn’t have anything to do with experience. Sometimes junior team members find it easy to adopt new ways of working while the seniors are reluctant to change their ways. Getting this right requires at least experience and some feeling.

Quick wins

So am I telling you that you can’t propose any changes at all? Not really. After all you were hired as lead so you get some credits from the get go. You can introduce some minor changes to the product or processes. Often some minor changes can help the team work more efficiently. Maybe eliminate some small repetitive tasks for example. Proposing some good changes can really help establish your position as lead. Also people are more receptive to minor changes then they are to big changes. Just be aware of how many of these changes the team can adopt.

Bigger changes

Before you start introducing big changes, I think it is important to take some time to reflect on past and present. During this time quickly gather information. Try and get to know about:

  1. the team;
  2. the system’s architecture;
  3. the technologies used;
  4. the history and past architectural/design decisions;
  5. The external systems your system interfaces with;
  6. the project stakeholders (business, privacy, security…)

Don’t be afraid to ask questions. A lot of information will be in the minds of the team members you work with. This information will help you prioritize and drive your future decisions. When you have settled in the team and have a good sense of the context you are operating in, you can start discussing the changes that are most urgent.

Conclusion

When introducing change  try to not be that bull in the china shop. Instead take some time to really get to know the history of a project, the team members who work on it and the context that you operate in. If you are able to identify some quick wins you can propose some of them to the team. Use the information that you gathered to prioritize and drive any major changes that you deem necessary and propose them at an appropriate moment. Consider the velocity at which your team can adopt these changes to prevent piling on to many changes at the same time.

PlantUML Pleasantness: Layout Elements With Hidden Lines

In a previous post we learned how to use a together block to keep elements together. We can also layout elements in a different way: using hidden lines. We define our elements and by using the keyword [hidden] in our line definition the elements are layout as if there was a line, but we don’t see it. This gives us great flexibility on how we layout our elements.

Continue reading

Implementing architectural fitness functions using Gradle, JUnit and code-assert

Architectural fitness functions

Inspired by Neal Ford’s presentation at our Change is the Only constant event I started experimenting with architectural fitness functions. An architectural fitness function provides an objective integrity assessment of some architectural characteristic(s).

If you want to take a deeper dive into evolutionary architectures including fitness functions take look at Neals book: Building Evolutionary Architectures: Support Constant Change.

Neal’s slides contained an example of verifying package dependencies from a Unit Test using JDepend.

 

Verifying code modularity

In this blog post we’ll elaborate on that approach and create a Unit Test that verifies that our code complies to the chosen packaging strategies using an alternative to JDepend named code-assert.

Continue reading