Microservice architecture diagram composition

A low code approach to composing microservice architecture diagrams from per service context diagrams.

Background

On a recent assignment I was one of multiple new engineers joining a start-up transitioning into a scale-up. The past two years had been spent rapidly developing to prove the concept, and while at the outset some diagrams were drawn up, at the time I joined these were outdated, lacking or even misleading. To help get ourselves and other new developers up to speed, Andrew Morgan and I set out to create architecture diagrams for the system as a whole.

Approach

In typical fashion these days, the system consists of microservices communicating with one-another in a mix of direct HTTP invocation and by publishing and subscribing to Kafka topics. In the absence of tracing information or tools to deduce the diagrams, we settled on a simple approach:

  1. Create context diagrams per service
    Using PlantUML we created diagrams for each service, in the root of the project repository. The diagrams contain the topics published/subscribed to, and direct HTTP invocations going out from the service, as well as any data stores and interactions with external services.
  2. Compose overall diagrams
    Using the GitLab API we gather all projects with a particular tag, and for those projects extract the PlantUML diagrams from the repository. These are then combined into a single overall diagram by stripping out the @start/enduml tags and simply appending them one after the other. By using distinct tags we’re able to create diagrams for services in active development, in production or zoom in on a particular aspect such as data processing. New projects are automatically picked up, and warnings are sent out when diagrams are missing.
  3. Expose diagrams as images
    A nightly pipeline creates images for each service individually, as well as a number of composite diagram images. The individual service diagrams are visualized in the GitLab repository README of each project, giving passing developers a quick glance of the context of a single service. The composed service diagrams are visualized in the README of the composition project, for easy reference elsewhere.

Compose individual diagrams

Implementation

The code consists of a Gradle task, a Gradle build file and a GitLab CI file. We place the task in the buildSrc directory, so we can easily use it in our project’s build.gradle file.

Gradle Task

buildSrc build.gradle

Project build.gradle

.gitlab-ci.yml

Strengths

We chose this approach as a simple way to document the overall architecture, in a landscape that’s still undergoing changes and needs to be kept up to date. Having the individual diagrams close to the code ensures the individual diagrams are easily maintained, while the overall diagrams always reflect the latest changes. These qualities are not as easily achieved when the diagram is further removed from the code. The low-tech approach allows anyone to quickly contribute diagrams, with minimal tooling.

Weaknesses

As the overall diagrams are composed from individual diagrams, a common set of guidelines should be enforced for a consistent overall view. Inconsistent use of arrow directionality for instance, can have big consequences for the overall diagram layout. Also, layout of the overall diagram is not stable with regards to previous versions in light of recent changes. Introduction of a new inter-service-dependency for example, can rearrange large parts of the diagram, which can confuse those familiar with previous versions.

Future extensions

GitLab PlantUML integration

Above we use separate files for the diagrams for simplicity, and render the diagrams using the PlantUML jar. However, GitLab can integrate with PlantUML directly to visualize diagrams present in any AsciiDoc and Markdown documents in a repository. This would allow one to make the diagrams part of the README for instance, with only slight changes to the diagram composition.

Other platforms

While the code above uses the GitLab API the same approach can just as easily be applied to different platforms such as GitHub.

Alternative diagrams

As briefly mentioned above, there are alternatives to generate overall system architecture diagrams. One such approach is Structurizer, which inspects the code to generate diagrams at multiple levels. Another, more deductive approach is to use the operational platform and available tracing information to compose diagrams based on runtime service dependencies. These and more are explored to extend and supplement the current approach.

JSHeroes 2018 Conference Report

In this report Sjoerd Valk (@Sjoerdus) and Martijn van der Wijst (@vanderwise) try to sum up their experiences in Cluj-Napoca, Romania. They stayed in Cluj during the 17th and 21st of April to attend the JSHeroes conference, a frontend conference all about JavaScript. Presented by heroes of the JavaScript open source community. Although most of the speakers admitted they weren’t really heroes, the very last speaker gave his talk in a hero-esque costume.

This year marks only the second edition of JSHeroes, but it’s organized really professional. Speakers are from all over the world, and the location is a chique hotel. A few kilometers out of the city center. Its grandeur reminds us a bit of the Grand Budapest Hotel. The name too by the way: the Grand Hotel Italia.
On Wednesday we followed an Angular Masterclass. Thursday and Friday are the JSHeroes conference days. The talks are just 30 minutes each. A quick pace and quite diverse topics. The popular frameworks (React, Angular and Vue) were covered much, but also webfonts, Codesandbox, and V8, the engine behind JavaScript.

Continue reading

Starting as a new technical lead – Shaping a Team

Congratulations! Someone has made the wise decision to hire you as the new technical lead. It is an exciting time. You start in a new environment, will be working with a new team and maybe even have to learn new technologies along the way. This can be quite challenging. In the first article I wrote about introducing change. In this second article I want to share my personal views on shaping great teams.

As team lead, there are certain values that are just part of me and they way I think about leadership. Fortunately, in JDriven I found a company that matches my values step by step. These values are #commit, #develop and #share. In this article I want to go into some of the values that I consider important to create a great team. Continue reading

Slim modular Java 9 runtime Docker image with Alpine Linux

With the release of Java 9, and the introduction of Project Jigsaw (the Java Platform Module System), we no longer have the need for a full-blown JRE to run our Java applications. It is now possible to construct a stripped-down Java Runtime, containing the minimum set of required modules. This allows us to create slim Docker containers without excess baggage.

The source code belonging to this blog post can be found at: https://github.com/rlippolis/java9-runtime-image

Continue reading

Starting as a new technical lead – Introducing change

Congratulations! Someone has made the wise decision to hire you as the new technical lead. It is an exciting time. You start in a new environment, will be working with a new team and maybe even have to learn new technologies along the way. This can be quite challenging. In this two-part article I want to share my personal views regarding Introducing change and shaping teams as a technical lead.

Introducing change

When starting in this new environment you probably bring lots of energy and want to leverage your experience to change things for the better. In my opinion introducing changes in a new environment requires some consideration.

Judging

As technical lead you are (hopefully) up to date with the latest principles and technology trends. As technical lead your life purpose is to “build the thing right” because “building the right thing” is someone else’s concern ;-) You quickly determine that the project you inherited has some some design/architectural issues that should be addressed promptly. So the next day you climb on a chair and give a passionate speech about how things need to change. Wait right there! Even if you are right, you need to consider some things.

First remember that you are judging the work of the current development team. They probably put their heart and soul into it. Secondly there are often influences from outside the team that drive technical decisions. The most obvious are deadline but there are others. Don’t be judgemental about the current state of development, because it  can quickly create a gap between you and the rest of the team.

Change velocity

Changing something requires some time for the team to adjust. This means you can’t pile change on change, even if they are beneficial. If you make to many changes you effectively try to change the team’s entire way of working.

Consider the change velocity of the team. This doesn’t have anything to do with experience. Sometimes junior team members find it easy to adopt new ways of working while the seniors are reluctant to change their ways. Getting this right requires at least experience and some feeling.

Quick wins

So am I telling you that you can’t propose any changes at all? Not really. After all you were hired as lead so you get some credits from the get go. You can introduce some minor changes to the product or processes. Often some minor changes can help the team work more efficiently. Maybe eliminate some small repetitive tasks for example. Proposing some good changes can really help establish your position as lead. Also people are more receptive to minor changes then they are to big changes. Just be aware of how many of these changes the team can adopt.

Bigger changes

Before you start introducing big changes, I think it is important to take some time to reflect on past and present. During this time quickly gather information. Try and get to know about:

  1. the team;
  2. the system’s architecture;
  3. the technologies used;
  4. the history and past architectural/design decisions;
  5. The external systems your system interfaces with;
  6. the project stakeholders (business, privacy, security…)

Don’t be afraid to ask questions. A lot of information will be in the minds of the team members you work with. This information will help you prioritize and drive your future decisions. When you have settled in the team and have a good sense of the context you are operating in, you can start discussing the changes that are most urgent.

Conclusion

When introducing change  try to not be that bull in the china shop. Instead take some time to really get to know the history of a project, the team members who work on it and the context that you operate in. If you are able to identify some quick wins you can propose some of them to the team. Use the information that you gathered to prioritize and drive any major changes that you deem necessary and propose them at an appropriate moment. Consider the velocity at which your team can adopt these changes to prevent piling on to many changes at the same time.

Awesome Asciidoctor: Use Diagram Block Macro To Include Diagrams

With the Asciidoctor diagram extension we can include diagrams that are written in plain text. For example PlantUML or Ditaa diagrams. The extension offers a block processor where we include the diagram definitions in our Asciidoctor document. But there is also a block macro processor. With the block macro processor we can refer to an external file. The file is processed and the resulting image is in our output document.

Continue reading

Groovy Goodness: Download Grab Dependencies In IntelliJ IDEA

In our Groovy scripts we can use the @Grab annotation. With this annotation we define dependencies for our script and they will be automatically downloaded and added to the class path when we run our script. When we use IntelliJ IDEA we can use a nice intention that is part of the IntelliJ IDEA Groovy support to download the dependencies from our IDE editor. IDEA downloads the dependencies and add it to the project module dependencies. This is useful, because this will also adds code completion for the classes in the dependency to our editor.

Continue reading

Serverless Java with AWS Lambda: Introduction

Just as we are over the crest of the microservice hype and can finally see how this architectural tool might (or might not) solve our problems the next hype is already here: serverless programming! In this first blog post I’m going to explain what serverless is, what it isn’t, and how it can change the way we create software. In the next posts I’m going to show a few simple examples using a well known ‘serverless’ platform: AWS Lambda.

Originally posted here.

Continue reading