May 2018 - JDriven

A low code approach to composing microservice architecture diagrams from per service context diagrams.

Background

On a recent assignment I was one of multiple new engineers joining a start-up transitioning into a scale-up. The past two years had been spent rapidly developing to prove the concept, and while at the outset some diagrams were drawn up, at the time I joined these were outdated, lacking or even misleading. To help get ourselves and other new developers up to speed, Andrew Morgan and I set out to create architecture diagrams for the system as a whole.

Approach

In typical fashion these days, the system consists of microservices communicating with one-another in a mix of direct HTTP invocation and by publishing and subscribing to Kafka topics. In the absence of tracing information or tools to deduce the diagrams, we settled on a simple approach:

Create context diagrams per service
Using PlantUML we created diagrams for each service, in the root of the project repository. The diagrams contain the topics published/subscribed to, and direct HTTP invocations going out from the service, as well as any data stores and interactions with external services.
Compose overall diagrams
Using the GitLab API we gather all projects with a particular tag, and for those projects extract the PlantUML diagrams from the repository. These are then combined into a single overall diagram by stripping out the @start/enduml tags and simply appending them one after the other. By using distinct tags we’re able to create diagrams for services in active development, in production or zoom in on a particular aspect such as data processing. New projects are automatically picked up, and warnings are sent out when diagrams are missing.
Expose diagrams as images
A nightly pipeline creates images for each service individually, as well as a number of composite diagram images. The individual service diagrams are visualized in the GitLab repository README of each project, giving passing developers a quick glance of the context of a single service. The composed service diagrams are visualized in the README of the composition project, for easy reference elsewhere.

Implementation

The code consists of a Gradle task, a Gradle build file and a GitLab CI file. We place the task in the buildSrc directory, so we can easily use it in our project’s build.gradle file.

Gradle Task


// File: buildSrc/src/main/groovy/com/jdriven/blog/RenderDiagrams.groovy
package com.jdriven.blog

import org.gradle.api.DefaultTask;
import org.gradle.api.tasks.*;
import java.nio.charset.Charset
import net.sourceforge.plantuml.*
import org.gitlab4j.api.*
import org.gitlab4j.api.models.*
import static groovy.io.FileType.*

class RenderDiagrams extends DefaultTask {

    @OutputDirectory
    File destinationDir = new File(project.buildDir, 'diagrams')

    RenderDiagrams() {
        group = 'Documentation'
        description = 'Download architecture diagrams and render as SVG'
        // Always run task.
        outputs.upToDateWhen { false }
    }

    @TaskAction
    void render() {
        downloadDiagrams()
        composeArchitectureDiagram()
        renderDiagrams()
    }

    private void downloadDiagrams() {
        // Token should have access to read all target projects
        final token = System.getProperty('gitlabtoken')
        if (!token) {
            throw new RuntimeException("Please provide a 'gitlabtoken' property")
        }

        final gitLabApi = new GitLabApi("https://gitlab.acme.org/", token)

        final missingArchitectureDiagrams = [] as Set
        getProjects(gitLabApi).each { project ->
                try {
                    logger.info "Downloading ${project.namespace.name}/${project.name} architecture.puml"
                    RepositoryFile file = gitLabApi.getRepositoryFileApi().getFile("architecture.puml", project.id, "master");
                    new File(destinationDir, "${project.name}.puml").text = new String(file.content.decodeBase64(), Charset.forName("UTF-8"))
                } catch (GitLabApiException e) {
                    missingArchitectureDiagrams << project.name
                }
            }

        // Simply print missing diagrams in this reduced example; Normally notify developers
        if (!missingArchitectureDiagrams.isEmpty()){
            logger.warn "MISSING ARCHITECTURE DIAGRAMS:"
            missingArchitectureDiagrams.toSorted().each { name -> println name }
        }
    }

    private getProjects(final GitLabApi gitLabApi) {
        Pager<Project> pageOfProjects = gitLabApi.getProjectApi().getProjects(500)
        List<Project> projects = []
        projects.addAll(pageOfProjects.first())
        while (pageOfProjects.hasNext()) {
            projects.addAll(pageOfProjects.next())
        }
        // Tag can refer to either a tag on target projects or a complete GitLab namespace
        final tag = System.getProperty('tag')?: "service"
        return projects.findAll { project -> tag in project.tagList || project.namespace.name == tag }
    }

    private void composeArchitectureDiagram() {
        final prefix = """\
        @startuml
        scale max 4096 width
        skinparam queue {
            BackgroundColor AliceBlue
            BorderColor blue
        }
        skinparam database {
            BackgroundColor LightPink
            BorderColor Red
        }
        """.stripIndent()
        final composed = new StringBuilder(prefix)

        final projectDiagramFile = { name ->
            name != 'composed-architecture-diagram.puml' &amp;&amp; name.endsWith('.puml')
        }
        destinationDir.eachFileMatch(FILES, projectDiagramFile) { diagramFile ->
            composed << "\n'${diagramFile.name}\n"
            composed << diagramFile.text.replaceAll("@startuml", "'startuml").replaceAll("@enduml", "'enduml")
        }
        composed << "\n@enduml"

        new File(destinationDir, "composed-architecture-diagram.puml").text = composed.toString()
    }

    private void renderDiagrams() {
        destinationDir.eachFileMatch(FILES, { name -> name.endsWith('.puml') }) { diagramFile ->
            final reader = new SourceStringReader(diagramFile.text)
            final imageFile = new File(destinationDir, diagramFile.name.replaceAll(".puml", ".svg"))
            reader.generateImage(new FileOutputStream(imageFile), new FileFormatOption(FileFormat.SVG))
        }
    }
}

100

101

102

103

104

105

// File: buildSrc/src/main/groovy/com/jdriven/blog/RenderDiagrams.groovy

package com.jdriven.blog

import org.gradle.api.DefaultTask;

import org.gradle.api.tasks.*;

import java.nio.charset.Charset

import net.sourceforge.plantuml.*

import org.gitlab4j.api.*

import org.gitlab4j.api.models.*

import static groovy.io.FileType.*

class RenderDiagrams extends DefaultTask {

@OutputDirectory

File destinationDir = new File(project.buildDir, 'diagrams')

RenderDiagrams() {

group = 'Documentation'

description = 'Download architecture diagrams and render as SVG'

// Always run task.

outputs.upToDateWhen { false }

}

@TaskAction

void render() {

downloadDiagrams()

composeArchitectureDiagram()

renderDiagrams()

}

private void downloadDiagrams() {

// Token should have access to read all target projects

final token = System.getProperty('gitlabtoken')

if (!token) {

throw new RuntimeException("Please provide a 'gitlabtoken' property")

}

final gitLabApi = new GitLabApi("https://gitlab.acme.org/", token)

final missingArchitectureDiagrams = [] as Set

getProjects(gitLabApi).each { project ->

try {

logger.info "Downloading ${project.namespace.name}/${project.name} architecture.puml"

RepositoryFile file = gitLabApi.getRepositoryFileApi().getFile("architecture.puml", project.id, "master");

new File(destinationDir, "${project.name}.puml").text = new String(file.content.decodeBase64(), Charset.forName("UTF-8"))

} catch (GitLabApiException e) {

missingArchitectureDiagrams << project.name

}

// Simply print missing diagrams in this reduced example; Normally notify developers

if (!missingArchitectureDiagrams.isEmpty()){

logger.warn "MISSING ARCHITECTURE DIAGRAMS:"

missingArchitectureDiagrams.toSorted().each { name -> println name }

}

private getProjects(final GitLabApi gitLabApi) {

Pager<Project> pageOfProjects = gitLabApi.getProjectApi().getProjects(500)

List<Project> projects = []

projects.addAll(pageOfProjects.first())

while (pageOfProjects.hasNext()) {

projects.addAll(pageOfProjects.next())

}

// Tag can refer to either a tag on target projects or a complete GitLab namespace

final tag = System.getProperty('tag')?: "service"

return projects.findAll { project -> tag in project.tagList || project.namespace.name == tag }

}

private void composeArchitectureDiagram() {

final prefix = """\

@startuml

scale max 4096 width

skinparam queue {

BackgroundColor AliceBlue

BorderColor blue

}

skinparam database {

BackgroundColor LightPink

BorderColor Red

}

""".stripIndent()

final composed = new StringBuilder(prefix)

final projectDiagramFile = { name ->

name != 'composed-architecture-diagram.puml' && name.endsWith('.puml')

}

destinationDir.eachFileMatch(FILES, projectDiagramFile) { diagramFile ->

composed << "\n'${diagramFile.name}\n"

composed << diagramFile.text.replaceAll("@startuml", "'startuml").replaceAll("@enduml", "'enduml")

}

composed << "\n@enduml"

new File(destinationDir, "composed-architecture-diagram.puml").text = composed.toString()

}

private void renderDiagrams() {

destinationDir.eachFileMatch(FILES, { name -> name.endsWith('.puml') }) { diagramFile ->

final reader = new SourceStringReader(diagramFile.text)

final imageFile = new File(destinationDir, diagramFile.name.replaceAll(".puml", ".svg"))

reader.generateImage(new FileOutputStream(imageFile), new FileFormatOption(FileFormat.SVG))

}

buildSrc build.gradle


// File: buildSrc/build.gradle
apply plugin: 'groovy'

repositories {
    mavenCentral()
}

dependencies {
    compile gradleApi()
    compile 'org.gitlab4j:gitlab4j-api:4.8.9'
    compile 'net.sourceforge.plantuml:plantuml:1.2018.3'
}

// File: buildSrc/build.gradle

apply plugin: 'groovy'

repositories {

mavenCentral()

}

dependencies {

compile gradleApi()

compile 'org.gitlab4j:gitlab4j-api:4.8.9'

compile 'net.sourceforge.plantuml:plantuml:1.2018.3'

}

Project build.gradle


// File: build.gradle
apply plugin: 'java'

task renderDiagrams(type: com.jdriven.blog.RenderDiagrams)

// File: build.gradle

apply plugin: 'java'

task renderDiagrams(type: com.jdriven.blog.RenderDiagrams)

.gitlab-ci.yml


stages:
  - create diagrams

## Prevent duplication via Anchors as per: https://docs.gitlab.com/ce/ci/yaml/README.html#anchors
.job_template: &amp;job_definition
  stage: create diagrams
  variables:
    TAG: $CI_JOB_NAME
  script:
    - apt-get update -y
    - apt-get install graphviz -y
    - ./gradlew clean renderDiagrams -Dgitlabtoken=${gitlabtoken} -Dtag=$TAG --stacktrace
  artifacts:
    name: "$CI_JOB_NAME $CI_JOB_ID"
    when: always
    expire_in: 1 week
    paths:
      - build

# All services; used to generate both an overall diagram and individual service diagrams
service:
  <<: *job_definition

# Diagrams that focus on part of the service; job name should match GitLab namespace or GitLab project tags
backend:
  <<: *job_definition
datascience:
  <<: *job_definition
production:
  <<: *job_definition

stages:

- create diagrams

## Prevent duplication via Anchors as per: https://docs.gitlab.com/ce/ci/yaml/README.html#anchors

.job_template: &job_definition

stage: create diagrams

variables:

TAG: $CI_JOB_NAME

script:

- apt-get update -y

- apt-get install graphviz -y

- ./gradlew clean renderDiagrams -Dgitlabtoken=${gitlabtoken} -Dtag=$TAG --stacktrace

artifacts:

name: "$CI_JOB_NAME $CI_JOB_ID"

when: always

expire_in: 1 week

paths:

- build

# All services; used to generate both an overall diagram and individual service diagrams

service:

<<: *job_definition

# Diagrams that focus on part of the service; job name should match GitLab namespace or GitLab project tags

backend:

<<: *job_definition

datascience:

<<: *job_definition

production:

<<: *job_definition

Strengths

We chose this approach as a simple way to document the overall architecture, in a landscape that’s still undergoing changes and needs to be kept up to date. Having the individual diagrams close to the code ensures the individual diagrams are easily maintained, while the overall diagrams always reflect the latest changes. These qualities are not as easily achieved when the diagram is further removed from the code. The low-tech approach allows anyone to quickly contribute diagrams, with minimal tooling.

Weaknesses

As the overall diagrams are composed from individual diagrams, a common set of guidelines should be enforced for a consistent overall view. Inconsistent use of arrow directionality for instance, can have big consequences for the overall diagram layout. Also, layout of the overall diagram is not stable with regards to previous versions in light of recent changes. Introduction of a new inter-service-dependency for example, can rearrange large parts of the diagram, which can confuse those familiar with previous versions.

Future extensions

GitLab PlantUML integration

Above we use separate files for the diagrams for simplicity, and render the diagrams using the PlantUML jar. However, GitLab can integrate with PlantUML directly to visualize diagrams present in any AsciiDoc and Markdown documents in a repository. This would allow one to make the diagrams part of the README for instance, with only slight changes to the diagram composition.

Other platforms

While the code above uses the GitLab API the same approach can just as easily be applied to different platforms such as GitHub.

Alternative diagrams

As briefly mentioned above, there are alternatives to generate overall system architecture diagrams. One such approach is Structurizer, which inspects the code to generate diagrams at multiple levels. Another, more deductive approach is to use the operational platform and available tracing information to compose diagrams based on runtime service dependencies. These and more are explored to extend and supplement the current approach.

Monthly Archives: May 2018

Gradle Goodness: Command Line Options For Custom Tasks

Microservice architecture diagram composition

Background

Approach

Implementation

Gradle Task

buildSrc build.gradle

Project build.gradle

.gitlab-ci.yml

Strengths

Weaknesses

Future extensions

GitLab PlantUML integration

Other platforms

Alternative diagrams

JSHeroes 2018 Conference Report