A low code approach to composing microservice architecture diagrams from per service context diagrams.
Background
On a recent assignment I was one of multiple new engineers joining a start-up transitioning into a scale-up. The past two years had been spent rapidly developing to prove the concept, and while at the outset some diagrams were drawn up, at the time I joined these were outdated, lacking or even misleading. To help get ourselves and other new developers up to speed, Andrew Morgan and I set out to create architecture diagrams for the system as a whole.
Approach
In typical fashion these days, the system consists of microservices communicating with one-another in a mix of direct HTTP invocation and by publishing and subscribing to Kafka topics. In the absence of tracing information or tools to deduce the diagrams, we settled on a simple approach:
- Create context diagrams per service
Using PlantUML we created diagrams for each service, in the root of the project repository. The diagrams contain the topics published/subscribed to, and direct HTTP invocations going out from the service, as well as any data stores and interactions with external services. - Compose overall diagrams
Using the GitLab API we gather all projects with a particular tag, and for those projects extract the PlantUML diagrams from the repository. These are then combined into a single overall diagram by stripping out the@start/endumltags and simply appending them one after the other. By using distinct tags we’re able to create diagrams for services in active development, in production or zoom in on a particular aspect such as data processing. New projects are automatically picked up, and warnings are sent out when diagrams are missing. - Expose diagrams as images
A nightly pipeline creates images for each service individually, as well as a number of composite diagram images. The individual service diagrams are visualized in the GitLab repository README of each project, giving passing developers a quick glance of the context of a single service. The composed service diagrams are visualized in the README of the composition project, for easy reference elsewhere.
Implementation
The code consists of a Gradle task, a Gradle build file and a GitLab CI file. We place the task in the buildSrc directory, so we can easily use it in our project’s build.gradle file.
Gradle Task
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 |
// File: buildSrc/src/main/groovy/com/jdriven/blog/RenderDiagrams.groovy package com.jdriven.blog import org.gradle.api.DefaultTask; import org.gradle.api.tasks.*; import java.nio.charset.Charset import net.sourceforge.plantuml.* import org.gitlab4j.api.* import org.gitlab4j.api.models.* import static groovy.io.FileType.* class RenderDiagrams extends DefaultTask { @OutputDirectory File destinationDir = new File(project.buildDir, 'diagrams') RenderDiagrams() { group = 'Documentation' description = 'Download architecture diagrams and render as SVG' // Always run task. outputs.upToDateWhen { false } } @TaskAction void render() { downloadDiagrams() composeArchitectureDiagram() renderDiagrams() } private void downloadDiagrams() { // Token should have access to read all target projects final token = System.getProperty('gitlabtoken') if (!token) { throw new RuntimeException("Please provide a 'gitlabtoken' property") } final gitLabApi = new GitLabApi("https://gitlab.acme.org/", token) final missingArchitectureDiagrams = [] as Set getProjects(gitLabApi).each { project -> try { logger.info "Downloading ${project.namespace.name}/${project.name} architecture.puml" RepositoryFile file = gitLabApi.getRepositoryFileApi().getFile("architecture.puml", project.id, "master"); new File(destinationDir, "${project.name}.puml").text = new String(file.content.decodeBase64(), Charset.forName("UTF-8")) } catch (GitLabApiException e) { missingArchitectureDiagrams << project.name } } // Simply print missing diagrams in this reduced example; Normally notify developers if (!missingArchitectureDiagrams.isEmpty()){ logger.warn "MISSING ARCHITECTURE DIAGRAMS:" missingArchitectureDiagrams.toSorted().each { name -> println name } } } private getProjects(final GitLabApi gitLabApi) { Pager<Project> pageOfProjects = gitLabApi.getProjectApi().getProjects(500) List<Project> projects = [] projects.addAll(pageOfProjects.first()) while (pageOfProjects.hasNext()) { projects.addAll(pageOfProjects.next()) } // Tag can refer to either a tag on target projects or a complete GitLab namespace final tag = System.getProperty('tag')?: "service" return projects.findAll { project -> tag in project.tagList || project.namespace.name == tag } } private void composeArchitectureDiagram() { final prefix = """\ @startuml scale max 4096 width skinparam queue { BackgroundColor AliceBlue BorderColor blue } skinparam database { BackgroundColor LightPink BorderColor Red } """.stripIndent() final composed = new StringBuilder(prefix) final projectDiagramFile = { name -> name != 'composed-architecture-diagram.puml' && name.endsWith('.puml') } destinationDir.eachFileMatch(FILES, projectDiagramFile) { diagramFile -> composed << "\n'${diagramFile.name}\n" composed << diagramFile.text.replaceAll("@startuml", "'startuml").replaceAll("@enduml", "'enduml") } composed << "\n@enduml" new File(destinationDir, "composed-architecture-diagram.puml").text = composed.toString() } private void renderDiagrams() { destinationDir.eachFileMatch(FILES, { name -> name.endsWith('.puml') }) { diagramFile -> final reader = new SourceStringReader(diagramFile.text) final imageFile = new File(destinationDir, diagramFile.name.replaceAll(".puml", ".svg")) reader.generateImage(new FileOutputStream(imageFile), new FileFormatOption(FileFormat.SVG)) } } } |
buildSrc build.gradle
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
// File: buildSrc/build.gradle apply plugin: 'groovy' repositories { mavenCentral() } dependencies { compile gradleApi() compile 'org.gitlab4j:gitlab4j-api:4.8.9' compile 'net.sourceforge.plantuml:plantuml:1.2018.3' } |
Project build.gradle
|
1 2 3 4 5 |
// File: build.gradle apply plugin: 'java' task renderDiagrams(type: com.jdriven.blog.RenderDiagrams) |
.gitlab-ci.yml
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
stages: - create diagrams ## Prevent duplication via Anchors as per: https://docs.gitlab.com/ce/ci/yaml/README.html#anchors .job_template: &job_definition stage: create diagrams variables: TAG: $CI_JOB_NAME script: - apt-get update -y - apt-get install graphviz -y - ./gradlew clean renderDiagrams -Dgitlabtoken=${gitlabtoken} -Dtag=$TAG --stacktrace artifacts: name: "$CI_JOB_NAME $CI_JOB_ID" when: always expire_in: 1 week paths: - build # All services; used to generate both an overall diagram and individual service diagrams service: <<: *job_definition # Diagrams that focus on part of the service; job name should match GitLab namespace or GitLab project tags backend: <<: *job_definition datascience: <<: *job_definition production: <<: *job_definition |
Strengths
We chose this approach as a simple way to document the overall architecture, in a landscape that’s still undergoing changes and needs to be kept up to date. Having the individual diagrams close to the code ensures the individual diagrams are easily maintained, while the overall diagrams always reflect the latest changes. These qualities are not as easily achieved when the diagram is further removed from the code. The low-tech approach allows anyone to quickly contribute diagrams, with minimal tooling.
Weaknesses
As the overall diagrams are composed from individual diagrams, a common set of guidelines should be enforced for a consistent overall view. Inconsistent use of arrow directionality for instance, can have big consequences for the overall diagram layout. Also, layout of the overall diagram is not stable with regards to previous versions in light of recent changes. Introduction of a new inter-service-dependency for example, can rearrange large parts of the diagram, which can confuse those familiar with previous versions.
Future extensions
GitLab PlantUML integration
Above we use separate files for the diagrams for simplicity, and render the diagrams using the PlantUML jar. However, GitLab can integrate with PlantUML directly to visualize diagrams present in any AsciiDoc and Markdown documents in a repository. This would allow one to make the diagrams part of the README for instance, with only slight changes to the diagram composition.
Other platforms
While the code above uses the GitLab API the same approach can just as easily be applied to different platforms such as GitHub.
Alternative diagrams
As briefly mentioned above, there are alternatives to generate overall system architecture diagrams. One such approach is Structurizer, which inspects the code to generate diagrams at multiple levels. Another, more deductive approach is to use the operational platform and available tracing information to compose diagrams based on runtime service dependencies. These and more are explored to extend and supplement the current approach.