Follow through GitLab deployments with Slack
Our team has a (not so) slight tendency to not immediately follow through with our deployments to production. We’ll create and review our changes, merge and deploy to staging, and dilligently test the changes there. And then… nothing happens.
It could be that something else needs our immediate attention, or someone else wants to confirm an issue is fixed; Or we might want to deploy at a different point in time as to not disrupt an ongoing process by a service restart. Any which way the result is the same: changes accumulate in staging, and with that the risk involved with the next production deployment.
To nudge ourselves to deploy to production more often we created a Slack App that gives us a daily report of such pending deployments. In this post I’ll showcase the code we use, and how to set up something similar yourself.
Background & Goal
We use GitLab CI for our build pipelines, which include deployment to various environments. By tracking our environments in GitLab we can easily see per service what was the last deployment to each environment, and what changes have since been made to the default branch. The same environments and deployments are also available through the GitLab API, which we’ll use to extract changes that have not yet gone to production.
Since our team uses Slack, we thought it easiest to remind ourselves every morning who made what changes that are still pending for deployment. That way it’s immediately obvious who’s in charge, and any impediments can be discussed in the morning stand-up meeting.
Project structure
The project is set up as a very basic Spring Boot application, which we run in GitLab CI through mvnw spring-boot:run.
This has the advantage that it can easily be run on a schedule rather than as a service, since we only need the reports periodically.
Aside from Spring Boot, the only dependencies we need are on org.gitlab4j:gitlab4j-api & com.slack.api:slack-api-client.
Configuration is handled through a mix of environment variables declared in .gitlab-ci.yml, variables declared per scheduled pipeline report, as well as CI / CD Variables for secret tokens.
View project pom.xml
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.4.4</version>
<relativePath /> <!-- lookup parent from repository -->
</parent>
<groupId>com.company.team</groupId>
<artifactId>pending-deployments</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>pending-deployments</name>
<description>Pending deployments</description>
<properties>
<java.version>16</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.gitlab4j</groupId>
<artifactId>gitlab4j-api</artifactId>
<version>4.15.7</version>
</dependency>
<dependency>
<groupId>com.slack.api</groupId>
<artifactId>slack-api-client</artifactId>
<version>1.7.1</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>View project .gitlab-ci.yml
image: openjdk:16
report:
only:
- schedules
variables:
gitlabhost: https://your.gitlab.host
targetenvironment: production
script:
- './mvnw $MAVEN_CLI_OPTS spring-boot:run'View project CI / CD Variables
View scheduled pipeline report
View Java ApplicationRunner outline
@Component
class ReportPendingDeployments implements ApplicationRunner {
private static final Logger log = LoggerFactory.getLogger(ReportPendingDeployments.class);
private final org.springframework.core.env.Environment systemEnv;
public ReportPendingDeployments(org.springframework.core.env.Environment systemEnv) {
this.systemEnv = systemEnv;
}
@Override
public void run(ApplicationArguments args) throws Exception {
String environmentName = systemEnv.getProperty("targetenvironment");
List<PendingDeployment> pendingDeployments = extractPendingDeploymentsFromGitLab(environmentName);
reportToSlack(environmentName, pendingDeployments);
}
[...]
}GitLab API
We use the excellent gitlab4j-api to extract projects, environments, deployments, and through those any pending changes.
Gitlab4j-api closely follows the GitLab V4 API resources, so it helps to look at documentation for both when developing.
We authenticate using a personal access token using the read_api scope, which we provide as environment variable through CI / CD Variables as shown above.
As our organization has split teams by programming language, we look at all recently updated projects using an argument programming language. For those projects we evaluate if there are any pending deployments, and if there are we report those to Slack.
private List<PendingDeployment> extractPendingDeploymentsFromGitLab(String environmentName)
throws GitLabApiException {
// Extract GitLab arguments
String personalAccessToken = systemEnv.getProperty("accesstoken");
if (personalAccessToken == null || personalAccessToken.isBlank()) {
throw new IllegalStateException("GitLab accesstoken required");
}
String hostUrl = systemEnv.getProperty("gitlabhost");
String programmingLanguage = systemEnv.getProperty("language");
try (GitLabApi api = new GitLabApi(hostUrl, personalAccessToken)) {
// Retrieve actively maintained projects in team language
return api.getProjectApi()
.getProjectsStream(new ProjectFilter()
.withArchived(false)
.withSimple(true)
.withProgrammingLanguage(programmingLanguage)
.withOrderBy(ProjectOrderBy.NAME)
.withSortOder(SortOrder.ASC))
.filter(project -> project.getLastActivityAt()
.after(Date.from(Instant.now().minus(Duration.ofDays(90)))))
.flatMap(project -> {
try {
// Evaluate project & return optional PendingDeployment
return evaluateProject(api, environmentName, project).stream();
} catch (GitLabApiException e) {
log.warn("Failed to evaluate {}", project.getPathWithNamespace(), e);
return Stream.empty();
}
})
.collect(Collectors.toList());
}
}Our evaluate function takes an argument project and environment, and using the GitLabApi determines whether there are any pending deployments for the project. If there are pending deployments, we call another function to create and return a Slack formatted markdown message.
private static Optional<PendingDeployment> evaluateProject(
GitLabApi api,
String environment,
Project project)
throws GitLabApiException {
// Get target project environment
Optional<Environment> envSimple = api.getEnvironmentsApi()
.getEnvironmentsStream(project.getId())
.filter(env -> env.getName().equalsIgnoreCase(environment))
.findFirst();
if (envSimple.isEmpty()) {
log.info("No environment {} found for {}", environment, project.getPathWithNamespace());
return Optional.empty();
}
// Get latest deployed sha
Deployment deployment = api.getEnvironmentsApi()
.getEnvironment(project.getId(), envSimple.get().getId())
.getLastDeployment();
if (deployment == null) {
log.info("No deployment found for environment {} for {}", environment, project.getPathWithNamespace());
return Optional.empty();
}
// Compare with default branch
String lastDeployedSha = deployment.getSha();
CompareResults compareResults = api.getRepositoryApi()
.compare(project.getId(),
lastDeployedSha,
project.getDefaultBranch());
if (compareResults.getCommits().size() == 0) {
log.info("No changes found comparing {} with {} for {}", environment,
project.getDefaultBranch(), project.getPathWithNamespace());
return Optional.empty();
}
// Report undeployed commits for project
var pendingDeployment = composeMessageForPendingDeployment(api, project, lastDeployedSha, compareResults);
log.info("{}", pendingDeployment);
return Optional.of(pendingDeployment);
}Our markdown formatting function extracts the pending commits, and for those commits the authors involved. URLs are composed to show the differences compared with the main branch, as well as the latest commit pipeline. Finally all input is combined to show a single succinct markdown formatted message per project.
private static PendingDeployment composeMessageForPendingDeployment(
GitLabApi api,
Project project,
String lastDeployedSha,
CompareResults compareResults)
throws GitLabApiException {
// Extract authors for pending changes
var authors = compareResults.getCommits().stream()
.map(Commit::getAuthorName)
.distinct()
.sorted()
.collect(Collectors.joining(" & "));
// Compose comparison view url
String compareUrl = "%s/-/compare/%s...%s".formatted(
project.getWebUrl(),
lastDeployedSha,
project.getDefaultBranch());
// Get last pipeline for commit
Pipeline lastPipeline = api.getCommitsApi()
.getCommit(project.getId(), compareResults.getCommit().getId())
.getLastPipeline();
// Get web url for last pipeline
String pipelineUrl = api.getPipelineApi()
.getPipeline(project.getId(), lastPipeline.getId())
.getWebUrl();
// Compose pending deployment
return new PendingDeployment(
project.getPathWithNamespace(),
compareResults.getCommits().size(),
authors,
compareUrl,
pipelineUrl);
}
record PendingDeployment(
String pathWithNamespace,
int numberOfCommits,
String authors,
String compareUrl,
String pipelineUrl) {
}All together we end up with a list of markdown messages each reporting on a project with pending deployments to our target environment.
Slack API
Once we’ve identified our pending deployments, we want to report these to Slack for visibility, such that the team can evaluate and deploy outstanding changes. To communicate with Slack we use the standard slack-api-client for Java. Given a token, a channel and a list of markdown messages we wish to report these to a single thread per day, to keep noise to a minimum.
private void reportToSlack(
String environmentName,
List<PendingDeployment> pendingDeployments)
throws Exception {
// Extract Slack arguments
String channel = systemEnv.getProperty("channel");
String slackToken = systemEnv.getProperty("slacktoken");
if (channel == null || slackToken == null || slackToken.isBlank() || pendingDeployments.isEmpty()) {
log.warn("Channel and slacktoken required");
return;
}
try (Slack slack = Slack.getInstance()) {
MethodsClient methods = slack.methods(slackToken);
// Start a thread
ChatPostMessageResponse response = methods
.chatPostMessage(req -> req
.channel(channel)
.text("%d projects have changes not yet deployed to %s"
.formatted(pendingDeployments.size(), environmentName)));
if (!response.isOk()) {
// e.g., "invalid_auth", "channel_not_found"
String errorCode = response.getError();
log.error("Failed to start thread: {}", errorCode);
return;
}
// Post markdown messages in thread
String threadTs = response.getMessage().getTs();
for (PendingDeployment pendingDeployment : pendingDeployments) {
try {
ChatPostMessageResponse threadResponse = methods
.chatPostMessage(req -> req
.threadTs(threadTs)
.channel(channel)
.text("%s has <%s|%d commits> by %s not yet <%s|deployed to %s>"
.formatted(
pendingDeployment.pathWithNamespace(),
pendingDeployment.compareUrl(),
pendingDeployment.numberOfCommits(),
pendingDeployment.authors(),
pendingDeployment.pipelineUrl(),
environmentName)));
if (!threadResponse.isOk()) {
log.warn("Failed to post message to thread: {}", threadResponse.getError());
}
} catch (IOException | SlackApiException e) {
log.warn("Failed to post message to thread: {}", e.getMessage());
}
}
}
}That concludes the code we need to both extract pending deployments from GitLab, as well as report those messages to Slack.
Slack App
We need to create a Slack App before we’re able to communicate with the Slack API to write chat messages.
Give your App a name; we chose "Pending Deployments", and add the App to your Slack Workspace.
Next you need to configure the Bot Token Scopes for your App to include the chat:write scope.
Copy the Bot User OAuth Token for your App, which starts with xoxb-, and store it as slacktoken in your GitLab CI / CD Variables.
Be sure to add the bot to any channel where you want to post messages, through for instance: /add @pending_deployments.
Pipeline Schedule
Finally, we can schedule multiple pipelines to each report projects using a given language to team Slack channels.
We’ve chosen daily early morning reminders, but choose whatever works best for your team and flow.
Notice how we only define variables for language and channel here, as they vary from schedule to schedule.
Conclusion
That’s all you need to extract projects with pending deployments and report those to Slack. We’ve found these gentle reminders nudge us daily to deploy our changes to production as needed.
