Running Dependabot on GitLab
As a consultant I find myself alternating between GitLab and GitHub about once a year, depending on the assignment. While I like GitLab a lot, there’s one thing I had sorely missed whenever I switch back from GitHub: Dependabot. Dependabot scans your project dependencies, and creates merge requests whenever updates are found. This provides you with an easy way to keep up to date on dependencies, and notifies you early if there are any incompatibilities.
Even though there are alternatives such as snyk.io and even GitLab’s own Dependency Scanning, those don’t always support enterprise or partner installations of GitLab, require GitLab Ultimate, or don’t support the full range of package managers that Dependabot supports.
Luckily though, there’s now a Dependabot for GitLab project. This project is based on the same Open Source Dependabot Core, so you can get the exact same automated dependency updates on both platforms.
In this blogpost I’ll walk you through how you can quickly roll out Dependabot on an existing GitLab installation, so you can start updating your dependencies automatically.
You might want to look into Running Renovate on GitLab.com instead, as Renovate is a feature rich alternative to Dependabot, with official support. |
Running modes
There are two ways of running Dependabot GitLab:
-
Standalone via scheduled GitLab pipelines
-
As a service deployed through Helm or Docker Compose.
The standalone version is the easiest to get started with, and what we will use in this blogpost. It is however quite limited, and requires the user to manage the scheduled pipelines. It’s a great way to explore whether your organization wants to adopt automated dependency updates.
The service installation is recommended, and with that you get webhook support to automatically add and remove repositories, and a web user interface to view currently configured dependency updates.
Dependabot Standalone installation
Standalone installation is easy enough; We import the dependabot-standalone project into our own GitLab instance or group, and configure the two required access tokens for both GitLab and GitHub.
The GitHub token might seem surprising at first; it is used to retrieve the release notes shown in the merge requests for supported dependencies.
You might need to override SETTINGS__GITLAB_URL
if you run on an alternative GitLab host.
A helpful addition is to also configure access to your private repositories, such that you also get notified for internal library updates. For Maven repositories this comes down to some additional environment variables:
SETTINGS__CREDENTIALS__MAVEN__REPO1__URL=maven_url
SETTINGS__CREDENTIALS__MAVEN__REPO1__USERNAME=maven_username
SETTINGS__CREDENTIALS__MAVEN__REPO1__PASSWORD=maven_password
That’s it in terms of installation for Dependabot Standalone; we’ll reuse this repository once it’s time to create scheduled pipelines.
Configure repository updates through .gitlab/dependabot.yml
Any repositories that wish to receive automated dependency update merge requests will need to be configured through a .gitlab/dependabot.yml
file,
which follows the same format as the official Dependabot documentation.
Some minor extensions are provided to automatically accept and merge requests.
When getting started with Dependabot you’ll likely want to try it out on a first few projects, with various dependency managers. For a sample Gradle project your configuration file might look something like this:
version: 2
updates:
- package-ecosystem: "gradle"
directory: "/"
schedule:
interval: "daily"
open-pull-requests-limit: 20
rebase-strategy: auto
auto-merge: true
You can configure a single repository for updates through multiple package-ecosystems, optionally setting the correct directory
argument when not located at the root.
Exclusions can be added to skip certain package updates, for fine tuning according to project specifics.
For details see the reference documentation.
Configuration in bulk
Once you’ve determined that you want to configure all repositories to use Dependabot, you will need to roll out the configuration files in bulk. To make that easier I’ve set up a small sample script which you can update to match your environment. The script assumes you have your SSH key setup to clone all repositories locally, superficially explores your project for a few known package managers, and creates a merge request per repository to add the configuration file.
View local script to create .gitlab/dependabot.yml merge requests for all repositories
#!/bin/bash
set -v
set -e
# Variables
API=https://your.gitlab.host/api/v4/
AFTER=2021-01-01T00:00.00Z
PROJECTS=projects.txt
# Gather up to 300 repositories
curl --header Private-Token:${SETTINGS__GITLAB_ACCESS_TOKEN} "${API}projects?simple=true&archived=false&order_by=name&sort=asc&last_activity_after=${AFTER}&per_page=100" | jq -r '.[] | "\(.path_with_namespace)"' > $PROJECTS
curl --header Private-Token:${SETTINGS__GITLAB_ACCESS_TOKEN} "${API}projects?simple=true&archived=false&order_by=name&sort=asc&last_activity_after=${AFTER}&per_page=100&page=2" | jq -r '.[] | "\(.path_with_namespace)"' >> $PROJECTS
curl --header Private-Token:${SETTINGS__GITLAB_ACCESS_TOKEN} "${API}projects?simple=true&archived=false&order_by=name&sort=asc&last_activity_after=${AFTER}&per_page=100&page=3" | jq -r '.[] | "\(.path_with_namespace)"' >> $PROJECTS
sort $PROJECTS
# Setup variables
folder=$(pwd)
targetfile=.gitlab/dependabot.yml
branch="configure-dependabot-gitlab"
gitpush="$folder/git-push.out"
truncate -s 0 "$gitpush"
# Loop over projects to create merge requests with .gitlab/dependabot.yml
while read -r p; do
cd "$folder";
echo -e "\n\n$p"
# Update/clone local repository
if [ -d "repos/$p" ]; then
cd "repos/$p"
git reset --hard
git checkout master || git checkout main
git pull -q
git branch -D "$branch" || true
else
cd "repos"
git clone "git@your.gitlab.host:$p.git" "$p"
cd "$p"
fi
# Start a new branch
git checkout -b "$branch"
# Shared header
mkdir -p "$(dirname $targetfile)"
echo 'version: 2
updates:' > $targetfile
# Maven
if [ -f pom.xml ]; then
echo ' - package-ecosystem: "maven"
directory: "/"
schedule:
interval: "daily"
open-pull-requests-limit: 20' >> $targetfile
fi
# Gradle
if [ -f build.gradle ]; then
echo ' - package-ecosystem: "gradle"
directory: "/"
schedule:
interval: "daily"
open-pull-requests-limit: 20' >> $targetfile
fi
# Docker
if [ -f Dockerfile ]; then
echo ' - package-ecosystem: "docker"
directory: "/"
schedule:
interval: "daily"' >> $targetfile
fi
# Nuget
if [ -f nuget.config ]; then
echo ' - package-ecosystem: "nuget"
directory: "/"
schedule:
interval: "daily"
open-pull-requests-limit: 20' >> $targetfile
fi
# Python
if [ -f setup.py ] || [ -f Pipfile ] || [ -f requirements.txt ]; then
echo ' - package-ecosystem: "pip"
directory: "/"
schedule:
interval: "daily"
open-pull-requests-limit: 20' >> $targetfile
fi
# NPM
if [ -f package.json ]; then
echo ' - package-ecosystem: "npm"
directory: "/"
schedule:
interval: "daily"
open-pull-requests-limit: 20' >> $targetfile
fi
# Only commit if this resulted in changes
if [[ ! $(git status --porcelain) ]]; then
continue;
fi
# Commit changes
git add $targetfile
git commit -m "Configure dependabot"
# Show differences compared to master
git --no-pager diff origin/master || git --no-pager diff origin/main
# Push branch
git push --force -u origin "$branch" \
-o merge_request.create \
-o merge_request.merge_when_pipeline_succeeds \
-o merge_request.remove_source_branch \
-o merge_request.label="dependabot" \
2>&1 | tee -a "$gitpush"
done < $PROJECTS
# Open all new pull requests in Chrome
grep 'merge_request' "$gitpush" | awk '{print $2}' | xargs google-chrome
Creating Scheduled pipelines
Once you’ve configured some, or all, repositories with a .gitlab/dependabot.yml
file, you will want to create scheduled pipelines that match the configuration files.
Creating scheduled pipelines is only needed when running in standalone mode. When running Dependabot in service mode, it will automatically detect repositories. |
While you could manually create scheduled pipelines to match the configuration within the repositories, it’s far easier to use the GitLab Pipeline schedules API.
I’ve created yet another script to quickly create scheduled pipelines, which is very similar to the bulk script above.
The script will again superficially inspect the .gitlab/dependabot.yml
files for a few known package managers, and create corresponding scheduled pipeline with the appropriate parameters.
You’ll notice the pipelines are created but not yet active, and not started immediately.
You can alter these options as you see fit, but note that you might quickly clog up your GitLab runners.
View script to create scheduled pipelines for all repositories containing .gitlab/dependabot.yml
#!/bin/bash
set -v
set -e
# Variables
API=https://your.gitlab.host/api/v4/
DEPENDABOT_STANDALONE_PROJECT_ID=400
AFTER=2021-01-01T00:00.00Z
PROJECTS=projects.txt
SOURCE_FILE='.gitlab%2Fdependabot.yml'
# Gather up to 300 repositories
curl --header Private-Token:${SETTINGS__GITLAB_ACCESS_TOKEN} "${API}projects?simple=true&archived=false&order_by=name&sort=asc&last_activity_after=${AFTER}&per_page=100" | jq -r '.[] | "\(.id),\(.path_with_namespace)"' > $PROJECTS
curl --header Private-Token:${SETTINGS__GITLAB_ACCESS_TOKEN} "${API}projects?simple=true&archived=false&order_by=name&sort=asc&last_activity_after=${AFTER}&per_page=100&page=2" | jq -r '.[] | "\(.id),\(.path_with_namespace)"' >> $PROJECTS
curl --header Private-Token:${SETTINGS__GITLAB_ACCESS_TOKEN} "${API}projects?simple=true&archived=false&order_by=name&sort=asc&last_activity_after=${AFTER}&per_page=100&page=3" | jq -r '.[] | "\(.id),\(.path_with_namespace)"' >> $PROJECTS
throttle() {
while [ "$(jobs | wc -l)" -ge 4 ]
do
sleep 1
done
}
download () {
local id=$1
local repo=$2
# Download source files
echo $id : $repo
mkdir -p "$(dirname $repo)"
! curl --silent --fail --header Private-Token:${SETTINGS__GITLAB_ACCESS_TOKEN} "${API}projects/${id}/repository/files/${SOURCE_FILE}/raw?ref=main" --output $repo
! curl --silent --fail --header Private-Token:${SETTINGS__GITLAB_ACCESS_TOKEN} "${API}projects/${id}/repository/files/${SOURCE_FILE}/raw?ref=master" --output $repo
# Warn on missing config files
if [ ! -f $repo ]; then
echo $id : $repo >> MISSING.txt
fi
}
# Download config files for each repository
while IFS=, read -r id path_with_namespace
do
throttle; download $id $path_with_namespace &
done < $PROJECTS
wait
# Sort MISSING files
if [ -f MISSING.txt ]; then
sort -o MISSING.txt MISSING.txt
fi
# Determine what to run for which project
grep -rl docker your-gitlab-group/ > docker.txt
grep -rl gradle your-gitlab-group/ > gradle.txt
grep -rl maven your-gitlab-group/ > maven.txt
grep -rl nuget your-gitlab-group/ > nuget.txt
grep -rl pip your-gitlab-group/ > pip.txt
grep -rl npm your-gitlab-group/ > npm.txt
# Retrieve existing pipeline schedules
SCHEDULES="${API}projects/${DEPENDABOT_STANDALONE_PROJECT_ID}/pipeline_schedules"
curl --header Private-Token:${SETTINGS__GITLAB_ACCESS_TOKEN} "${SCHEDULES}?per_page=100" | jq -r '.[] | "\(.id),\(.description)"' > existing.txt
curl --header Private-Token:${SETTINGS__GITLAB_ACCESS_TOKEN} "${SCHEDULES}?per_page=100&page=2" | jq -r '.[] | "\(.id),\(.description)"' >> existing.txt
sort existing.txt
schedule() {
local package_manager=$1
local path=$2
local directory=$3
# Skip if already present
if ! grep -q "$path $package_manager" existing.txt; then
# Create schedule
echo "Creating $path $package_manager schedule"
schedule_id=$(curl --request POST --header Private-Token:${SETTINGS__GITLAB_ACCESS_TOKEN} \
--form description="$path $package_manager" \
--form ref="master" \
--form cron="0 4 * * 2" \
--form active="false" \
"${SCHEDULES}" | jq -r '.id')
# Add required parameters
curl --request POST --header Private-Token:${SETTINGS__GITLAB_ACCESS_TOKEN} --form "key=PROJECT_PATH" --form "value=$path" "${SCHEDULES}/${schedule_id}/variables"
curl --request POST --header Private-Token:${SETTINGS__GITLAB_ACCESS_TOKEN} --form "key=PACKAGE_MANAGER_SET" --form "value=$package_manager" "${SCHEDULES}/${schedule_id}/variables"
curl --request POST --header Private-Token:${SETTINGS__GITLAB_ACCESS_TOKEN} --form "key=DIRECTORY" --form "value=$directory" "${SCHEDULES}/${schedule_id}/variables"
# Start immediately
#curl --request POST --header Private-Token:${SETTINGS__GITLAB_ACCESS_TOKEN} "${SCHEDULES}/${schedule_id}/play"
else
echo "Skipping $path $package_manager as it already exists"
fi
}
# Add scheduled tasks for Docker
while read -r path
do
schedule docker "$path" /
done < docker.txt
# Add scheduled tasks for gradle
while read -r path
do
schedule gradle "$path" /
done < gradle.txt
# Add scheduled tasks for maven
while read -r path
do
schedule maven "$path" /
done < maven.txt
# Add scheduled tasks for pip
while read -r path
do
schedule pip "$path" /
done < pip.txt
# Add scheduled tasks for nuget
while read -r path
do
schedule nuget "$path" /
done < nuget.txt
# Add scheduled tasks for npm
while read -r path
do
schedule npm "$path" /
done < npm.txt
Given how similar the script to create .gitlab/dependabot.yml
merge requests, and the script to create a scheduled pipeline are,
one could easily combine these two into one to immediately create the scheduled pipeline with the correct arguments.
I’ve chosen to keep them separately here however, as creating scheduled pipelines is only necessary when running in standalone mode.
The script to create the scheduled pipelines can even be run from within GitLab itself.
For that you can extend the .gitlab-ci.yml
of the dependabot-standalone project with the following.
schedule:
when: manual
except:
- schedules
image: alpine:3.12
script:
- apk add --no-cache bash curl jq
- ./schedule.sh
artifacts:
when: always
paths:
- ./*.txt
This will allow you to create new scheduled pipelines by running a new pipeline of the dependabot-standalone project for the master branch, and manually starting the schedule job.
Conclusion
As this blogpost has shown, you no longer have to miss out on automated dependency updates through Dependabot when using GitLab. With the provided samples and scripts it’s easy to get started for a few, or all, repositories. This makes it a whole lot easier to keep your projects up to date, keep Common Vulnerabilities and Exposures out and keep developers happy.