As a consultant I find myself alternating between GitLab and GitHub about once a year, depending on the assignment. While I like GitLab a lot, there’s one thing I had sorely missed whenever I switch back from GitHub: Dependabot. Dependabot scans your project dependencies, and creates merge requests whenever updates are found. This provides you with an easy way to keep up to date on dependencies, and notifies you early if there are any incompatibilities.

Even though there are alternatives such as snyk.io and even GitLab’s own Dependency Scanning, those don’t always support enterprise or partner installations of GitLab, require GitLab Ultimate, or don’t support the full range of package managers that Dependabot supports.

Luckily though, there’s now a Dependabot for GitLab project. This project is based on the same Open Source Dependabot Core, so you can get the exact same automated dependency updates on both platforms.

In this blogpost I’ll walk you through how you can quickly roll out Dependabot on an existing GitLab installation, so you can start updating your dependencies automatically.

Running modes

There are two ways of running Dependabot GitLab:

  1. Standalone via scheduled GitLab pipelines

  2. As a service deployed through Helm or Docker Compose.

The standalone version is the easiest to get started with, and what we will use in this blogpost. It is however quite limited, and requires the user to manage the scheduled pipelines. It’s a great way to explore whether your organization wants to adopt automated dependency updates.

The service installation is recommended, and with that you get webhook support to automatically add and remove repositories, and a web user interface to view currently configured dependency updates.

Dependabot Standalone installation

Standalone installation is easy enough; We import the dependabot-standalone project into our own GitLab instance or group, and configure the two required access tokens for both GitLab and GitHub. The GitHub token might seem surprising at first; it is used to retrieve the release notes shown in the merge requests for supported dependencies. You might need to override SETTINGS__GITLAB_URL if you run on an alternative GitLab host.

A helpful addition is to also configure access to your private repositories, such that you also get notified for internal library updates. For Maven repositories this comes down to some additional environment variables:

SETTINGS__CREDENTIALS__MAVEN__REPO1__URL=maven_url
SETTINGS__CREDENTIALS__MAVEN__REPO1__USERNAME=maven_username
SETTINGS__CREDENTIALS__MAVEN__REPO1__PASSWORD=maven_password

That’s it in terms of installation for Dependabot Standalone; we’ll reuse this repository once it’s time to create scheduled pipelines.

Configure repository updates through .gitlab/dependabot.yml

Any repositories that wish to receive automated dependency update merge requests will need to be configured through a .gitlab/dependabot.yml file, which follows the same format as the official Dependabot documentation. Some minor extensions are provided to automatically accept and merge requests.

When getting started with Dependabot you’ll likely want to try it out on a first few projects, with various dependency managers. For a sample Gradle project your configuration file might look something like this:

version: 2
updates:
  - package-ecosystem: "gradle"
    directory: "/"
    schedule:
      interval: "daily"
    open-pull-requests-limit: 20
    rebase-strategy: auto
    auto-merge: true

You can configure a single repository for updates through multiple package-ecosystems, optionally setting the correct directory argument when not located at the root. Exclusions can be added to skip certain package updates, for fine tuning according to project specifics. For details see the reference documentation.

Configuration in bulk

Once you’ve determined that you want to configure all repositories to use Dependabot, you will need to roll out the configuration files in bulk. To make that easier I’ve set up a small sample script which you can update to match your environment. The script assumes you have your SSH key setup to clone all repositories locally, superficially explores your project for a few known package managers, and creates a merge request per repository to add the configuration file.

View local script to create .gitlab/dependabot.yml merge requests for all repositories
#!/bin/bash
set -v
set -e

# Variables
API=https://your.gitlab.host/api/v4/
AFTER=2021-01-01T00:00.00Z
PROJECTS=projects.txt

# Gather up to 300 repositories
curl --header Private-Token:${SETTINGS__GITLAB_ACCESS_TOKEN} "${API}projects?simple=true&archived=false&order_by=name&sort=asc&last_activity_after=${AFTER}&per_page=100" | jq -r '.[] | "\(.path_with_namespace)"' > $PROJECTS
curl --header Private-Token:${SETTINGS__GITLAB_ACCESS_TOKEN} "${API}projects?simple=true&archived=false&order_by=name&sort=asc&last_activity_after=${AFTER}&per_page=100&page=2" | jq -r '.[] | "\(.path_with_namespace)"' >> $PROJECTS
curl --header Private-Token:${SETTINGS__GITLAB_ACCESS_TOKEN} "${API}projects?simple=true&archived=false&order_by=name&sort=asc&last_activity_after=${AFTER}&per_page=100&page=3" | jq -r '.[] | "\(.path_with_namespace)"' >> $PROJECTS
sort $PROJECTS

# Setup variables
folder=$(pwd)
targetfile=.gitlab/dependabot.yml
branch="configure-dependabot-gitlab"
gitpush="$folder/git-push.out"
truncate -s 0 "$gitpush"

# Loop over projects to create merge requests with .gitlab/dependabot.yml
while read -r p; do
  cd "$folder";
  echo -e "\n\n$p"

  # Update/clone local repository
  if [ -d "repos/$p" ]; then
    cd "repos/$p"
    git reset --hard
    git checkout master || git checkout main
    git pull -q
    git branch -D "$branch" || true
  else
    cd "repos"
    git clone "git@your.gitlab.host:$p.git" "$p"
    cd "$p"
  fi

  # Start a new branch
  git checkout -b "$branch"

  # Shared header
  mkdir -p "$(dirname $targetfile)"
  echo 'version: 2
updates:' > $targetfile

  # Maven
  if [ -f pom.xml ]; then
    echo '  - package-ecosystem: "maven"
    directory: "/"
    schedule:
      interval: "daily"
    open-pull-requests-limit: 20' >> $targetfile
  fi

  # Gradle
  if [ -f build.gradle ]; then
    echo '  - package-ecosystem: "gradle"
    directory: "/"
    schedule:
      interval: "daily"
    open-pull-requests-limit: 20' >> $targetfile
  fi

  # Docker
  if [ -f Dockerfile ]; then
    echo '  - package-ecosystem: "docker"
    directory: "/"
    schedule:
      interval: "daily"' >> $targetfile
  fi

  # Nuget
  if [ -f nuget.config ]; then
    echo '  - package-ecosystem: "nuget"
    directory: "/"
    schedule:
      interval: "daily"
    open-pull-requests-limit: 20' >> $targetfile
  fi

  # Python
  if [ -f setup.py ] || [ -f Pipfile ] || [ -f requirements.txt ]; then
    echo '  - package-ecosystem: "pip"
    directory: "/"
    schedule:
      interval: "daily"
    open-pull-requests-limit: 20' >> $targetfile
  fi

  # NPM
  if [ -f package.json ]; then
    echo '  - package-ecosystem: "npm"
    directory: "/"
    schedule:
      interval: "daily"
    open-pull-requests-limit: 20' >> $targetfile
  fi

  # Only commit if this resulted in changes
  if [[ ! $(git status --porcelain) ]]; then
    continue;
  fi

  # Commit changes
  git add $targetfile
  git commit -m "Configure dependabot"

  # Show differences compared to master
  git --no-pager diff origin/master || git --no-pager diff origin/main
  # Push branch
  git push --force -u origin "$branch" \
    -o merge_request.create \
    -o merge_request.merge_when_pipeline_succeeds \
    -o merge_request.remove_source_branch \
    -o merge_request.label="dependabot" \
    2>&1 | tee -a "$gitpush"

done < $PROJECTS

# Open all new pull requests in Chrome
grep 'merge_request' "$gitpush" | awk '{print $2}' | xargs google-chrome

Creating Scheduled pipelines

Once you’ve configured some, or all, repositories with a .gitlab/dependabot.yml file, you will want to create scheduled pipelines that match the configuration files.

Creating scheduled pipelines is only needed when running in standalone mode. When running Dependabot in service mode, it will automatically detect repositories.

While you could manually create scheduled pipelines to match the configuration within the repositories, it’s far easier to use the GitLab Pipeline schedules API. I’ve created yet another script to quickly create scheduled pipelines, which is very similar to the bulk script above. The script will again superficially inspect the .gitlab/dependabot.yml files for a few known package managers, and create corresponding scheduled pipeline with the appropriate parameters. You’ll notice the pipelines are created but not yet active, and not started immediately. You can alter these options as you see fit, but note that you might quickly clog up your GitLab runners.

View script to create scheduled pipelines for all repositories containing .gitlab/dependabot.yml
#!/bin/bash
set -v
set -e

# Variables
API=https://your.gitlab.host/api/v4/
DEPENDABOT_STANDALONE_PROJECT_ID=400
AFTER=2021-01-01T00:00.00Z
PROJECTS=projects.txt
SOURCE_FILE='.gitlab%2Fdependabot.yml'

# Gather up to 300 repositories
curl --header Private-Token:${SETTINGS__GITLAB_ACCESS_TOKEN} "${API}projects?simple=true&archived=false&order_by=name&sort=asc&last_activity_after=${AFTER}&per_page=100" | jq -r '.[] | "\(.id),\(.path_with_namespace)"' > $PROJECTS
curl --header Private-Token:${SETTINGS__GITLAB_ACCESS_TOKEN} "${API}projects?simple=true&archived=false&order_by=name&sort=asc&last_activity_after=${AFTER}&per_page=100&page=2" | jq -r '.[] | "\(.id),\(.path_with_namespace)"' >> $PROJECTS
curl --header Private-Token:${SETTINGS__GITLAB_ACCESS_TOKEN} "${API}projects?simple=true&archived=false&order_by=name&sort=asc&last_activity_after=${AFTER}&per_page=100&page=3" | jq -r '.[] | "\(.id),\(.path_with_namespace)"' >> $PROJECTS

throttle() {
	while [ "$(jobs | wc -l)" -ge 4 ]
	do
		sleep 1
	done
}

download () {
  local id=$1
  local repo=$2

  # Download source files
  echo $id : $repo
  mkdir -p "$(dirname $repo)"
  ! curl --silent --fail --header Private-Token:${SETTINGS__GITLAB_ACCESS_TOKEN} "${API}projects/${id}/repository/files/${SOURCE_FILE}/raw?ref=main" --output $repo
  ! curl --silent --fail --header Private-Token:${SETTINGS__GITLAB_ACCESS_TOKEN} "${API}projects/${id}/repository/files/${SOURCE_FILE}/raw?ref=master" --output $repo

  # Warn on missing config files
  if [ ! -f $repo ]; then
    echo $id : $repo >> MISSING.txt
  fi
}

# Download config files for each repository
while IFS=, read -r id path_with_namespace
do
  throttle; download $id $path_with_namespace &
done < $PROJECTS
wait

# Sort MISSING files
if [ -f MISSING.txt ]; then
  sort -o MISSING.txt MISSING.txt
fi

# Determine what to run for which project
grep -rl docker your-gitlab-group/ > docker.txt
grep -rl gradle your-gitlab-group/ > gradle.txt
grep -rl maven your-gitlab-group/ > maven.txt
grep -rl nuget your-gitlab-group/ > nuget.txt
grep -rl pip your-gitlab-group/ > pip.txt
grep -rl npm your-gitlab-group/ > npm.txt

# Retrieve existing pipeline schedules
SCHEDULES="${API}projects/${DEPENDABOT_STANDALONE_PROJECT_ID}/pipeline_schedules"
curl --header Private-Token:${SETTINGS__GITLAB_ACCESS_TOKEN} "${SCHEDULES}?per_page=100" | jq -r '.[] | "\(.id),\(.description)"' > existing.txt
curl --header Private-Token:${SETTINGS__GITLAB_ACCESS_TOKEN} "${SCHEDULES}?per_page=100&page=2" | jq -r '.[] | "\(.id),\(.description)"' >> existing.txt
sort existing.txt

schedule() {
  local package_manager=$1
  local path=$2
  local directory=$3

  # Skip if already present
  if ! grep -q "$path $package_manager" existing.txt; then
    # Create schedule
    echo "Creating  $path $package_manager schedule"
    schedule_id=$(curl --request POST --header Private-Token:${SETTINGS__GITLAB_ACCESS_TOKEN} \
      --form description="$path $package_manager" \
      --form ref="master" \
      --form cron="0 4 * * 2" \
      --form active="false" \
      "${SCHEDULES}" | jq -r '.id')
    # Add required parameters
    curl --request POST --header Private-Token:${SETTINGS__GITLAB_ACCESS_TOKEN} --form "key=PROJECT_PATH" --form "value=$path" "${SCHEDULES}/${schedule_id}/variables"
    curl --request POST --header Private-Token:${SETTINGS__GITLAB_ACCESS_TOKEN} --form "key=PACKAGE_MANAGER_SET" --form "value=$package_manager" "${SCHEDULES}/${schedule_id}/variables"
    curl --request POST --header Private-Token:${SETTINGS__GITLAB_ACCESS_TOKEN} --form "key=DIRECTORY" --form "value=$directory" "${SCHEDULES}/${schedule_id}/variables"
    # Start immediately
    #curl --request POST --header Private-Token:${SETTINGS__GITLAB_ACCESS_TOKEN} "${SCHEDULES}/${schedule_id}/play"
  else
    echo "Skipping $path $package_manager as it already exists"
  fi
}

# Add scheduled tasks for Docker
while read -r path
do
  schedule docker "$path" /
done < docker.txt
# Add scheduled tasks for gradle
while read -r path
do
  schedule gradle "$path" /
done < gradle.txt
# Add scheduled tasks for maven
while read -r path
do
  schedule maven "$path" /
done < maven.txt
# Add scheduled tasks for pip
while read -r path
do
  schedule pip "$path" /
done < pip.txt
# Add scheduled tasks for nuget
while read -r path
do
  schedule nuget "$path" /
done < nuget.txt
# Add scheduled tasks for npm
while read -r path
do
  schedule npm "$path" /
done < npm.txt

Given how similar the script to create .gitlab/dependabot.yml merge requests, and the script to create a scheduled pipeline are, one could easily combine these two into one to immediately create the scheduled pipeline with the correct arguments. I’ve chosen to keep them separately here however, as creating scheduled pipelines is only necessary when running in standalone mode.

The script to create the scheduled pipelines can even be run from within GitLab itself. For that you can extend the .gitlab-ci.yml of the dependabot-standalone project with the following.

Listing 1. dependabot-standalone/.gitlab-ci.yml
schedule:
  when: manual
  except:
    - schedules
  image: alpine:3.12
  script:
    - apk add --no-cache bash curl jq
    - ./schedule.sh
  artifacts:
    when: always
    paths:
      - ./*.txt

This will allow you to create new scheduled pipelines by running a new pipeline of the dependabot-standalone project for the master branch, and manually starting the schedule job.

Conclusion

As this blogpost has shown, you no longer have to miss out on automated dependency updates through Dependabot when using GitLab. With the provided samples and scripts it’s easy to get started for a few, or all, repositories. This makes it a whole lot easier to keep your projects up to date, keep Common Vulnerabilities and Exposures out and keep developers happy.

shadow-left