Reuse Gradle Build Cache on GitLab
The Gradle Build Cache is particularly well suited to speed up your CI/CD build times. But to set it up properly in GitLab you need to get a few things exactly right. This blogpost will guide you through the steps, as well as provide you with some background.
Gradle Build Cache
The Gradle build cache is a cache mechanism that aims to save time by reusing outputs produced by other builds. The build cache works by storing (locally or remotely) build outputs and allowing builds to fetch these outputs from the cache when it is determined that inputs have not changed, avoiding the expensive work of regenerating them.
There’s a few takeaways in the documentation that are of importance:
-
By default, the build cache is not enabled. You have to enable it explicitly.
-
Your builds should work well with Up-to-date checks (AKA Incremental Build).
-
Task output caching is automatically enabled when you enable the build cache.
Caching in GitLab CI/CD
Caching in GitLab is also well supported and documented.
A cache is one or more files that a job downloads and saves. Subsequent jobs that use the same cache don’t have to download the files again, so they execute more quickly.
Takeaways here are to follow the good caching practices mentioned, as well as picking a suitable caching key.
Optimizing our workflow
Our goal is to maximize our chances of reusing task outputs. Therefore our task inputs should ideally match as often as possible. At minimum we want our builds on the main branch to match the immediately preceding build task outputs, just after merging a merge request. To make this work we should configure our projects to use fast-forward merges only, as seen below. That way there are never any changes in the build on the main branch as compared to the build on the feature branch.
Our builds typically use the Gradle Wrapper for reproducibility, and as such we also want to cache the wrapper between builds.
Configure the build
With all the components & workflow listed, we now want to configure our build to cache the proper files. We want to share our cache between builds across all branches, so a shared key independent of branch is in order. This is where we deviate from the default Gradle .gitlab-ci.yml template, as that maintains a separate cache per branch.
See the .gitlab-ci.yml cache keyword reference for further details.
build:
cache:
key:
files:
- gradle/wrapper/gradle-wrapper.properties (1)
paths:
- cache/caches/ (2)
- cache/notifications/ (3)
- cache/wrapper/ (4)
script:
- ./gradlew --build-cache --gradle-user-home cache/ check (5)| 1 | It only makes sense to cache a single version of the Gradle Wrapper, which is why we use the gradle-wrapper.properties file as our cache key.
That way the cache is invalidated whenever the Gradle Wrapper version is updated. |
| 2 | We want GitLab to cache a number of paths; firstly we want to upload and restore the Gradle Build Cache folder. |
| 3 | When invoking the Gradle Wrapper the first time, we get a welcome message listing the recent release features.
To ensure it’s only displayed once, a file is stored under notifications/7.3/release-features.rendered.
By caching that file, we ensure the message is not shown on every build. |
| 4 | The Gradle Wrapper binaries are cached between builds, after being downloaded the first time. |
| 5 | Finally, we invoke our Gradle wrapper script with two feature flags:
|
Running the build
With the above build step in place, any builds on the main branch after a merge will have a build output similar to the following.
> Getting source from Git repository (1)
Fetching changes...
Reinitialized existing Git repository in /builds/abcde/.git/
Checking out a06e128d as master...
Removing .gradle/
Removing build/
Removing cache/
Skipping Git submodules setup
> Restoring cache (2)
Checking cache for 20cc834f826addfeaacf2d79110f8338ce1b4539-11...
Downloading cache.zip from https://gitlab-runners-cachebucket-...
Successfully extracted cache
> Executing "step_script" stage of the job script
$ ./gradlew --build-cache --gradle-user-home cache/ check
> Task :compileJava FROM-CACHE
> Task :processResources
> Task :classes
> Task :compileTestJava FROM-CACHE
> Task :processTestResources
> Task :testClasses
> Task :test FROM-CACHE
> Task :check UP-TO-DATE
BUILD SUCCESSFUL in 10s
8 actionable tasks: 4 executed, 4 from cache (3)
> Saving cache for successful job (4)
Creating cache 20cc834f826addfeaacf2d79110f8338ce1b4539-11...
cache/caches/: found 10267 matching files and directories
cache/notifications/: found 3 matching files and directories
cache/wrapper/: found 246 matching files and directories
Uploading cache.zip to https://gitlab-runners-cachebucket-...
Created cache
Job succeeded| 1 | The build will start with downloading our project source files, and clearing out any Git ignored files. |
| 2 | GitLab will download and extract the previously stored project cache. |
| 3 | When all goes well, running our build will log as UP-TO-DATE, with task outputs taken from cache. |
| 4 | Finally, the new build cache is once again uploaded for reuse in subsequent builds. |
Limitations
Now of course it can’t all be smooth sailing, and this approach is no different.
This setup assumes a project with limited feature branches active in parallel, as all builds share a single cache. When builds interleave it’s possible that only parts of the cache can be reused from one build to the next.
Another limitation could be the time to create, upload and download the cache itself. There can only be a speed improvement if the project is slower to build than the time needed for the cache. For smaller projects it might make sense to only cache the Gradle Wrapper for instance, and not download and upload the other files in each build.
Tested using Gradle 7.3 & GitLab 14.4.
