OWASP DependencyCheck Plugin Central Cache

Since this month, anyone using the OWASP dependency check plugin from Jeremy Long(*1) needs to upgrade to version 9. The older versions are no longer supported and could fail to work.

It is also recommended to get an NVD api key(*2), else the NVD update can take a very long time. The NVD is the U.S. government repository of standards based vulnerability management data represented using the Security Content Automation Protocol (SCAP) (*3). Since most of us are using a build environment, we don’t want to create a key for every project, but if we do not, we might get rate-limit errors.

This is why NVD suggest to use a local (or centralised) cache of the nvd files. In this post I will explain how to do that.

Get the NVD files to your (local) machine

This is done by using the Open Vulnerability Project’s(*4) cli tool: vulnz. It is a java based tool so you need to build and install it. The Open Vulnerability Project is a collection of Java libraries and a CLI to work with various vulnerability data-sources (NVD, GitHub Security Advisories, CISA Known Exploited Vulnerablity Catalog, FIRST Exploit Prediction Scoring System (EPSS), etc.). Jeremy Long is one of the main contributors of this project.

git clone git@github.com:jeremylong/Open-Vulnerability-Project.git
cd Open-Vulnerability-Project
./gradlew vulnz:build
cd vulnz/build/libs

Now you have a java file that can be used to get the files needed. The first time you call this, it might take up to 10 minutes to run.

./vulnz.jar cve --apikey=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx --cache --directory ./cache

If you get an out-of-memory error, you can try running it with a bit more memory:

java -Xmx2g -jar ./vulnz.jar cve --apikey=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx --cache --directory ./cache

Now if you cd into the ./cache folder, it should look something like this:

jbrugman@computername cache % ls -ltr
total 238672
-rw-r--r--  1 jbrugman  staff   3798839 18 jan 16:17 nvdcve-2012.json.gz
-rw-r--r--  1 jbrugman  staff       143 18 jan 16:17 nvdcve-2012.meta
-rw-r--r--  1 jbrugman  staff  14982741 18 jan 16:17 nvdcve-2023.json.gz
-rw-r--r--  1 jbrugman  staff       145 18 jan 16:17 nvdcve-2023.meta
-rw-r--r--  1 jbrugman  staff   3312453 18 jan 16:17 nvdcve-2011.json.gz
-rw-r--r--  1 jbrugman  staff       143 18 jan 16:17 nvdcve-2011.meta
-rw-r--r--  1 jbrugman  staff  13119491 18 jan 16:17 nvdcve-2022.json.gz
-rw-r--r--  1 jbrugman  staff       144 18 jan 16:17 nvdcve-2022.meta
-rw-r--r--  1 jbrugman  staff   3330878 18 jan 16:17 nvdcve-2010.json.gz
-rw-r--r--  1 jbrugman  staff       143 18 jan 16:17 nvdcve-2010.meta
-rw-r--r--  1 jbrugman  staff  12532086 18 jan 16:17 nvdcve-2021.json.gz
-rw-r--r--  1 jbrugman  staff       145 18 jan 16:17 nvdcve-2021.meta
-rw-r--r--  1 jbrugman  staff   9058667 18 jan 16:17 nvdcve-2020.json.gz
-rw-r--r--  1 jbrugman  staff       143 18 jan 16:17 nvdcve-2020.meta
-rw-r--r--  1 jbrugman  staff   4055630 18 jan 16:17 nvdcve-2009.json.gz
-rw-r--r--  1 jbrugman  staff       143 18 jan 16:17 nvdcve-2009.meta
-rw-r--r--  1 jbrugman  staff   2962182 18 jan 16:17 nvdcve-2008.json.gz
-rw-r--r--  1 jbrugman  staff       143 18 jan 16:17 nvdcve-2008.meta
-rw-r--r--  1 jbrugman  staff   7924987 18 jan 16:17 nvdcve-2019.json.gz
-rw-r--r--  1 jbrugman  staff       142 18 jan 16:17 nvdcve-2019.meta
-rw-r--r--  1 jbrugman  staff   2914030 18 jan 16:17 nvdcve-2007.json.gz
-rw-r--r--  1 jbrugman  staff       143 18 jan 16:17 nvdcve-2007.meta
-rw-r--r--  1 jbrugman  staff   7020129 18 jan 16:17 nvdcve-2018.json.gz
-rw-r--r--  1 jbrugman  staff       143 18 jan 16:17 nvdcve-2018.meta
-rw-r--r--  1 jbrugman  staff   2637365 18 jan 16:17 nvdcve-2006.json.gz
-rw-r--r--  1 jbrugman  staff       142 18 jan 16:17 nvdcve-2006.meta
-rw-r--r--  1 jbrugman  staff   6403948 18 jan 16:17 nvdcve-2017.json.gz
-rw-r--r--  1 jbrugman  staff       143 18 jan 16:17 nvdcve-2017.meta
-rw-r--r--  1 jbrugman  staff   1956119 18 jan 16:17 nvdcve-2005.json.gz
-rw-r--r--  1 jbrugman  staff       143 18 jan 16:17 nvdcve-2005.meta
-rw-r--r--  1 jbrugman  staff   3233184 18 jan 16:17 nvdcve-2016.json.gz
-rw-r--r--  1 jbrugman  staff       142 18 jan 16:17 nvdcve-2016.meta
-rw-r--r--  1 jbrugman  staff   1076293 18 jan 16:17 nvdcve-2004.json.gz
-rw-r--r--  1 jbrugman  staff       142 18 jan 16:17 nvdcve-2004.meta
-rw-r--r--  1 jbrugman  staff   2634851 18 jan 16:17 nvdcve-2015.json.gz
-rw-r--r--  1 jbrugman  staff       143 18 jan 16:17 nvdcve-2015.meta
-rw-r--r--  1 jbrugman  staff    598141 18 jan 16:17 nvdcve-2003.json.gz
-rw-r--r--  1 jbrugman  staff       140 18 jan 16:17 nvdcve-2003.meta
-rw-r--r--  1 jbrugman  staff   3943878 18 jan 16:17 nvdcve-2014.json.gz
-rw-r--r--  1 jbrugman  staff       142 18 jan 16:17 nvdcve-2014.meta
-rw-r--r--  1 jbrugman  staff   1663893 18 jan 16:17 nvdcve-2002.json.gz
-rw-r--r--  1 jbrugman  staff       143 18 jan 16:17 nvdcve-2002.meta
-rw-r--r--  1 jbrugman  staff   4185442 18 jan 16:17 nvdcve-2013.json.gz
-rw-r--r--  1 jbrugman  staff       141 18 jan 16:17 nvdcve-2013.meta
-rw-r--r--  1 jbrugman  staff    709660 18 jan 16:17 nvdcve-2024.json.gz
-rw-r--r--  1 jbrugman  staff       140 18 jan 16:17 nvdcve-2024.meta
-rw-r--r--  1 jbrugman  staff    635158 18 jan 16:17 nvdcve-modified.json.gz
-rw-r--r--  1 jbrugman  staff       140 18 jan 16:17 nvdcve-modified.meta
-rw-r--r--  1 jbrugman  staff      1169 18 jan 16:17 cache.properties

So nice, we have the files we need to have our dependency check running again. Since I want to test it on my local machine, I just run a simple webserver from within this cache folder. For convenience, I use the default https.server that’s part of python.

python3 -m http.server --directory .

Configure your application

I use gradle in this project; all examples are based on it.

First edit your build.gradle file, to add the dependency:

plugins {
  id "org.owasp.dependencycheck" version "9.0.9"
  id "java"
}

Now we need to configure(*5) the plugin, so it gets the files from the proper location:

dependencyCheck {
  failBuildOnCVSS = 7
  failOnError = true
  suppressionFile = './owasp/suppressions.xml'

  // Make sure the nvd files are fetched from the own central server.
  nvd {
    datafeedUrl="http://localhost:8000/nvdcve-{0}.json.gz"
  }
}

That’s it, just configure the datafeedUrl! Now test it:

# clean your local cache
./gradlew dependencyCheckPurge

> Configure project :
Inferred project: splashmicroservice, version: 0.295.0-dev.16.uncommitted+combine.prs.branch.ae18b48

BUILD SUCCESSFUL in 1s
1 actionable task: 1 executed

# run the dependency check
./gradlew dependencyCheckAggregate --no-daemon --stacktrace

...

Well, if everything works as intended, you should see in your webserver log that it gets the files needed to fill the h2 database that the plugin uses.

# log from the python webserver
::ffff:127.0.0.1 - - [18/Jan/2024 16:19:13] "GET /cache.properties HTTP/1.1" 200 -
::ffff:127.0.0.1 - - [18/Jan/2024 16:19:13] "GET /nvdcve-2012.json.gz HTTP/1.1" 200 -
::ffff:127.0.0.1 - - [18/Jan/2024 16:19:13] "GET /nvdcve-2023.json.gz HTTP/1.1" 200 -
::ffff:127.0.0.1 - - [18/Jan/2024 16:19:14] "GET /nvdcve-2011.json.gz HTTP/1.1" 200 -
::ffff:127.0.0.1 - - [18/Jan/2024 16:19:14] "GET /nvdcve-2022.json.gz HTTP/1.1" 200 -
::ffff:127.0.0.1 - - [18/Jan/2024 16:19:14] "GET /nvdcve-2010.json.gz HTTP/1.1" 200 -
::ffff:127.0.0.1 - - [18/Jan/2024 16:19:14] "GET /nvdcve-2021.json.gz HTTP/1.1" 200 -
::ffff:127.0.0.1 - - [18/Jan/2024 16:19:14] "GET /nvdcve-2020.json.gz HTTP/1.1" 200 -
::ffff:127.0.0.1 - - [18/Jan/2024 16:19:14] "GET /nvdcve-2009.json.gz HTTP/1.1" 200 -
::ffff:127.0.0.1 - - [18/Jan/2024 16:19:14] "GET /nvdcve-2008.json.gz HTTP/1.1" 200 -
::ffff:127.0.0.1 - - [18/Jan/2024 16:19:14] "GET /nvdcve-2019.json.gz HTTP/1.1" 200 -
::ffff:127.0.0.1 - - [18/Jan/2024 16:19:14] "GET /nvdcve-2007.json.gz HTTP/1.1" 200 -
::ffff:127.0.0.1 - - [18/Jan/2024 16:19:14] "GET /nvdcve-2018.json.gz HTTP/1.1" 200 -
::ffff:127.0.0.1 - - [18/Jan/2024 16:19:14] "GET /nvdcve-2006.json.gz HTTP/1.1" 200 -
::ffff:127.0.0.1 - - [18/Jan/2024 16:19:14] "GET /nvdcve-2017.json.gz HTTP/1.1" 200 -
::ffff:127.0.0.1 - - [18/Jan/2024 16:19:14] "GET /nvdcve-2005.json.gz HTTP/1.1" 200 -
::ffff:127.0.0.1 - - [18/Jan/2024 16:19:14] "GET /nvdcve-2016.json.gz HTTP/1.1" 200 -
::ffff:127.0.0.1 - - [18/Jan/2024 16:19:14] "GET /nvdcve-2004.json.gz HTTP/1.1" 200 -
::ffff:127.0.0.1 - - [18/Jan/2024 16:19:14] "GET /nvdcve-2015.json.gz HTTP/1.1" 200 -
::ffff:127.0.0.1 - - [18/Jan/2024 16:19:14] "GET /nvdcve-2003.json.gz HTTP/1.1" 200 -
::ffff:127.0.0.1 - - [18/Jan/2024 16:19:14] "GET /nvdcve-2014.json.gz HTTP/1.1" 200 -
::ffff:127.0.0.1 - - [18/Jan/2024 16:19:14] "GET /nvdcve-2002.json.gz HTTP/1.1" 200 -
::ffff:127.0.0.1 - - [18/Jan/2024 16:19:14] "GET /nvdcve-2013.json.gz HTTP/1.1" 200 -
::ffff:127.0.0.1 - - [18/Jan/2024 16:19:14] "GET /nvdcve-2024.json.gz HTTP/1.1" 200 -

All that’s left to do is set up the server, so it gets those files on a daily base, and make sure your config points to that central managed server. You could for example use an s3 bucket on AWS to host your files and use that toe serve your cached nvd files. The configuration would then look like this:

dependencyCheck {
  failBuildOnCVSS = 7
  failOnError = true
  suppressionFile = '.circleci/owasp/suppressions.xml'
  nvd {
    datafeedUrl="https://mycompany-owasp-files.s3.eu-west-1.amazonaws.com/nvdcve-{0}.json.gz"
  }
}

Recapitulation

If you have your CI/CD set up to handle your build, you hardly have to change anything. All you need to do is:

Create a cron-job to update your nvd-files on a daily basis using the vulnz cli tool
Host those files on a (secure) webserver that your CI/CD environment can reach
Alter the configuration of your build.gradle to point to that given webserver

Sources / Links

(*1) DependencyCheck plugin
(*2) Request NVD api key
(*3) NVD (National Vulnerability Database)
(*4) Open Vulnerability Project
(*5) Dependency Check Configuration

Get the NVD files to your (local) machine

Configure your application

Recapitulation

Sources / Links

Other interesting blogs about OWASP related topics