From ba564a6aed32c8fd245cd12d44ff06c0475eb933 Mon Sep 17 00:00:00 2001 From: Dale Visser Date: Mon, 3 Aug 2015 17:15:45 -0400 Subject: [PATCH] Added page to site that documents how to take daily snapshots of the NVD, and run D-C cli against those snapshots. --- src/site/markdown/data/cachenvd.md | 93 ++++++++++++++++++++ src/site/resources/general/dep-check-date.sh | 11 +++ src/site/resources/general/nvd_download.sh | 5 ++ src/site/site.xml | 1 + 4 files changed, 110 insertions(+) create mode 100644 src/site/markdown/data/cachenvd.md create mode 100755 src/site/resources/general/dep-check-date.sh create mode 100755 src/site/resources/general/nvd_download.sh diff --git a/src/site/markdown/data/cachenvd.md b/src/site/markdown/data/cachenvd.md new file mode 100644 index 000000000..fdfb3f69b --- /dev/null +++ b/src/site/markdown/data/cachenvd.md @@ -0,0 +1,93 @@ +Snapshotting the NVD +==================== + +The [Mirroring the NVD from NIST](./mirrornvd.html) topic describes briefly +how to use the [Nist-Data-Mirror](https://github.com/stevespringett/nist-data-mirror/) +project to cache the NVD locally and run Dependency Check (D-C) against the +local cache. + +This topic goes into a bit more depth with the [cli](../dependency-check-cli/index.html) +client, focusing on the following use case. + +1. You wish to have daily local snapshots of the NVD, so that +2. in order to compare later runs of D-C with earlier runs, you can compare + "apples with apples". + +In other words: It is sometimes desirable to run a comparison D-C analysis +against the same NVD snapshot that an earlier D-C report used. + +In the steps below, concrete examples will be given assuming an Ubuntu Linux +system. Hopefully, enough explanation is provided that the steps can easily be +translated to other systems. + +Build Nist-Data-Mirror +---------------------- + +1. Perform a "git clone" of [Nist-Data-Mirror](https://github.com/stevespringett/nist-data-mirror/) +2. Install gradle, if necessary. See [here](http://gradle.org/gradle-download/) + or your Linux distributions package management system. (e.g., + `sudo apt-get install gradle`). +3. Follow the [build instructions](https://github.com/stevespringett/nist-data-mirror/blob/master/README.md#user-content-build). + You will be left with a build artifact called `nist-data-mirror-1.0.0.jar`. + +Set Up a Daily NVD Download Job +------------------------------- + +On Linux, the way to do this using the [cron daemon](http://linux.die.net/man/8/cron). +"Cron jobs" are configured by invoking [crontab](http://linux.die.net/man/5/crontab). +For example, invoke `crontab -e` to add a line like the following to your +crontab file: + + 4 5 * * * ~/.local/bin/nvd_download.sh ~/NVD ~/.local/jars + +This would run a job on your system at 4:05 AM daily to run the +[nvd_download.sh](general/nvd_download.sh) shell script with the two given +arguments. The script is simple: + +```sh +#!/bin/sh +NVD_ROOT=$1/`date -I` +JAR_PATH=$2/nist-data-mirror-1.0.0.jar +java -jar $JAR_PATH $NVD_ROOT +rm $NVD_ROOT/*.xml # D-C works directly with .gz files anyway. +``` + +Nist-Data-Mirror will automatically create the directory, download the +.xml.gz files, and extract the .xml files alongside them. Given the parameters +in the cron example above, the new directory will be `~/NVD/2015-08-03` if +executed on August 3rd, 2015. The download for 2015-08-03 pulled 47 +MiB, and took up a total of 668 MiB after extracting from the compressed +archive format. It turns out that D-C works directly with the .xml.gz files, +so the above script preserves disk space by deleting the .xml files. + +Invoke the Command-Line Using a Specific Daily Snapshot +------------------------------------------------------- + +An example script named [dep-check-date.sh](general/dep-check-date.sh) is +shown below, which facilitates a D-C scan against an arbitrary NVD snapshot: + +```sh +#!/bin/sh +CLI_LOCATION=~/.local/dependency-check-1.2.11 +CLI_SCRIPT=$CLI_LOCATION/bin/dependency-check.sh +NVD_PATH=$1/`date -I -d $2` +NVD=file://$NVD_PATH +shift 2 # We've used the first two params. The rest go to CLI_SCRIPT. +$CLI_SCRIPT --cveUrl20Base $NVD/nvdcve-2.0-%d.xml.gz \ + --cveUrl12Base $NVD/nvdcve-%d.xml.gz \ + --cveUrl20Modified $NVD/nvdcve-2.0-Modified.xml.gz \ + --cveUrl12Modified $NVD/nvdcve-Modified.xml.gz \ + --data $NVD_PATH $@ +``` + +The script takes advantage of the `date` command's ability to parse a variety +of date formats. The following invokation would successfully point to the +`~/NVD/2015-08-03` folder. + + $ ./dep-check-date.sh ~/NVD "08/03/2015" -app Foo -scan /path/to/Foo --out ~/DCreports/FooFollowup/ + +If today happened to be August 4th, 2015, `"yesterday"` would also have worked +as well. Also notice the usage of the `--data` parameter. This places the +H2 database file directly in the folder alongside the .xml.gz files. This is +critical, so that D-C doesn't run against another version of the database, +like the usual default in `$CLI_LOCATION/data`. \ No newline at end of file diff --git a/src/site/resources/general/dep-check-date.sh b/src/site/resources/general/dep-check-date.sh new file mode 100755 index 000000000..21130bf8d --- /dev/null +++ b/src/site/resources/general/dep-check-date.sh @@ -0,0 +1,11 @@ +#!/bin/sh +CLI_LOCATION=~/.local/dependency-check-1.2.11 +CLI_SCRIPT=$CLI_LOCATION/bin/dependency-check.sh +NVD_PATH=$1/`date -I -d $2` +NVD=file://$NVD_PATH +shift 2 # We've used the first two params. The rest go to CLI_SCRIPT. +$CLI_SCRIPT --cveUrl20Base $NVD/nvdcve-2.0-%d.xml.gz \ + --cveUrl12Base $NVD/nvdcve-%d.xml.gz \ + --cveUrl20Modified $NVD/nvdcve-2.0-Modified.xml.gz \ + --cveUrl12Modified $NVD/nvdcve-Modified.xml.gz \ + --data $NVD_PATH $@ \ No newline at end of file diff --git a/src/site/resources/general/nvd_download.sh b/src/site/resources/general/nvd_download.sh new file mode 100755 index 000000000..5af32b5b9 --- /dev/null +++ b/src/site/resources/general/nvd_download.sh @@ -0,0 +1,5 @@ +#!/bin/sh +NVD_ROOT=$1/`date -I` +JAR_PATH=$2/nist-data-mirror-1.0.0.jar +java -jar $JAR_PATH $NVD_ROOT +rm $NVD_ROOT/*.xml # D-C works directly with .gz files anyway. \ No newline at end of file diff --git a/src/site/site.xml b/src/site/site.xml index 6d0a9d3ed..ac5b51b5d 100644 --- a/src/site/site.xml +++ b/src/site/site.xml @@ -87,6 +87,7 @@ Copyright (c) 2013 Jeremy Long. All Rights Reserved. +