Multiple documentation updates

README.md

@@ -65,10 +65,10 @@ I decided to use PostgreSQL, because:
 
 The application also needs read-only access to vulnerability database maintained by OWASP Dependency Check. ODC currently supports H2 and MySQL. However, there are multiple issues with H2 for this usage. The first one issue is concurrent access. The concurrent access probably could have been somehow configured, but ODC uses different case for MySQL and H2 table names and column names. This makes it hard to support both at the same time.
 
-## Development notes
+## Running multiple instances behind load-balancer
+
+Although Play framework is designed to allow running one application in multiple instances in order to balance the load, applications might break the share-nothing approach, which is also the case of this application. Such applications might not work properly when running in multiple instances behind a load balancer. We simply did not consider to be so important for this application, but if it is your concern, we are happy to accept pull requests that address this issue. There are basically two points with shared state:
+
+* Cache. The cache can be purged manually, which might cause inconsistent behavior. For example, when you purge the cache on one server instance and you are switched to another instance then. This might be avoidable by adding some timestamp to some cookie, but this is not implemented.
+* Cron task locking. The lock is held in an instance, not in database. This behavior is good when recovering from a crash, but you should not run the cron job on multiple servers at the same time. Not following this advice might lead to multiple exports (e.g., emails or issue tracker items) for one event and even to some crashes when a worker tries to add a ticket ID for already exported issue. Maybe no pull code change is desired and it should be administrator's responsibility to run the cron task on one machine only.
 
-### Naming
-* Library × Identifier × PlainLibraryIdentifier – should be renamed
-    * Identifier is the most verbose one, it comes from OWASP Dependency Check.
-    * Library is a record stored in our database.
-    * PlainLibraryIdentifier is just version-less (e.g. `"$groupId:$artifactId"`) library identifier.

conf/application.conf.-example

@@ -1,4 +1,6 @@
-# This is the main configuration file for the application.
+# This configuration file is intended for development mode. If you want an example configuration for peoduction, look at /production.conf-example
+#
+#
 # ~~~~~
 
 # Secret key
@@ -14,10 +16,14 @@ play.crypto.secret = "changeme"
 # ~~~~~
 play.i18n.langs = [ "en" ]
 
-play.modules.enabled += "modules.ConfigModule"
-play.modules.enabled += "modules.SilhouetteModule"
+app{
+  host = "localhost" # You have to configure the host there. If you don't do so, all accesses via host will be prohibited. This is a protection against DNS rebind attacks.
+  secure = false # Use true iff you use HTTPS
+}
 
 yssdc{
+  # Anyone who knows the cron key can start periodic tasks
+  cronKey="{{ lookup('cron_token', 'play_secret length=64') }}"
   bamboo{
     url = …
   }
@@ -28,6 +34,25 @@ yssdc{
       password = …
     }
   }
+  export{
+    # Optional section: export to issue tracker
+    issueTracker{
+      provider: "jira"
+      server: "http://…"
+      projectId = 10000
+      vulnerabilityIssueType = 10100
+      authentication {
+        type = "credentials"
+        user = "…"
+        password = "…"
+      }
+    }
+    # Optional section: email notifications
+    email{
+      from = "info@example.com"
+      noSubscriberContact = "foobar@example.com"
+    }
+  }
   projects = {jobId:humanReadableName, …}
   teams = […]
   exclusions{
@@ -75,7 +100,7 @@ slick.dbs.odc {
   driver = "slick.driver.MySQLDriver$"
   db {
     url = "jdbc:mysql://127.0.0.1/dependencycheck"
-    # Those credentials are default in ODC (but you might have changed them):
+    # These credentials are default in ODC (but you might have changed them):
     user = "dcuser"
     password = "DC-Pass1337!"
   }
@@ -115,3 +140,11 @@ silhouette {
   }
 }
 
+play{
+  # needed if you want this app to send emails
+  mailer{
+    mock = true # If mock is true, mails are not actually sent, but just logged.
+    // host = "…"
+  }
+}
+

documentation-debt.md

@@ -0,0 +1,6 @@
+# Documentation debt
+
+* more detailed installation howto
+* how to configure cron jobs
+* external authentication service API
+* what artifacts are expected from Bamboo build?

extensibility.md

@@ -0,0 +1,48 @@
+# Extensibility
+
+## Using other issue tracker than JIRA
+
+The code is ready for that. You simply can:
+
+1. implement corresponding interface (i.e. `services.IssueTrackerService`)
+2. implement config parsing
+3. send a pull-request :)
+
+Note that issue tracking is a fresh feature and there might be some interface changes in the future in order to implement new features.
+
+## Adding another export than issue trackers and e-mail
+
+This might be an easy task, but it depends how are the outputs structured. E-mail notifications and issue tracker export uses much common code and new platform for export might also utilize this common code.
+
+## Using other build server than Bamboo
+
+There might be some (rather minor) issues, because our code might be slightly bound to Bamboo. We wish to help with refactoring needed for adding another build server if our code is too specific for Bamboo somewhere.
+
+## Using other vulnerability database than NVD
+
+In such case, you will probably want extend OWASP Dependency Check first. Once this is done, little or no work has to be done in this analyzer:
+
+* Other vulnerability database might theoretically add some new fields. In such case, it would be useful to parse them from the report and display them.
+* In some rare cases, ODC Analyzer provides an external link to NVD. This might be useful to refactor.
+
+## Using other scanner than OWASP Dependency Check
+
+We don't believe we will extend it in this way. However, if you have a good reason for adding other scanner than ODC, we might consider it. Adding such scanner might, however, depend on its output format. If it is similar enough to XML output from OWASP Dependency Check, it is more likely to be added.
+
+## Adding a classes in Java
+
+While most codebase is in Scala, it is acceptable to use Java (or maybe even other JVM language like Groovy, Ceylon, Kotlin, …) for some cases. General rules:
+
+* One functionality should be implemented in one language.
+* We don't want to introduce much external libraries (including standard libraries) for minor functionality.
+
+OK:
+
+* to implement export to another issue tracker in Java
+* to have export to one issue tracker implemented in one language and export to other issue tracker in another issue tracker in another language
+
+Discouraged:
+
+* to mix multiple languages in implementation of export to one issue tracker
+
+None of there rules is strict, but they are general rule-of-thumb.

functional-debt.md

@@ -0,0 +1,5 @@
+# Functional debt
+
+* Tagging requires having admin privileges. However, no privilege levels are implemented now, so no one can add a tag.
+* The application does not support adding tags.
+* The homepage should be redesigned.

production.conf-example

@@ -21,7 +21,7 @@ app{
 
 yssdc{
   # Anyone who knows the cron key can start periodic tasks
-  cronKey="{{ lookup('cron_token', 'play_secret length=64') }}"
+  cronKey="{{ lookup('password', 'cron_token length=64') }}"
   bamboo{
     url = …
   }

technical-debt.md

@@ -0,0 +1,49 @@
+# Technical debt
+
+All the technical debt should be documented here.
+
+## Checking for new versions
+This code calls some OWASP Dependency Check API. However:
+
+* I don't think that the API can be considered as public.
+* The ODC now connects to an empty H2 database, although the database is not used at all.
+
+## Caching
+
+* Caching might require large amount of RAM. In some cases, this might cause OOMs.
+* Expensive computation might be performed multiple times when another cache-miss is done before the computation is done. This might be solved by switching to spray-caching, which will however probably need a custom backend.
+* Cache does not explicitly expire. The current workaround is using the cron job for periodic refresh.
+* Manual cache control makes it inconsistent when multiple instances behind a load-balancer are used. (Note that if you require such setup, you will probably want to do few adjustments. We are OK with consulting such changes and accepting some reasonable pull-request for supporting such functionality, but we don't want to implement it ourselves, because we don't need such setup.)
+
+## Long-running operations
+
+When data is being refreshed from the build server, it might take a long time. There is no user-friendly indication of such operation. In some cases, it might even cause infamous 502 proxy timeout errors.
+
+Workaround: Use the cron task for periodic refresh.
+
+## Support for other build servers than Bamboo
+
+The ODC Analyzer was originally designed to download data from Bamboo and other build servers were not taken into the account. The code has been refactored since then. We are OK to help with further refactoring that might be needed to add support for other build servers. This will probably not be much work.
+
+## Authentication
+
+External authentication uses the e-mail as identifier and ignores username. Maybe this should be made more flexible.
+
+Note that this was a quick hack that allows e-mail notifications.
+
+## Naming
+* Library × Identifier × PlainLibraryIdentifier – should be renamed
+    * Identifier is the most verbose one, it comes from OWASP Dependency Check.
+    * Library is a record stored in our database.
+    * PlainLibraryIdentifier is just version-less (e.g. `"$groupId:$artifactId"`) library identifier.
+* Some other naming issues are described in code under TODOs.
+
+## Lack of automated tests
+
+The lack of automated tests is mainly caused by economical view. It is useful to write cheap tests for likely defects. However, it seems that most issues come from integration with external environment and it is hard to write automated tests for that, while typical candidates for unit tests is a code that does not break very much.
+
+### Some candidates for automated tests:
+
+* Exporting vulnerabilities to external systems (like mail and issue tracker).
+* Parsing output from ODC. (Maybe some integration test would be useful in order to check compatibility with newer ODC version. In contrast, unit tests are likely to be rather useless in this area.)
+* Authentication – this is area for future refactoring.