Polish test result running and reporting (#738)

Changes:
* Move class `TestResults` to package `org.pkl.core`, because it is a public class (it's the result of `Evaluator#evaluateTest`)
* Change examples to treat individual examples as assertions in the same test. Previously, they were considered different tests with an incrementing number. This better aligns with how facts are treated.
* Change `TestResults` to be a record, and introduce builders.
* Remove "module" test result section (it is not really a section).
* Add javadoc to `TestResults`
* Formatting fix: prefix ✍️ emoji just like we do the  and  emojis 
* Consider writing examples as failures, not successes. `pkl test` will exit with code 10 if the only failing tests are due to writing examples.
This commit is contained in:
Daniel Chao
2024-10-28 21:05:13 -07:00
committed by GitHub
parent 666f8c3939
commit acd2222534
15 changed files with 851 additions and 708 deletions

View File

@@ -60,12 +60,14 @@ constructor(
val evaluator = builder.build()
evaluator.use {
var failed = false
var isExampleWrittenFailure = true
val moduleNames = mutableSetOf<String>()
for ((idx, moduleUri) in sources.withIndex()) {
try {
val results = evaluator.evaluateTest(uri(moduleUri), testOptions.overwrite)
if (!failed) {
failed = results.failed()
isExampleWrittenFailure = results.isExampleWrittenFailure.and(isExampleWrittenFailure)
}
SimpleReport().report(results, consoleWriter)
if (sources.size > 1 && idx != sources.size - 1) {
@@ -101,7 +103,8 @@ constructor(
}
}
if (failed) {
throw CliTestException(ErrorMessages.create("testsFailed"))
val exitCode = if (isExampleWrittenFailure) 10 else 1
throw CliTestException(ErrorMessages.create("testsFailed"), exitCode)
}
}
}