[PR #738] [MERGED] Polish test result running and reporting #702

New Issue

adam · 2025-12-30T01:26:15+01:00

adam commented

2025-12-30 01:26:15 +01:00

📋 Pull Request Information

Original PR: https://github.com/apple/pkl/pull/738
Author: @bioball
Created: 10/24/2024
Status: ✅ Merged
Merged: 10/29/2024
Merged by: @bioball

Base: main ← Head: polish-tests

📝 Commits (6)

704eadc Polish test result running and reporting
fc75b85 Consider writing tests as failures
2eaec7b rename stdErr to logs
9bd95b7 Present numbers next to failing examples
d2e7e5a Add source location to expected output
fb795c9 Update thrown example exit code, fix Windows tests

📊 Changes

15 files changed (+852 additions, -709 deletions)

View changed files

📝 docs/modules/pkl-cli/pages/index.adoc (+4 -0)
📝 pkl-cli/src/main/kotlin/org/pkl/cli/CliTestRunner.kt (+4 -1)
📝 pkl-cli/src/test/kotlin/org/pkl/cli/CliProjectPackagerTest.kt (+18 -0)
📝 pkl-cli/src/test/kotlin/org/pkl/cli/CliTestRunnerTest.kt (+120 -17)
📝 pkl-commons-cli/src/main/kotlin/org/pkl/commons/cli/CliTestException.kt (+1 -1)
📝 pkl-core/src/main/java/org/pkl/core/Evaluator.java (+2 -3)
📝 pkl-core/src/main/java/org/pkl/core/EvaluatorImpl.java (+0 -1)
➕ pkl-core/src/main/java/org/pkl/core/TestResults.java (+301 -0)
➖ pkl-core/src/main/java/org/pkl/core/runtime/TestResults.java (+0 -358)
📝 pkl-core/src/main/java/org/pkl/core/runtime/TestRunner.java (+193 -126)
📝 pkl-core/src/main/java/org/pkl/core/stdlib/test/report/JUnitReport.java (+36 -32)
📝 pkl-core/src/main/java/org/pkl/core/stdlib/test/report/SimpleReport.java (+30 -24)
📝 pkl-core/src/main/java/org/pkl/core/stdlib/test/report/TestReport.java (+1 -1)
📝 pkl-core/src/test/kotlin/org/pkl/core/EvaluateTestsTest.kt (+28 -28)
📝 pkl-gradle/src/test/kotlin/org/pkl/gradle/TestsTest.kt (+114 -117)

📄 Description

Changes:

Move class TestResults to package org.pkl.core, because it is a public class (it's the result of Evaluator#evaluateTest)
Change examples to treat individual examples as assertions in the same test. Previously, they were considered different tests with an incrementing number. This better aligns with how facts are treated.
Change TestResults to be a record, and introduce builders.
Remove "module" test result section (it is not really a section).
Add javadoc to TestResults
Formatting fix: prefix ✍️ emoji just like we do the ❌ and ✅ emojis
Consider writing examples as failures, not successes. pkl test will exit with code 2 if the only failing tests are due to writing examples. Primary motivation: not checking in pkl-expected.pcf files should not be a passing test in CI. The expected output file is part of the specification of the test, and users are meant to review them and check them into source control.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/apple/pkl/pull/738 **Author:** [@bioball](https://github.com/bioball) **Created:** 10/24/2024 **Status:** ✅ Merged **Merged:** 10/29/2024 **Merged by:** [@bioball](https://github.com/bioball) **Base:** `main` ← **Head:** `polish-tests` --- ### 📝 Commits (6) - [`704eadc`](https://github.com/apple/pkl/commit/704eadca8e6adbbe19c8c4c3c1e3141cbdf95c83) Polish test result running and reporting - [`fc75b85`](https://github.com/apple/pkl/commit/fc75b859e6c163dc75ed91a7ab06a81e44db2a15) Consider writing tests as failures - [`2eaec7b`](https://github.com/apple/pkl/commit/2eaec7b3fb9f395d20327b6f66d5aa3ba7c2322d) rename stdErr to logs - [`9bd95b7`](https://github.com/apple/pkl/commit/9bd95b7accb7a18315d040cf1b1b64a237e1decc) Present numbers next to failing examples - [`d2e7e5a`](https://github.com/apple/pkl/commit/d2e7e5af867ad91ff45a2742fe2e1dd4460a2506) Add source location to expected output - [`fb795c9`](https://github.com/apple/pkl/commit/fb795c984555584e21fc2987dc1d3e4b8d6bae7e) Update thrown example exit code, fix Windows tests ### 📊 Changes **15 files changed** (+852 additions, -709 deletions) <details> <summary>View changed files</summary> 📝 `docs/modules/pkl-cli/pages/index.adoc` (+4 -0) 📝 `pkl-cli/src/main/kotlin/org/pkl/cli/CliTestRunner.kt` (+4 -1) 📝 `pkl-cli/src/test/kotlin/org/pkl/cli/CliProjectPackagerTest.kt` (+18 -0) 📝 `pkl-cli/src/test/kotlin/org/pkl/cli/CliTestRunnerTest.kt` (+120 -17) 📝 `pkl-commons-cli/src/main/kotlin/org/pkl/commons/cli/CliTestException.kt` (+1 -1) 📝 `pkl-core/src/main/java/org/pkl/core/Evaluator.java` (+2 -3) 📝 `pkl-core/src/main/java/org/pkl/core/EvaluatorImpl.java` (+0 -1) ➕ `pkl-core/src/main/java/org/pkl/core/TestResults.java` (+301 -0) ➖ `pkl-core/src/main/java/org/pkl/core/runtime/TestResults.java` (+0 -358) 📝 `pkl-core/src/main/java/org/pkl/core/runtime/TestRunner.java` (+193 -126) 📝 `pkl-core/src/main/java/org/pkl/core/stdlib/test/report/JUnitReport.java` (+36 -32) 📝 `pkl-core/src/main/java/org/pkl/core/stdlib/test/report/SimpleReport.java` (+30 -24) 📝 `pkl-core/src/main/java/org/pkl/core/stdlib/test/report/TestReport.java` (+1 -1) 📝 `pkl-core/src/test/kotlin/org/pkl/core/EvaluateTestsTest.kt` (+28 -28) 📝 `pkl-gradle/src/test/kotlin/org/pkl/gradle/TestsTest.kt` (+114 -117) </details> ### 📄 Description Changes: * Move class `TestResults` to package `org.pkl.core`, because it is a public class (it's the result of `Evaluator#evaluateTest`) * Change examples to treat individual examples as assertions in the same test. Previously, they were considered different tests with an incrementing number. This better aligns with how facts are treated. * Change `TestResults` to be a record, and introduce builders. * Remove "module" test result section (it is not really a section). * Add javadoc to `TestResults` * Formatting fix: prefix ✍️ emoji just like we do the ❌ and ✅ emojis * Consider writing examples as failures, not successes. `pkl test` will exit with code 2 if the only failing tests are due to writing examples. Primary motivation: not checking in `pkl-expected.pcf` files should not be a passing test in CI. The expected output file is part of the specification of the test, and users are meant to review them and check them into source control. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

adam added the pull-request label 2025-12-30 01:26:15 +01:00

adam closed this issue

2025-12-30 01:26:15 +01:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/pkl#702