This introduces a new `Bytes` standard library class, for working with
binary data.
* Add Bytes class to the standard library
* Change CLI to eval `output.bytes`
* Change code generators to map Bytes to respective underlying type
* Add subscript and concat operator support
* Add binary encoding for Bytes
* Add PCF and Plist rendering for Bytes
Co-authored-by: Kushal Pisavadia <kushi.p@gmail.com>
Some systems require junit report to be in a single file. For example `bazel` https://bazel.build/reference/test-encyclopedia needs single file to be available in `XML_OUTPUT_FILE` path.
To avoid implementing junit aggregation in pkl wrappers in different places this PR instead adds a `--junit-aggregate-reports` flag to return all junit reports as a single file.
Additional flag `--junit-aggregate-suite-name` is added to allow overriding global test suite name from default `pkl-tests`
When specifying a directory instead of a file for the output, pkl won't crash anymore, but instead output an error message, suggesting --multiple-file-output-path.
This adds logic to build and publish the other executables related to Pkl.
These are:
* pkl-doc
* pkl-codegen-kotlin
* pkl-codegen-java
pkl-codegen-kotlin and pkl-codegen-java are published as executable JARs, whereas pkldoc is published both as an executable JAR, and also native executables (matching the set of os/arch supported by Pkl).
The reason this only publishes executable JARs for pkl-codegen-kotlin and pkl-codegen-java is because we expect that the Java requirement is not a problem for these users, and that the native executable provides negligible added value.
As part of this, the following changes are made:
* Introduce `pklJavaExecutable` plugin, which sets up building and publishing of executable JAR.
* Introduce `pklNativeExecutable` plugin, which sets up building and publishing of native executables.
* Introduce `NativeImageBuild` Gradle task, which knows how to build native-image executables.
* Introduce `ExecutableSpec` extension, for projects that publish executables to configure how those executables should be published.
* `./griddles buildNative`, by default, will only build the executable of the host OS/Arch, and will no longer cross-build.
* The target arch of `./gradlew buildNative` can be changed using `-Dpkl.targetArch=<aarch64|amd64>`.
* On linux/amd64 only, with `./gradlew buildNative`, a statically linked executable can be built using `-Dpkl.musl=true`
* Make `javaExecutable` a dependency of `assemble`
* Make `testStartJavaExecutable` a dependency of `check`
* Change name `pklNativeBuild` to `pklNativeLifecycle` to better match the plugin's purpose
* Remove Truffle SVM classes from main source set (don't publish these classes as part of the pkl-cli JAR)
* Change CircleCI definition to publish new executables
* Change CircleCI definition to call `buildNative` instead of individual task names
This bumps Clikt from version 3 to version 5, which, among other things, improves
the help text formatting with colors.
Also:
* Add `--version` flag to pkldoc, pkl-codegen-java, pkl-codegen-kotlin
* Add help text to pkldoc, pkl-codegen-java, pkl-codegen-kotlin
When we updated spotless's Java and Kotlin formatter, we changed the underlying
formatting rules.
However, due to spotless ratcheting, these formatting changes don't get applied unless a file
gets touched in a commit.
To avoid future PRs introducing lines of change that aren't related to the intention of the PR,
this is a one-time format of all files.
Making these classes caused native-image to statically initialize
them at build time, which included CLI argument default values
(like working dir).
This turns them back into classes.
Co-authored-by: Islon Scherer <islonscherer@gmail.com>
- update Kotlin from 1.7.10 to 2.0.21
- Kotlin 1.6 dependencies in Gradle lock files are expected because kotlinc,
which is also used by some tests, internally uses some 1.6 dependencies
for backwards compatibility reasons.
- update kotlinx-html and kotlinx-serialization
- adapt Kotlin code where necessary
- use Kotlin stdlib Path APIs where possible
- fix IntelliJ Kotlin inspection warnings
- reformat code with `./gradlew spotlessApply`
- ktfmt adds lots of trailing commas
- Add workaround to fix IntelliJ "unresolved reference" errors
Any thrown Pkl Errors are colored in the simple test report!
Also:
* Refactor `TextFormatter` to be more generic; rename to `TextFormattingStringBuilder`
* Adjust test report slightly (no emojis, add more spacing).
* Introduce `ColorTheme` class.
* Make stack frame descriptors colored as "faint"
Also: this changes the summary so it summarizes _all_ modules, rather than a summary per module.
---------
Co-authored-by: Islon Scherer <islonscherer@gmail.com>
Co-authored-by: Philip K.F. Hölzenspies <holzensp@gmail.com>
* Encapsulate message transport by removing `ExternalReaderProcess.getTransport` and adding `getModuleResolver` and `getResourceResolver` methods
* Reuse `Random` instances within `ExternalReaderProcessImpl` and module/resource resolvers
* Externalize all `ExternalReaderProcessException` messages
* Add some missing doc comments to `ModuleKeyFactories` and `ResourceReaders` methods for external readers
* Move org.pkl.core.util.Readers to org.pkl.core.Readers
Changes:
* Move class `TestResults` to package `org.pkl.core`, because it is a public class (it's the result of `Evaluator#evaluateTest`)
* Change examples to treat individual examples as assertions in the same test. Previously, they were considered different tests with an incrementing number. This better aligns with how facts are treated.
* Change `TestResults` to be a record, and introduce builders.
* Remove "module" test result section (it is not really a section).
* Add javadoc to `TestResults`
* Formatting fix: prefix ✍️ emoji just like we do the ❌ and ✅ emojis
* Consider writing examples as failures, not successes. `pkl test` will exit with code 10 if the only failing tests are due to writing examples.
This adds a new feature, which allows Pkl to read resources and modules from external processes.
Follows the design laid out in SPICE-0009.
Also, this moves most of the messaging API into pkl-core
Write annotations to project metadata, and provide them to pkl-doc\
The following annotations have meaning for pkl-doc:
* `@Unlisted`: hide package from documentation site
* `@Deprecated`: add deprecated information
This adds a new feature to build a dependency graph of Pkl programs, following the SPICE outlined in https://github.com/apple/pkl-evolution/pull/2.
It adds:
* CLI command `pkl analyze imports`
* Java API `org.pkl.core.Analyzer`
* Pkl stdlib module `pkl:analyze`
* pkl-gradle extension `analyze`
In addition, it also changes the Gradle plugin such that `transitiveModules` is by default computed from the import graph.
This changes how the language performs typechecks for mappings and
listings.
Currently, Pkl will shallow-force any Mapping and Listing to check it
the type parameter (e.g. Listing<Person> means each element is checked
to be an instance of Person).
This changes the language to check each member's type when the member
is accessed.
This also adjust test runner to handle thrown errors from within tests.
With the change to make mapping/listing typechecks lazy, we can now
correctly handle thrown errors from within a single test case.
This adjusts the test runner to consider any thrown errors as a failure
for that specific test case.
* Add `getEffectiveOutputFiles` and `getEffectiveOutputDirs` to `EvalTask`, and mark them as output files/dirs so they are tracked by Gradle. This enables implicit dependency tracking between two tasks.
* Fix usage of `file()` notation in Gradle scripts when on Windows.
To make error messages from Pkl eval easier to read, this change uses
the Jansi library to colour the output, making it quicker and easier to
scan error messages and understand what's happened.
The Jansi library also detects if the CLI output is a terminal capable
of handling colours, and will automatically strip out escape codes if
the output won't support them (e.g. piping the output somewhere else).
This addresses an issue where network requests may fail if cert revocation checks
error, which may occur due to availability issues, or due to lack of internet access.
Revocation checking can still be enabled by setting JVM property com.sun.net.ssl.checkRevocation if on the JVM.
Also:
* Load built-in certs from resources, and move them to pkl-commons-cli
* Fix an issue where HttpInitException is not caught when loading a module
Rationale: "proxy" can mean very different things (e.g. java.lang.reflect.Proxy in Java).
This makes the flag name more specific.
CLI:
* `--proxy` -> `--http-proxy`
* `--no-proxy` -> `--http-no-proxy`
Gradle:
* `proxyAddress` -> `httpProxy`
* `noProxy` -> `httpNoProxy`
Instead of bundling Pkl's built-in CA certificates as a class path resource and loading them at runtime,
pass them to the native image compiler as the default SSL context's trust store.
This results in faster SSL initialization and is more consistent with how default certificates
are handled when running on the JVM.
Further related improvements:
- Remove HttpClientBuilder methods `addDefaultCliCertificates` and `addBuiltInCertificates`.
- Remove pkl-certs subproject and the optional dependencies on it.
- Move `PklCARoots.pem` to `pkl-cli/src/certs`.
- Fix certificate related error messages that were missing an argument.
- Prevent PklBugException if initialization of `CliBaseOptions.httpClient` fails.
- Add ability to set CA certificates as a byte array
- Add CA certificates option to message passing API
* Add `--proxy` and `--no-proxy` CLI flags
* Add property `http` to `pkl:settings`
* Move `EvaluatorSettings` from `pkl:Project` to its own module and add property `http`
* Add support for proxying in server mode, and through Gradle
* Add `setProxy()` to `HttpClient`
* Add documentation
This adds support for Windows.
The in-language path separator is still `/`, to ensure Pkl programs are cross-platform.
Log lines are written using CRLF endings on Windows.
Modules that are combined with `--module-output-separator` uses LF endings to ensure
consistent rendering across platforms.
`jpkl` does not work on Windows as a direct executable.
However, it can work with `java -jar jpkl`.
Additional details:
* Adjust git settings for Windows
* Add native executable for pkl cli
* Add jdk17 windows Gradle check in CI
* Adjust CI test reports to be staged within Gradle rather than by shell script.
* Fix: encode more characters that are not safe Windows paths
* Skip running tests involving symbolic links on Windows (these require administrator privileges to run).
* Introduce custom implementation of `IoUtils.relativize`
* Allow Gradle to initialize ExecutableJar `Property` values
* Add Gradle flag to enable remote JVM debugging
Co-authored-by: Philip K.F. Hölzenspies <holzensp@gmail.com>
GenericUrl is a catch-all that uses URL.openConnection().
Since we now have special handling of HTTP urls, it makes more sense to
put it in its own module key.
This changes the file paths to use characters that are safe for Windows.
Channges the output of the following:
* Package cache directory
* Generated pkl-doc files
* Kotlin generated code
Unsafe characters are encoded as (<hex>).
For example, the colon character `:` is encoded as `(3a)`.
Additionally, this changes the cache directory prefix (package-1 to
package-2).
Follows the design of https://github.com/apple/pkl-evolution/pull/3
- Change HttpClient to follow all redirects except HTTPS to HTTP.
- Run language snippet tests with --no-cache and real PackageServer
instead of pre-seeded cache.
This increases HTTP test coverage and enables testing of package redirects.
- Change PackageServer to return 301 for request paths starting with /HTTP301/
and 307 for request paths starting with /HTTP307/.
- Update some outdated test package checksums that apparently weren't verified.
Changes the packager to exclude any empty directories.
This change means that pkl project package for an already published packages will fail. The packager checks for an existing package at this version, and compares checksums. It will then error if the checksum has changed.
This is technically a breaking change, albeit a minor one. The workaround is to publish new versions of packages.
Published packages should still be compatible with Pkl 0.25.
- Fix and clean up the pkl-commons-test build script.
- Change tests to read test packages/certs directly from
the file system instead of packaging and reading them
from the class path.
- Update expected checksums of some test packages.
- Fix a conflict between Pkl's and Gradle's
Kotlin libraries in the pkl-gradle project.
- Fix build deprecation warnings.
- Ensure Gradle distribution integrity with `distributionSha256Sum`.
- Manually verify integrity of Gradle wrapper added by this commit.
This is a comprehensive solution to the "flaky PackageServer tests"
problem. It rules out port conflicts and imposes no limits on test
parallelism. The same solution can be used for other test servers
in the future.
Major changes:
- Turn `PackageServer` from a singleton into a class that is
instantiated per test class or test method.
- Start the server the first time its `port` property is read.
Bind the server to an ephemeral port instead of port 12110.
- For every test that uses `PackageServer`, pass the server port to
`--test-port`, `HttpClient.Builder.setTestPort`, the `CliBaseOptions`
or `ExecutorOptions` constructor, or the Gradle plugin's `testPort` property.
Wire all of these to `RequestRewritingClient`'s `testPort` constructor parameter.
- Enhance `RequestRewritingClient` to replace port 12110 with `testPort`
in request URIs unless `testPort` is -1 (its default).
- Introduce `ExecutorOptions.Builder`.
This makes executor options more comfortable to create
and allows to hide options such as `testPort`.
- Deprecate the `ExecutorOptions` constructor to steer users towards the builder.
- Get rid of `ExecutorOptions2`, which is no longer needed.
- Clean up `EmbeddedExecutorTest` with the help of the builder.
Moving to java.net.http.HttpClient brings many benefits, including
HTTP/2 support and the ability to make asynchronous requests.
Major additions and changes:
- Introduce a lightweight org.pkl.core.http.HttpClient API.
This keeps some flexibility and allows to enforce behavior
such as setting the User-Agent header.
- Provide an implementation that delegates to java.net.http.HttpClient.
- Use HttpClient for all HTTP(s) requests across the codebase.
This required adding an HttpClient parameter to constructors and
factory methods of multiple classes, some of which are public APIs.
- Manage CA certificates per HTTP client instead of per JVM.
This makes it unnecessary to set JVM-wide system/security properties
and default SSLSocketFactory's.
- Add executor v2 options to the executor SPI
- Add pkl-certs as a new artifact, and remove certs from pkl-commons-cli artifact
Each HTTP client maintains its own connection pool and SSLContext.
For efficiency reasons, It's best to reuse clients whenever feasible.
To avoid memory leaks, clients are not stored in static fields.
HTTP clients are expensive to create. For this reason,
EvaluatorBuilder defaults to a "lazy" client that creates the underlying
java.net.http.HttpClient on the first send (which may never happen).
