Sometimes we see a thrown IllegalStateException when CPU profiling,
where the underlying cause is a ConcurrentModificationException in
the underlying GraalVM internals.
This attempts to minimize the chances of this exception being thrown
by starting proiling after the evaluator and its context have been
created.
Motivation:
- VmObjectLike.iterateMemberValues() is an internal iterator whose loop body is a Java lambda.
It suffers from the following limitations:
- `continue` and `break` cannot be used.
- `return` becomes an (awkward) replacement for `continue` and `break`.
It cannot be used to return from the enclosing method.
- Mutating variables requires hoisting them (MutableBoolean, etc.).
- Node.execute() calls are very slow because they are behind a Truffle boundary.
- Callers of iterateMemberValues() do not specify what and how they want to iterate.
This obscures the iteration code and makes it difficult (often impossible) to optimize.
Changes:
- Add interface VmObjectCursor, an external iterator that iterates over VmObject members.
Its design is inspired by org.graalvm.collections.MapCursor.
- Add VmObjectCursor implementations specialized for different members, VmObject subclasses, and cursor options.
- Replace VmObjectLike.iterateMemberValues() with the following methods that return VmObjectCursor:
- properties()
- properties(CursorOption)
- elements()
- elements(CursorOption)
- entries()
- entries(CursorOption)
- members
- members(CursorOption)
- Replace all usages of iterateMemberValues() with one of the above methods.
Before: object.iterateMembersValues((key, member, value) -> { if (member.isProp()) {...} });
After : for (var cursor = object.properties(); cursor.advance(); ) {...}
Result:
- Iteration code is much simpler and clearer.
- Iteration code that calls Node.execute() is much faster because it is no longer behind a Truffle boundary.
- Other iteration code is potentially faster and can be better optimized in the future.
Co-authored-by: translatenix <119817707+translatenix@users.noreply.github.com>
Co-authored-by: Jen Basch <jbasch94@gmail.com>
This introduces a way to run CPU profiling.
The profiler produces output in pprof format.
CLI users can enable CPU profiling via command line flags.
Profiling is otherwise not offered as an API.
Co-authored-by: Jen Basch <jbasch94@gmail.com>
Co-authored-by: Islon Scherer <islonscherer@gmail.com>
* Add tasks to generate source jars for java executables
* native source bundles
* source bundles in separate plugin
* Wire up sourceBundle task
* Produce bundles on a per-variant basis (Java fat jar, each native variant)
* Introduce `pklNativeSourceBundle` and `pklJvmSourceBundle` plugins for handling fat jar vs. GraalVM bits
* Download compiled classes list, and exclude Java sources for executable stuff specifically
* Include native sources when appropriate (GraalVM, JDK, zlib, musl, libpkl)
---------
Co-authored-by: Jen Basch <jbasch@apple.com>
This adds an option which allows users to control exactly how
`@ConfigurationProperties` annotations are attached to classes.
This helps when:
* Spring Boot users don't have a single root class, but multiple
classes that are injected in disparate places
* Pkl's module structure doesn't match the resulting `.properties`
layout
This changes it to render the qualified name instead of the simple name.
* Module classes render the module name
* Classes within module render the module name and the class name joined by `#`
* Rendering of members of `pkl:base` does not change
For example:
* `new k8s.core.v1.Service {}` instead of `new ModuleClass {}`.
* `new k8s.core.v1.Service#ServiceSpec {}` instead of `new ServiceSpec {}`
* `new pkl.Project {}` instead of `new ModuleClass {}`
This affects the following:
* `toString()`
* `trace()`
* Error messages
* REPL
Patterns that originate from the language runtime will add
`Pattern.UNICODE_CHARACTER_CLASS | Pattern.UNICODE_CASE` as implicit
regex flags, so we need to do the same on the binary decoding side.
* Compute a derived regex expression with `\z` anchor
* Ensure derived regex can't be messed up with comments (in extended mode) or quote start
Follow up to #1880
This adds logic to UnresolvedTypeNode to create a `UnionOfStringLiterals`
type node if a member is a typealias who is a string or a string literal union.
Also, this fixes an issue where marking a duplicate string literal with the
default marker throws an index out of bounds exception.
This makes various improvements to help truffle do partial
evaluation of code related to typechecking.
* Split `UnionTypeNode` into `UnionTypeNodeExploded` and
`UnionTypeNodeLooped` to avoid gigantic graphs. This especially helps
in the case of typealiases, because they are inlined into their usage
sites.
* Move constraint-failure exception construction in
`TypeConstraintNode` behind `@TruffleBoundary` helper methods
instead of `transferToInterpreterAndInvalidate`, since building the
exception is a rare, off-hot-path operation.
* Move `ClassNode`'s one-time class-initialization logic behind a
`@TruffleBoundary` method (`initClass`), store `VmLanguage` as a
field instead of recomputing it via `VmLanguage.get(this)`, and use
`transferToInterpreterAndInvalidate` since class init never needs to
re-run on the compiled path.
* Pass `VmLanguage` explicitly into `TypeTestNode` and `ClassNode`
(via `AstBuilder`) rather than resolving it with
`VmLanguage.get(this)` on a hot path.
* Fix various issues where the partial evaluator determines that a frame
should have been materialized but was not.
* Pre-initialize call targets in `ObjectMember.setMemberNode()` and
`FunctionLiteralNode` so the first invocation doesn't trigger a
compilation invalidation.
* Fix truffle node tree issues (don't mark `@RootNode` as child,
deepCopy instead of passing an already adopted node).
* Remove unnecessary interpreter transfers when throwing type mismatch exceptions
This changes the `join` method in `List`/`Set`/`Listing` to
call the overridden `toString()` method.
This is a breaking change, but is considered a bugfix.
Fix this printed warning during java compilation:
"Note: Some input files use unchecked or unsafe operations. Recompile with -Xlint:unchecked for details"
* Add `-Xlint:unchecked` as a javac flag
* Fix the various warnings
This changes the logic for inferring `new {}` expressions in method
arguments to using regular frame slots, instead of auxiliary slots.
Overall, this is a performance improvement: no looking up auxiliary slot
numbers, which makes this code much more PE friendly.
* At parse time, get or create a frame slot that holds the method
* Introduce `ArgInfo` record (preserve primitive int) over using `Pair`
* Remove unneeded `isImplicitNewExpr` method; this is not needed because
we can just keep track of the method slot on the frame descriptor
builder.
This changes code so that the in-language `toString()` doesn't use
Java's `Object.toString()`.
Additionally, the existing `toString()` implementation is switched over
to using `VmValueRenderer`, which does not force (leaves `?` in place of
unforced values).
This helps improve ergonomics when working on the language (can enable
automatically calling `toString()` on debug values).
Additionally, external tools might call toString(); for example,
Truffle's compiler sometimes will stringify node fields (e.g.
when -Dengine.CompilationFailureAction=Throw) is set.
Fix invalid prefix when generating spring boot config
In Spring Boot, the prefix passed to `@ConfigurationProperties` must
be in "canonical form".
The presence of this annotation allows spring boot to directly inject
properties into a class, but is not actually needed.
The correct behavior here is to omit this annotation if the property
name does not match Spring Boot's canonical form.
* In GeneratorWhenNode, need to move individual iterations into their
own methods so that the @ExplodeLoop sees a constant number of
iterations whenever that method is called.
* In the other nodes, the lambda passed to
`forceAndIterateMemberValues` needs to capture the materialized frame,
not the virtual frame.
This skips some unnecessary steps when evaluating annotations that
have no attached object bodies by returning an EmptyObjectLiteralNode
instead of a PropertiesLiteralNode
GraalVM's partial evaulator can't handle recursive calls into the same
node.
The current implementation around evaluating `VmNull` as a parent causes
the partial evaluator to bail out, leaving Pkl stuck in interpreter
mode.
This rewrites the various object literals to add individual
specializations for each type of default value we can see from a VmNull
parent.
Also:
* Fix `isTypeObjectClass` impl
* Fix bug when amending with generator object literal node with object params
The truffle compiler currently _always_ bails out of compilation for
`VmUtils.getFrame()` calls, because it determines that some branch
of this code can possibly return a materialized frame.
This adjusts the code so that the hot path (levelsUp == 0)
does not touch `VmUtils.getFrame()`, and avoids unnecessarily
materializing the frame.
This rewrites the listing/mapping typecast nodes to be more friendly
to partial-evaluation.
This is because the compiler bails out of deeply recursive calls.
This moves the protobuf module from experimental to deprecated
Rationale:
* This is very seldom used, if used at all
* Not fully fleshed out (this really should be paired with a protoc-gen-pkl, which doesn't exist)
* There is a ProtoJSON wire format that we can already render to
This changes logic that previously read/wrote from `~/.pkl` to use
XDG base directories (all OSes), and Known Folders locations on Windows.
For example, Pkl will look for `settings.pkl` in:
1. `$XDG_CONFIG_HOME/pkl/settings.pkl`
2. `%APPDATA/pkl/settings.pkl`
3. `~/.pkl/settings.pkl`
4. Path pkl/settings/pkl within `$XDG_CONFIG_DIRS`
5. `/etc/xdg/pkl/settings.pkl`
---------
Co-authored-by: Florin Ungur <florin@florinungur.com>
This adds a `libpkl-static.pc` so pkg-config users can link either
statically or dynamically via `pkg-config --libs libpkl`, or
`pkg-config --libs libpkl-static`.
Additionally, this fixes an issue where the replace tokens wasn't
doing anything, leaving the `@version@` token intact in the archive.
Additionally, this adds an `-install_name` when building the dylib for macOS.
Co-authored-by: Islon Scherer <islonscherer@gmail.com>
* Fix possible SIGSEGV from host process.
- Require that the same `pkl_exec_t` be used in the same OS thread; returning meaningful error if this fails
* Remove `System.exitProcess(1)` logic in NativeTransport; this would kill the host process too and isn't an appropriate action for a received ProtocolException
* Allow multiple calls to pkl_init without pkl_close; this limitation doesn't really make any sense
* Prefer calling methods defined in graal_isolate.h, instead of creating the same method via CEntryPoint
* Add a CMakeLists.txt so that CLion can follow the code and provide proper diagnostics
* Improve doc comments; describe params as either in or out
This adds a new Antora module for libpkl, and updates the language
bindings documentation.
Co-authored-by: Jen Basch <jbasch94@gmail.com>
Co-authored-by: Islon Scherer <islonscherer@gmail.com>
This simplifies the resolution of names in typealias constraints.
This removes the existing logic around swapping out the frame's
owner/receiver, and instead favors resolving the variable at parse time.
* During variable resolution, create read variable nodes that read off of
a receiver node
* Introduce `GetTypeAliasModuleNode` that provides the typealias's
enclosing module
* Inject the enclosing module during typealias instantiation
* Fix issue where the DeployJob and GithubRelease jobs download
artifacts using a glob that only matches one artifact
* Make NativeImageBuild task only produce outputs as declared by build;
ensure there's no build_artifacts.txt, sources/ dir, etc.
This uses native-image to generate a C library for Pkl.
This generated library from native-image is wrapped with our own library,
in `pkl.h`.
This produces a static and a dynamic library for each os/arch variant
that Pkl currently supports.
Co-authored-by: Kushal Pisavadia <kushal.p@apple.com>
Co-authored-by: Jen Basch <jbasch94@gmail.com>
Co-authored-by: Islon Scherer <i_desouzascherer@apple.com>
This fixes a regression where local members cannot be seen by the
expression evaluator.
Because variable are now parse-time resolved, this builds a truffle node
by first constructing a synthetic module of all defined local members,
and calls AstBuilder on this synthetic module first.
This changes local asset path resolution so that `..` segments at the
dependency root just resolves to the dependency root.
This makes import resolution work the same between local and remote dependencies.
This changes the following:
1. If given a union type, the member must exist on each member of that type
2. If accessing a member off `Reference<D, Null>`, give a `Reference<D, Null>`
Also:
* Improve error messages thrown during member access.
* Add test around accessing members off of a function type
Stdlib modules are singletons that are shared across multiple evaluators.
This improves thread safety by evaluating its `output.bytes` during
initialization, which initializes truffle nodes (e.g. TypeTestNode),
and also initializes the member cache of the module output of `pkl:base`
This addresses many IDE warnings resulting from the switch to JSpecify.
Also, this changes the behavior of exporting VmObject; there's no place
in our code that does not force a VmObject prior to export, so
the existing logic around handling nullable values has been removed.
This adjusts the doc comments in ref.pkl
* Fix incorrect code snippets
* Clean up examples
* Remove some sections
* Make phrasing consistent with the rest of the stdlib
The changes made in this commit are good, but we're going to kick this out
to the next release.
This is because:
1. There's a couple more issues around the `abstract` modifier that is not
implemented yet, and need design considerations
2. These are breaking changes, and we want to minimize the amount of breakages
for users.
3. The main branch is still the develop branch for Pkl 0.32
We will apply a re-revert of this commit after Pkl 0.32 is released.
These methods aren't implemented in `Set`, and don't really make sense because `Set` types can't be accessed by index.
Note: although this removes methods, this actually isn't a breaking change:
1. Calling `Set.findIndex()` currently throws an error around "cannot invoke abstract method"
2. `List` and `Set` are the only subclasses of `Collection`.
The following code isn't breaking at runtime, although static analysis tooling (like our IDE plugins) will now flag this as an error:
```pkl
myCollection: Collection
idx = myCollection.indexOf(1)
```
This makes various improvements to the handling of frame slot vars, and
includes some bug fixes introduced by
https://github.com/apple/pkl/pull/1622
* Refactor SymbolTable to track for-generator and parameter slots in
each scope
* Execute let expressions in their own root node in some places
* Unify how frame slots are managed; they are all represented as
`FrameSlotVariable`, created in `AstBuilder`, and passed into
`SymbolTable`.
* Fix how let expressions are executed in custom this scopes (introduce
a new root node when needed)
This implements HTTP redirect following ourselves.
The goal is:
1. All I/O is checked against `--allowed-resources` and
`--allowed-modules`, including HTTP redirects
2. HTTP rewrite rules can affect redirect following
3. HTTP headers can affect redirect following
---------
Co-authored-by: Islon Scherer <islonscherer@gmail.com>
This reverts the commits that enabled Gradle's configuration cache
feature.
IMO: this feature is too hard to use. We don't know if a task is valid
for the configuration cache until it runs, and it's very hard to tell if
something is safe when authoring Gradle code.
For example, our publish tasks are currently failing; I don't know how I
would fix this without running the publish task again on my dev machine.
Also, some of our build scripts become more brittle because of this; for
example, see
https://github.com/apple/pkl/blob/bb07589eae0b3195a589559a3245cbc12c29b394/build-logic/src/main/kotlin/BuildInfo.kt#L291-L296
This introduces breaking changes for external readers are loaded:
1. In PklProject, relative paths are resolved relative to the enclosing
PklProject file (make behavior consistent with how other settings work)
2. Make CLI flags blow away any settings set on a PklProject
3. Introduce a new `workingDir` property, which defaults to the
PklProject dir
The overall goal is to make this behavior consistent with how other
settings work.
For example, relative paths for other evaluator settings are already
relative to the project directory.
Additionally, in every other case, CLI flags will overwrite any setting
set within PklProject.
This introduces an optimization to write let expressions to frame slots
of the current frame, rather than transforming them into lambdas. As a
result, let expressions are _much_ faster to execute.
Fixes the following error:
```
A problem was found with the configuration of task ':pkl-core:sourcesJar' (type 'Jar').
Deprecated Gradle features were used in this build, making it incompatible with Gradle 10.
- Gradle detected a problem with the following location: '/home/runner/work/pkl/pkl/pkl-core/build/generated/sources/baseModuleMembers'.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
Reason: Task ':pkl-core:sourcesJar' uses this output of task ':pkl-core:generateBaseModuleMembers' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed.
For more on this, please refer to https://docs.gradle.org/9.5.1/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
Possible solutions:
94 actionable tasks: 76 executed, 18 from cache
1. Declare task ':pkl-core:generateBaseModuleMembers' as an input of ':pkl-core:sourcesJar'.
2. Declare an explicit dependency on ':pkl-core:generateBaseModuleMembers' from ':pkl-core:sourcesJar' using Task#dependsOn.
3. Declare an explicit dependency on ':pkl-core:generateBaseModuleMembers' from ':pkl-core:sourcesJar' using Task#mustRunAfter.
```
- Remove single usage of @immutable without replacement
- Remove HttpClient's usages of @threadsafe without replacement
- Replace javax.annotation.concurrent.GuardedBy
with com.google.errorprone.annotations.concurrent.GuardedBy
Also:
- Remove redundant final modifiers from members of a final class
---------
Co-authored-by: odenix <self@odenix.org>
Daniel Chao
