Add analyze imports libs (SPICE-0001) (#695)

This adds a new feature to build a dependency graph of Pkl programs, following the SPICE outlined in https://github.com/apple/pkl-evolution/pull/2.

It adds:
* CLI command `pkl analyze imports`
* Java API `org.pkl.core.Analyzer`
* Pkl stdlib module `pkl:analyze`
* pkl-gradle extension `analyze`

In addition, it also changes the Gradle plugin such that `transitiveModules` is by default computed from the import graph.
This commit is contained in:
Daniel Chao
2024-10-23 14:36:57 -07:00
committed by GitHub
parent eb3891b21f
commit ce25cb8ef0
53 changed files with 2054 additions and 53 deletions

View File

@@ -5407,7 +5407,7 @@ package {
packageZipUrl = "https://example.com/\(name)/\(name)@\(version).zip" // <4> packageZipUrl = "https://example.com/\(name)/\(name)@\(version).zip" // <4>
} }
---- ----
<1> The display name of the package. For display purposes only. <1> The display name of the package.For display purposes only.
<2> The package URI, without the version part. <2> The package URI, without the version part.
<3> The version of the package. <3> The version of the package.
<4> The URL to download the package's ZIP file. <4> The URL to download the package's ZIP file.
@@ -5415,8 +5415,9 @@ package {
The package itself is created by the command xref:pkl-cli:index.adoc#command-project-package[`pkl project package`]. The package itself is created by the command xref:pkl-cli:index.adoc#command-project-package[`pkl project package`].
This command only prepares artifacts to be published. This command only prepares artifacts to be published.
Once the artifacts are prepared, they are expected to be uploaded to an HTTPS server such that the ZIP asset can be downloaded at path `packageZipUrl`, and the metadata can be downloaded at `+https://<package uri>+`. Once the artifacts are prepared, they are expected to be uploaded to an HTTPS server such that the ZIP asset can be downloaded at path `packageZipUrl`, and the metadata can be downloaded at `+https://<package uri>+`.
[[local-dependencies]]
==== Local dependencies ==== Local dependencies
A project can depend on a local project as a dependency. A project can depend on a local project as a dependency.

View File

@@ -558,14 +558,64 @@ package already exists in the cache directory, this command is a no-op.
This command accepts <<common-options,common options>>. This command accepts <<common-options,common options>>.
[[command-analyze-imports]]
=== `pkl analyze imports`
*Synopsis*: `pkl analyze imports [<modules>]`
This command builds a graph of imports declared in the provided modules.
This is a lower level command that is meant to be useful for Pkl-related tooling.
For example, this command feeds into the xref:pkl-gradle:index.adoc[] to determine if tasks are considered up-to-date or not.
This command produces an object with two properties, `imports` and `resolvedImports`.
The `imports` property is a mapping of a module's absolute URI, to the set of imports declared within that module.
The `resolvedImports` property is a mapping of a module's absolute URI (as stated in `imports`), to the resolved absolute URI that might be useful for fetching the module's contents.
For example, a xref:language-reference:index.adoc#local-dependencies[local dependency] import will have an in-language URI with scheme `projectpackage:`, and may have resolved URI with scheme `file:` (assuming that the project is file-based).
Examples:
[source,shell]
----
# Analyze the imports of a single module
pkl analyze imports myModule.pkl
# Same as the previous command, but output in JSON.
pkl analyze imports -f json myModule.pkl
# Analyze imports of all modules declared within src/
pkl analyze imports src/*.pkl
----
<modules>::
The absolute or relative URIs of the modules to analyze. Relative URIs are resolved against the working directory.
==== Options
.-f, --format
[%collapsible]
====
Same meaning as <<format>> in <<command-eval>>.
====
.-o, --output-path
[%collapsible]
====
Same meaning as <<output-path>> in <<command-eval>>.
====
This command also takes <<common-options,common options>>.
[[common-options]] [[common-options]]
=== Common options === Common options
The <<command-eval>>, <<command-test>>, <<command-repl>>, <<command-project-resolve>>, <<command-project-package>>, and <<command-download-package>> commands support the following common options: The <<command-eval>>, <<command-test>>, <<command-repl>>, <<command-project-resolve>>, <<command-project-package>>, <<command-download-package>>, and <<command-analyze-imports>> commands support the following common options:
include::../../pkl-cli/partials/cli-common-options.adoc[] include::../../pkl-cli/partials/cli-common-options.adoc[]
The <<command-eval>>, <<command-test>>, <<command-repl>>, and <<command-download-package>> commands also take the following options: The <<command-eval>>, <<command-test>>, <<command-repl>>, <<command-download-package>>, and <<command-analyze-imports>> commands also take the following options:
include::../../pkl-cli/partials/cli-project-options.adoc[] include::../../pkl-cli/partials/cli-project-options.adoc[]

View File

@@ -7,7 +7,6 @@ Comma-separated list of URI patterns that determine which modules can be loaded
Patterns are matched against the beginning of module URIs. Patterns are matched against the beginning of module URIs.
(File paths have been converted to `file:` URLs at this stage.) (File paths have been converted to `file:` URLs at this stage.)
At least one pattern needs to match for a module to be loadable. At least one pattern needs to match for a module to be loadable.
Both source modules and transitive modules are subject to this check.
==== ====
[[allowed-resources]] [[allowed-resources]]

View File

@@ -102,7 +102,6 @@ pkl {
evaluators { evaluators {
evalPkl { evalPkl {
sourceModules.add(file("module1.pkl")) sourceModules.add(file("module1.pkl"))
transitiveModules.from file("module2.pkl")
outputFile = layout.buildDirectory.file("module1.yaml") outputFile = layout.buildDirectory.file("module1.yaml")
outputFormat = "yaml" outputFormat = "yaml"
} }
@@ -118,7 +117,6 @@ pkl {
evaluators { evaluators {
register("evalPkl") { register("evalPkl") {
sourceModules.add(file("module1.pkl")) sourceModules.add(file("module1.pkl"))
transitiveModules.from(file("module2.pkl"))
outputFile.set(layout.buildDirectory.file("module1.yaml")) outputFile.set(layout.buildDirectory.file("module1.yaml"))
outputFormat.set("yaml") outputFormat.set("yaml")
} }
@@ -127,9 +125,6 @@ pkl {
---- ----
==== ====
To guarantee correct Gradle up-to-date behavior,
`transitiveModules` needs to contain all module files transitively referenced by `sourceModules`.
For each declared evaluator, the Pkl plugin creates an equally named task. For each declared evaluator, the Pkl plugin creates an equally named task.
Hence the above evaluator can be run with: Hence the above evaluator can be run with:
@@ -691,3 +686,61 @@ The project directories to create packages for.
Common properties: Common properties:
include::../partials/gradle-common-properties.adoc[] include::../partials/gradle-common-properties.adoc[]
[[analyze-imports]]
== Analyze Imports
This feature is the Gradle analogy for the xref:pkl-cli:index.adoc#command-analyze-imports[analyze imports] command in the CLI. It builds a graph of imports of the provided source modules.
=== Usage
[tabs]
====
build.gradle::
+
[source,groovy]
----
pkl {
analyzers {
imports {
appConfig {
sourceModules.add(file("src/main/resources/appConfig.pkl"))
}
}
}
}
----
build.gradle.kts::
+
[source,kotlin]
----
pkl {
analyzers {
imports {
register("appConfig") {
sourceModules.add(file("src/main/resources/appConfig.pkl"))
}
}
}
}
----
====
=== Configuration Options
.outputFormat: Property<String>
[%collapsible]
====
Same meaning as <<output-format,outputFormat>> in <<module-evaluation>>.
====
.outputFile: RegularFileProperty<String>
[%collapsible]
====
Same meaning as <<output-file,outputFile>> in <<module-evaluation>>.
====
Common properties:
include::../partials/gradle-modules-properties.adoc[]

View File

@@ -7,7 +7,6 @@ URI patterns that determine which modules can be loaded and evaluated.
Patterns are matched against the beginning of module URIs. Patterns are matched against the beginning of module URIs.
(File paths have been converted to `file:` URLs at this stage.) (File paths have been converted to `file:` URLs at this stage.)
At least one pattern needs to match for a module to be loadable. At least one pattern needs to match for a module to be loadable.
Both source modules and transitive modules are subject to this check.
==== ====
.allowedResources: ListProperty<String> .allowedResources: ListProperty<String>

View File

@@ -20,11 +20,17 @@ This property accepts the following types to represent a module:
.transitiveModules: ConfigurableFileCollection .transitiveModules: ConfigurableFileCollection
[%collapsible] [%collapsible]
==== ====
Default: `files()` (empty collection) + Default: [computed by pkl-gradle] +
Example 1: `transitiveModules.from files("module1.pkl", "module2.pkl")` + Example 1: `transitiveModules.from files("module1.pkl", "module2.pkl")` +
Example 2: `+transitiveModules.from fileTree("config").include("**/*.pkl")+` + Example 2: `+transitiveModules.from fileTree("config").include("**/*.pkl")+` +
File paths of modules that are directly or indirectly used by source modules. File paths of modules that are directly or indirectly used by source modules.
Setting this option enables correct Gradle up-to-date checks, which ensures that your Pkl tasks are executed if any of the transitive files are modified; it does not affect evaluation otherwise.
This property, along with `sourceModules`, is the set of input files used to determine whether this task is up-to-date or not.
By default, Pkl computes this property by analyzing the imports of the source modules.
Setting this property explicitly causes Pkl to skip the analyze imports step.
Including source modules in `transitiveModules` is permitted but not required. Including source modules in `transitiveModules` is permitted but not required.
Relative paths are resolved against the project directory. Relative paths are resolved against the project directory.
==== ====

View File

@@ -0,0 +1,79 @@
/**
* Copyright © 2024 Apple Inc. and the Pkl project authors. All rights reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.pkl.cli
import java.io.Writer
import org.pkl.commons.cli.CliCommand
import org.pkl.commons.createParentDirectories
import org.pkl.commons.writeString
import org.pkl.core.ModuleSource
import org.pkl.core.module.ModuleKeyFactories
class CliImportAnalyzer
@JvmOverloads
constructor(
private val options: CliImportAnalyzerOptions,
private val consoleWriter: Writer = System.out.writer()
) : CliCommand(options.base) {
override fun doRun() {
val rendered = render()
if (options.outputPath != null) {
options.outputPath.createParentDirectories()
options.outputPath.writeString(rendered)
} else {
consoleWriter.write(rendered)
consoleWriter.flush()
}
}
// language=pkl
private val sourceModule =
ModuleSource.text(
"""
import "pkl:analyze"
local importStrings = read*("prop:pkl.analyzeImports.**").toMap().values.toSet()
output {
value = analyze.importGraph(importStrings)
renderer {
converters {
[Map] = (it) -> it.toMapping()
[Set] = (it) -> it.toListing()
}
}
}
"""
.trimIndent()
)
private fun render(): String {
val builder = evaluatorBuilder().setOutputFormat(options.outputFormat)
try {
return builder
.apply {
for ((idx, sourceModule) in options.base.normalizedSourceModules.withIndex()) {
addExternalProperty("pkl.analyzeImports.$idx", sourceModule.toString())
}
}
.build()
.use { it.evaluateOutputText(sourceModule) }
} finally {
ModuleKeyFactories.closeQuietly(builder.moduleKeyFactories)
}
}
}

View File

@@ -0,0 +1,34 @@
/**
* Copyright © 2024 Apple Inc. and the Pkl project authors. All rights reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.pkl.cli
import java.nio.file.Path
import org.pkl.commons.cli.CliBaseOptions
data class CliImportAnalyzerOptions(
/** Base options shared between CLI commands. */
val base: CliBaseOptions,
/** The file path where the output file is placed. */
val outputPath: Path? = null,
/**
* The output format to generate.
*
* These accept the same options as [CliEvaluatorOptions.outputFormat].
*/
val outputFormat: String? = null,
)

View File

@@ -34,7 +34,8 @@ internal fun main(args: Array<String>) {
ServerCommand(helpLink), ServerCommand(helpLink),
TestCommand(helpLink), TestCommand(helpLink),
ProjectCommand(helpLink), ProjectCommand(helpLink),
DownloadPackageCommand(helpLink) DownloadPackageCommand(helpLink),
AnalyzeCommand(helpLink),
) )
.main(args) .main(args)
} }

View File

@@ -0,0 +1,66 @@
/**
* Copyright © 2024 Apple Inc. and the Pkl project authors. All rights reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.pkl.cli.commands
import com.github.ajalt.clikt.core.NoOpCliktCommand
import com.github.ajalt.clikt.core.subcommands
import com.github.ajalt.clikt.parameters.options.option
import com.github.ajalt.clikt.parameters.types.path
import java.nio.file.Path
import org.pkl.cli.CliImportAnalyzer
import org.pkl.cli.CliImportAnalyzerOptions
import org.pkl.commons.cli.commands.ModulesCommand
import org.pkl.commons.cli.commands.single
class AnalyzeCommand(helpLink: String) :
NoOpCliktCommand(
name = "analyze",
help = "Commands related to static analysis",
epilog = "For more information, visit $helpLink"
) {
init {
subcommands(AnalyzeImportsCommand(helpLink))
}
companion object {
class AnalyzeImportsCommand(helpLink: String) :
ModulesCommand(
name = "imports",
helpLink = helpLink,
help = "Prints the the graph of modules imported by the input module(s)."
) {
private val outputPath: Path? by
option(
names = arrayOf("-o", "--output-path"),
metavar = "<path>",
help = "File path where the output file is placed."
)
.path()
.single()
override fun run() {
val options =
CliImportAnalyzerOptions(
base = baseOptions.baseOptions(modules, projectOptions),
outputFormat = baseOptions.format,
outputPath = outputPath
)
CliImportAnalyzer(options).run()
}
}
}
}

View File

@@ -0,0 +1,142 @@
/**
* Copyright © 2024 Apple Inc. and the Pkl project authors. All rights reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.pkl.cli
import java.net.URI
import java.nio.file.Path
import org.assertj.core.api.Assertions.assertThat
import org.assertj.core.api.Assertions.assertThatCode
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.io.TempDir
import org.pkl.commons.cli.CliBaseOptions
import org.pkl.commons.writeString
import org.pkl.core.OutputFormat
import org.pkl.core.util.StringBuilderWriter
class CliImportAnalyzerTest {
@Test
fun `write to console writer`(@TempDir tempDir: Path) {
val file = tempDir.resolve("test.pkl").writeString("import \"bar.pkl\"")
val otherFile = tempDir.resolve("bar.pkl").writeString("")
val baseOptions = CliBaseOptions(sourceModules = listOf(file.toUri()))
val sb = StringBuilder()
val analyzer = CliImportAnalyzer(CliImportAnalyzerOptions(baseOptions), StringBuilderWriter(sb))
analyzer.run()
assertThat(sb.toString())
.isEqualTo(
"""
imports {
["${otherFile.toUri()}"] {}
["${file.toUri()}"] {
new {
uri = "${otherFile.toUri()}"
}
}
}
resolvedImports {
["${otherFile.toUri()}"] = "${otherFile.toRealPath().toUri()}"
["${file.toUri()}"] = "${file.toRealPath().toUri()}"
}
"""
.trimIndent()
)
}
@Test
fun `different output format`(@TempDir tempDir: Path) {
val file = tempDir.resolve("test.pkl").writeString("import \"bar.pkl\"")
val otherFile = tempDir.resolve("bar.pkl").writeString("")
val baseOptions = CliBaseOptions(sourceModules = listOf(file.toUri()))
val sb = StringBuilder()
val analyzer =
CliImportAnalyzer(
CliImportAnalyzerOptions(baseOptions, outputFormat = OutputFormat.JSON.toString()),
StringBuilderWriter(sb)
)
analyzer.run()
assertThat(sb.toString())
.isEqualTo(
"""
{
"imports": {
"${otherFile.toUri()}": [],
"${file.toUri()}": [
{
"uri": "${otherFile.toUri()}"
}
]
},
"resolvedImports": {
"${otherFile.toUri()}": "${otherFile.toRealPath().toUri()}",
"${file.toUri()}": "${file.toRealPath().toUri()}"
}
}
"""
.trimIndent()
)
}
@Test
fun `write to output file`(@TempDir tempDir: Path) {
val file = tempDir.resolve("test.pkl").writeString("import \"bar.pkl\"")
val otherFile = tempDir.resolve("bar.pkl").writeString("")
val outputPath = tempDir.resolve("imports.pcf")
val baseOptions = CliBaseOptions(sourceModules = listOf(file.toUri()))
val analyzer = CliImportAnalyzer(CliImportAnalyzerOptions(baseOptions, outputPath = outputPath))
analyzer.run()
assertThat(outputPath)
.hasContent(
"""
imports {
["${otherFile.toUri()}"] {}
["${file.toUri()}"] {
new {
uri = "${otherFile.toUri()}"
}
}
}
resolvedImports {
["${otherFile.toUri()}"] = "${otherFile.toRealPath().toUri()}"
["${file.toUri()}"] = "${file.toRealPath().toUri()}"
}
"""
.trimIndent()
)
}
@Test
fun `invalid syntax in module`(@TempDir tempDir: Path) {
val file = tempDir.resolve("test.pkl").writeString("foo = bar(]")
assertThatCode {
CliImportAnalyzer(
CliImportAnalyzerOptions(
CliBaseOptions(sourceModules = listOf(file.toUri()), settings = URI("pkl:settings"))
)
)
.run()
}
.hasMessageContaining(
"""
Pkl Error
Found a syntax error when parsing module `${file.toUri()}`.
"""
.trimIndent()
)
}
}

View File

@@ -27,6 +27,7 @@ import org.junit.jupiter.api.assertThrows
import org.junit.jupiter.api.condition.DisabledOnOs import org.junit.jupiter.api.condition.DisabledOnOs
import org.junit.jupiter.api.condition.OS import org.junit.jupiter.api.condition.OS
import org.junit.jupiter.api.io.TempDir import org.junit.jupiter.api.io.TempDir
import org.pkl.cli.commands.AnalyzeCommand
import org.pkl.cli.commands.EvalCommand import org.pkl.cli.commands.EvalCommand
import org.pkl.cli.commands.RootCommand import org.pkl.cli.commands.RootCommand
import org.pkl.commons.writeString import org.pkl.commons.writeString
@@ -34,7 +35,8 @@ import org.pkl.commons.writeString
class CliMainTest { class CliMainTest {
private val evalCmd = EvalCommand("") private val evalCmd = EvalCommand("")
private val cmd = RootCommand("pkl", "pkl version 1", "").subcommands(evalCmd) private val analyzeCommand = AnalyzeCommand("")
private val cmd = RootCommand("pkl", "pkl version 1", "").subcommands(evalCmd, analyzeCommand)
@Test @Test
fun `duplicate CLI option produces meaningful error message`(@TempDir tempDir: Path) { fun `duplicate CLI option produces meaningful error message`(@TempDir tempDir: Path) {

View File

@@ -0,0 +1,120 @@
/**
* Copyright © 2024 Apple Inc. and the Pkl project authors. All rights reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.pkl.core;
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import java.nio.file.Path;
import java.util.Collection;
import java.util.List;
import java.util.Map;
import org.graalvm.polyglot.Context;
import org.pkl.core.http.HttpClient;
import org.pkl.core.http.HttpClientInitException;
import org.pkl.core.module.ModuleKeyFactory;
import org.pkl.core.module.ProjectDependenciesManager;
import org.pkl.core.packages.PackageLoadError;
import org.pkl.core.packages.PackageResolver;
import org.pkl.core.project.DeclaredDependencies;
import org.pkl.core.runtime.ModuleResolver;
import org.pkl.core.runtime.ResourceManager;
import org.pkl.core.runtime.VmContext;
import org.pkl.core.runtime.VmException;
import org.pkl.core.runtime.VmImportAnalyzer;
import org.pkl.core.runtime.VmUtils;
import org.pkl.core.util.Nullable;
/** Utility library for static analysis of Pkl programs. */
public class Analyzer {
private final StackFrameTransformer transformer;
private final SecurityManager securityManager;
private final @Nullable Path moduleCacheDir;
private final @Nullable DeclaredDependencies projectDependencies;
private final ModuleResolver moduleResolver;
private final HttpClient httpClient;
public Analyzer(
StackFrameTransformer transformer,
SecurityManager securityManager,
Collection<ModuleKeyFactory> moduleKeyFactories,
@Nullable Path moduleCacheDir,
@Nullable DeclaredDependencies projectDependencies,
HttpClient httpClient) {
this.transformer = transformer;
this.securityManager = securityManager;
this.moduleCacheDir = moduleCacheDir;
this.projectDependencies = projectDependencies;
this.moduleResolver = new ModuleResolver(moduleKeyFactories);
this.httpClient = httpClient;
}
/**
* Builds a graph of imports from the provided source modules.
*
* <p>For details, see {@link ImportGraph}.
*/
public ImportGraph importGraph(URI... sources) {
var context = createContext();
try {
context.enter();
var vmContext = VmContext.get(null);
return VmImportAnalyzer.analyze(sources, vmContext);
} catch (SecurityManagerException
| IOException
| URISyntaxException
| PackageLoadError
| HttpClientInitException e) {
throw new PklException(e.getMessage(), e);
} catch (PklException err) {
throw err;
} catch (VmException err) {
throw err.toPklException(transformer);
} catch (Exception e) {
throw new PklBugException(e);
} finally {
context.leave();
context.close();
}
}
private Context createContext() {
var packageResolver =
PackageResolver.getInstance(
securityManager, HttpClient.builder().buildLazily(), moduleCacheDir);
return VmUtils.createContext(
() -> {
VmContext vmContext = VmContext.get(null);
vmContext.initialize(
new VmContext.Holder(
transformer,
securityManager,
httpClient,
moduleResolver,
new ResourceManager(securityManager, List.of()),
Loggers.stdErr(),
Map.of(),
Map.of(),
moduleCacheDir,
null,
packageResolver,
projectDependencies == null
? null
: new ProjectDependenciesManager(
projectDependencies, moduleResolver, securityManager)));
});
}
}

View File

@@ -433,6 +433,10 @@ public final class EvaluatorBuilder {
return this; return this;
} }
public @Nullable DeclaredDependencies getProjectDependencies() {
return this.dependencies;
}
/** /**
* Given a project, sets its dependencies, and also applies any evaluator settings if set. * Given a project, sets its dependencies, and also applies any evaluator settings if set.
* *

View File

@@ -0,0 +1,111 @@
/**
* Copyright © 2024 Apple Inc. and the Pkl project authors. All rights reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.pkl.core;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.Map;
import java.util.Set;
import java.util.TreeMap;
import java.util.TreeSet;
import org.pkl.core.util.json.Json;
import org.pkl.core.util.json.Json.FormatException;
import org.pkl.core.util.json.Json.JsArray;
import org.pkl.core.util.json.Json.JsObject;
import org.pkl.core.util.json.Json.JsonParseException;
import org.pkl.core.util.json.Json.MappingException;
/**
* Java representation of {@code pkl.analyze#ImportGraph}.
*
* @param imports The graph of imports declared within the program.
* <p>Each key is a module inside the program, and each value is the module URIs declared as
* imports inside that module. The set of all dependent modules within a program is the set of
* keys in this map.
* @param resolvedImports A mapping of a module's in-language URI, and the URI that it resolves to.
* <p>For example, a local package dependency is represented with scheme {@code
* projectpackage:}, and (typically) resolves to a {@code file:} scheme.
*/
public record ImportGraph(Map<URI, Set<Import>> imports, Map<URI, URI> resolvedImports) {
/**
* Java representation of {@code pkl.analyze#Import}.
*
* @param uri The absolute URI of the import.
*/
public record Import(URI uri) implements Comparable<Import> {
@Override
public int compareTo(Import o) {
return uri.compareTo(o.uri());
}
}
/** Parses the provided JSON into an import graph. */
public static ImportGraph parseFromJson(String input) throws JsonParseException {
var parsed = Json.parseObject(input);
var imports = parseImports(parsed.getObject("imports"));
var resolvedImports = parseResolvedImports(parsed.getObject("resolvedImports"));
return new ImportGraph(imports, resolvedImports);
}
private static Map<URI, Set<Import>> parseImports(Json.JsObject jsObject)
throws JsonParseException {
var ret = new TreeMap<URI, Set<Import>>();
for (var entry : jsObject.entrySet()) {
try {
var key = new URI(entry.getKey());
var value = entry.getValue();
var set = new TreeSet<Import>();
if (!(value instanceof JsArray array)) {
throw new FormatException("array", value.getClass());
}
for (var elem : array) {
if (!(elem instanceof JsObject importObj)) {
throw new FormatException("object", elem.getClass());
}
set.add(parseImport(importObj));
}
ret.put(key, set);
} catch (URISyntaxException e) {
throw new MappingException(entry.getKey(), e);
}
}
return ret;
}
private static ImportGraph.Import parseImport(Json.JsObject jsObject) throws JsonParseException {
var uri = jsObject.getURI("uri");
return new Import(uri);
}
private static Map<URI, URI> parseResolvedImports(Json.JsObject jsObject)
throws JsonParseException {
var ret = new TreeMap<URI, URI>();
for (var entry : jsObject.entrySet()) {
try {
var key = new URI(entry.getKey());
var value = entry.getValue();
if (!(value instanceof String str)) {
throw new FormatException("string", value.getClass());
}
var valueUri = new URI(str);
ret.put(key, valueUri);
} catch (URISyntaxException e) {
throw new MappingException(entry.getKey(), e);
}
}
return ret;
}
}

View File

@@ -21,9 +21,12 @@ import java.io.IOException;
import java.util.ArrayList; import java.util.ArrayList;
import java.util.Collections; import java.util.Collections;
import java.util.List; import java.util.List;
import org.pkl.core.ast.builder.ImportsAndReadsParser.Entry;
import org.pkl.core.module.ModuleKey; import org.pkl.core.module.ModuleKey;
import org.pkl.core.module.ResolvedModuleKey; import org.pkl.core.module.ResolvedModuleKey;
import org.pkl.core.parser.LexParseException;
import org.pkl.core.parser.Parser; import org.pkl.core.parser.Parser;
import org.pkl.core.parser.antlr.PklLexer;
import org.pkl.core.parser.antlr.PklParser.ImportClauseContext; import org.pkl.core.parser.antlr.PklParser.ImportClauseContext;
import org.pkl.core.parser.antlr.PklParser.ImportExprContext; import org.pkl.core.parser.antlr.PklParser.ImportExprContext;
import org.pkl.core.parser.antlr.PklParser.ModuleExtendsOrAmendsClauseContext; import org.pkl.core.parser.antlr.PklParser.ModuleExtendsOrAmendsClauseContext;
@@ -31,8 +34,8 @@ import org.pkl.core.parser.antlr.PklParser.ReadExprContext;
import org.pkl.core.parser.antlr.PklParser.SingleLineStringLiteralContext; import org.pkl.core.parser.antlr.PklParser.SingleLineStringLiteralContext;
import org.pkl.core.runtime.VmExceptionBuilder; import org.pkl.core.runtime.VmExceptionBuilder;
import org.pkl.core.runtime.VmUtils; import org.pkl.core.runtime.VmUtils;
import org.pkl.core.util.IoUtils;
import org.pkl.core.util.Nullable; import org.pkl.core.util.Nullable;
import org.pkl.core.util.Pair;
/** /**
* Collects module uris and resource uris imported within a module. * Collects module uris and resource uris imported within a module.
@@ -46,17 +49,29 @@ import org.pkl.core.util.Pair;
* <li>read expressions * <li>read expressions
* </ul> * </ul>
*/ */
public final class ImportsAndReadsParser public class ImportsAndReadsParser extends AbstractAstBuilder<@Nullable List<Entry>> {
extends AbstractAstBuilder<@Nullable List<Pair<String, SourceSection>>> {
public record Entry(
boolean isModule,
boolean isGlob,
boolean isExtends,
boolean isAmends,
String stringValue,
SourceSection sourceSection) {}
/** Parses a module, and collects all imports and reads. */ /** Parses a module, and collects all imports and reads. */
public static @Nullable List<Pair<String, SourceSection>> parse( public static @Nullable List<Entry> parse(
ModuleKey moduleKey, ResolvedModuleKey resolvedModuleKey) throws IOException { ModuleKey moduleKey, ResolvedModuleKey resolvedModuleKey) throws IOException {
var parser = new Parser(); var parser = new Parser();
var text = resolvedModuleKey.loadSource(); var text = resolvedModuleKey.loadSource();
var source = VmUtils.createSource(moduleKey, text); var source = VmUtils.createSource(moduleKey, text);
var importListParser = new ImportsAndReadsParser(source); var importListParser = new ImportsAndReadsParser(source);
return parser.parseModule(text).accept(importListParser); try {
return parser.parseModule(text).accept(importListParser);
} catch (LexParseException e) {
var moduleName = IoUtils.inferModuleName(moduleKey);
throw VmUtils.toVmException(e, source, moduleName);
}
} }
public ImportsAndReadsParser(Source source) { public ImportsAndReadsParser(Source source) {
@@ -69,29 +84,35 @@ public final class ImportsAndReadsParser
} }
@Override @Override
public List<Pair<String, SourceSection>> visitModuleExtendsOrAmendsClause( public @Nullable List<Entry> visitModuleExtendsOrAmendsClause(
ModuleExtendsOrAmendsClauseContext ctx) { ModuleExtendsOrAmendsClauseContext ctx) {
var importStr = doVisitSingleLineConstantStringPart(ctx.stringConstant().ts); var importStr = doVisitSingleLineConstantStringPart(ctx.stringConstant().ts);
var sourceSection = createSourceSection(ctx.stringConstant()); var sourceSection = createSourceSection(ctx.stringConstant());
return Collections.singletonList(Pair.of(importStr, sourceSection)); return Collections.singletonList(
new Entry(
true, false, ctx.EXTENDS() != null, ctx.AMENDS() != null, importStr, sourceSection));
} }
@Override @Override
public List<Pair<String, SourceSection>> visitImportClause(ImportClauseContext ctx) { public List<Entry> visitImportClause(ImportClauseContext ctx) {
var importStr = doVisitSingleLineConstantStringPart(ctx.stringConstant().ts); var importStr = doVisitSingleLineConstantStringPart(ctx.stringConstant().ts);
var sourceSection = createSourceSection(ctx.stringConstant()); var sourceSection = createSourceSection(ctx.stringConstant());
return Collections.singletonList(Pair.of(importStr, sourceSection)); return Collections.singletonList(
new Entry(
true, ctx.t.getType() == PklLexer.IMPORT_GLOB, false, false, importStr, sourceSection));
} }
@Override @Override
public List<Pair<String, SourceSection>> visitImportExpr(ImportExprContext ctx) { public List<Entry> visitImportExpr(ImportExprContext ctx) {
var importStr = doVisitSingleLineConstantStringPart(ctx.stringConstant().ts); var importStr = doVisitSingleLineConstantStringPart(ctx.stringConstant().ts);
var sourceSection = createSourceSection(ctx.stringConstant()); var sourceSection = createSourceSection(ctx.stringConstant());
return Collections.singletonList(Pair.of(importStr, sourceSection)); return Collections.singletonList(
new Entry(
true, ctx.t.getType() == PklLexer.IMPORT_GLOB, false, false, importStr, sourceSection));
} }
@Override @Override
public List<Pair<String, SourceSection>> visitReadExpr(ReadExprContext ctx) { public List<Entry> visitReadExpr(ReadExprContext ctx) {
var expr = ctx.expr(); var expr = ctx.expr();
if (!(expr instanceof SingleLineStringLiteralContext slCtx)) { if (!(expr instanceof SingleLineStringLiteralContext slCtx)) {
return Collections.emptyList(); return Collections.emptyList();
@@ -111,20 +132,26 @@ public final class ImportsAndReadsParser
} else { } else {
return Collections.emptyList(); return Collections.emptyList();
} }
return Collections.singletonList(Pair.of(importString, createSourceSection(slCtx))); return Collections.singletonList(
new Entry(
false,
ctx.t.getType() == PklLexer.READ_GLOB,
false,
false,
importString,
createSourceSection(slCtx)));
} }
@Override @Override
protected @Nullable List<Pair<String, SourceSection>> aggregateResult( protected @Nullable List<Entry> aggregateResult(
@Nullable List<Pair<String, SourceSection>> aggregate, @Nullable List<Entry> aggregate, @Nullable List<Entry> nextResult) {
@Nullable List<Pair<String, SourceSection>> nextResult) {
if (aggregate == null || aggregate.isEmpty()) { if (aggregate == null || aggregate.isEmpty()) {
return nextResult; return nextResult;
} }
if (nextResult == null || nextResult.isEmpty()) { if (nextResult == null || nextResult.isEmpty()) {
return aggregate; return aggregate;
} }
var ret = new ArrayList<Pair<String, SourceSection>>(aggregate.size() + nextResult.size()); var ret = new ArrayList<Entry>(aggregate.size() + nextResult.size());
ret.addAll(aggregate); ret.addAll(aggregate);
ret.addAll(nextResult); ret.addAll(nextResult);
return ret; return ret;

View File

@@ -199,6 +199,10 @@ public final class ProjectDependenciesManager {
} }
} }
public DeclaredDependencies getDeclaredDependencies() {
return declaredDependencies;
}
public Dependency getResolvedDependency(PackageUri packageUri) { public Dependency getResolvedDependency(PackageUri packageUri) {
var dep = getProjectDeps().get(CanonicalPackageUri.fromPackageUri(packageUri)); var dep = getProjectDeps().get(CanonicalPackageUri.fromPackageUri(packageUri));
if (dep == null) { if (dep == null) {

View File

@@ -15,7 +15,6 @@
*/ */
package org.pkl.core.project; package org.pkl.core.project;
import com.oracle.truffle.api.source.SourceSection;
import java.io.IOException; import java.io.IOException;
import java.io.OutputStream; import java.io.OutputStream;
import java.io.UncheckedIOException; import java.io.UncheckedIOException;
@@ -64,7 +63,6 @@ import org.pkl.core.util.GlobResolver;
import org.pkl.core.util.GlobResolver.InvalidGlobPatternException; import org.pkl.core.util.GlobResolver.InvalidGlobPatternException;
import org.pkl.core.util.IoUtils; import org.pkl.core.util.IoUtils;
import org.pkl.core.util.Nullable; import org.pkl.core.util.Nullable;
import org.pkl.core.util.Pair;
/** /**
* Given a list of project directories, prepares artifacts to be published as a package. * Given a list of project directories, prepares artifacts to be published as a package.
@@ -397,8 +395,8 @@ public final class ProjectPackager {
return; return;
} }
for (var importContext : imports) { for (var importContext : imports) {
var importStr = importContext.first; var importStr = importContext.stringValue();
var sourceSection = importContext.second; var sourceSection = importContext.sourceSection();
if (isAbsoluteImport(importStr)) { if (isAbsoluteImport(importStr)) {
continue; continue;
} }
@@ -440,7 +438,7 @@ public final class ProjectPackager {
} }
} }
private @Nullable List<Pair<String, SourceSection>> getImportsAndReads(Path pklModulePath) { private @Nullable List<ImportsAndReadsParser.Entry> getImportsAndReads(Path pklModulePath) {
try { try {
var moduleKey = ModuleKeys.file(pklModulePath.toUri()); var moduleKey = ModuleKeys.file(pklModulePath.toUri());
var resolvedModuleKey = ResolvedModuleKeys.file(moduleKey, moduleKey.getUri(), pklModulePath); var resolvedModuleKey = ResolvedModuleKeys.file(moduleKey, moduleKey.getUri(), pklModulePath);

View File

@@ -0,0 +1,53 @@
/**
* Copyright © 2024 Apple Inc. and the Pkl project authors. All rights reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.pkl.core.runtime;
import com.oracle.truffle.api.CompilerDirectives.TruffleBoundary;
import java.net.URI;
public class AnalyzeModule extends StdLibModule {
private static final VmTyped instance = VmUtils.createEmptyModule();
static {
loadModule(URI.create("pkl:analyze"), instance);
}
public static VmTyped getModule() {
return instance;
}
public static VmClass getImportGraphClass() {
return AnalyzeModule.ImportGraphClass.instance;
}
public static VmClass getImportClass() {
return AnalyzeModule.ImportClass.instance;
}
private static final class ImportGraphClass {
static final VmClass instance = loadClass("ImportGraph");
}
private static final class ImportClass {
static final VmClass instance = loadClass("Import");
}
@TruffleBoundary
private static VmClass loadClass(String className) {
var theModule = getModule();
return (VmClass) VmUtils.readMember(theModule, Identifier.get(className));
}
}

View File

@@ -85,6 +85,8 @@ public final class ModuleCache {
// some standard library modules are cached as static singletons // some standard library modules are cached as static singletons
// and hence aren't parsed/initialized anew for every evaluator // and hence aren't parsed/initialized anew for every evaluator
switch (moduleName) { switch (moduleName) {
case "analyze":
return AnalyzeModule.getModule();
case "base": case "base":
// always needed // always needed
return BaseModule.getModule(); return BaseModule.getModule();

View File

@@ -94,6 +94,7 @@ public abstract class VmException extends AbstractTruffleException {
public enum Kind { public enum Kind {
EVAL_ERROR, EVAL_ERROR,
UNDEFINED_VALUE, UNDEFINED_VALUE,
WRAPPED,
BUG BUG
} }

View File

@@ -53,6 +53,7 @@ public final class VmExceptionBuilder {
private @Nullable Object receiver; private @Nullable Object receiver;
private @Nullable Map<CallTarget, StackFrame> insertedStackFrames; private @Nullable Map<CallTarget, StackFrame> insertedStackFrames;
private VmException wrappedException;
public static class MultilineValue { public static class MultilineValue {
private final Iterable<?> lines; private final Iterable<?> lines;
@@ -332,6 +333,12 @@ public final class VmExceptionBuilder {
return this; return this;
} }
public VmExceptionBuilder wrapping(VmException nestedException) {
this.wrappedException = nestedException;
this.kind = VmException.Kind.WRAPPED;
return this;
}
public VmExceptionBuilder withInsertedStackFrames( public VmExceptionBuilder withInsertedStackFrames(
Map<CallTarget, StackFrame> insertedStackFrames) { Map<CallTarget, StackFrame> insertedStackFrames) {
this.insertedStackFrames = insertedStackFrames; this.insertedStackFrames = insertedStackFrames;
@@ -383,6 +390,19 @@ public final class VmExceptionBuilder {
memberName, memberName,
hint, hint,
effectiveInsertedStackFrames); effectiveInsertedStackFrames);
case WRAPPED ->
new VmWrappedEvalException(
message,
cause,
isExternalMessage,
messageArguments,
programValues,
location,
sourceSection,
memberName,
hint,
effectiveInsertedStackFrames,
wrappedException);
}; };
} }

View File

@@ -19,6 +19,7 @@ import com.oracle.truffle.api.CompilerDirectives.TruffleBoundary;
import java.io.PrintWriter; import java.io.PrintWriter;
import java.net.URLEncoder; import java.net.URLEncoder;
import java.nio.charset.StandardCharsets; import java.nio.charset.StandardCharsets;
import java.util.stream.Collectors;
import org.pkl.core.Release; import org.pkl.core.Release;
import org.pkl.core.util.ErrorMessages; import org.pkl.core.util.ErrorMessages;
import org.pkl.core.util.Nullable; import org.pkl.core.util.Nullable;
@@ -46,7 +47,7 @@ public final class VmExceptionRenderer {
if (exception instanceof VmBugException bugException) { if (exception instanceof VmBugException bugException) {
renderBugException(bugException, builder); renderBugException(bugException, builder);
} else { } else {
renderException(exception, builder); renderException(exception, builder, true);
} }
} }
@@ -66,13 +67,13 @@ public final class VmExceptionRenderer {
.replaceAll("\\+", "%20")); .replaceAll("\\+", "%20"));
builder.append("\n\n"); builder.append("\n\n");
renderException(exception, builder); renderException(exception, builder, true);
builder.append('\n').append(Release.current().versionInfo()).append("\n\n"); builder.append('\n').append(Release.current().versionInfo()).append("\n\n");
exceptionToReport.printStackTrace(new PrintWriter(new StringBuilderWriter(builder))); exceptionToReport.printStackTrace(new PrintWriter(new StringBuilderWriter(builder)));
} }
private void renderException(VmException exception, StringBuilder builder) { private void renderException(VmException exception, StringBuilder builder, boolean withHeader) {
var header = " Pkl Error "; var header = " Pkl Error ";
String message; String message;
@@ -94,7 +95,16 @@ public final class VmExceptionRenderer {
message = exception.getMessage(); message = exception.getMessage();
} }
builder.append(header).append('\n').append(message).append('\n'); if (withHeader) {
builder.append(header).append('\n');
}
builder.append(message).append('\n');
if (exception instanceof VmWrappedEvalException vmWrappedEvalException) {
var sb = new StringBuilder();
renderException(vmWrappedEvalException.getWrappedException(), sb, false);
hint = sb.toString().lines().map((it) -> ">\t" + it).collect(Collectors.joining("\n"));
}
// include cause's message unless it's the same as this exception's message // include cause's message unless it's the same as this exception's message
if (exception.getCause() != null) { if (exception.getCause() != null) {

View File

@@ -0,0 +1,125 @@
/**
* Copyright © 2024 Apple Inc. and the Pkl project authors. All rights reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.pkl.core.runtime;
import com.oracle.truffle.api.CompilerDirectives.TruffleBoundary;
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.TreeMap;
import java.util.TreeSet;
import org.pkl.core.ImportGraph;
import org.pkl.core.ImportGraph.Import;
import org.pkl.core.SecurityManager;
import org.pkl.core.SecurityManagerException;
import org.pkl.core.ast.builder.ImportsAndReadsParser;
import org.pkl.core.ast.builder.ImportsAndReadsParser.Entry;
import org.pkl.core.util.GlobResolver;
import org.pkl.core.util.GlobResolver.InvalidGlobPatternException;
import org.pkl.core.util.GlobResolver.ResolvedGlobElement;
import org.pkl.core.util.IoUtils;
public class VmImportAnalyzer {
@TruffleBoundary
public static ImportGraph analyze(URI[] moduleUris, VmContext context)
throws IOException, URISyntaxException, SecurityManagerException {
var imports = new TreeMap<URI, Set<ImportGraph.Import>>();
var resolvedImports = new TreeMap<URI, URI>();
for (var moduleUri : moduleUris) {
analyzeSingle(moduleUri, context, imports, resolvedImports);
}
return new ImportGraph(imports, resolvedImports);
}
@TruffleBoundary
private static void analyzeSingle(
URI moduleUri,
VmContext context,
Map<URI, Set<ImportGraph.Import>> imports,
Map<URI, URI> resolvedImports)
throws IOException, URISyntaxException, SecurityManagerException {
var moduleResolver = context.getModuleResolver();
var securityManager = context.getSecurityManager();
var importsInModule = collectImports(moduleUri, moduleResolver, securityManager);
imports.put(moduleUri, importsInModule);
resolvedImports.put(
moduleUri, moduleResolver.resolve(moduleUri).resolve(securityManager).getUri());
for (var imprt : importsInModule) {
if (imports.containsKey(imprt.uri())) {
continue;
}
analyzeSingle(imprt.uri(), context, imports, resolvedImports);
}
}
private static Set<ImportGraph.Import> collectImports(
URI moduleUri, ModuleResolver moduleResolver, SecurityManager securityManager)
throws IOException, URISyntaxException, SecurityManagerException {
var moduleKey = moduleResolver.resolve(moduleUri);
var resolvedModuleKey = moduleKey.resolve(securityManager);
List<Entry> importsAndReads;
try {
importsAndReads = ImportsAndReadsParser.parse(moduleKey, resolvedModuleKey);
} catch (VmException err) {
throw new VmExceptionBuilder()
.evalError("cannotAnalyzeBecauseSyntaxError", moduleKey.getUri())
.wrapping(err)
.build();
}
if (importsAndReads == null) {
return Set.of();
}
var result = new TreeSet<ImportGraph.Import>();
for (var entry : importsAndReads) {
if (!entry.isModule()) {
continue;
}
if (entry.isGlob()) {
var theModuleKey =
moduleResolver.resolve(moduleKey.resolveUri(IoUtils.toUri(entry.stringValue())));
try {
var elements =
GlobResolver.resolveGlob(
securityManager,
theModuleKey,
moduleKey,
moduleKey.getUri(),
entry.stringValue());
var globImports =
elements.values().stream()
.map(ResolvedGlobElement::getUri)
.map(ImportGraph.Import::new)
.toList();
result.addAll(globImports);
} catch (InvalidGlobPatternException e) {
throw new VmExceptionBuilder()
.evalError("invalidGlobPattern", entry.stringValue())
.withSourceSection(entry.sourceSection())
.build();
}
} else {
var resolvedUri =
IoUtils.resolve(securityManager, moduleKey, IoUtils.toUri(entry.stringValue()));
result.add(new Import(resolvedUri));
}
}
return result;
}
}

View File

@@ -0,0 +1,59 @@
/**
* Copyright © 2024 Apple Inc. and the Pkl project authors. All rights reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.pkl.core.runtime;
import com.oracle.truffle.api.CallTarget;
import com.oracle.truffle.api.nodes.Node;
import com.oracle.truffle.api.source.SourceSection;
import java.util.List;
import java.util.Map;
import org.pkl.core.StackFrame;
import org.pkl.core.util.Nullable;
public class VmWrappedEvalException extends VmEvalException {
private final VmException wrappedException;
public VmWrappedEvalException(
String message,
@Nullable Throwable cause,
boolean isExternalMessage,
Object[] messageArguments,
List<ProgramValue> programValues,
@Nullable Node location,
@Nullable SourceSection sourceSection,
@Nullable String memberName,
@Nullable String hint,
Map<CallTarget, StackFrame> insertedStackFrames,
VmException wrappedException) {
super(
message,
cause,
isExternalMessage,
messageArguments,
programValues,
location,
sourceSection,
memberName,
hint,
insertedStackFrames);
this.wrappedException = wrappedException;
}
public VmException getWrappedException() {
return wrappedException;
}
}

View File

@@ -0,0 +1,99 @@
/**
* Copyright © 2024 Apple Inc. and the Pkl project authors. All rights reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.pkl.core.stdlib.analyze;
import com.oracle.truffle.api.CompilerDirectives.TruffleBoundary;
import com.oracle.truffle.api.dsl.Specialization;
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import org.pkl.core.ImportGraph;
import org.pkl.core.ImportGraph.Import;
import org.pkl.core.SecurityManagerException;
import org.pkl.core.packages.PackageLoadError;
import org.pkl.core.runtime.AnalyzeModule;
import org.pkl.core.runtime.VmContext;
import org.pkl.core.runtime.VmImportAnalyzer;
import org.pkl.core.runtime.VmMap;
import org.pkl.core.runtime.VmSet;
import org.pkl.core.runtime.VmTyped;
import org.pkl.core.stdlib.ExternalMethod1Node;
import org.pkl.core.stdlib.VmObjectFactory;
public final class AnalyzeNodes {
private AnalyzeNodes() {}
private static VmObjectFactory<Import> importFactory =
new VmObjectFactory<Import>(AnalyzeModule::getImportClass)
.addStringProperty("uri", (it) -> it.uri().toString());
private static VmObjectFactory<ImportGraph> importGraphFactory =
new VmObjectFactory<ImportGraph>(AnalyzeModule::getImportGraphClass)
.addMapProperty(
"imports",
graph -> {
var builder = VmMap.builder();
for (var entry : graph.imports().entrySet()) {
var vmSetBuilder = VmSet.EMPTY.builder();
for (var imprt : entry.getValue()) {
vmSetBuilder.add(importFactory.create(imprt));
}
builder.add(entry.getKey().toString(), vmSetBuilder.build());
}
return builder.build();
})
.addMapProperty(
"resolvedImports",
graph -> {
var builder = VmMap.builder();
for (var entry : graph.resolvedImports().entrySet()) {
builder.add(entry.getKey().toString(), entry.getValue().toString());
}
return builder.build();
});
public abstract static class importGraph extends ExternalMethod1Node {
@Specialization
@TruffleBoundary
protected Object eval(@SuppressWarnings("unused") VmTyped self, VmSet moduleUris) {
var uris = new URI[moduleUris.getLength()];
var idx = 0;
for (var moduleUri : moduleUris) {
URI uri;
try {
uri = new URI((String) moduleUri);
} catch (URISyntaxException e) {
throw exceptionBuilder()
.evalError("invalidModuleUri", moduleUri)
.withHint(e.getMessage())
.build();
}
if (!uri.isAbsolute()) {
throw exceptionBuilder().evalError("cannotAnalyzeRelativeModuleUri", moduleUri).build();
}
uris[idx] = uri;
idx++;
}
var context = VmContext.get(this);
try {
var results = VmImportAnalyzer.analyze(uris, context);
return importGraphFactory.create(results);
} catch (IOException | URISyntaxException | SecurityManagerException | PackageLoadError e) {
throw exceptionBuilder().withCause(e).build();
}
}
}
}

View File

@@ -0,0 +1,4 @@
@NonnullByDefault
package org.pkl.core.stdlib.analyze;
import org.pkl.core.util.NonnullByDefault;

View File

@@ -635,6 +635,9 @@ Expected an exception, but none was thrown.
cannotEvaluateRelativeModuleUri=\ cannotEvaluateRelativeModuleUri=\
Cannot evaluate relative module URI `{0}`. Cannot evaluate relative module URI `{0}`.
cannotAnalyzeRelativeModuleUri=\
Cannot analyze relative module URI `{0}`.
invalidModuleUri=\ invalidModuleUri=\
Module URI `{0}` has invalid syntax. Module URI `{0}` has invalid syntax.
@@ -1060,3 +1063,6 @@ To fix this problem, add dependendy `org.pkl:pkl-certs`.
# suppress inspection "HttpUrlsUsage" # suppress inspection "HttpUrlsUsage"
malformedProxyAddress=\ malformedProxyAddress=\
Malformed proxy URI (expecting `http://<host>[:<port>]`): `{0}`. Malformed proxy URI (expecting `http://<host>[:<port>]`): `{0}`.
cannotAnalyzeBecauseSyntaxError=\
Found a syntax error when parsing module `{0}`.

View File

@@ -0,0 +1 @@
import "b.pkl"

View File

@@ -0,0 +1 @@
import "cyclicalB.pkl"

View File

@@ -0,0 +1 @@
import "cyclicalA.pkl"

View File

@@ -0,0 +1 @@
import* "[ab].pkl"

View File

@@ -0,0 +1,33 @@
amends "../snippetTest.pkl"
import "pkl:analyze"
import "pkl:reflect"
import ".../input-helper/analyze/a.pkl"
import ".../input-helper/analyze/cyclicalA.pkl"
import ".../input-helper/analyze/globImport.pkl"
examples {
["basic"] {
analyze.importGraph(Set(reflect.Module(a).uri))
}
["cycles"] {
analyze.importGraph(Set(reflect.Module(cyclicalA).uri))
}
["globs"] {
analyze.importGraph(Set(reflect.Module(globImport).uri))
}
["packages"] {
analyze.importGraph(Set("package://localhost:0/birds@0.5.0#/Bird.pkl"))
}
}
output {
renderer {
// mimick result of `pkl analyze imports` CLI command
converters {
[Map] = (it) -> it.toMapping()
[Set] = (it) -> it.toListing()
}
}
}

View File

@@ -0,0 +1,3 @@
import "pkl:analyze"
result = analyze.importGraph(Set("http://localhost:0/foo.pkl"))

View File

@@ -0,0 +1,3 @@
import "pkl:analyze"
result = analyze.importGraph(Set("foo <>"))

View File

@@ -0,0 +1,3 @@
import "pkl:analyze"
result = analyze.importGraph(Set("foo.pkl"))

View File

@@ -0,0 +1,79 @@
examples {
["basic"] {
new {
imports {
["file:///$snippetsDir/input-helper/analyze/a.pkl"] {
new {
uri = "file:///$snippetsDir/input-helper/analyze/b.pkl"
}
}
["file:///$snippetsDir/input-helper/analyze/b.pkl"] {}
}
resolvedImports {
["file:///$snippetsDir/input-helper/analyze/a.pkl"] = "file:///$snippetsDir/input-helper/analyze/a.pkl"
["file:///$snippetsDir/input-helper/analyze/b.pkl"] = "file:///$snippetsDir/input-helper/analyze/b.pkl"
}
}
}
["cycles"] {
new {
imports {
["file:///$snippetsDir/input-helper/analyze/cyclicalA.pkl"] {
new {
uri = "file:///$snippetsDir/input-helper/analyze/cyclicalB.pkl"
}
}
["file:///$snippetsDir/input-helper/analyze/cyclicalB.pkl"] {
new {
uri = "file:///$snippetsDir/input-helper/analyze/cyclicalA.pkl"
}
}
}
resolvedImports {
["file:///$snippetsDir/input-helper/analyze/cyclicalA.pkl"] = "file:///$snippetsDir/input-helper/analyze/cyclicalA.pkl"
["file:///$snippetsDir/input-helper/analyze/cyclicalB.pkl"] = "file:///$snippetsDir/input-helper/analyze/cyclicalB.pkl"
}
}
}
["globs"] {
new {
imports {
["file:///$snippetsDir/input-helper/analyze/a.pkl"] {
new {
uri = "file:///$snippetsDir/input-helper/analyze/b.pkl"
}
}
["file:///$snippetsDir/input-helper/analyze/b.pkl"] {}
["file:///$snippetsDir/input-helper/analyze/globImport.pkl"] {
new {
uri = "file:///$snippetsDir/input-helper/analyze/a.pkl"
}
new {
uri = "file:///$snippetsDir/input-helper/analyze/b.pkl"
}
}
}
resolvedImports {
["file:///$snippetsDir/input-helper/analyze/a.pkl"] = "file:///$snippetsDir/input-helper/analyze/a.pkl"
["file:///$snippetsDir/input-helper/analyze/b.pkl"] = "file:///$snippetsDir/input-helper/analyze/b.pkl"
["file:///$snippetsDir/input-helper/analyze/globImport.pkl"] = "file:///$snippetsDir/input-helper/analyze/globImport.pkl"
}
}
}
["packages"] {
new {
imports {
["package://localhost:0/birds@0.5.0#/Bird.pkl"] {
new {
uri = "package://localhost:0/fruit@1.0.5#/Fruit.pkl"
}
}
["package://localhost:0/fruit@1.0.5#/Fruit.pkl"] {}
}
resolvedImports {
["package://localhost:0/birds@0.5.0#/Bird.pkl"] = "package://localhost:0/birds@0.5.0#/Bird.pkl"
["package://localhost:0/fruit@1.0.5#/Fruit.pkl"] = "package://localhost:0/fruit@1.0.5#/Fruit.pkl"
}
}
}
}

View File

@@ -0,0 +1,10 @@
Pkl Error
HTTP/1.1 header parser received no bytes
x | result = analyze.importGraph(Set("http://localhost:0/foo.pkl"))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
at analyzeInvalidHttpModule#result (file:///$snippetsDir/input/errors/analyzeInvalidHttpModule.pkl)
xxx | text = renderer.renderDocument(value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
at pkl.base#Module.output.text (pkl:base)

View File

@@ -0,0 +1,12 @@
Pkl Error
Module URI `foo <>` has invalid syntax.
x | result = analyze.importGraph(Set("foo <>"))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
at analyzeInvalidModuleUri#result (file:///$snippetsDir/input/errors/analyzeInvalidModuleUri.pkl)
Illegal character in path at index 3: foo <>
xxx | text = renderer.renderDocument(value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
at pkl.base#Module.output.text (pkl:base)

View File

@@ -0,0 +1,10 @@
Pkl Error
Cannot analyze relative module URI `foo.pkl`.
x | result = analyze.importGraph(Set("foo.pkl"))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
at analyzeRelativeModuleUri#result (file:///$snippetsDir/input/errors/analyzeRelativeModuleUri.pkl)
xxx | text = renderer.renderDocument(value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
at pkl.base#Module.output.text (pkl:base)

View File

@@ -6,6 +6,7 @@ x | import "pkl:nonExisting"
at cannotFindStdLibModule#nonExisting (file:///$snippetsDir/input/errors/cannotFindStdLibModule.pkl) at cannotFindStdLibModule#nonExisting (file:///$snippetsDir/input/errors/cannotFindStdLibModule.pkl)
Available standard library modules: Available standard library modules:
pkl:analyze
pkl:base pkl:base
pkl:Benchmark pkl:Benchmark
pkl:DocPackageInfo pkl:DocPackageInfo

View File

@@ -0,0 +1,318 @@
/**
* Copyright © 2024 Apple Inc. and the Pkl project authors. All rights reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.pkl.core
import java.net.URI
import java.nio.file.Path
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.Test
import org.junit.jupiter.api.io.TempDir
import org.pkl.commons.createParentDirectories
import org.pkl.commons.test.PackageServer
import org.pkl.commons.writeString
import org.pkl.core.http.HttpClient
import org.pkl.core.module.ModuleKeyFactories
import org.pkl.core.project.Project
class AnalyzerTest {
private val simpleAnalyzer =
Analyzer(
StackFrameTransformers.defaultTransformer,
SecurityManagers.defaultManager,
listOf(ModuleKeyFactories.file, ModuleKeyFactories.standardLibrary, ModuleKeyFactories.pkg),
null,
null,
HttpClient.dummyClient()
)
@Test
fun `simple case`(@TempDir tempDir: Path) {
val file =
tempDir
.resolve("test.pkl")
.writeString(
"""
amends "pkl:base"
import "pkl:json"
myProp = import("pkl:xml")
"""
.trimIndent()
)
.toUri()
val result = simpleAnalyzer.importGraph(file)
assertThat(result.imports)
.containsEntry(
file,
setOf(
ImportGraph.Import(URI("pkl:base")),
ImportGraph.Import(URI("pkl:json")),
ImportGraph.Import(URI("pkl:xml"))
)
)
}
@Test
fun `glob imports`(@TempDir tempDir: Path) {
val file1 =
tempDir
.resolve("file1.pkl")
.writeString(
"""
import* "*.pkl"
"""
.trimIndent()
)
.toUri()
val file2 = tempDir.resolve("file2.pkl").writeString("foo = 1").toUri()
val file3 = tempDir.resolve("file3.pkl").writeString("bar = 1").toUri()
val result = simpleAnalyzer.importGraph(file1)
assertThat(result.imports)
.isEqualTo(
mapOf(
file1 to
setOf(ImportGraph.Import(file1), ImportGraph.Import(file2), ImportGraph.Import(file3)),
file2 to emptySet(),
file3 to emptySet()
),
)
}
@Test
fun `cyclical imports`(@TempDir tempDir: Path) {
val file1 = tempDir.resolve("file1.pkl").writeString("import \"file2.pkl\"").toUri()
val file2 = tempDir.resolve("file2.pkl").writeString("import \"file1.pkl\"").toUri()
val result = simpleAnalyzer.importGraph(file1)
assertThat(result.imports)
.isEqualTo(
mapOf(file1 to setOf(ImportGraph.Import(file2)), file2 to setOf(ImportGraph.Import(file1)))
)
}
@Test
fun `package imports`(@TempDir tempDir: Path) {
val analyzer =
Analyzer(
StackFrameTransformers.defaultTransformer,
SecurityManagers.defaultManager,
listOf(ModuleKeyFactories.file, ModuleKeyFactories.standardLibrary, ModuleKeyFactories.pkg),
tempDir.resolve("packages"),
null,
HttpClient.dummyClient(),
)
PackageServer.populateCacheDir(tempDir.resolve("packages"))
val file1 =
tempDir
.resolve("file1.pkl")
.writeString("import \"package://localhost:0/birds@0.5.0#/Bird.pkl\"")
.toUri()
val result = analyzer.importGraph(file1)
assertThat(result.imports)
.isEqualTo(
mapOf(
file1 to setOf(ImportGraph.Import(URI("package://localhost:0/birds@0.5.0#/Bird.pkl"))),
URI("package://localhost:0/birds@0.5.0#/Bird.pkl") to
setOf(ImportGraph.Import(URI("package://localhost:0/fruit@1.0.5#/Fruit.pkl"))),
URI("package://localhost:0/fruit@1.0.5#/Fruit.pkl") to emptySet()
)
)
}
@Test
fun `project dependency imports`(@TempDir tempDir: Path) {
tempDir
.resolve("PklProject")
.writeString(
"""
amends "pkl:Project"
dependencies {
["birds"] { uri = "package://localhost:0/birds@0.5.0" }
}
"""
.trimIndent()
)
tempDir
.resolve("PklProject.deps.json")
.writeString(
"""
{
"schemaVersion": 1,
"resolvedDependencies": {
"package://localhost:0/birds@0": {
"type": "remote",
"uri": "projectpackage://localhost:0/birds@0.5.0",
"checksums": {
"sha256": "${'$'}skipChecksumVerification"
}
},
"package://localhost:0/fruit@1": {
"type": "remote",
"uri": "projectpackage://localhost:0/fruit@1.0.5",
"checksums": {
"sha256": "${'$'}skipChecksumVerification"
}
}
}
}
"""
.trimIndent()
)
val project = Project.loadFromPath(tempDir.resolve("PklProject"))
PackageServer.populateCacheDir(tempDir.resolve("packages"))
val analyzer =
Analyzer(
StackFrameTransformers.defaultTransformer,
SecurityManagers.defaultManager,
listOf(
ModuleKeyFactories.file,
ModuleKeyFactories.standardLibrary,
ModuleKeyFactories.pkg,
ModuleKeyFactories.projectpackage
),
tempDir.resolve("packages"),
project.dependencies,
HttpClient.dummyClient()
)
val file1 =
tempDir
.resolve("file1.pkl")
.writeString(
"""
import "@birds/Bird.pkl"
"""
.trimIndent()
)
.toUri()
val result = analyzer.importGraph(file1)
assertThat(result.imports)
.isEqualTo(
mapOf(
file1 to
setOf(ImportGraph.Import(URI("projectpackage://localhost:0/birds@0.5.0#/Bird.pkl"))),
URI("projectpackage://localhost:0/birds@0.5.0#/Bird.pkl") to
setOf(ImportGraph.Import(URI("projectpackage://localhost:0/fruit@1.0.5#/Fruit.pkl"))),
URI("projectpackage://localhost:0/fruit@1.0.5#/Fruit.pkl") to emptySet()
)
)
assertThat(result.resolvedImports)
.isEqualTo(
mapOf(
file1 to file1.realPath(),
URI("projectpackage://localhost:0/birds@0.5.0#/Bird.pkl") to
URI("projectpackage://localhost:0/birds@0.5.0#/Bird.pkl"),
URI("projectpackage://localhost:0/fruit@1.0.5#/Fruit.pkl") to
URI("projectpackage://localhost:0/fruit@1.0.5#/Fruit.pkl")
)
)
}
@Test
fun `local project dependency import`(@TempDir tempDir: Path) {
val pklProject =
tempDir
.resolve("project1/PklProject")
.createParentDirectories()
.writeString(
"""
amends "pkl:Project"
dependencies {
["birds"] = import("../birds/PklProject")
}
"""
.trimIndent()
)
tempDir
.resolve("birds/PklProject")
.createParentDirectories()
.writeString(
"""
amends "pkl:Project"
package {
name = "birds"
version = "1.0.0"
packageZipUrl = "https://localhost:0/foo.zip"
baseUri = "package://localhost:0/birds"
}
"""
.trimIndent()
)
val birdModule = tempDir.resolve("birds/bird.pkl").writeString("name = \"Warbler\"")
pklProject.parent
.resolve("PklProject.deps.json")
.writeString(
"""
{
"schemaVersion": 1,
"resolvedDependencies": {
"package://localhost:0/birds@1": {
"type": "local",
"uri": "projectpackage://localhost:0/birds@1.0.0",
"path": "../birds"
}
}
}
"""
.trimIndent()
)
val mainPkl =
pklProject.parent
.resolve("main.pkl")
.writeString(
"""
import "@birds/bird.pkl"
"""
.trimIndent()
)
val project = Project.loadFromPath(pklProject)
val analyzer =
Analyzer(
StackFrameTransformers.defaultTransformer,
SecurityManagers.defaultManager,
listOf(
ModuleKeyFactories.file,
ModuleKeyFactories.standardLibrary,
ModuleKeyFactories.pkg,
ModuleKeyFactories.projectpackage
),
tempDir.resolve("packages"),
project.dependencies,
HttpClient.dummyClient()
)
val result = analyzer.importGraph(mainPkl.toUri())
val birdUri = URI("projectpackage://localhost:0/birds@1.0.0#/bird.pkl")
assertThat(result.imports)
.isEqualTo(
mapOf(mainPkl.toUri() to setOf(ImportGraph.Import(birdUri)), birdUri to emptySet()),
)
assertThat(result.resolvedImports)
.isEqualTo(
mapOf(
mainPkl.toUri() to mainPkl.toRealPath().toUri(),
birdUri to birdModule.toRealPath().toUri()
)
)
}
private fun URI.realPath() = Path.of(this).toRealPath().toUri()
}

View File

@@ -18,8 +18,11 @@ package org.pkl.core.ast.builder
import java.net.URI import java.net.URI
import org.assertj.core.api.Assertions.assertThat import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.Test import org.junit.jupiter.api.Test
import org.junit.jupiter.api.assertThrows
import org.pkl.core.SecurityManagers import org.pkl.core.SecurityManagers
import org.pkl.core.StackFrameTransformers
import org.pkl.core.module.ModuleKeys import org.pkl.core.module.ModuleKeys
import org.pkl.core.runtime.VmException
class ImportsAndReadsParserTest { class ImportsAndReadsParserTest {
@Test @Test
@@ -27,13 +30,13 @@ class ImportsAndReadsParserTest {
val moduleText = val moduleText =
""" """
amends "foo.pkl" amends "foo.pkl"
import "bar.pkl" import "bar.pkl"
import "bazzy/buz.pkl" import "bazzy/buz.pkl"
res1 = import("qux.pkl") res1 = import("qux.pkl")
res2 = import*("qux/*.pkl") res2 = import*("qux/*.pkl")
class MyClass { class MyClass {
res3 { res3 {
res4 { res4 {
@@ -48,7 +51,7 @@ class ImportsAndReadsParserTest {
val moduleKey = ModuleKeys.synthetic(URI("repl:text"), moduleText) val moduleKey = ModuleKeys.synthetic(URI("repl:text"), moduleText)
val imports = val imports =
ImportsAndReadsParser.parse(moduleKey, moduleKey.resolve(SecurityManagers.defaultManager)) ImportsAndReadsParser.parse(moduleKey, moduleKey.resolve(SecurityManagers.defaultManager))
assertThat(imports?.map { it.first }) assertThat(imports?.map { it.stringValue })
.hasSameElementsAs( .hasSameElementsAs(
listOf( listOf(
"foo.pkl", "foo.pkl",
@@ -62,4 +65,31 @@ class ImportsAndReadsParserTest {
) )
) )
} }
@Test
fun `invalid syntax`() {
val moduleText =
"""
not valid Pkl syntax
"""
.trimIndent()
val moduleKey = ModuleKeys.synthetic(URI("repl:text"), moduleText)
val err =
assertThrows<VmException> {
ImportsAndReadsParser.parse(moduleKey, moduleKey.resolve(SecurityManagers.defaultManager))
}
assertThat(err.toPklException(StackFrameTransformers.defaultTransformer))
.hasMessage(
"""
Pkl Error
Mismatched input: `<EOF>`. Expected one of: `{`, `=`, `:`
1 | not valid Pkl syntax
^
at text (repl:text)
"""
.trimIndent()
)
}
} }

View File

@@ -0,0 +1,28 @@
/**
* Copyright © 2024 Apple Inc. and the Pkl project authors. All rights reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.pkl.gradle;
import org.gradle.api.Action;
import org.gradle.api.NamedDomainObjectContainer;
import org.pkl.gradle.spec.AnalyzeImportsSpec;
public interface PklAnalyzerCommands {
NamedDomainObjectContainer<AnalyzeImportsSpec> getImports();
default void imports(Action<? super NamedDomainObjectContainer<AnalyzeImportsSpec>> action) {
action.execute(getImports());
}
}

View File

@@ -39,6 +39,9 @@ public interface PklExtension {
@Nested @Nested
PklProjectCommands getProject(); PklProjectCommands getProject();
@Nested
PklAnalyzerCommands getAnalyzers();
default void evaluators(Action<? super NamedDomainObjectContainer<EvalSpec>> action) { default void evaluators(Action<? super NamedDomainObjectContainer<EvalSpec>> action) {
action.execute(getEvaluators()); action.execute(getEvaluators());
} }
@@ -64,4 +67,8 @@ public interface PklExtension {
default void project(Action<? super PklProjectCommands> action) { default void project(Action<? super PklProjectCommands> action) {
action.execute(getProject()); action.execute(getProject());
} }
default void analyzers(Action<? super PklAnalyzerCommands> action) {
action.execute(getAnalyzers());
}
} }

View File

@@ -17,6 +17,7 @@ package org.pkl.gradle;
import java.io.File; import java.io.File;
import java.lang.reflect.InvocationTargetException; import java.lang.reflect.InvocationTargetException;
import java.nio.file.Files;
import java.util.LinkedHashSet; import java.util.LinkedHashSet;
import java.util.List; import java.util.List;
import java.util.Optional; import java.util.Optional;
@@ -36,8 +37,12 @@ import org.gradle.language.base.plugins.LifecycleBasePlugin;
import org.gradle.plugins.ide.idea.model.IdeaModel; import org.gradle.plugins.ide.idea.model.IdeaModel;
import org.gradle.util.GradleVersion; import org.gradle.util.GradleVersion;
import org.pkl.cli.CliEvaluatorOptions; import org.pkl.cli.CliEvaluatorOptions;
import org.pkl.core.ImportGraph;
import org.pkl.core.OutputFormat;
import org.pkl.core.util.IoUtils; import org.pkl.core.util.IoUtils;
import org.pkl.core.util.LateInit; import org.pkl.core.util.LateInit;
import org.pkl.core.util.Nullable;
import org.pkl.gradle.spec.AnalyzeImportsSpec;
import org.pkl.gradle.spec.BasePklSpec; import org.pkl.gradle.spec.BasePklSpec;
import org.pkl.gradle.spec.CodeGenSpec; import org.pkl.gradle.spec.CodeGenSpec;
import org.pkl.gradle.spec.EvalSpec; import org.pkl.gradle.spec.EvalSpec;
@@ -48,6 +53,7 @@ import org.pkl.gradle.spec.PkldocSpec;
import org.pkl.gradle.spec.ProjectPackageSpec; import org.pkl.gradle.spec.ProjectPackageSpec;
import org.pkl.gradle.spec.ProjectResolveSpec; import org.pkl.gradle.spec.ProjectResolveSpec;
import org.pkl.gradle.spec.TestSpec; import org.pkl.gradle.spec.TestSpec;
import org.pkl.gradle.task.AnalyzeImportsTask;
import org.pkl.gradle.task.BasePklTask; import org.pkl.gradle.task.BasePklTask;
import org.pkl.gradle.task.CodeGenTask; import org.pkl.gradle.task.CodeGenTask;
import org.pkl.gradle.task.EvalTask; import org.pkl.gradle.task.EvalTask;
@@ -87,6 +93,7 @@ public class PklPlugin implements Plugin<Project> {
configureTestTasks(extension.getTests()); configureTestTasks(extension.getTests());
configureProjectPackageTasks(extension.getProject().getPackagers()); configureProjectPackageTasks(extension.getProject().getPackagers());
configureProjectResolveTasks(extension.getProject().getResolvers()); configureProjectResolveTasks(extension.getProject().getResolvers());
configureAnalyzeImportsTasks(extension.getAnalyzers().getImports());
} }
private void configureProjectPackageTasks(NamedDomainObjectContainer<ProjectPackageSpec> specs) { private void configureProjectPackageTasks(NamedDomainObjectContainer<ProjectPackageSpec> specs) {
@@ -128,6 +135,21 @@ public class PklPlugin implements Plugin<Project> {
}); });
} }
private void configureAnalyzeImportsTasks(NamedDomainObjectContainer<AnalyzeImportsSpec> specs) {
specs.all(
spec -> {
configureBaseSpec(spec);
spec.getOutputFormat().convention(OutputFormat.PCF.toString());
var analyzeImportsTask = createTask(AnalyzeImportsTask.class, spec);
analyzeImportsTask.configure(
task -> {
task.getOutputFormat().set(spec.getOutputFormat());
task.getOutputFile().set(spec.getOutputFile());
configureModulesTask(task, spec, null);
});
});
}
private void configureEvalTasks(NamedDomainObjectContainer<EvalSpec> specs) { private void configureEvalTasks(NamedDomainObjectContainer<EvalSpec> specs) {
specs.all( specs.all(
spec -> { spec -> {
@@ -141,7 +163,7 @@ public class PklPlugin implements Plugin<Project> {
// and the working directory is set to the project directory, // and the working directory is set to the project directory,
// so this path works correctly. // so this path works correctly.
.file("%{moduleDir}/%{moduleName}.%{outputFormat}")); .file("%{moduleDir}/%{moduleName}.%{outputFormat}"));
spec.getOutputFormat().convention("pcf"); spec.getOutputFormat().convention(OutputFormat.PCF.toString());
spec.getModuleOutputSeparator() spec.getModuleOutputSeparator()
.convention(CliEvaluatorOptions.Companion.getDefaults().getModuleOutputSeparator()); .convention(CliEvaluatorOptions.Companion.getDefaults().getModuleOutputSeparator());
spec.getExpression() spec.getExpression()
@@ -431,20 +453,75 @@ public class PklPlugin implements Plugin<Project> {
task.getHttpNoProxy().set(spec.getHttpNoProxy()); task.getHttpNoProxy().set(spec.getHttpNoProxy());
} }
private <T extends ModulesTask, S extends ModulesSpec> void configureModulesTask(T task, S spec) { private List<File> getTransitiveModules(AnalyzeImportsTask analyzeTask) {
var outputFile = analyzeTask.getOutputFile().get().getAsFile().toPath();
try {
var contents = Files.readString(outputFile);
ImportGraph importGraph = ImportGraph.parseFromJson(contents);
var imports = importGraph.resolvedImports().values();
return imports.stream()
.filter((it) -> it.getScheme().equalsIgnoreCase("file"))
.map(File::new)
.toList();
} catch (Exception e) {
throw new RuntimeException(e);
}
}
private <T extends ModulesTask, S extends ModulesSpec> void configureModulesTask(
T task, S spec, @Nullable TaskProvider<AnalyzeImportsTask> analyzeImportsTask) {
configureBaseTask(task, spec); configureBaseTask(task, spec);
task.getSourceModules().set(spec.getSourceModules()); task.getSourceModules().set(spec.getSourceModules());
task.getTransitiveModules().from(spec.getTransitiveModules());
task.getNoProject().set(spec.getNoProject()); task.getNoProject().set(spec.getNoProject());
task.getProjectDir().set(spec.getProjectDir()); task.getProjectDir().set(spec.getProjectDir());
task.getOmitProjectSettings().set(spec.getOmitProjectSettings()); task.getOmitProjectSettings().set(spec.getOmitProjectSettings());
if (!spec.getTransitiveModules().isEmpty()) {
task.getTransitiveModules().set(spec.getTransitiveModules());
} else if (analyzeImportsTask != null) {
task.dependsOn(analyzeImportsTask);
task.getTransitiveModules().set(analyzeImportsTask.map(this::getTransitiveModules));
}
} }
private <T extends ModulesTask> TaskProvider<T> createModulesTask( private TaskProvider<AnalyzeImportsTask> createAnalyzeImportsTask(ModulesSpec spec) {
Class<T> taskClass, ModulesSpec spec) { var outputFile =
project
.getLayout()
.getBuildDirectory()
.file("pkl-gradle/imports/" + spec.getName() + ".json");
return project return project
.getTasks() .getTasks()
.register(spec.getName(), taskClass, task -> configureModulesTask(task, spec)); .register(
spec.getName() + "GatherImports",
AnalyzeImportsTask.class,
task -> {
configureModulesTask(task, spec, null);
task.setDescription("Compute the set of imports declared by input modules");
task.setGroup("build");
task.getOutputFormat().set(OutputFormat.JSON.toString());
task.getOutputFile().set(outputFile);
});
}
/**
* Implicitly also create a task of type {@link AnalyzeImportsTask}, postfixing the spec name with
* {@code "GatherImports"}.
*
* <p>The resulting task depends on the analyze task, and configures its own input files based on
* the result of analysis.
*
* <p>The end result is that the task automatically has correct up-to-date checks without users
* needing to manually provide transitive modules.
*/
private <T extends ModulesTask> TaskProvider<T> createModulesTask(
Class<T> taskClass, ModulesSpec spec) {
var analyzeImportsTask = createAnalyzeImportsTask(spec);
return project
.getTasks()
.register(
spec.getName(),
taskClass,
task -> configureModulesTask(task, spec, analyzeImportsTask));
} }
private <T extends BasePklTask> TaskProvider<T> createTask(Class<T> taskClass, BasePklSpec spec) { private <T extends BasePklTask> TaskProvider<T> createTask(Class<T> taskClass, BasePklSpec spec) {

View File

@@ -0,0 +1,26 @@
/**
* Copyright © 2024 Apple Inc. and the Pkl project authors. All rights reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.pkl.gradle.spec;
import org.gradle.api.file.RegularFileProperty;
import org.gradle.api.provider.Property;
/** Configuration options for import analyzers. Documented in user manual. */
public interface AnalyzeImportsSpec extends ModulesSpec {
RegularFileProperty getOutputFile();
Property<String> getOutputFormat();
}

View File

@@ -0,0 +1,52 @@
/**
* Copyright © 2024 Apple Inc. and the Pkl project authors. All rights reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.pkl.gradle.task;
import java.io.File;
import org.gradle.api.file.RegularFileProperty;
import org.gradle.api.provider.Property;
import org.gradle.api.provider.Provider;
import org.gradle.api.tasks.Input;
import org.gradle.api.tasks.Optional;
import org.gradle.api.tasks.OutputFile;
import org.pkl.cli.CliImportAnalyzer;
import org.pkl.cli.CliImportAnalyzerOptions;
public abstract class AnalyzeImportsTask extends ModulesTask {
@OutputFile
@Optional
public abstract RegularFileProperty getOutputFile();
@Input
public abstract Property<String> getOutputFormat();
private final Provider<CliImportAnalyzer> cliImportAnalyzerProvider =
getProviders()
.provider(
() ->
new CliImportAnalyzer(
new CliImportAnalyzerOptions(
getCliBaseOptions(),
mapAndGetOrNull(getOutputFile(), it -> it.getAsFile().toPath()),
mapAndGetOrNull(getOutputFormat(), it -> it))));
@Override
protected void doRunTask() {
//noinspection ResultOfMethodCallIgnored
getOutputs().getPreviousOutputFiles().forEach(File::delete);
cliImportAnalyzerProvider.get().run();
}
}

View File

@@ -25,7 +25,6 @@ import java.util.List;
import java.util.Map; import java.util.Map;
import java.util.stream.Collectors; import java.util.stream.Collectors;
import org.gradle.api.InvalidUserDataException; import org.gradle.api.InvalidUserDataException;
import org.gradle.api.file.ConfigurableFileCollection;
import org.gradle.api.file.DirectoryProperty; import org.gradle.api.file.DirectoryProperty;
import org.gradle.api.file.FileCollection; import org.gradle.api.file.FileCollection;
import org.gradle.api.provider.ListProperty; import org.gradle.api.provider.ListProperty;
@@ -49,7 +48,7 @@ public abstract class ModulesTask extends BasePklTask {
public abstract ListProperty<Object> getSourceModules(); public abstract ListProperty<Object> getSourceModules();
@InputFiles @InputFiles
public abstract ConfigurableFileCollection getTransitiveModules(); public abstract ListProperty<File> getTransitiveModules();
private final Map<List<Object>, Pair<List<File>, List<URI>>> parsedSourceModulesCache = private final Map<List<Object>, Pair<List<File>, List<URI>>> parsedSourceModulesCache =
new HashMap<>(); new HashMap<>();

View File

@@ -0,0 +1,108 @@
/**
* Copyright © 2024 Apple Inc. and the Pkl project authors. All rights reserved.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* https://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.pkl.gradle
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.Test
class AnalyzeImportsTest : AbstractTest() {
@Test
fun `write to console`() {
writeFile("input.pkl", "")
writeFile(
"build.gradle",
"""
plugins {
id "org.pkl-lang"
}
pkl {
analyzers {
imports {
analyzeMyImports {
sourceModules = ["input.pkl"]
}
}
}
}
"""
.trimIndent()
)
val result = runTask("analyzeMyImports")
assertThat(result.output).contains("imports {")
}
@Test
fun `output file`() {
writeFile("input.pkl", "")
writeFile(
"build.gradle",
"""
plugins {
id "org.pkl-lang"
}
pkl {
analyzers {
imports {
analyzeMyImports {
sourceModules = ["input.pkl"]
outputFile = file("myFile.pcf")
}
}
}
}
"""
.trimIndent()
)
runTask("analyzeMyImports")
assertThat(testProjectDir.resolve("myFile.pcf")).exists()
}
@Test
fun `output format`() {
writeFile("input.pkl", "")
writeFile(
"build.gradle",
"""
plugins {
id "org.pkl-lang"
}
pkl {
analyzers {
imports {
analyzeMyImports {
sourceModules = ["input.pkl"]
outputFormat = "json"
}
}
}
}
"""
.trimIndent()
)
val result = runTask("analyzeMyImports")
assertThat(result.output)
.contains(
"""
{
"imports": {
"""
.trimIndent()
)
}
}

View File

@@ -839,6 +839,62 @@ class EvaluatorsTest : AbstractTest() {
) )
} }
@Test
fun `implicit dependency tracking for declared imports`() {
writePklFile("import \"shared.pkl\"")
writeFile("shared.pkl", "foo = 1")
writeBuildFile("json")
val result1 = runTask("evalTest")
assertThat(result1.task(":evalTest")!!.outcome).isEqualTo(TaskOutcome.SUCCESS)
// evalTest should be up-to-date now
val result2 = runTask("evalTest")
assertThat(result2.task(":evalTest")!!.outcome).isEqualTo(TaskOutcome.UP_TO_DATE)
// update transitive module with new contents
writeFile("shared.pkl", "foo = 2")
// evalTest should be out-of-date and need to run again
val result3 = runTask("evalTest")
assertThat(result3.task(":evalTest")!!.outcome).isEqualTo(TaskOutcome.SUCCESS)
// running again should be up-to-date again
val result4 = runTask("evalTest")
assertThat(result4.task(":evalTest")!!.outcome).isEqualTo(TaskOutcome.UP_TO_DATE)
}
@Test
fun `explicit dependency tracking using transitive modules`() {
writePklFile("import \"shared.pkl\"")
writeFile("shared.pkl", "foo = 1")
writeFile("shared2.pkl", "foo = 1")
// intentionally use wrong transitive module
writeBuildFile(
"json",
additionalContents =
"""
transitiveModules.from(files("shared2.pkl"))
"""
.trimIndent()
)
val result1 = runTask("evalTest")
assertThat(result1.task(":evalTest")!!.outcome).isEqualTo(TaskOutcome.SUCCESS)
// evalTest should be up-to-date now
val result2 = runTask("evalTest")
assertThat(result2.task(":evalTest")!!.outcome).isEqualTo(TaskOutcome.UP_TO_DATE)
// update transitive module with new contents
writeFile("shared2.pkl", "foo = 2")
// evalTest should be out-of-date and need to run again
val result5 = runTask("evalTest")
assertThat(result5.task(":evalTest")!!.outcome).isEqualTo(TaskOutcome.SUCCESS)
// the "GatherImports" task did not run
assertThat(result5.task(":evalTestGatherImports")).isNull()
}
private fun writeBuildFile( private fun writeBuildFile(
// don't use `org.pkl.core.OutputFormat` // don't use `org.pkl.core.OutputFormat`
// because test compile class path doesn't contain pkl-core // because test compile class path doesn't contain pkl-core

67
stdlib/analyze.pkl Normal file
View File

@@ -0,0 +1,67 @@
//===----------------------------------------------------------------------===//
// Copyright © 2024 Apple Inc. and the Pkl project authors. All rights reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// https://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//===----------------------------------------------------------------------===//
/// A library for statically analyzing Pkl modules.
///
/// These tools differentiate from [pkl:reflect][reflect] in that they parse Pkl modules, but do not
/// execute any code within these modules.
@Since { version = "0.27.0" }
@ModuleInfo { minPklVersion = "0.27.0" }
module pkl.analyze
// used by doc comments
import "pkl:reflect"
/// Given a set of Pkl module URIs, returns a graph of imports declared by these modules.
///
/// The resulting graph includes transitive imports.
external function importGraph(moduleUris: Set<Uri>): ImportGraph
/// The graph of imports declared (directly and transitively) by the modules passed to
/// [importGraph()].
class ImportGraph {
/// The imports declared within a Pkl program.
///
/// Each entry maps a module URI to the set of imports declared in that module.
///
/// The set of all modules in the graph can be obtained via its [keys][Map.keys].
imports: Map<Uri, Set<Import>>
/// Mappings of modules from their in-language URI, to their resolved URI.
///
/// A module's in-language URI is the form used within Pkl source code.
/// For example, modulepath-based modules have form `modulepath:/path/to/my/module.pkl`.
///
/// A module's resolved URI is the form used to load the module's contents.
/// The same modulepath module might have form
/// `jar:file:///path/to/file.zip!/path/to/my/module.pkl` if Pkl run with
/// `--module-path /path/to/file.zip`.
///
/// Dependency-notation imports, such as `"@myPackage/myModule.pkl"`, are represented as
/// in-language URIs with scheme `projectpackage:`.
/// In the case of local project dependenecies, they will be local URIs resolved from the project
/// file URI (in normal cases, `file:` URIs).
resolvedImports: Map<Uri, Uri>(keys == imports.keys)
}
/// An import as declared inside a module.
class Import {
/// The absolute (in-language) URI of the import.
///
/// Dependency notation URIs (such as `import "@foo/bar"`) are resolved to package URIs with
/// scheme `projectpackage:`.
uri: Uri
}