diff --git a/docs/modules/ROOT/partials/component-attributes.adoc b/docs/modules/ROOT/partials/component-attributes.adoc index 002141394..fa7979db4 100644 --- a/docs/modules/ROOT/partials/component-attributes.adoc +++ b/docs/modules/ROOT/partials/component-attributes.adoc @@ -74,6 +74,8 @@ endif::[] :uri-stdlib-jsonnetModule: {uri-pkl-stdlib-docs}/jsonnet :uri-stdlib-reflectModule: {uri-pkl-stdlib-docs}/reflect :uri-stdlib-mathModule: {uri-pkl-stdlib-docs}/math +:uri-stdlib-netModule: {uri-pkl-stdlib-docs}/net +:uri-stdlib-net-Url: {uri-stdlib-netModule}/Url :uri-stdlib-xmlModule: {uri-pkl-stdlib-docs}/xml :uri-stdlib-protobufModule: {uri-pkl-stdlib-docs}/protobuf :uri-stdlib-pklbinaryModule: {uri-pkl-stdlib-docs}/pklbinary diff --git a/docs/modules/release-notes/pages/0.33.adoc b/docs/modules/release-notes/pages/0.33.adoc index 75566a830..fccfd1837 100644 --- a/docs/modules/release-notes/pages/0.33.adoc +++ b/docs/modules/release-notes/pages/0.33.adoc @@ -9,7 +9,77 @@ include::partial$intro.adoc[] == Highlights [small]#💖# -=== XXX +=== New standard library module: `pkl:net` + +This release introduces a new standard library module, link:{uri-stdlib-netModule}[`pkl:net`], which brings native URL support to Pkl (https://github.com/apple/pkl/pull/1834[#1834]). + +Its centerpiece is link:{uri-stdlib-net-Url}[`net.Url`], a URL as defined by https://www.rfc-editor.org/rfc/rfc3986[RFC 3986]. +A string is parsed with `net.Url()`, which accepts an absolute URL as well as a relative reference, and which throws if the string cannot be parsed: + +[source,pkl] +---- +import "pkl:net" + +local url = net.Url("https://user:pw@example.com:8080/a/b c?q=1&q=2#f") + +scheme = url.scheme // "https" +rawUserInfo = url.rawUserInfo // "user:pw" +authority = url.authority // "user:pw@example.com:8080" +rawHost = url.rawHost // "example.com" +rawPort = url.rawPort // "8080" +port = url.port // 8080 +rawPath = url.rawPath // "/a/b%20c" +path = url.path // "/a/b c" +pathSegments = url.pathSegments // List("a", "b c") +rawQuery = url.rawQuery // "q=1&q=2" +queryParameters = url.queryParameters // Mapping { ["q"] = Listing { "1"; "2" } } +rawFragment = url.rawFragment // "f" +---- + +Parsing is lenient: whatever cannot appear literally in a component is percent-encoded as its UTF-8 bytes, and whatever is already encoded is kept as written. +Every component is held percent-encoded in its `raw` property, whether the input was encoded or not; the percent-decoded form of each is available as `userInfo`, `host`, `path`, `query`, and `fragment`. +The derived `pathSegments` and `queryParameters` are percent-decoded too. + +A URL can also be built from its components, or derived from another URL by amending it. +A component given this way has to be percent-encoded already: + +[source,pkl] +---- +built = new net.Url { + scheme = "https" + rawHost = "example.com" + rawPath = "/a%20b" +}.toString() // "https://example.com/a%20b" + +local base = net.Url("https://example.com/a?q=1") + +amended = (base) { rawQuery = null }.toString() // "https://example.com/a" +---- + +Each component is type-checked, so a URL built this way cannot be malformed. +For example, a `rawPort` can only hold digits, only a URL with a `rawHost` can have one, and a `rawPath` of `"/a b"` is rejected. +The decoded components are `fixed`, so they cannot be amended. + +A relative reference can be resolved against an absolute URL, which also removes dot segments from the result's path: + +[source,pkl] +---- +resolved = net.Url("https://example.com/a/b").resolve("../c").toString() // "https://example.com/c" +---- + +Comparing two URLs with `==` compares their components as written. +To instead compare them as the *syntax-based normalization* of RFC 3986, where scheme and host are case-insensitive and percent-encoding is normalized, use `equals()`: + +[source,pkl] +---- +same = net.Url("HTTPS://Example.com/%7Euser").equals(net.Url("https://example.com/~user")) // true +---- + +The module also provides standalone helpers: `net.encodeUrlComponent()` and `net.decodeUrlComponent()` for percent-encoding, and `net.buildQuery()` and `net.parseQuery()` for `application/x-www-form-urlencoded` queries. + +Alongside this module, `pkl:base` gains link:{uri-stdlib-String}#isValidUrl[`String.isValidUrl`], which tells whether a string is a valid absolute URL. +It is stricter than `net.Url()`, which also parses a relative reference, and which percent-encodes whatever cannot appear literally in a URL component. +The `Uri` typealias is now defined in terms of it, which is a <>. == Noteworthy [small]#🎶# @@ -51,6 +121,30 @@ For example, setting `XDG_CACHE_HOME` will configure the cache directory. Note that the existing `~/.pkl/cache` directory is ignored, so Pkl will download packages to populate its cache if configured to do so. +=== Java Library Changes + +==== `pkl.net#Url` maps to `java.net.URI` + +The JVM bindings now understand the new `pkl.net#Url` class (https://github.com/apple/pkl/pull/1834[#1834]). + +The Java and Kotlin code generators generate a `java.net.URI` for a property whose type is `net.Url`. + +Two conversions are added to `org.pkl.config.java.mapper.Conversions`, both part of the preconfigured set: + +* `pUrlToString` -- gives the same result as `Url.toString()` +* `pUrlToURI` -- throws `ConversionException` if the URL cannot be represented as a `java.net.URI` + +=== Standard Library Changes + +==== `pkl:net` + +* New module introduced. + +==== `pkl:base` + +* New property: link:{uri-stdlib-String}#isValidUrl[`String.isValidUrl`] +* Changed typealias: link:{uri-stdlib-Uri}[`Uri`] is now `String(isValidUrl)` (<>) + == Breaking Changes [small]#💔# === Removal of macOS amd64 (Intel) Binaries @@ -126,6 +220,26 @@ res = """ """ ---- +[[uri-validates]] +=== `Uri` validates its value + +The link:{uri-stdlib-Uri}[`Uri`] typealias in `pkl:base` is now `String(isValidUrl)` instead of a plain `String` (https://github.com/apple/pkl/pull/1834[#1834]). + +A value typed as `Uri` must be a valid absolute URL according to https://www.rfc-editor.org/rfc/rfc3986[RFC 3986]. +This means that a relative reference, or a URL containing a character that must be percent-encoded, no longer type-checks: + +[source,pkl] +---- +good: Uri = "https://example.com/some%20path" + +bad1: Uri = "./some/path" // throws: not absolute +bad2: Uri = "https://example.com/some path" // throws: space must be percent-encoded +---- + +This affects every standard library member typed as `Uri`, among them link:{uri-stdlib-Resource}#uri[`Resource.uri`], link:{uri-stdlib-analyzeModule}#importGraph()[`analyze.importGraph()`], and the URI-valued properties of `pkl:Project`, `pkl:EvaluatorSettings` and `pkl:DocPackageInfo`. + +As a consequence, an invalid URI is now reported as a type constraint violation at the point where it is set, rather than as an error raised later by whatever consumes it. + == Bug Fixes [small]#🐜# * XXX diff --git a/pkl-codegen-java/src/main/kotlin/org/pkl/codegen/java/JavaCodeGenerator.kt b/pkl-codegen-java/src/main/kotlin/org/pkl/codegen/java/JavaCodeGenerator.kt index 7a6762276..c4f0adcfd 100644 --- a/pkl-codegen-java/src/main/kotlin/org/pkl/codegen/java/JavaCodeGenerator.kt +++ b/pkl-codegen-java/src/main/kotlin/org/pkl/codegen/java/JavaCodeGenerator.kt @@ -845,6 +845,7 @@ class JavaCodeGenerator( PClassInfo.Class -> PCLASS.nullableIf(nullable) PClassInfo.Regex -> PATTERN.nullableIf(nullable) PClassInfo.Version -> VERSION.nullableIf(nullable) + PClassInfo.Url -> URI.nullableIf(nullable) else -> when { !classInfo.isStandardLibraryClass -> pClass.toJavaPoetName().nullableIf(nullable) diff --git a/pkl-codegen-java/src/test/kotlin/org/pkl/codegen/java/JavaCodeGeneratorTest.kt b/pkl-codegen-java/src/test/kotlin/org/pkl/codegen/java/JavaCodeGeneratorTest.kt index 17e68f578..2b335a52b 100644 --- a/pkl-codegen-java/src/test/kotlin/org/pkl/codegen/java/JavaCodeGeneratorTest.kt +++ b/pkl-codegen-java/src/test/kotlin/org/pkl/codegen/java/JavaCodeGeneratorTest.kt @@ -1159,6 +1159,52 @@ class JavaCodeGeneratorTest { ) } + @Test + fun `pkl_net Url`() { + val javaCode = + generateJavaCode( + """ + module mod + + import "pkl:net" + + url: net.Url + nullableUrl: net.Url? + listing: Listing + mapping: Mapping + + class Foo { + url: net.Url + } + """ + .trimIndent() + ) + + assertThat(javaCode) + .compilesSuccessfully() + .contains("import java.net.URI;") + .contains( + """ + |public final class Mod { + | public final @NonNull URI url; + | + | public final URI nullableUrl; + | + | public final @NonNull List<@NonNull URI> listing; + | + | public final @NonNull Map<@NonNull String, @NonNull URI> mapping; + """ + .trimMargin() + ) + .contains( + """ + | public static final class Foo { + | public final @NonNull URI url; + """ + .trimMargin() + ) + } + @Test fun `nullable properties`() { var javaCode = diff --git a/pkl-codegen-kotlin/src/main/kotlin/org/pkl/codegen/kotlin/KotlinCodeGenerator.kt b/pkl-codegen-kotlin/src/main/kotlin/org/pkl/codegen/kotlin/KotlinCodeGenerator.kt index 1733663da..508f59933 100644 --- a/pkl-codegen-kotlin/src/main/kotlin/org/pkl/codegen/kotlin/KotlinCodeGenerator.kt +++ b/pkl-codegen-kotlin/src/main/kotlin/org/pkl/codegen/kotlin/KotlinCodeGenerator.kt @@ -703,6 +703,7 @@ class KotlinCodeGenerator( PClassInfo.Class -> PCLASS PClassInfo.Regex -> REGEX PClassInfo.Version -> VERSION + PClassInfo.Url -> URI else -> when { !classInfo.isStandardLibraryClass -> pClass.toKotlinPoetName() diff --git a/pkl-codegen-kotlin/src/test/kotlin/org/pkl/codegen/kotlin/KotlinCodeGeneratorTest.kt b/pkl-codegen-kotlin/src/test/kotlin/org/pkl/codegen/kotlin/KotlinCodeGeneratorTest.kt index 64f7202a8..c48321843 100644 --- a/pkl-codegen-kotlin/src/test/kotlin/org/pkl/codegen/kotlin/KotlinCodeGeneratorTest.kt +++ b/pkl-codegen-kotlin/src/test/kotlin/org/pkl/codegen/kotlin/KotlinCodeGeneratorTest.kt @@ -1186,6 +1186,52 @@ class KotlinCodeGeneratorTest { ) } + @Test + fun `pkl_net Url`() { + val kotlinCode = + generateKotlinCode( + """ + module mod + + import "pkl:net" + + url: net.Url + nullableUrl: net.Url? + listing: Listing + mapping: Mapping + + class Foo { + url: net.Url + } + """ + .trimIndent() + ) + + assertThat(kotlinCode) + .compilesSuccessfully() + .isEqualTo( + """ + import java.net.URI + import kotlin.String + import kotlin.collections.List + import kotlin.collections.Map + + data class Mod( + val url: URI, + val nullableUrl: URI?, + val listing: List, + val mapping: Map + ) { + data class Foo( + val url: URI + ) + } + + """ + .trimIndent() + ) + } + @Test fun `user defined type aliases`() { val kotlinCode = diff --git a/pkl-config-java/src/main/java/org/pkl/config/java/mapper/Conversions.java b/pkl-config-java/src/main/java/org/pkl/config/java/mapper/Conversions.java index 87c1ec0b0..77b54716e 100644 --- a/pkl-config-java/src/main/java/org/pkl/config/java/mapper/Conversions.java +++ b/pkl-config-java/src/main/java/org/pkl/config/java/mapper/Conversions.java @@ -1,5 +1,5 @@ /* - * Copyright © 2024-2025 Apple Inc. and the Pkl project authors. All rights reserved. + * Copyright © 2024-2026 Apple Inc. and the Pkl project authors. All rights reserved. * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -265,6 +265,36 @@ public final class Conversions { } }); + /** + * Conversion from {@code pkl.net#Url} to {@link String}. Gives the same result as {@code + * Url.toString()}. + */ + public static final Conversion pUrlToString = + Conversion.of(PClassInfo.Url, String.class, (value, mapper) -> serializeUrl(value)); + + /** + * Conversion from {@code pkl.net#Url} to {@link URI}. Throws {@link ConversionException} if the + * URL cannot be represented as a {@link URI}, which is the case for a host that is an IPvFuture + * address. + */ + public static final Conversion pUrlToURI = + Conversion.of( + PClassInfo.Url, + URI.class, + (value, mapper) -> { + var url = serializeUrl(value); + try { + return new URI(url); + } catch (URISyntaxException e) { + throw new ConversionException( + String.format("Failed to convert `pkl.net#Url` `%s` to `java.net.URI`.", url), e); + } + }); + + private static String serializeUrl(PObject url) { + return (String) url.getProperty("stringValue"); + } + /** * Identity conversions used when the Java representation of the Pkl type matches the target type * or when the target type is {@link Object}. @@ -323,7 +353,9 @@ public final class Conversions { pDurationToDuration, pVersionToVersion, pVersionToString, - pStringToVersion); + pStringToVersion, + pUrlToString, + pUrlToURI); /** All conversions defined in this class. */ public static final Collection> all = collectAll(); diff --git a/pkl-config-java/src/test/java/org/pkl/config/java/mapper/PUrlToStringTest.java b/pkl-config-java/src/test/java/org/pkl/config/java/mapper/PUrlToStringTest.java new file mode 100644 index 000000000..d3e91fafd --- /dev/null +++ b/pkl-config-java/src/test/java/org/pkl/config/java/mapper/PUrlToStringTest.java @@ -0,0 +1,66 @@ +/* + * Copyright © 2026 Apple Inc. and the Pkl project authors. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.pkl.config.java.mapper; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.pkl.core.ModuleSource.modulePath; + +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.Test; +import org.pkl.core.Evaluator; +import org.pkl.core.PModule; + +public class PUrlToStringTest { + private static final Evaluator evaluator = Evaluator.preconfigured(); + + private static final PModule module = + evaluator.evaluate(modulePath("org/pkl/config/java/mapper/PUrlToURITest.pkl")); + + private static final ValueMapper mapper = ValueMapperBuilder.preconfigured().build(); + + @AfterAll + public static void afterAll() { + evaluator.close(); + } + + @Test + public void absolute() { + assertThat(map("absolute")).isEqualTo("https://user:pw@example.com:8080/a/b?q=1#frag"); + } + + @Test + public void percentEncoding() { + assertThat(map("literalSpace")).isEqualTo("https://example.com/a%20b"); + assertThat(map("nonAscii")).isEqualTo("https://example.com/%C3%A9"); + } + + @Test + public void relative() { + assertThat(map("relative")).isEqualTo("/foo/bar"); + assertThat(map("colonSegment")).isEqualTo("./a:b"); + } + + @Test + public void ipvFuture() { + assertThat(map("ipvFuture")).isEqualTo("http://[v1.fe80::a+en1]/"); + } + + private static String map(String propertyName) { + var url = module.get(propertyName); + assert url != null; + return mapper.map(url, String.class); + } +} diff --git a/pkl-config-java/src/test/java/org/pkl/config/java/mapper/PUrlToURITest.java b/pkl-config-java/src/test/java/org/pkl/config/java/mapper/PUrlToURITest.java new file mode 100644 index 000000000..990c68558 --- /dev/null +++ b/pkl-config-java/src/test/java/org/pkl/config/java/mapper/PUrlToURITest.java @@ -0,0 +1,92 @@ +/* + * Copyright © 2026 Apple Inc. and the Pkl project authors. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.pkl.config.java.mapper; + +import static org.assertj.core.api.Assertions.*; +import static org.pkl.core.ModuleSource.modulePath; + +import java.net.URI; +import java.net.URISyntaxException; +import org.junit.jupiter.api.AfterAll; +import org.junit.jupiter.api.Test; +import org.pkl.core.Evaluator; +import org.pkl.core.PModule; + +public class PUrlToURITest { + private static final Evaluator evaluator = Evaluator.preconfigured(); + + private static final PModule module = + evaluator.evaluate(modulePath("org/pkl/config/java/mapper/PUrlToURITest.pkl")); + + private static final ValueMapper mapper = ValueMapperBuilder.preconfigured().build(); + + @AfterAll + public static void afterAll() { + evaluator.close(); + } + + @Test + public void absolute() { + assertThat(map("absolute")) + .isEqualTo(URI.create("https://user:pw@example.com:8080/a/b?q=1#frag")); + } + + @Test + public void percentEncoding() { + // characters that cannot appear literally are encoded, and what is already encoded is kept + assertThat(map("literalSpace")).isEqualTo(URI.create("https://example.com/a%20b")); + assertThat(map("encodedSpace")).isEqualTo(URI.create("https://example.com/a%20b")); + assertThat(map("encodedSlash")).isEqualTo(URI.create("https://example.com/a%2Fb")); + assertThat(map("nonAscii")).isEqualTo(URI.create("https://example.com/%C3%A9")); + } + + @Test + public void relative() { + assertThat(map("relative")).isEqualTo(URI.create("/foo/bar")); + // a first segment holding a ":" has to be preceded by a dot-segment, or it would be read as a + // scheme + assertThat(map("colonSegment")).isEqualTo(URI.create("./a:b")); + } + + @Test + public void authority() { + assertThat(map("rootless")).isEqualTo(URI.create("mailto:user@example.com")); + assertThat(map("noAuthority")).isEqualTo(URI.create("file:/tmp")); + assertThat(map("emptyAuthority")).isEqualTo(URI.create("file:///tmp")); + assertThat(map("emptyQuery")).isEqualTo(URI.create("https://example.com/?")); + assertThat(map("ipv6Zone")).isEqualTo(URI.create("http://[fe80::1%25eth0]/p")); + } + + @Test + public void constructed() { + assertThat(map("constructed")).isEqualTo(URI.create("https://example.com/a%20b?x%20y")); + } + + @Test + public void ipvFuture() { + // `java.net.URI` doesn't know IPvFuture addresses + assertThatThrownBy(() -> map("ipvFuture")) + .isInstanceOf(ConversionException.class) + .hasMessageContaining("http://[v1.fe80::a+en1]/") + .hasCauseInstanceOf(URISyntaxException.class); + } + + private static URI map(String propertyName) { + var url = module.get(propertyName); + assert url != null; + return mapper.map(url, URI.class); + } +} diff --git a/pkl-config-java/src/test/resources/org/pkl/config/java/mapper/PUrlToURITest.pkl b/pkl-config-java/src/test/resources/org/pkl/config/java/mapper/PUrlToURITest.pkl new file mode 100644 index 000000000..5e801def1 --- /dev/null +++ b/pkl-config-java/src/test/resources/org/pkl/config/java/mapper/PUrlToURITest.pkl @@ -0,0 +1,16 @@ +import "pkl:net" + +absolute = net.Url("https://user:pw@example.com:8080/a/b?q=1#frag") +literalSpace = net.Url("https://example.com/a b") +encodedSpace = net.Url("https://example.com/a%20b") +encodedSlash = net.Url("https://example.com/a%2Fb") +nonAscii = net.Url("https://example.com/é") +relative = net.Url("/foo/bar") +rootless = net.Url("mailto:user@example.com") +noAuthority = net.Url("file:/tmp") +emptyAuthority = net.Url("file:///tmp") +emptyQuery = net.Url("https://example.com/?") +ipv6Zone = net.Url("http://[fe80::1%25eth0]/p") +ipvFuture = net.Url("http://[v1.fe80::a+en1]/") +constructed = new net.Url { scheme = "https"; rawHost = "example.com"; rawPath = "/a%20b"; rawQuery = "x%20y" } +colonSegment = new net.Url { rawPath = "./a:b" } diff --git a/pkl-core/src/main/java/org/pkl/core/PClassInfo.java b/pkl-core/src/main/java/org/pkl/core/PClassInfo.java index abc9b40dd..6c1f33437 100644 --- a/pkl-core/src/main/java/org/pkl/core/PClassInfo.java +++ b/pkl-core/src/main/java/org/pkl/core/PClassInfo.java @@ -37,6 +37,7 @@ public final class PClassInfo implements Serializable { public static final URI pklBaseUri = URI.create("pkl:base"); public static final URI pklSemverUri = URI.create("pkl:semver"); + public static final URI pklNetUri = URI.create("pkl:net"); public static final URI pklSettingsUri = URI.create("pkl:settings"); public static final URI pklProjectUri = URI.create("pkl:Project"); public static final URI pklRefUri = URI.create("pkl:ref"); @@ -82,6 +83,8 @@ public final class PClassInfo implements Serializable { pklBaseClassInfo("ModuleInfo", PObject.class); public static final PClassInfo Version = new PClassInfo<>("pkl.semver", "Version", PObject.class, pklSemverUri); + public static final PClassInfo Url = + new PClassInfo<>("pkl.net", "Url", PObject.class, pklNetUri); public static final PClassInfo Project = new PClassInfo<>("pkl.Project", MODULE_CLASS_NAME, PObject.class, pklProjectUri); public static final PClassInfo Settings = diff --git a/pkl-core/src/main/java/org/pkl/core/ast/internal/GetParsedUrlNode.java b/pkl-core/src/main/java/org/pkl/core/ast/internal/GetParsedUrlNode.java new file mode 100644 index 000000000..12c2ea777 --- /dev/null +++ b/pkl-core/src/main/java/org/pkl/core/ast/internal/GetParsedUrlNode.java @@ -0,0 +1,44 @@ +/* + * Copyright © 2026 Apple Inc. and the Pkl project authors. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.pkl.core.ast.internal; + +import com.oracle.truffle.api.dsl.Cached; +import com.oracle.truffle.api.dsl.NeverDefault; +import com.oracle.truffle.api.dsl.Specialization; +import com.oracle.truffle.api.nodes.IndirectCallNode; +import org.pkl.core.ast.PklNode; +import org.pkl.core.runtime.VmTyped; +import org.pkl.core.util.url.UrlFactory; +import org.pkl.core.util.url.UrlParser.Parsed; + +public abstract class GetParsedUrlNode extends PklNode { + @NeverDefault + public static GetParsedUrlNode create() { + return GetParsedUrlNodeGen.create(); + } + + @Specialization(guards = "url.hasExtraStorage()") + protected Parsed evalCached(VmTyped url) { + return (Parsed) url.getExtraStorage(); + } + + @Specialization(replaces = "evalCached") + protected Parsed eval(VmTyped url, @Cached("create()") IndirectCallNode callNode) { + return UrlFactory.read(url, callNode); + } + + public abstract Parsed execute(VmTyped url); +} diff --git a/pkl-core/src/main/java/org/pkl/core/runtime/Identifier.java b/pkl-core/src/main/java/org/pkl/core/runtime/Identifier.java index 9009ccf17..d311b2b70 100644 --- a/pkl-core/src/main/java/org/pkl/core/runtime/Identifier.java +++ b/pkl-core/src/main/java/org/pkl/core/runtime/Identifier.java @@ -147,6 +147,15 @@ public final class Identifier implements Comparable { // members of pkl.yaml public static final Identifier MAX_COLLECTION_ALIASES = get("maxCollectionAliases"); + // members of pkl.net + public static final Identifier RAW_USER_INFO = get("rawUserInfo"); + public static final Identifier RAW_PATH = get("rawPath"); + public static final Identifier RAW_QUERY = get("rawQuery"); + public static final Identifier SCHEME = get("scheme"); + public static final Identifier RAW_HOST = get("rawHost"); + public static final Identifier RAW_PORT = get("rawPort"); + public static final Identifier RAW_FRAGMENT = get("rawFragment"); + // members of pkl.Command public static final Identifier OPTIONS = get("options"); public static final Identifier PARENT = get("parent"); diff --git a/pkl-core/src/main/java/org/pkl/core/runtime/ModuleCache.java b/pkl-core/src/main/java/org/pkl/core/runtime/ModuleCache.java index 13e8971a8..48a1b165a 100644 --- a/pkl-core/src/main/java/org/pkl/core/runtime/ModuleCache.java +++ b/pkl-core/src/main/java/org/pkl/core/runtime/ModuleCache.java @@ -98,6 +98,8 @@ public final class ModuleCache { return JsonnetModule.getModule(); case "math": return MathModule.getModule(); + case "net": + return NetModule.getModule(); case "pklbinary": return PklBinaryModule.getModule(); case "platform": diff --git a/pkl-core/src/main/java/org/pkl/core/runtime/NetModule.java b/pkl-core/src/main/java/org/pkl/core/runtime/NetModule.java new file mode 100644 index 000000000..b677d46e2 --- /dev/null +++ b/pkl-core/src/main/java/org/pkl/core/runtime/NetModule.java @@ -0,0 +1,47 @@ +/* + * Copyright © 2026 Apple Inc. and the Pkl project authors. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.pkl.core.runtime; + +import com.oracle.truffle.api.CompilerDirectives.TruffleBoundary; +import java.net.URI; + +public final class NetModule extends StdLibModule { + private static final VmTyped instance = VmUtils.createEmptyModule(); + + static { + loadModule(URI.create("pkl:net"), instance); + } + + private NetModule() {} + + public static VmTyped getModule() { + return instance; + } + + public static VmClass getUrlClass() { + return UrlClass.instance; + } + + private static final class UrlClass { + static final VmClass instance = loadClass("Url"); + } + + @TruffleBoundary + private static VmClass loadClass(String className) { + var theModule = getModule(); + return (VmClass) VmUtils.readMember(theModule, Identifier.get(className)); + } +} diff --git a/pkl-core/src/main/java/org/pkl/core/stdlib/base/StringNodes.java b/pkl-core/src/main/java/org/pkl/core/stdlib/base/StringNodes.java index f55a8033b..f8af6ea49 100644 --- a/pkl-core/src/main/java/org/pkl/core/stdlib/base/StringNodes.java +++ b/pkl-core/src/main/java/org/pkl/core/stdlib/base/StringNodes.java @@ -34,6 +34,7 @@ import org.pkl.core.util.GlobResolver; import org.pkl.core.util.GlobResolver.InvalidGlobPatternException; import org.pkl.core.util.Pair; import org.pkl.core.util.StringUtils; +import org.pkl.core.util.url.UrlParser; import org.pkl.parser.Lexer; import org.pkl.parser.ParserError; import org.pkl.parser.Token; @@ -197,6 +198,14 @@ public final class StringNodes { } } + public abstract static class isValidUrl extends ExternalPropertyNode { + @Specialization + @TruffleBoundary + protected boolean eval(String self) { + return UrlParser.isValidUrl(self); + } + } + public abstract static class chars extends ExternalPropertyNode { @Specialization @TruffleBoundary diff --git a/pkl-core/src/main/java/org/pkl/core/stdlib/net/NetNodes.java b/pkl-core/src/main/java/org/pkl/core/stdlib/net/NetNodes.java new file mode 100644 index 000000000..dac03b1f4 --- /dev/null +++ b/pkl-core/src/main/java/org/pkl/core/stdlib/net/NetNodes.java @@ -0,0 +1,87 @@ +/* + * Copyright © 2026 Apple Inc. and the Pkl project authors. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.pkl.core.stdlib.net; + +import com.oracle.truffle.api.CompilerDirectives.TruffleBoundary; +import com.oracle.truffle.api.dsl.Specialization; +import org.pkl.core.runtime.VmListing; +import org.pkl.core.runtime.VmMapping; +import org.pkl.core.runtime.VmTyped; +import org.pkl.core.stdlib.ExternalMethod1Node; +import org.pkl.core.util.url.PercentEncoder; +import org.pkl.core.util.url.UrlFactory; + +/** Backing nodes for {@code pkl:net}'s module-level members. */ +public final class NetNodes { + private NetNodes() {} + + public abstract static class Url extends ExternalMethod1Node { + @Specialization + protected Object eval(@SuppressWarnings("unused") VmTyped self, String input) { + return UrlFactory.create(UrlFactory.parseOrThrow(input, this)); + } + } + + public abstract static class encodeUrlComponent extends ExternalMethod1Node { + @Specialization + @TruffleBoundary + protected String eval(@SuppressWarnings("unused") VmTyped self, String value) { + var out = new StringBuilder(value.length()); + PercentEncoder.encodeComponent(out, value); + return out.toString(); + } + } + + public abstract static class decodeUrlComponent extends ExternalMethod1Node { + @Specialization + protected String eval(@SuppressWarnings("unused") VmTyped self, String value) { + UrlFactory.checkPercentEncoding(value, "cannotDecodeUrlComponent", this); + return PercentEncoder.decode(value); + } + } + + public abstract static class buildQuery extends ExternalMethod1Node { + @Specialization + @TruffleBoundary + protected String eval(@SuppressWarnings("unused") VmTyped self, VmMapping parameters) { + var out = new StringBuilder(); + parameters.forceAndIterateMemberValues( + (name, def, values) -> { + ((VmListing) values) + .forceAndIterateMemberValues( + (index, valueDef, value) -> { + if (!out.isEmpty()) { + out.append('&'); + } + PercentEncoder.encodeForm(out, (String) name); + out.append('='); + PercentEncoder.encodeForm(out, (String) value); + return true; + }); + return true; + }); + return out.toString(); + } + } + + public abstract static class parseQuery extends ExternalMethod1Node { + @Specialization + protected VmMapping eval(@SuppressWarnings("unused") VmTyped self, String query) { + UrlFactory.checkPercentEncoding(query, "cannotParseQuery", this); + return UrlFactory.createQueryParameters(query); + } + } +} diff --git a/pkl-core/src/main/java/org/pkl/core/stdlib/net/UrlNodes.java b/pkl-core/src/main/java/org/pkl/core/stdlib/net/UrlNodes.java new file mode 100644 index 000000000..c73adf4a5 --- /dev/null +++ b/pkl-core/src/main/java/org/pkl/core/stdlib/net/UrlNodes.java @@ -0,0 +1,225 @@ +/* + * Copyright © 2026 Apple Inc. and the Pkl project authors. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.pkl.core.stdlib.net; + +import com.oracle.truffle.api.CompilerDirectives.TruffleBoundary; +import com.oracle.truffle.api.dsl.Cached; +import com.oracle.truffle.api.dsl.Specialization; +import com.oracle.truffle.api.nodes.IndirectCallNode; +import org.jspecify.annotations.Nullable; +import org.pkl.core.ast.internal.GetParsedUrlNode; +import org.pkl.core.ast.internal.GetParsedUrlNodeGen; +import org.pkl.core.runtime.VmList; +import org.pkl.core.runtime.VmNull; +import org.pkl.core.runtime.VmTyped; +import org.pkl.core.stdlib.ExternalMethod0Node; +import org.pkl.core.stdlib.ExternalMethod1Node; +import org.pkl.core.stdlib.ExternalMethod3Node; +import org.pkl.core.stdlib.ExternalPropertyNode; +import org.pkl.core.util.url.UrlFactory; +import org.pkl.core.util.url.UrlParser; +import org.pkl.core.util.url.UrlParser.Parsed; + +/** Backing nodes for {@code pkl:net}'s {@code Url} class. */ +public final class UrlNodes { + private UrlNodes() {} + + public abstract static class authority extends ExternalPropertyNode { + @Specialization(guards = "self.hasExtraStorage()") + protected Object evalCached(VmTyped self) { + var parsed = (Parsed) self.getExtraStorage(); + return VmNull.lift(parsed.authority()); + } + + @Specialization(guards = "!self.hasExtraStorage()") + protected Object eval(VmTyped self, @Cached("create()") IndirectCallNode callNode) { + return VmNull.lift(UrlFactory.readAuthority(self, callNode)); + } + } + + public abstract static class hostKind extends ExternalPropertyNode { + @Specialization(guards = "self.hasExtraStorage()") + protected Object evalCached(VmTyped self) { + var parsed = (Parsed) self.getExtraStorage(); + return getHostKind(parsed.host()); + } + + @Specialization(guards = "!self.hasExtraStorage()") + protected Object eval(VmTyped self, @Cached("create()") IndirectCallNode callNode) { + return getHostKind(UrlFactory.readHost(self, callNode)); + } + + private static Object getHostKind(@Nullable String host) { + return host == null ? VmNull.withoutDefault() : UrlParser.hostKind(host); + } + } + + public abstract static class zoneId extends ExternalPropertyNode { + @Specialization(guards = "self.hasExtraStorage()") + protected Object evalCached(VmTyped self) { + var parsed = (Parsed) self.getExtraStorage(); + return getZoneId(parsed.host()); + } + + @Specialization(guards = "!self.hasExtraStorage()") + protected Object eval(VmTyped self, @Cached("create()") IndirectCallNode callNode) { + return getZoneId(UrlFactory.readHost(self, callNode)); + } + + private static Object getZoneId(@Nullable String host) { + return VmNull.lift(host == null ? null : UrlParser.zoneId(host)); + } + } + + public abstract static class pathSegments extends ExternalPropertyNode { + @Specialization(guards = "self.hasExtraStorage()") + protected VmList evalCached(VmTyped self) { + var parsed = (Parsed) self.getExtraStorage(); + return VmList.create(UrlParser.segments(parsed.path())); + } + + @Specialization(guards = "!self.hasExtraStorage()") + protected VmList eval(VmTyped self, @Cached("create()") IndirectCallNode callNode) { + return VmList.create(UrlParser.segments(UrlFactory.readPath(self, callNode))); + } + } + + public abstract static class queryParameters extends ExternalPropertyNode { + @Specialization(guards = "self.hasExtraStorage()") + protected Object evalCached(VmTyped self) { + var parsed = (Parsed) self.getExtraStorage(); + return getQueryParameters(parsed.query()); + } + + @Specialization(guards = "!self.hasExtraStorage()") + protected Object eval(VmTyped self, @Cached("create()") IndirectCallNode callNode) { + return getQueryParameters(UrlFactory.readQuery(self, callNode)); + } + + private static Object getQueryParameters(@Nullable String query) { + return query == null ? VmNull.withoutDefault() : UrlFactory.createQueryParameters(query); + } + } + + public abstract static class toString extends ExternalMethod0Node { + @Child private GetParsedUrlNode getParsedUrlNode = GetParsedUrlNodeGen.create(); + + @Specialization + protected String eval(VmTyped self) { + var parsed = getParsedUrlNode.execute(self); + return parsed.serialize(); + } + } + + public abstract static class resolve extends ExternalMethod1Node { + @Child private GetParsedUrlNode getParsedNode = GetParsedUrlNode.create(); + + @Specialization + protected Object evalString(VmTyped self, String ref) { + return resolve(getParsedNode.execute(self), UrlFactory.parseOrThrow(ref, this)); + } + + @Specialization + protected Object eval(VmTyped self, VmTyped ref) { + return resolve(getParsedNode.execute(self), getParsedNode.execute(ref)); + } + + @SuppressWarnings("MethodNameSameAsClassName") + @TruffleBoundary + private Object resolve(Parsed base, Parsed ref) { + if (base.scheme() == null) { + throw exceptionBuilder() + .evalError("cannotResolveAgainstRelativeUrl", base.serialize()) + .build(); + } + return UrlFactory.create(UrlParser.resolve(base, ref)); + } + } + + public abstract static class normalized extends ExternalPropertyNode { + @Child private GetParsedUrlNode getSelfNode = GetParsedUrlNode.create(); + + @Specialization + protected VmTyped eval(VmTyped self) { + return UrlFactory.create(UrlParser.normalize(getSelfNode.execute(self))); + } + } + + public abstract static class equals extends ExternalMethod1Node { + @Child private GetParsedUrlNode getParsedNode = GetParsedUrlNode.create(); + + @Specialization + protected boolean eval(VmTyped self, VmTyped other) { + return UrlParser.isEquivalent(getParsedNode.execute(self), getParsedNode.execute(other)); + } + } + + // The constraints of `Url`. Each one only accepts a component that is already percent-encoded, + // and is otherwise the same check the parser makes, so that a URL written or amended by hand is + // held to exactly the standard of the parser's output. + + public abstract static class isValidUserInfo extends ExternalMethod1Node { + @Specialization + @TruffleBoundary + protected boolean eval(@SuppressWarnings("unused") VmTyped self, String value) { + return UrlParser.isValidUserInfo(value); + } + } + + public abstract static class isValidQuery extends ExternalMethod1Node { + @Specialization + @TruffleBoundary + protected boolean eval(@SuppressWarnings("unused") VmTyped self, String value) { + return UrlParser.isValidQueryOrFragment(value); + } + } + + public abstract static class isValidFragment extends ExternalMethod1Node { + @Specialization + @TruffleBoundary + protected boolean eval(@SuppressWarnings("unused") VmTyped self, String value) { + return UrlParser.isValidQueryOrFragment(value); + } + } + + public abstract static class isValidScheme extends ExternalMethod1Node { + @Specialization + @TruffleBoundary + protected boolean eval(@SuppressWarnings("unused") VmTyped self, String value) { + return UrlParser.isValidScheme(value); + } + } + + public abstract static class isValidHost extends ExternalMethod1Node { + @Specialization + @TruffleBoundary + protected boolean eval(@SuppressWarnings("unused") VmTyped self, String value) { + return UrlParser.isValidHost(value); + } + } + + public abstract static class isValidPath extends ExternalMethod3Node { + @Specialization + @TruffleBoundary + protected boolean eval( + @SuppressWarnings("unused") VmTyped self, + String value, + boolean hasScheme, + boolean hasAuthority) { + return UrlParser.isValidPath(value, hasScheme, hasAuthority); + } + } +} diff --git a/pkl-core/src/main/java/org/pkl/core/stdlib/net/package-info.java b/pkl-core/src/main/java/org/pkl/core/stdlib/net/package-info.java new file mode 100644 index 000000000..9fa279e49 --- /dev/null +++ b/pkl-core/src/main/java/org/pkl/core/stdlib/net/package-info.java @@ -0,0 +1,19 @@ +/* + * Copyright © 2026 Apple Inc. and the Pkl project authors. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +@NullMarked +package org.pkl.core.stdlib.net; + +import org.jspecify.annotations.NullMarked; diff --git a/pkl-core/src/main/java/org/pkl/core/util/url/PercentEncoder.java b/pkl-core/src/main/java/org/pkl/core/util/url/PercentEncoder.java new file mode 100644 index 000000000..8cbc8216d --- /dev/null +++ b/pkl-core/src/main/java/org/pkl/core/util/url/PercentEncoder.java @@ -0,0 +1,226 @@ +/* + * Copyright © 2026 Apple Inc. and the Pkl project authors. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.pkl.core.util.url; + +import com.oracle.truffle.api.CompilerDirectives.TruffleBoundary; +import java.io.ByteArrayOutputStream; +import java.net.URLDecoder; +import java.net.URLEncoder; +import java.nio.charset.StandardCharsets; +import java.util.function.IntPredicate; + +/** + * Percent-encoding, and the character classes of RFC 3986's ABNF. + * + *

Each {@code IntPredicate} here holds for the characters that may appear literally in a + * component. Anything else has to be percent-encoded to appear there at all. + */ +public final class PercentEncoder { + private PercentEncoder() {} + + /** unreserved = ALPHA / DIGIT / "-" / "." / "_" / "~" */ + static final IntPredicate UNRESERVED = PercentEncoder::isUnreserved; + + /** userinfo = *( unreserved / pct-encoded / sub-delims / ":" ) */ + static final IntPredicate USERINFO = c -> isUnreserved(c) || isSubDelim(c) || c == ':'; + + /** reg-name = *( unreserved / pct-encoded / sub-delims ) */ + static final IntPredicate REG_NAME = c -> isUnreserved(c) || isSubDelim(c); + + /** {@code *( pchar / "/" )}, which covers every shape of path. */ + static final IntPredicate PATH = c -> isPchar(c) || c == '/'; + + /** {@code query} and {@code fragment} share a set: {@code *( pchar / "/" / "?" )}. */ + static final IntPredicate QUERY_OR_FRAGMENT = c -> isPchar(c) || c == '/' || c == '?'; + + private static boolean isUnreserved(int c) { + return isAlpha(c) || isDigit(c) || c == '-' || c == '.' || c == '_' || c == '~'; + } + + /** sub-delims = "!" / "$" / "&" / "'" / "(" / ")" / "*" / "+" / "," / ";" / "=" */ + private static boolean isSubDelim(int c) { + return switch (c) { + case '!', '$', '&', '\'', '(', ')', '*', '+', ',', ';', '=' -> true; + default -> false; + }; + } + + /** pchar = unreserved / pct-encoded / sub-delims / ":" / "@" */ + private static boolean isPchar(int c) { + return isUnreserved(c) || isSubDelim(c) || c == ':' || c == '@'; + } + + static boolean isAlpha(int c) { + return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z'); + } + + static boolean isDigit(int c) { + return c >= '0' && c <= '9'; + } + + @SuppressWarnings("BooleanMethodIsAlwaysInverted") + static boolean isHexDigit(int c) { + return isDigit(c) || (c >= 'a' && c <= 'f') || (c >= 'A' && c <= 'F'); + } + + /** + * Percent-encodes every character of {@code value} that is not {@code allowed}. + * + *

A {@code %} is passed through rather than encoded, so this leaves an already percent-encoded + * value alone. Callers are expected to have checked that every {@code %} begins a percent-encoded + * octet; see {@link UrlParser#percentEncodingFailure}. + */ + static String encode(String value, IntPredicate allowed) { + if (isEncoded(value, allowed)) { + return value; + } + var out = new StringBuilder(value.length()); + value + .codePoints() + .forEach( + codePoint -> { + if (codePoint == '%' || allowed.test(codePoint)) { + out.appendCodePoint(codePoint); + } else { + encodeUtf8(codePoint, out); + } + }); + return out.toString(); + } + + static boolean isEncoded(String value, IntPredicate allowed) { + return value.codePoints().allMatch(c -> c == '%' || allowed.test(c)); + } + + /** + * Returns the percent-encoded {@code value} in the form section 6.2.2 compares it in. + * + *

This makes a component that was written in a different but equivalent way compare equal. + */ + static String normalize(String value) { + if (value.indexOf('%') < 0) { + return value; + } + var out = new StringBuilder(value.length()); + var i = 0; + while (i < value.length()) { + var octet = octetAt(value, i); + if (octet < 0) { + out.append(value.charAt(i)); + i++; + } else { + if (isUnreserved(octet)) { + out.append((char) octet); + } else { + out.append('%').append(toUpperHexDigit(octet >> 4)).append(toUpperHexDigit(octet & 0xF)); + } + i += 3; + } + } + return out.toString(); + } + + private static int octetAt(String value, int index) { + if (value.charAt(index) != '%' + || index + 2 >= value.length() + || !isHexDigit(value.charAt(index + 1)) + || !isHexDigit(value.charAt(index + 2))) { + return -1; + } + return (Character.digit(value.charAt(index + 1), 16) << 4) + | Character.digit(value.charAt(index + 2), 16); + } + + /** UTF-8 percent-encodes {@code codePoint} and appends the result to {@code out}. */ + static void encodeUtf8(int codePoint, StringBuilder out) { + var bytes = new String(Character.toChars(codePoint)).getBytes(StandardCharsets.UTF_8); + for (var b : bytes) { + out.append('%'); + out.append(toUpperHexDigit((b >> 4) & 0xF)); + out.append(toUpperHexDigit(b & 0xF)); + } + } + + /** + * Appends {@code value} to {@code out}, percent-encoding everything that is not {@link + * #UNRESERVED}, {@code %} included. + * + *

Unlike {@link #encode}, this treats its input as entirely undecoded, which is what a value + * that is about to be interpolated into a URL component is. + */ + public static void encodeComponent(StringBuilder out, String value) { + value + .codePoints() + .forEach( + codePoint -> { + if (isUnreserved(codePoint)) { + out.appendCodePoint(codePoint); + } else { + encodeUtf8(codePoint, out); + } + }); + } + + private static char toUpperHexDigit(int value) { + return Character.toUpperCase(Character.forDigit(value, 16)); + } + + /** + * Percent-decodes {@code input} and interprets the decoded bytes as UTF-8. + * + *

Callers are expected to have checked that every {@code %} begins a percent-encoded octet; + * see {@link UrlParser#percentEncodingFailure}. One that does not is kept as-is. + */ + @TruffleBoundary + public static String decode(String input) { + if (input.indexOf('%') < 0) { + return input; + } + var in = input.getBytes(StandardCharsets.UTF_8); + var bytes = new ByteArrayOutputStream(in.length); + for (var i = 0; i < in.length; i++) { + if (in[i] != '%' || i + 2 >= in.length || !isHexDigit(in[i + 1]) || !isHexDigit(in[i + 2])) { + bytes.write(in[i]); + } else { + bytes.write((Character.digit(in[i + 1], 16) << 4) | Character.digit(in[i + 2], 16)); + i += 2; + } + } + return bytes.toString(StandardCharsets.UTF_8); + } + + /** + * Percent-decodes {@code input} as an {@code application/x-www-form-urlencoded}. + * + *

Callers are expected to have checked that every {@code %} begins a percent-encoded octet; + * see {@link UrlParser#percentEncodingFailure}. {@link URLDecoder} throws on one that does not. + */ + @TruffleBoundary + static String decodeForm(String input) { + return URLDecoder.decode(input, StandardCharsets.UTF_8); + } + + /** + * Appends {@code value} to {@code out} as an {@code application/x-www-form-urlencoded}. + * + *

The inverse of {@link #decodeForm}. + */ + @TruffleBoundary + public static void encodeForm(StringBuilder out, String value) { + out.append(URLEncoder.encode(value, StandardCharsets.UTF_8)); + } +} diff --git a/pkl-core/src/main/java/org/pkl/core/util/url/UrlFactory.java b/pkl-core/src/main/java/org/pkl/core/util/url/UrlFactory.java new file mode 100644 index 000000000..70e9f7dd2 --- /dev/null +++ b/pkl-core/src/main/java/org/pkl/core/util/url/UrlFactory.java @@ -0,0 +1,139 @@ +/* + * Copyright © 2026 Apple Inc. and the Pkl project authors. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.pkl.core.util.url; + +import com.oracle.truffle.api.CompilerDirectives.TruffleBoundary; +import com.oracle.truffle.api.nodes.IndirectCallNode; +import com.oracle.truffle.api.nodes.Node; +import org.jspecify.annotations.Nullable; +import org.pkl.core.runtime.Identifier; +import org.pkl.core.runtime.NetModule; +import org.pkl.core.runtime.VmExceptionBuilder; +import org.pkl.core.runtime.VmMapping; +import org.pkl.core.runtime.VmNull; +import org.pkl.core.runtime.VmObjectBuilder; +import org.pkl.core.runtime.VmObjectLike; +import org.pkl.core.runtime.VmTyped; +import org.pkl.core.runtime.VmUtils; +import org.pkl.core.stdlib.VmObjectFactory; +import org.pkl.core.util.url.UrlParser.Parsed; +import org.pkl.core.util.url.UrlParser.Result; + +/** Converts between a parsed URL and {@code pkl:net}'s {@code Url}. */ +public final class UrlFactory { + private UrlFactory() {} + + private static final VmObjectFactory factory = + new VmObjectFactory(NetModule::getUrlClass) + .addProperty("scheme", parsed -> VmNull.lift(parsed.scheme())) + .addProperty("rawUserInfo", parsed -> VmNull.lift(parsed.userInfo())) + .addProperty("rawHost", parsed -> VmNull.lift(parsed.host())) + .addProperty("rawPort", parsed -> VmNull.lift(parsed.port())) + .addStringProperty("rawPath", Parsed::path) + .addProperty("rawQuery", parsed -> VmNull.lift(parsed.query())) + .addProperty("rawFragment", parsed -> VmNull.lift(parsed.fragment())); + + public static VmTyped create(Parsed parsed) { + return factory.create(parsed); + } + + /** Parses {@code input} as a URI reference. */ + @TruffleBoundary + public static Parsed parseOrThrow(String input, Node node) { + var result = UrlParser.parse(input); + if (result instanceof Result.Failure failure) { + throw new VmExceptionBuilder() + .withLocation(node) + .evalError("cannotParseUrl", input) + .withHint(failure.hint()) + .build(); + } + return ((Result.Success) result).url(); + } + + @TruffleBoundary + public static void checkPercentEncoding(String value, String messageKey, Node node) { + var failure = UrlParser.percentEncodingFailure(value); + if (failure != null) { + throw new VmExceptionBuilder() + .withLocation(node) + .evalError(messageKey, value) + .withHint(failure.hint()) + .build(); + } + } + + /** Reads the components back off {@code url}. */ + public static Parsed read(VmTyped url, IndirectCallNode callNode) { + if (url.hasExtraStorage()) { + return (Parsed) url.getExtraStorage(); + } + var parsed = + new Parsed( + readNullableString(url, Identifier.SCHEME, callNode), + readNullableString(url, Identifier.RAW_USER_INFO, callNode), + readNullableString(url, Identifier.RAW_HOST, callNode), + readNullableString(url, Identifier.RAW_PORT, callNode), + (String) VmUtils.readMember(url, Identifier.RAW_PATH, callNode), + readNullableString(url, Identifier.RAW_QUERY, callNode), + readNullableString(url, Identifier.RAW_FRAGMENT, callNode)); + url.setExtraStorage(parsed); + return parsed; + } + + private static @Nullable String readNullableString( + VmObjectLike url, Identifier name, IndirectCallNode callNode) { + return (String) VmNull.unwrap(VmUtils.readMember(url, name, callNode)); + } + + public static String readPath(VmObjectLike url, IndirectCallNode callNode) { + return (String) VmUtils.readMember(url, Identifier.RAW_PATH, callNode); + } + + public static @Nullable String readHost(VmObjectLike url, IndirectCallNode callNode) { + return readNullableString(url, Identifier.RAW_HOST, callNode); + } + + public static @Nullable String readQuery(VmObjectLike url, IndirectCallNode callNode) { + return readNullableString(url, Identifier.RAW_QUERY, callNode); + } + + /** The parameters of {@code query}, as {@code Mapping>}. */ + @TruffleBoundary + public static VmMapping createQueryParameters(String query) { + var builder = new VmObjectBuilder(); + for (var parameter : UrlParser.queryParameters(query).entrySet()) { + var values = new VmObjectBuilder(parameter.getValue().size()); + for (var value : parameter.getValue()) { + values.addElement(value); + } + builder.addEntry(parameter.getKey(), values.toListing()); + } + return builder.toMapping(); + } + + /** Reads the authority off {@code url}, or {@code null} if it has none. */ + public static @Nullable String readAuthority(VmObjectLike url, IndirectCallNode callNode) { + var host = readNullableString(url, Identifier.RAW_HOST, callNode); + if (host == null) { + return null; + } + return UrlParser.serializeAuthority( + readNullableString(url, Identifier.RAW_USER_INFO, callNode), + host, + readNullableString(url, Identifier.RAW_PORT, callNode)); + } +} diff --git a/pkl-core/src/main/java/org/pkl/core/util/url/UrlParser.java b/pkl-core/src/main/java/org/pkl/core/util/url/UrlParser.java new file mode 100644 index 000000000..07e371682 --- /dev/null +++ b/pkl-core/src/main/java/org/pkl/core/util/url/UrlParser.java @@ -0,0 +1,849 @@ +/* + * Copyright © 2026 Apple Inc. and the Pkl project authors. All rights reserved. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.pkl.core.util.url; + +import com.oracle.truffle.api.CompilerDirectives.TruffleBoundary; +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.function.IntPredicate; +import org.jspecify.annotations.Nullable; + +/** + * A parser for RFC 3986 URI references. + * + *

Both shapes of {@code URI-reference} (section 4.1) are accepted: a {@code URI}, which states a + * scheme, and a {@code relative-ref}, which does not. + * + *

Reference resolution is not part of parsing; see {@link #resolve}. + * + *

An IPv6 address may carry the zone identifier of RFC 6874, as in {@code [fe80::1%25eth0]}.. + */ +@SuppressWarnings("JavadocLinkAsPlainText") +public final class UrlParser { + private UrlParser() {} + + /** + * The components of a URI reference, each percent-encoded, so that it holds nothing that cannot + * appear literally in it. + * + *

{@code scheme} is {@code null} for a relative reference. {@code host} is {@code null} when + * the reference has no authority. {@code userInfo} and {@code port} are only ever set alongside a + * host. + */ + public record Parsed( + @Nullable String scheme, + @Nullable String userInfo, + @Nullable String host, + @Nullable String port, + String path, + @Nullable String query, + @Nullable String fragment) { + + /** Serializes these components (section 5.3). */ + @TruffleBoundary + public String serialize() { + var sb = new StringBuilder(); + if (scheme != null) { + sb.append(scheme).append(':'); + } + if (host != null) { + sb.append("//").append(serializeAuthority(userInfo, host, port)); + } + sb.append(path); + if (query != null) { + sb.append('?').append(query); + } + if (fragment != null) { + sb.append('#').append(fragment); + } + return sb.toString(); + } + + public @Nullable String authority() { + return host == null ? null : serializeAuthority(userInfo, host, port); + } + } + + /** Serializes an authority (section 3.2) from its percent-encoded components. */ + @TruffleBoundary + static String serializeAuthority(@Nullable String userInfo, String host, @Nullable String port) { + var sb = new StringBuilder(); + if (userInfo != null) { + sb.append(userInfo).append('@'); + } + sb.append(host); + if (port != null) { + sb.append(':').append(port); + } + return sb.toString(); + } + + // Parsing (https://www.rfc-editor.org/rfc/rfc3986#section-3) + + /** The outcome of parsing. */ + sealed interface Result { + record Success(Parsed url) implements Result {} + + record Failure(String hint) implements Result {} + } + + private static @Nullable String encodeOptional(@Nullable String component, IntPredicate allowed) { + return component == null ? null : PercentEncoder.encode(component, allowed); + } + + /** + * Parses {@code input} as a URI reference, percent-encoding whatever its components cannot hold + * literally. + */ + static Result parse(String input) { + return parseAsWritten(input, true); + } + + private static Result parseAsWritten(String input, boolean encode) { + var length = input.length(); + + // scheme = ALPHA *( ALPHA / DIGIT / "+" / "-" / "." ) ":" + // Absent a scheme, this is a relative reference and any ":" belongs to the path. + String scheme = null; + var pointer = 0; + if (length > 0 && PercentEncoder.isAlpha(input.charAt(0))) { + var end = 1; + while (end < length && isSchemeChar(input.charAt(end))) { + end++; + } + if (end < length && input.charAt(end) == ':') { + scheme = input.substring(0, end); + pointer = end + 1; + } + } + + // authority = [ userinfo "@" ] host [ ":" port ] + String userInfo = null; + String host = null; + String port = null; + if (input.startsWith("//", pointer)) { + pointer += 2; + var end = pointer; + while (end < length && !isAuthorityTerminator(input.charAt(end))) { + end++; + } + var authority = input.substring(pointer, end); + pointer = end; + + // the userinfo runs to the last "@", because "@" cannot appear literally in a host + var at = authority.lastIndexOf('@'); + if (at >= 0) { + userInfo = authority.substring(0, at); + var failure = percentEncodingFailure(userInfo); + if (failure != null) { + return failure; + } + } + host = authority.substring(at + 1); + + var colon = portSeparator(host); + if (colon >= 0) { + port = host.substring(colon + 1); + host = host.substring(0, colon); + if (!isPort(port)) { + return new Result.Failure("The port `" + port + "` can only hold digits."); + } + } + var failure = hostFailure(host); + if (failure != null) { + return failure; + } + } + + var pathEnd = pointer; + while (pathEnd < length && input.charAt(pathEnd) != '?' && input.charAt(pathEnd) != '#') { + pathEnd++; + } + var path = input.substring(pointer, pathEnd); + var pathFailure = pathFailure(path, scheme != null, host != null); + if (pathFailure != null) { + return pathFailure; + } + pointer = pathEnd; + + String query = null; + if (pointer < length && input.charAt(pointer) == '?') { + var end = input.indexOf('#', pointer + 1); + if (end < 0) { + end = length; + } + query = input.substring(pointer + 1, end); + var failure = percentEncodingFailure(query); + if (failure != null) { + return failure; + } + pointer = end; + } + + String fragment = null; + if (pointer < length && input.charAt(pointer) == '#') { + fragment = input.substring(pointer + 1); + var failure = percentEncodingFailure(fragment); + if (failure != null) { + return failure; + } + } + + if (encode) { + return new Result.Success( + new Parsed( + scheme, + encodeOptional(userInfo, PercentEncoder.USERINFO), + // an IP literal has already been validated, and none of its characters may be encoded + host == null || isIpLiteral(host) + ? host + : PercentEncoder.encode(host, PercentEncoder.REG_NAME), + port, + PercentEncoder.encode(path, PercentEncoder.PATH), + encodeOptional(query, PercentEncoder.QUERY_OR_FRAGMENT), + encodeOptional(fragment, PercentEncoder.QUERY_OR_FRAGMENT))); + } + return new Result.Success(new Parsed(scheme, userInfo, host, port, path, query, fragment)); + } + + // Reference resolution (https://www.rfc-editor.org/rfc/rfc3986#section-5.2.2) + + /** + * Resolves {@code ref} against {@code base}. + * + *

This is the strict form of the transform, which never reinterprets a reference that repeats + * the base's scheme as a relative one. + * + *

{@code base} is expected to state a scheme, as section 5.1 requires. Resolving against one + * that does not is not defined, and would only produce another relative reference. + */ + public static Parsed resolve(Parsed base, Parsed ref) { + if (ref.scheme() != null) { + return new Parsed( + ref.scheme(), + ref.userInfo(), + ref.host(), + ref.port(), + removeDotSegments(ref.path(), ref.host() != null), + ref.query(), + ref.fragment()); + } + if (ref.host() != null) { + return new Parsed( + base.scheme(), + ref.userInfo(), + ref.host(), + ref.port(), + removeDotSegments(ref.path()), + ref.query(), + ref.fragment()); + } + String path; + String query; + if (ref.path().isEmpty()) { + path = base.path(); + query = ref.query() != null ? ref.query() : base.query(); + } else { + path = + removeDotSegments( + ref.path().charAt(0) == '/' ? ref.path() : merge(base, ref.path()), + base.host() != null); + query = ref.query(); + } + return new Parsed( + base.scheme(), base.userInfo(), base.host(), base.port(), path, query, ref.fragment()); + } + + /** https://www.rfc-editor.org/rfc/rfc3986#section-5.2.3 */ + private static String merge(Parsed base, String path) { + if (base.host() != null && base.path().isEmpty()) { + return "/" + path; + } + var lastSlash = base.path().lastIndexOf('/'); + return lastSlash < 0 ? path : base.path().substring(0, lastSlash + 1) + path; + } + + private static String removeDotSegments(String path, boolean hasAuthority) { + var result = removeDotSegments(path); + return !hasAuthority && result.startsWith("//") ? "/." + result : result; + } + + /** https://www.rfc-editor.org/rfc/rfc3986#section-5.2.4 */ + static String removeDotSegments(String path) { + var out = new StringBuilder(path.length()); + var length = path.length(); + var pointer = 0; + while (pointer < length) { + if (path.startsWith("../", pointer)) { + pointer += 3; + } else if (path.startsWith("./", pointer)) { + pointer += 2; + } else if (path.startsWith("/./", pointer)) { + pointer += 2; + } else if (pointer + 2 == length && path.startsWith("/.", pointer)) { + out.append('/'); + pointer = length; + } else if (path.startsWith("/../", pointer)) { + removeLastSegment(out); + pointer += 3; + } else if (pointer + 3 == length && path.startsWith("/..", pointer)) { + removeLastSegment(out); + out.append('/'); + pointer = length; + } else if (pointer + 1 == length && path.charAt(pointer) == '.') { + pointer = length; + } else if (pointer + 2 == length && path.startsWith("..", pointer)) { + pointer = length; + } else { + // move the first segment, along with any leading "/", to the output + var end = path.indexOf('/', path.charAt(pointer) == '/' ? pointer + 1 : pointer); + if (end < 0) { + end = length; + } + out.append(path, pointer, end); + pointer = end; + } + } + return out.toString(); + } + + private static void removeLastSegment(StringBuilder out) { + var lastSlash = out.lastIndexOf("/"); + out.setLength(Math.max(lastSlash, 0)); + } + + // Derived views + + /** The percent-decoded segments of {@code path}. */ + @TruffleBoundary + public static List segments(String path) { + if (path.isEmpty()) { + return List.of(); + } + var segments = new ArrayList(); + var start = path.charAt(0) == '/' ? 1 : 0; + while (true) { + var slash = path.indexOf('/', start); + if (slash < 0) { + segments.add(PercentEncoder.decode(path.substring(start))); + return segments; + } + segments.add(PercentEncoder.decode(path.substring(start, slash))); + start = slash + 1; + } + } + + /** + * The percent-decoded parameters of {@code query}, which is read as {@code + * application/x-www-form-urlencoded}. + * + *

A name that repeats collects the values of all its occurrences, in the order they are + * written. A parameter that states no {@code =} holds no value, and contributes nothing. + */ + static Map> queryParameters(String query) { + if (query.isEmpty()) { + return Map.of(); + } + var parameters = new LinkedHashMap>(); + var start = 0; + while (start < query.length()) { + var end = query.indexOf('&', start); + if (end < 0) { + end = query.length(); + } + var separator = query.indexOf('=', start); + // an empty pair ("a=1&&b=2") and a bare name ("a=1&b") both hold no parameter + if (end > start && separator >= 0 && separator < end) { + var name = PercentEncoder.decodeForm(query.substring(start, separator)); + var value = PercentEncoder.decodeForm(query.substring(separator + 1, end)); + parameters.computeIfAbsent(name, ignored -> new ArrayList<>()).add(value); + } + start = end + 1; + } + return parameters; + } + + /** + * Whether {@code left} and {@code right} identify the same resource + * (https://www.rfc-editor.org/rfc/rfc3986#section-6). + * + *

Both are put through the syntax-based normalization of section 6.2.2 first, so that two ways + * of writing the same URL compare equal. A default port is kept. + */ + @TruffleBoundary + public static boolean isEquivalent(Parsed left, Parsed right) { + return normalize(left).equals(normalize(right)); + } + + /** + * The syntax-based normalization of {@code url} + * (https://www.rfc-editor.org/rfc/rfc3986#section-6.2.2). + * + *

Nothing that would take knowing the scheme is normalized, so a default port is kept. + */ + @TruffleBoundary + public static Parsed normalize(Parsed url) { + return new Parsed( + url.scheme() == null ? null : toLowerAscii(url.scheme()), + normalizeOptional(url.userInfo()), + url.host() == null ? null : normalizeHost(url.host()), + url.port(), + normalizePath(url), + normalizeOptional(url.query()), + normalizeOptional(url.fragment())); + } + + private static @Nullable String normalizeOptional(@Nullable String component) { + return component == null ? null : PercentEncoder.normalize(component); + } + + private static String normalizeHost(String host) { + if (!isIpLiteral(host)) { + // the hex digits of a percent-encoded octet are uppercase even in a component that is + // otherwise folded to lower case (section 6.2.2.1) + return toLowerAsciiOutsideOctets(PercentEncoder.normalize(host)); + } + // an IP literal holds nothing that may be encoded, and its zone identifier, unlike the address + // in front of it, names an interface and is case-sensitive + var zone = host.indexOf("%25"); + return zone < 0 + ? toLowerAscii(host) + : toLowerAscii(host.substring(0, zone)) + host.substring(zone); + } + + private static String normalizePath(Parsed url) { + var path = PercentEncoder.normalize(url.path()); + if (path.isEmpty()) { + // a URL with an authority and no path names the same resource as one whose path is "/" + // (section 6.2.3) + return url.host() == null ? path : "/"; + } + // dot segments are only removable from an absolute path + return path.charAt(0) == '/' ? removeDotSegments(path, url.host() != null) : path; + } + + static Result.@Nullable Failure percentEncodingFailure(String input) { + for (var i = input.indexOf('%'); i >= 0; i = input.indexOf('%', i + 3)) { + if (i + 2 >= input.length() + || !PercentEncoder.isHexDigit(input.charAt(i + 1)) + || !PercentEncoder.isHexDigit(input.charAt(i + 2))) { + var octet = input.substring(i, Math.min(i + 3, input.length())); + return new Result.Failure("`" + octet + "` is not a percent-encoded octet."); + } + } + return null; + } + + /** + * Whether {@code input} is already percent-encoded for a component that holds {@code allowed} + * literally: it holds nothing else, and every {@code %} begins a percent-encoded octet. + */ + private static boolean isEncodedComponent(String input, IntPredicate allowed) { + return PercentEncoder.isEncoded(input, allowed) && percentEncodingFailure(input) == null; + } + + // The checks below are stricter than parsing, which encodes whatever a component cannot hold + // literally: each of them only accepts a component that is already encoded. + + /** Whether {@code input} is a percent-encoded {@code userinfo}. */ + public static boolean isValidUserInfo(String input) { + return isEncodedComponent(input, PercentEncoder.USERINFO); + } + + /** Whether {@code input} is a percent-encoded {@code query} or {@code fragment}. */ + public static boolean isValidQueryOrFragment(String input) { + return isEncodedComponent(input, PercentEncoder.QUERY_OR_FRAGMENT); + } + + /** + * Whether {@code input} is an absolute URL that is already written the way RFC 3986 spells one + * out. Backs {@code String.isValidUrl}. + * + *

This is stricter than {@link #parse}, which is lenient in the two ways text taken from the + * outside world usually needs it to be: + * + *

    + *
  • a relative reference, such as {@code ./foo}, parses but is not a URL + *
  • nothing is encoded here, so a character that its component cannot hold literally has to + * already be percent-encoded: {@code http://example.com/some path} is not a URL, but {@code + * http://example.com/some%20path} is + *
+ */ + public static boolean isValidUrl(String input) { + if (!(parseAsWritten(input, false) instanceof Result.Success success)) { + return false; + } + var parsed = success.url(); + return parsed.scheme() != null + && (parsed.userInfo() == null || isValidUserInfo(parsed.userInfo())) + && (parsed.host() == null || isValidHost(parsed.host())) + && isValidPath(parsed.path(), true, parsed.host() != null) + && (parsed.query() == null || isValidQueryOrFragment(parsed.query())) + && (parsed.fragment() == null || isValidQueryOrFragment(parsed.fragment())); + } + + /** Whether {@code input} is a scheme. Unlike parsing, the trailing {@code :} is not accepted. */ + public static boolean isValidScheme(String input) { + if (input.isEmpty() || !PercentEncoder.isAlpha(input.charAt(0))) { + return false; + } + for (var i = 1; i < input.length(); i++) { + if (!isSchemeChar(input.charAt(i))) { + return false; + } + } + return true; + } + + static Result.@Nullable Failure hostFailure(String host) { + if (isIpLiteral(host)) { + if (host.length() < 3) { + return new Result.Failure("The IP literal `" + host + "` is empty."); + } + var address = host.substring(1, host.length() - 1); + // "v" cannot begin an IPv6address, as it is not a hex digit, so it tells the two forms apart + if (address.charAt(0) == 'v' || address.charAt(0) == 'V') { + return isIpvFuture(address) + ? null + : new Result.Failure("`" + address + "` is not an IPvFuture address."); + } + // IPv6addrz = IPv6address "%25" ZoneID (https://www.rfc-editor.org/rfc/rfc6874#section-2) + var zone = address.indexOf("%25"); + if (zone < 0) { + if (isIpv6Address(address)) { + return null; + } + return address.indexOf('%') < 0 + ? new Result.Failure("`" + address + "` is not an IPv6 address.") + : new Result.Failure( + "A zone identifier must be preceded by `%25`, as in `[fe80::1%25eth0]`."); + } + if (zone == 0) { + return new Result.Failure( + "The IP literal `" + host + "` states a zone identifier but no address."); + } + var address6 = address.substring(0, zone); + if (!isIpv6Address(address6)) { + return new Result.Failure("`" + address6 + "` is not an IPv6 address."); + } + var zoneId = address.substring(zone + 3); + if (zoneId.isEmpty()) { + return new Result.Failure("The zone identifier of `" + host + "` is empty."); + } + return isZoneId(zoneId) + ? null + : new Result.Failure( + "The zone identifier `" + + zoneId + + "` may only hold unreserved characters and percent-encoded octets."); + } + if (host.startsWith("[")) { + return host.indexOf(']') < 0 + ? new Result.Failure("The IP literal `" + host + "` is missing a closing `]`.") + : new Result.Failure( + "The IP literal `" + host + "` has trailing characters after its `]`."); + } + if (host.indexOf('[') >= 0 || host.indexOf(']') >= 0) { + return new Result.Failure( + "A registered name cannot contain `[` or `]`; percent-encode them, or enclose an IP" + + " address in brackets."); + } + if (host.indexOf(':') >= 0) { + return new Result.Failure( + "A registered name cannot contain `:`; percent-encode it, or enclose an IPv6 address in brackets."); + } + return percentEncodingFailure(host); + } + + /** + * What {@code host} is: {@code "ipv6"} or {@code "ipvFuture"} for an IP literal, {@code "ipv4"} + * for an {@code IPv4address}, and {@code "name"} for any other, registered, name (section 3.2.2). + */ + public static String hostKind(String host) { + if (isIpLiteral(host)) { + var c = host.charAt(1); + return c == 'v' || c == 'V' ? "ipvFuture" : "ipv6"; + } + return isIpv4Address(host) ? "ipv4" : "name"; + } + + /** + * The percent-decoded zone identifier of {@code host}, or {@code null} if it is not an IPv6 + * address that states one. + */ + @TruffleBoundary + public static @Nullable String zoneId(String host) { + if (!isIpLiteral(host)) { + return null; + } + // "%" cannot appear in an IPvFuture, so it can only begin a zone identifier + var zone = host.indexOf("%25"); + return zone < 0 ? null : PercentEncoder.decode(host.substring(zone + 3, host.length() - 1)); + } + + /** Whether {@code host} is a percent-encoded {@code host}. */ + public static boolean isValidHost(String host) { + return hostFailure(host) == null + && (isIpLiteral(host) || PercentEncoder.isEncoded(host, PercentEncoder.REG_NAME)); + } + + /** + * Whether {@code address} is an {@code IPv6address}: eight 16-bit pieces, the last two of which + * may be written as an {@code IPv4address}, and a single {@code "::"} that stands for one or more + * pieces of zeros (section 3.2.2). + */ + private static boolean isIpv6Address(String address) { + var elision = address.indexOf("::"); + if (elision < 0) { + return countIpv6Pieces(address) == 8; + } + if (address.indexOf("::", elision + 1) >= 0) { + return false; + } + var head = address.substring(0, elision); + var tail = address.substring(elision + 2); + // an IPv4address can only end the address, so it cannot sit in front of the "::" + var headPieces = head.isEmpty() ? 0 : head.indexOf('.') >= 0 ? -1 : countIpv6Pieces(head); + var tailPieces = tail.isEmpty() ? 0 : countIpv6Pieces(tail); + return headPieces >= 0 && tailPieces >= 0 && headPieces + tailPieces <= 7; + } + + private static int countIpv6Pieces(String input) { + var pieces = 0; + var start = 0; + while (true) { + var colon = input.indexOf(':', start); + if (colon < 0) { + if (input.indexOf('.', start) >= 0) { + return isIpv4Address(input.substring(start)) ? pieces + 2 : -1; + } + return isH16(input, start, input.length()) ? pieces + 1 : -1; + } + if (!isH16(input, start, colon)) { + return -1; + } + pieces++; + start = colon + 1; + } + } + + private static boolean isH16(String input, int start, int end) { + if (end <= start || end - start > 4) { + return false; + } + for (var i = start; i < end; i++) { + if (!PercentEncoder.isHexDigit(input.charAt(i))) { + return false; + } + } + return true; + } + + /** IPv4address = dec-octet "." dec-octet "." dec-octet "." dec-octet */ + @TruffleBoundary + private static boolean isIpv4Address(String input) { + var octets = 0; + var start = 0; + while (true) { + var dot = input.indexOf('.', start); + var end = dot < 0 ? input.length() : dot; + if (!isDecOctet(input, start, end)) { + return false; + } + octets++; + if (dot < 0) { + return octets == 4; + } + start = dot + 1; + } + } + + /** A decimal number between 0 and 255, with no leading zeros. */ + private static boolean isDecOctet(String input, int start, int end) { + var length = end - start; + if (length < 1 || length > 3 || (length > 1 && input.charAt(start) == '0')) { + return false; + } + var value = 0; + for (var i = start; i < end; i++) { + var c = input.charAt(i); + if (!PercentEncoder.isDigit(c)) { + return false; + } + value = value * 10 + (c - '0'); + } + return value <= 255; + } + + /** Whether {@code zoneId} is a {@code ZoneID}: {@code 1*( unreserved / pct-encoded )}. */ + private static boolean isZoneId(String zoneId) { + if (zoneId.isEmpty()) { + return false; + } + for (var i = 0; i < zoneId.length(); i++) { + var c = zoneId.charAt(i); + if (c == '%') { + if (i + 2 >= zoneId.length() + || !PercentEncoder.isHexDigit(zoneId.charAt(i + 1)) + || !PercentEncoder.isHexDigit(zoneId.charAt(i + 2))) { + return false; + } + i += 2; + } else if (!PercentEncoder.UNRESERVED.test(c)) { + return false; + } + } + return true; + } + + /** IPvFuture = "v" 1*HEXDIG "." 1*( unreserved / sub-delims / ":" ) */ + private static boolean isIpvFuture(String address) { + var dot = address.indexOf('.'); + if (dot < 2 || dot == address.length() - 1) { + return false; + } + for (var i = 1; i < dot; i++) { + if (!PercentEncoder.isHexDigit(address.charAt(i))) { + return false; + } + } + for (var i = dot + 1; i < address.length(); i++) { + // the tail of an IPvFuture is drawn from the same set as a userinfo + if (!PercentEncoder.USERINFO.test(address.charAt(i))) { + return false; + } + } + return true; + } + + static Result.@Nullable Failure pathFailure( + String path, boolean hasScheme, boolean hasAuthority) { + var failure = percentEncodingFailure(path); + if (failure != null) { + return failure; + } + if (hasAuthority) { + // path-abempty + return path.isEmpty() || path.charAt(0) == '/' + ? null + : new Result.Failure("A path that follows an authority must start with `/`."); + } + if (path.startsWith("//")) { + return new Result.Failure("A path that follows no authority cannot start with `//`."); + } + if (!hasScheme && startsWithColonSegment(path)) { + // path-noscheme (section 4.2) + return new Result.Failure( + "The first segment of a relative path cannot contain `:`; precede it with `./`, as in" + + " `./a:b`."); + } + // path-absolute / path-rootless / path-empty + return null; + } + + /** + * Whether {@code path} is percent-encoded, and can sit next to a scheme and an authority, or next + * to no scheme and no authority, as {@code hasScheme} and {@code hasAuthority} say. + */ + public static boolean isValidPath(String path, boolean hasScheme, boolean hasAuthority) { + return pathFailure(path, hasScheme, hasAuthority) == null + && PercentEncoder.isEncoded(path, PercentEncoder.PATH); + } + + private static boolean isIpLiteral(String host) { + return host.startsWith("[") && host.endsWith("]"); + } + + /** Whether the first segment of {@code path} holds a {@code ":"}. */ + private static boolean startsWithColonSegment(String path) { + var colon = path.indexOf(':'); + if (colon < 0) { + return false; + } + var slash = path.indexOf('/'); + return slash < 0 || colon < slash; + } + + /** + * The index of the {@code ":"} that separates the host from the port in {@code hostAndPort}, or + * {@code -1} if it states no port. + */ + private static int portSeparator(String hostAndPort) { + if (hostAndPort.startsWith("[")) { + var close = hostAndPort.indexOf(']'); + if (close < 0) { + // an unterminated IP literal; let host validation reject it + return -1; + } + var next = close + 1; + return next < hostAndPort.length() && hostAndPort.charAt(next) == ':' ? next : -1; + } + return hostAndPort.lastIndexOf(':'); + } + + /** + * Whether {@code input} is a {@code port}: {@code *DIGIT}, so it may be empty, and its leading + * zeros are part of it. + */ + private static boolean isPort(String input) { + for (var i = 0; i < input.length(); i++) { + if (!PercentEncoder.isDigit(input.charAt(i))) { + return false; + } + } + return true; + } + + private static boolean isSchemeChar(int c) { + return PercentEncoder.isAlpha(c) + || PercentEncoder.isDigit(c) + || c == '+' + || c == '-' + || c == '.'; + } + + /** The gen-delims that end an authority and start the component after it. */ + private static boolean isAuthorityTerminator(int c) { + return c == '/' || c == '?' || c == '#'; + } + + private static String toLowerAscii(String input) { + var out = new StringBuilder(input.length()); + for (var i = 0; i < input.length(); i++) { + var c = input.charAt(i); + out.append((char) ((c >= 'A' && c <= 'Z') ? c + 0x20 : c)); + } + return out.toString(); + } + + /** Lowercases {@code input}, leaving the hex digits of its percent-encoded octets as written. */ + private static String toLowerAsciiOutsideOctets(String input) { + var out = new StringBuilder(input.length()); + for (var i = 0; i < input.length(); i++) { + var c = input.charAt(i); + if (c == '%') { + out.append(input, i, i + 3); + i += 2; + } else { + out.append((char) ((c >= 'A' && c <= 'Z') ? c + 0x20 : c)); + } + } + return out.toString(); + } +} diff --git a/pkl-core/src/main/resources/org/pkl/core/errorMessages.properties b/pkl-core/src/main/resources/org/pkl/core/errorMessages.properties index 055283e09..c18a2868c 100644 --- a/pkl-core/src/main/resources/org/pkl/core/errorMessages.properties +++ b/pkl-core/src/main/resources/org/pkl/core/errorMessages.properties @@ -809,6 +809,20 @@ Error parsing YAML document: The number of aliases for collection nodes exceeds \n\ To increase the allowed maximum, set `YamlRenderer.maxCollectionAliases`. +cannotParseUrl=\ +Cannot parse `{0}` as a URL. + +cannotDecodeUrlComponent=\ +Cannot percent-decode `{0}`. + +cannotParseQuery=\ +Cannot parse `{0}` as a query. + +cannotResolveAgainstRelativeUrl=\ +Cannot resolve a reference against `{0}`, which is a relative URL.\n\ +\n\ +Only a URL that states a scheme can be resolved against. + evaluationTimedOut=\ Evaluation timed out after {0,number,#.##} second{0,choice,0 net.Url(input)) == "Cannot parse `\(input)` as a URL." + +local function resolved(ref: String): String = net.Url(rfcBase).resolve(ref).toString() + +local gClef = "\u{1D11E}" + +facts { + ["components"] { + componentsOf("https://user:pw@example.com:8080/a/b?q=1#f") + == List("https", "user:pw", "example.com", "8080", "/a/b", "q=1", "f") + componentsOf("https://example.com") + == List("https", null, "example.com", null, "", null, null) + componentsOf("https://example.com/a?#") + == List("https", null, "example.com", null, "/a", "", "") + componentsOf("mailto:user@example.com") + == List("mailto", null, null, null, "user@example.com", null, null) + + // a null host is not an empty one + componentsOf("file:/tmp") == List("file", null, null, null, "/tmp", null, null) + componentsOf("file:///tmp") == List("file", null, "", null, "/tmp", null, null) + + // an empty port is still one + componentsOf("http://example.com:/a") == List("http", null, "example.com", "", "/a", null, null) + + // the userinfo runs to the last "@", so the extra one lands in it and is encoded + componentsOf("http://a@b@example.com/") == List("http", "a%40b", "example.com", null, "/", null, null) + } + + ["parse(): keeps what is already encoded as it was written"] { + net.Url("HTTPS://example.com").scheme == "HTTPS" + net.Url("http://EXAMPLE.com/A").rawHost == "EXAMPLE.com" + roundTrips("http://EXAMPLE.com/A") + + net.Url("https://example.com:443/a").port == 443 + roundTrips("https://example.com:443/a") + // a port is held as written, leading zeros and all + net.Url("x://foo:001").rawPort == "001" + net.Url("x://foo:001").port == 1 + net.Url("http://example.com/a/b/../c/./d").rawPath == "/a/b/../c/./d" + roundTrips("http://example.com/a/b/../c/./d") + net.Url("http://example.com/%41%2Fb").rawPath == "/%41%2Fb" + roundTrips("http://example.com/%41%2fb") + roundTrips("http://0x7f.0.0.1/") + roundTrips("http://1.2.3.4./") + roundTrips("http://[2001:db8:0:0:0:0:0:1]/") + } + + ["parse(): round-trips valid URIs"] { + roundTrips("https://example.com/a?q=1#f") + roundTrips("http://1.2.3.4/") + roundTrips("http://[2001:db8::1]/") + roundTrips("http://[v7.future]/") + roundTrips("file:///a/b") + roundTrips("foo://example.com/a") + roundTrips("mailto:user@example.com") + roundTrips("urn:isbn:0451450523") + roundTrips("https://example.com/a%20b?q=%C3%A9#f%2F") + roundTrips("https://example.com") + roundTrips("https://example.com/a?#") + roundTrips("http://a/b/c/d;p?q") + roundTrips("foo:") + // every sub-delim, ":" and "@" may appear literally in a path, query and fragment + roundTrips("http://example.com/!$&'()*+,;=:@?!$&'()*+,;=:@/?#!$&'()*+,;=:@/?") + } + + ["parse(): encodes what it cannot keep"] { + net.Url("http://example.com/a b").rawPath == "/a%20b" + net.Url("http://example.com/a%20b").rawPath == "/a%20b" + parsed("http://example.com/a b") == "http://example.com/a%20b" + parsed("https://example.com/ac") == "https://example.com/a%3Cb%3Ec" + parsed("https://example.com/?a b") == "https://example.com/?a%20b" + parsed("https://example.com/#a b") == "https://example.com/#a%20b" + parsed("https://example.com/münchen?q=café") == "https://example.com/m%C3%BCnchen?q=caf%C3%A9" + parsed("https://example.com/a" + gClef) == "https://example.com/a%F0%9D%84%9E" + parsed("http://münchen.de/") == "http://m%C3%BCnchen.de/" + parsed("http://exa mple.com/") == "http://exa%20mple.com/" + parsed("http://a@b@example.com/") == "http://a%40b@example.com/" + parsed("http://u:p@example.com/") == "http://u:p@example.com/" + // a "#" after the one that starts the fragment is part of it + net.Url("https://example.com/#a#b").rawFragment == "a%23b" + + // the components are encoded, not just their serialization + componentsOf("https://u v@exa mple.com/a b?c d#e f") + == List("https", "u%20v", "exa%20mple.com", null, "/a%20b", "c%20d", "e%20f") + + // once encoded, the result round-trips + roundTrips("http://example.com/a%20b") + parsed(parsed("http://example.com/a b")) == "http://example.com/a%20b" + net.Url("http://example.com/a b") == net.Url("http://example.com/a%20b") + } + + ["decoded components"] { + local url = net.Url("https://u%20v@m%C3%BCnchen.de/a%20b/c?q=a%20b+c#f%20g") + url.userInfo == "u v" + url.host == "münchen.de" + url.path == "/a b/c" + // a query is not read as a form here, so a "+" is kept + url.query == "q=a b+c" + url.fragment == "f g" + + // it makes no difference whether the input was encoded + net.Url("https://example.com/a b").path == "/a b" + net.Url("https://example.com/a%20b").path == "/a b" + + // an absent component has no decoded form either + local bare = net.Url("mailto:user@example.com") + bare.userInfo == null + bare.host == null + bare.path == "user@example.com" + bare.query == null + bare.fragment == null + net.Url("https://example.com").path == "" + + // an encoded "/" decodes to one, which only pathSegments tells apart + net.Url("https://example.com/a%2Fb").path == "/a/b" + net.Url("https://example.com/a%2Fb").pathSegments == List("a/b") + + net.Url("http://[fe80::1%25eth0]/").host == "[fe80::1%eth0]" + + // the decoded components follow an amendment + (url) { rawPath = "/x%20y" }.path == "/x y" + (url) { rawQuery = null }.query == null + } + + ["parse(): failing"] { + // a "%" that does not begin a percent-encoded octet + throws("http://example.com/a%zz") + throws("http://example.com/a%2") + throws("http://example.com/a%") + throws("http://example.com/?a%zz") + throws("http://example.com/#a%zz") + throws("http://u%zz@example.com/") + throws("http://exa%zzmple.com/") + + // a ":" in an authority always separates a port, so what follows has to be one + throws("https://example.com:port/") + throws("https://example.com:-1/") + throws("https://example.com:+1/") + + // and only the last one separates it, so any earlier ":" is left in a registered name, which + // cannot hold one; an IPv6 address has to be bracketed + throws("http://::/foo") + throws("http://:::/foo") + throws("http://::1/foo") + throws("http://a:b:8080/") + // a single ":" still separates a port, which may be empty + roundTrips("http://:/") + net.Url("http://:/").port == null + roundTrips("http://[::]/foo") + roundTrips("http://[::1]/foo") + // an encoded ":" is a registered name that holds one, and is left alone + roundTrips("http://%3A%3A/foo") + + // a malformed IP literal + throws("http://[::1") + throws("http://[::1x]/") + throws("http://[]/") + throws("http://[::1]x/") + + // an IPv6 address is held to its grammar, not just to the characters it is built from + roundTrips("http://[::ffff:192.0.2.1]/") + throws("http://[1]/") + throws("http://[1::2::3]/") + throws("http://[1:2:3:4:5:6:7:8:9]/") + throws("http://[::192.0.2.256]/") + // the same holds for a relative reference, which is checked no less than an absolute URL + throws("a/%zz") + throws("//example.com:port/a") + throws("//[::1x]/a") + } + + ["parse(): RFC 6874 zone identifiers"] { + roundTrips("http://[fe80::1%25eth0]/") + roundTrips("http://[fe80::1%25eth0]:8080/a?q#f") + net.Url("http://[fe80::1%25eth0]/").rawHost == "[fe80::1%25eth0]" + roundTrips("http://[fe80::1%25a-b._~1]/") + roundTrips("http://[fe80::1%25%41]/") + throws("http://[fe80::1%eth0]/") + throws("http://[fe80::1%65th0]/") + throws("http://[fe80::1%25]/") + throws("http://[%25eth0]/") + throws("http://[fe80::1%25eth 0]/") + throws("http://[fe80::1%25eth%zz]/") + throws("http://[fe80::1%25a:b]/") + throws("http://[v7.future%25eth0]/") + roundTrips("http://fe80--1%25eth0/") + } + + ["parse(): relative references"] { + componentsOf("a/foo") == List(null, null, null, null, "a/foo", null, null) + componentsOf("../c") == List(null, null, null, null, "../c", null, null) + componentsOf("/a/b?q#f") == List(null, null, null, null, "/a/b", "q", "f") + componentsOf("//example.com/a") == List(null, null, "example.com", null, "/a", null, null) + componentsOf("?q") == List(null, null, null, null, "", "q", null) + componentsOf("#f") == List(null, null, null, null, "", null, "f") + componentsOf("") == List(null, null, null, null, "", null, null) + + roundTrips("a/foo") + roundTrips("../c") + roundTrips("/a/b?q#f") + roundTrips("//example.com/a") + roundTrips("//user@example.com:8080") + roundTrips("?q") + roundTrips("#f") + roundTrips("") + + // almost any text is a well-formed relative reference, so it is the scheme, and not parsing + // succeeding, that tells a URL apart from arbitrary text + net.Url("not a url").scheme == null + parsed("not a url") == "not%20a%20url" + + // a hand-built relative URL is a parsed one + new net.Url { rawPath = "a/foo" } == net.Url("a/foo") + new net.Url { rawPath = "a/foo" }.toString() == "a/foo" + new net.Url { rawPath = "a/foo" }.pathSegments == List("a", "foo") + module.catch(() -> new net.Url { rawPath = "a:b" }.toString()) + .startsWith("Type constraint `isValidPath(this, scheme != null, rawHost != null)` violated.") + } + + ["resolve(): RFC 3986, section 5.4.1 normal examples"] { + resolved("g:h") == "g:h" + resolved("g") == "http://a/b/c/g" + resolved("./g") == "http://a/b/c/g" + resolved("g/") == "http://a/b/c/g/" + resolved("/g") == "http://a/g" + resolved("//g") == "http://g" + resolved("?y") == "http://a/b/c/d;p?y" + resolved("g?y") == "http://a/b/c/g?y" + resolved("#s") == "http://a/b/c/d;p?q#s" + resolved("g#s") == "http://a/b/c/g#s" + resolved("g?y#s") == "http://a/b/c/g?y#s" + resolved(";x") == "http://a/b/c/;x" + resolved("g;x") == "http://a/b/c/g;x" + resolved("g;x?y#s") == "http://a/b/c/g;x?y#s" + resolved("") == "http://a/b/c/d;p?q" + resolved(".") == "http://a/b/c/" + resolved("./") == "http://a/b/c/" + resolved("..") == "http://a/b/" + resolved("../") == "http://a/b/" + resolved("../g") == "http://a/b/g" + resolved("../..") == "http://a/" + resolved("../../") == "http://a/" + resolved("../../g") == "http://a/g" + } + + ["resolve(): RFC 3986, section 5.4.2 abnormal examples"] { + // a path cannot be backed out of its root + resolved("../../../g") == "http://a/g" + resolved("../../../../g") == "http://a/g" + resolved("/./g") == "http://a/g" + resolved("/../g") == "http://a/g" + + // only a complete "." or ".." segment is a dot segment + resolved("g.") == "http://a/b/c/g." + resolved(".g") == "http://a/b/c/.g" + resolved("g..") == "http://a/b/c/g.." + resolved("..g") == "http://a/b/c/..g" + + resolved("./../g") == "http://a/b/g" + resolved("./g/.") == "http://a/b/c/g/" + resolved("g/./h") == "http://a/b/c/g/h" + resolved("g/../h") == "http://a/b/c/h" + resolved("g;x=1/./y") == "http://a/b/c/g;x=1/y" + resolved("g;x=1/../y") == "http://a/b/c/y" + + // dot segments are a path notion; a query or fragment keeps them + resolved("g?y/./x") == "http://a/b/c/g?y/./x" + resolved("g?y/../x") == "http://a/b/c/g?y/../x" + resolved("g#s/./x") == "http://a/b/c/g#s/./x" + resolved("g#s/../x") == "http://a/b/c/g#s/../x" + + resolved("http:g") == "http:g" + } + + ["resolve(): other cases"] { + resolved("../c") == "http://a/b/c" + net.Url("mailto:user@example.com").resolve("#f").toString() + == "mailto:user@example.com#f" + net.Url("foo://example.com/a/b").resolve("../c").toString() == "foo://example.com/c" + net.Url("file:///a/b").resolve("c").toString() == "file:///a/c" + // a reference that cannot be parsed cannot be resolved + module.catch(() -> net.Url(rfcBase).resolve("%zz")) == "Cannot parse `%zz` as a URL." + net.Url("http://a/b/c/").resolve(new net.Url { rawPath = "d%20e" }).toString() + == "http://a/b/c/d%20e" + // resolve a string or object should be the same + net.Url("http://a/b/c/").resolve("d e").rawPath == "/b/c/d%20e" + net.Url("http://a/b/c/").resolve(new net.Url { rawPath = "d%20e" }).rawPath == "/b/c/d%20e" + // resolving is the way to set a component without encoding it first + net.Url("http://a/b/c/").resolve("/x y").toString() == "http://a/x%20y" + // a path without an authority that would start with "//" is prefixed with "/." + net.Url("foo:/a").resolve("..//x").toString() == "foo:/.//x" + net.Url("foo:/a").resolve("..//x") == new net.Url { scheme = "foo"; rawPath = "/.//x" } + } + + ["new Url"] { + new net.Url { scheme = "https"; rawHost = "example.com" }.toString() == "https://example.com" + new net.Url { scheme = "https"; rawHost = "example.com"; rawPath = "/a/b" }.toString() + == "https://example.com/a/b" + new net.Url { + scheme = "https" + rawUserInfo = "user:pw" + rawHost = "example.com" + rawPort = "8080" + rawPath = "/a/b" + rawQuery = "q=1" + rawFragment = "f" + }.toString() == "https://user:pw@example.com:8080/a/b?q=1#f" + + // a part has to be given encoded + new net.Url { scheme = "https"; rawHost = "example.com"; rawPath = "/a%20b" }.toString() + == "https://example.com/a%20b" + new net.Url { scheme = "https"; rawHost = "m%C3%BCnchen.de" }.toString() == "https://m%C3%BCnchen.de" + module.catch(() -> new net.Url { scheme = "https"; rawHost = "example.com"; rawPath = "/a b" }.toString()) + .startsWith("Type constraint `isValidPath(this, scheme != null, rawHost != null)` violated.") + module.catch(() -> new net.Url { scheme = "https"; rawHost = "münchen.de" }.toString()) + .startsWith("Type constraint `isValidHost(this)` violated.") + module.catch(() -> new net.Url { scheme = "https"; rawHost = "example.com"; rawUserInfo = "a@b" }.toString()) + .startsWith("Type constraint `isValidUserInfo(this)` violated.") + module.catch(() -> new net.Url { scheme = "https"; rawHost = "example.com"; rawQuery = "a b" }.toString()) + .startsWith("Type constraint `isValidQuery(this)` violated.") + module.catch(() -> new net.Url { scheme = "https"; rawHost = "example.com"; rawFragment = "a#b" }.toString()) + .startsWith("Type constraint `isValidFragment(this)` violated.") + module.catch(() -> new net.Url { scheme = "https"; rawHost = "example.com"; rawPath = "/a%zz" }.toString()) + .startsWith("Type constraint `isValidPath(this, scheme != null, rawHost != null)` violated.") + module.catch(() -> new net.Url { scheme = "https"; rawHost = "example.com"; rawPort = "8a" }.toString()) + .startsWith(##"Type constraint `matches(Regex(#"\d*"#))` violated."##) + new net.Url { scheme = "x"; rawHost = "foo"; rawPort = "001" }.toString() == "x://foo:001" + + // a URL without an authority + new net.Url { scheme = "mailto"; rawPath = "user@example.com" }.toString() == "mailto:user@example.com" + new net.Url { scheme = "foo" }.toString() == "foo:" + + new net.Url { scheme = "https"; rawHost = "example.com"; rawPath = "/a" } + == net.Url("https://example.com/a") + + // a host is held to the same rules as a parsed one: a registered name cannot hold a ":", but + // an IP literal and an encoded ":" are fine + module.catch(() -> new net.Url { scheme = "http"; rawHost = ":" }.toString()) + .startsWith("Type constraint `isValidHost(this)` violated.") + module.catch(() -> new net.Url { scheme = "http"; rawHost = "::" }.toString()) + .startsWith("Type constraint `isValidHost(this)` violated.") + new net.Url { scheme = "http"; rawHost = "[::]" }.toString() == "http://[::]" + new net.Url { scheme = "http"; rawHost = "%3A" }.toString() == "http://%3A" + } + + ["amending a Url"] { + local url = net.Url("https://user:pw@example.com:8080/a/b?q=1#f") + + (url) { rawPath = "/c" }.toString() == "https://user:pw@example.com:8080/c?q=1#f" + (url) { rawQuery = null }.toString() == "https://user:pw@example.com:8080/a/b#f" + (url) { rawFragment = null; rawQuery = null }.toString() == "https://user:pw@example.com:8080/a/b" + (url) { rawPort = "0080" }.port == 80 + (url) { rawUserInfo = null }.toString() == "https://example.com:8080/a/b?q=1#f" + (url) { scheme = "http" }.toString() == "http://user:pw@example.com:8080/a/b?q=1#f" + + // the derived views follow the amendment + (url) { rawPath = "/c/d%20e" }.pathSegments == List("c", "d e") + // an amended part has to be encoded + module.catch(() -> (url) { rawPath = "/a b" }.toString()) + .startsWith("Type constraint `isValidPath(this, scheme != null, rawHost != null)` violated.") + (url) { rawPath = "/a%20b" }.toString() == "https://user:pw@example.com:8080/a%20b?q=1#f" + } + + ["hostKind and zoneId"] { + net.Url("http://[v1.fe80::a+en1]/").hostKind == "ipvFuture" + net.Url("http://192.0.2.256/").hostKind == "name" + net.Url("file:///tmp").hostKind == "name" + net.Url("http://[fe80::1%25en%2F1]/").zoneId == "en/1" + new net.Url { scheme = "http"; rawHost = "[fe80::1%25eth0]" }.zoneId == "eth0" + } + + ["pathSegments"] { + net.Url("https://example.com/a/b").pathSegments == List("a", "b") + net.Url("https://example.com/").pathSegments == List("") + net.Url("https://example.com/a/").pathSegments == List("a", "") + net.Url("https://example.com").pathSegments == List() + net.Url("https://example.com/a/b%20c").pathSegments == List("a", "b c") + net.Url("https://example.com/m%C3%BCnchen").pathSegments == List("münchen") + // a path is split before its segments are decoded, so an encoded "/" stays in its segment + net.Url("https://example.com/a%2Fb").pathSegments == List("a/b") + // a rootless path has segments too + net.Url("mailto:user@example.com").pathSegments == List("user@example.com") + net.Url("urn:isbn:0451450523").pathSegments == List("isbn:0451450523") + } + + ["queryParameters"] { + net.Url("https://example.com?a=1&&b=2").queryParameters + == new Mapping> { ["a"] { "1" }; ["b"] { "2" } } + net.Url("https://example.com?").queryParameters == new Mapping {} + net.Url("https://example.com").queryParameters == null + // a bare name holds no value, so it contributes nothing + net.Url("https://example.com?a").queryParameters == new Mapping {} + net.Url("https://example.com?a=").queryParameters == new Mapping> { ["a"] { "" } } + net.Url("https://example.com?=1").queryParameters == new Mapping> { [""] { "1" } } + // a name that repeats holds every value it is given, in order + net.Url("https://example.com?a=1&a=2").queryParameters == new Mapping> { ["a"] { "1"; "2" } } + net.Url("https://example.com?a&a=2").queryParameters == new Mapping> { ["a"] { "2" } } + net.Url("https://example.com?a=&a=2").queryParameters == new Mapping> { ["a"] { ""; "2" } } + net.Url("https://example.com?a=1&b=2&a=3").queryParameters + == new Mapping> { ["a"] { "1"; "3" }; ["b"] { "2" } } + net.Url("https://example.com?a%20b=c%20d").queryParameters == new Mapping> { ["a b"] { "c d" } } + net.Url("https://example.com?a+b=c+d").queryParameters == new Mapping> { ["a b"] { "c d" } } + } + + ["equality"] { + net.Url("https://example.com/a") == net.Url("https://example.com/a") + net.Url("https://example.com/a") != net.Url("https://example.com/b") + // nothing is normalized, so these are distinct URLs + net.Url("https://example.com/a") != net.Url("https://example.com:443/a") + net.Url("x://foo:001") != net.Url("x://foo:1") + net.Url("https://example.com/a") != net.Url("https://EXAMPLE.com/a") + net.Url("https://example.com/%41") != net.Url("https://example.com/A") + net.Url("https://example.com/a") != net.Url("HTTPS://example.com/a") + } + + ["normalized"] { + local function normalized(input: String): String = net.Url(input).normalized.toString() + + normalized("HTTPS://EXAMPLE.com/A") == "https://example.com/A" + normalized("http://[2001:DB8::1]/") == "http://[2001:db8::1]/" + normalized("http://[FE80::1%25eTH0]/") == "http://[fe80::1%25eTH0]/" + normalized("https://example.com/%7Euser") == "https://example.com/~user" + normalized("https://U%7Ev@example.com/?q=%7E#%7E") == "https://U~v@example.com/?q=~#~" + normalized("https://example.com/a%2fb") == "https://example.com/a%2Fb" + normalized("https://münchen.de/") == "https://m%C3%BCnchen.de/" + normalized("https://example.com/a/./b/../c") == "https://example.com/a/c" + normalized("foo:/bar/baz") == "foo:/bar/baz" + normalized("file:///../a") == "file:///a" + normalized("file:///foo/%2E%2E/bar") == "file:///bar" + normalized("a/../b") == "a/../b" + normalized("mailto:user@example.com") == "mailto:user@example.com" + normalized("https://example.com") == "https://example.com/" + normalized("foo:") == "foo:" + normalized("https://example.com:443/") == "https://example.com:443/" + normalized("https://example.com/?#") == "https://example.com/?#" + normalized("//EXAMPLE.com/a/../b") == "//example.com/b" + normalized("") == "" + normalized("foo:/..//x") == "foo:/.//x" + // normalizing is idempotent + net.Url("HTTPS://Example.com/a/../%7Eb c").normalized.normalized + == net.Url("HTTPS://Example.com/a/../%7Eb c").normalized + + net.Url("HTTPS://Example.com/%7Euser").normalized == net.Url("https://example.com/~user").normalized + net.Url("HTTPS://Example.com/%7Euser").normalized == net.Url("https://example.com/~user") + net.Url("https://example.com/a%2Fb").normalized != net.Url("https://example.com/a/b").normalized + + net.Url("HTTPS://User%7E@EXAMPLE.com:8080/a/../%7Eb?q=%7E#%7E").normalized + == new net.Url { + scheme = "https" + rawUserInfo = "User~" + rawHost = "example.com" + rawPort = "8080" + rawPath = "/~b" + rawQuery = "q=~" + rawFragment = "~" + } + } + + ["equals()"] { + local function equal(left: String, right: String): Boolean = + net.Url(left).equals(net.Url(right)) + + // the scheme and host are case-insensitive + equal("FILE:///foo%20bar/baz", "file:///foo bar/baz") + equal("https://EXAMPLE.com/a", "https://example.com/a") + equal("http://[2001:DB8::1]/a", "http://[2001:db8::1]/a") + // but nothing else is + !equal("https://example.com/A", "https://example.com/a") + + // percent-encoding is normalized in every component + equal("https://u%20v@example.com/?q=a b#a b", "https://u v@example.com/?q=a%20b#a%20b") + equal("https://example.com/m%C3%BCnchen", "https://example.com/münchen") + // an octet standing for a reserved character means something else decoded, so it is kept + !equal("https://example.com/a%2Fb", "https://example.com/a/b") + + // dot segments are removed from an absolute path + equal("file:///foo/bar", "file:///foo/./bar") + equal("file:///foo/bar", "file:///foo/baz/../bar") + equal("file:///foo/bar/", "file:///foo/bar/baz/../") + // percent-encoding is normalized first, so an encoded dot segment is one too + equal("file:///foo/%2E%2E/bar", "file:///bar") + // a trailing "/" is part of the path, and a path ending in ".." keeps one + !equal("file:///foo/bar", "file:///foo/bar/") + // a rootless path is compared as written + !equal("../a", "a") + !equal("../a", "../../a") + equal("a/b", "a/b") + + // a URL with an authority and no path is one whose path is "/" + equal("https://example.com", "https://example.com/") + // but a URL without an authority is not + !equal("foo:", "foo:/") + + // a default port is only dropped by scheme-based normalization, which this does not do + !equal("https://example.com:443/a", "https://example.com/a") + } +} diff --git a/pkl-core/src/test/files/LanguageSnippetTests/input/api/urlHelpers.pkl b/pkl-core/src/test/files/LanguageSnippetTests/input/api/urlHelpers.pkl new file mode 100644 index 000000000..e59066351 --- /dev/null +++ b/pkl-core/src/test/files/LanguageSnippetTests/input/api/urlHelpers.pkl @@ -0,0 +1,98 @@ +amends "../snippetTest.pkl" + +import "pkl:net" + +local gClef = "\u{1D11E}" + +facts { + ["encodeUrlComponent()"] { + net.encodeUrlComponent("a b&c") == "a%20b%26c" + net.encodeUrlComponent("münchen") == "m%C3%BCnchen" + // only the unreserved characters are left alone + net.encodeUrlComponent("aZ09-._~") == "aZ09-._~" + net.encodeUrlComponent("!'()*") == "%21%27%28%29%2A" + net.encodeUrlComponent("/?#[]@:;=+$,%") == "%2F%3F%23%5B%5D%40%3A%3B%3D%2B%24%2C%25" + net.encodeUrlComponent("a" + gClef + "b") == "a%F0%9D%84%9Eb" + // an encoded component is safe to drop into any component of a URL + new net.Url { + scheme = "https" + rawHost = "example.com" + rawPath = "/" + net.encodeUrlComponent("a/b") + }.pathSegments == List("a/b") + } + + ["decodeUrlComponent()"] { + net.decodeUrlComponent("a%20b%26c") == "a b&c" + net.decodeUrlComponent("m%C3%BCnchen") == "münchen" + // "+" is not a space; that is an application/x-www-form-urlencoded convention + net.decodeUrlComponent("a+b") == "a+b" + // a literal "%" has to be written as "%25" + net.decodeUrlComponent("100%25") == "100%" + net.decodeUrlComponent("a%F0%9D%84%9Eb") == "a" + gClef + "b" + net.decodeUrlComponent("%FF") == "\u{FFFD}" + net.decodeUrlComponent(net.encodeUrlComponent("a b&c=d?e#f/münchen+%")) == "a b&c=d?e#f/münchen+%" + net.decodeUrlComponent(net.encodeUrlComponent(gClef)) == gClef + } + + ["buildQuery()"] { + net.buildQuery(new Mapping> { ["a"] { "1" }; ["b"] { "2" } }) == "a=1&b=2" + net.buildQuery(new Mapping {}) == "" + // a name is written once per value it holds + net.buildQuery(new Mapping> { ["a"] { "1"; "2" } }) == "a=1&a=2" + net.buildQuery(new Mapping> { ["a"] { "1"; "2" }; ["b"] { "3" } }) == "a=1&a=2&b=3" + // one that holds no values is left out entirely + net.buildQuery(new Mapping> { ["a"] {} }) == "" + net.buildQuery(new Mapping> { ["a"] {}; ["b"] { "1" } }) == "b=1" + net.buildQuery(new Mapping> { ["a"] { "" } }) == "a=" + net.buildQuery(new Mapping> { [""] { "1" } }) == "=1" + net.buildQuery(new Mapping> { ["a b"] { "c d" } }) == "a+b=c+d" + net.buildQuery(new Mapping> { ["a&b"] { "1=2" } }) == "a%26b=1%3D2" + // only the ASCII alphanumerics and "*-._" are left alone, as WHATWG's serializer does + net.buildQuery(new Mapping> { ["q"] { "aZ09*-._" } }) == "q=aZ09*-._" + net.buildQuery(new Mapping> { ["q"] { "~!'()" } }) == "q=%7E%21%27%28%29" + + local parameters = new Mapping> { + ["a b"] { "c&d" } + ["e"] { "1"; "2" } + ["f"] { "" } + ["münchen"] { "a+b" } + } + net.Url("https://example.com?" + net.buildQuery(parameters)).queryParameters == parameters + (net.Url("https://example.com/a?q=1")) { + rawQuery = net.buildQuery(new Mapping> { ["a"] { "b c" } }) + }.toString() == "https://example.com/a?a=b+c" + } + + ["parseQuery()"] { + net.parseQuery("a=1&b=2") == new Mapping> { ["a"] { "1" }; ["b"] { "2" } } + net.parseQuery("") == new Mapping {} + net.parseQuery("a=1&&b=2") == new Mapping> { ["a"] { "1" }; ["b"] { "2" } } + // a bare name holds no value, so it contributes nothing + net.parseQuery("a") == new Mapping {} + net.parseQuery("a&b=1") == new Mapping> { ["b"] { "1" } } + net.parseQuery("a=") == new Mapping> { ["a"] { "" } } + net.parseQuery("=1") == new Mapping> { [""] { "1" } } + // a name that repeats holds every value it is given, in order + net.parseQuery("a=1&a=2") == new Mapping> { ["a"] { "1"; "2" } } + net.parseQuery("a=1&b=2&a=3") + == new Mapping> { ["a"] { "1"; "3" }; ["b"] { "2" } } + // only the first "=" separates a name from its value + net.parseQuery("a=1=2") == new Mapping> { ["a"] { "1=2" } } + net.parseQuery("a%20b=c%20d") == new Mapping> { ["a b"] { "c d" } } + net.parseQuery("a+b=c+d") == new Mapping> { ["a b"] { "c d" } } + net.parseQuery("q=m%C3%BCnchen") == new Mapping> { ["q"] { "münchen" } } + net.parseQuery("q=%7E*") == new Mapping> { ["q"] { "~*" } } + + // this is how a Url reads its own query + local url = net.Url("https://example.com?a=1&a=2&b=3") + net.parseQuery(url.rawQuery!!) == url.queryParameters + // and the inverse of buildQuery() + local parameters = new Mapping> { + ["a b"] { "c&d" } + ["e"] { "1"; "2" } + ["f"] { "" } + ["münchen"] { "a+b" } + } + net.parseQuery(net.buildQuery(parameters)) == parameters + } +} diff --git a/pkl-core/src/test/files/LanguageSnippetTests/input/errors/cannotDecodeUrlComponent.pkl b/pkl-core/src/test/files/LanguageSnippetTests/input/errors/cannotDecodeUrlComponent.pkl new file mode 100644 index 000000000..47b23f4d8 --- /dev/null +++ b/pkl-core/src/test/files/LanguageSnippetTests/input/errors/cannotDecodeUrlComponent.pkl @@ -0,0 +1,3 @@ +import "pkl:net" + +res = net.decodeUrlComponent("100%") diff --git a/pkl-core/src/test/files/LanguageSnippetTests/input/errors/cannotDecodeUrlComponent2.pkl b/pkl-core/src/test/files/LanguageSnippetTests/input/errors/cannotDecodeUrlComponent2.pkl new file mode 100644 index 000000000..41811b4b7 --- /dev/null +++ b/pkl-core/src/test/files/LanguageSnippetTests/input/errors/cannotDecodeUrlComponent2.pkl @@ -0,0 +1,3 @@ +import "pkl:net" + +res = net.decodeUrlComponent("foo%zzbar") diff --git a/pkl-core/src/test/files/LanguageSnippetTests/input/errors/cannotParseQuery.pkl b/pkl-core/src/test/files/LanguageSnippetTests/input/errors/cannotParseQuery.pkl new file mode 100644 index 000000000..26b725b4e --- /dev/null +++ b/pkl-core/src/test/files/LanguageSnippetTests/input/errors/cannotParseQuery.pkl @@ -0,0 +1,3 @@ +import "pkl:net" + +res = net.parseQuery("a=1&b=100%") diff --git a/pkl-core/src/test/files/LanguageSnippetTests/input/errors/cannotResolveAgainstRelativeUrl.pkl b/pkl-core/src/test/files/LanguageSnippetTests/input/errors/cannotResolveAgainstRelativeUrl.pkl new file mode 100644 index 000000000..11c7e12c9 --- /dev/null +++ b/pkl-core/src/test/files/LanguageSnippetTests/input/errors/cannotResolveAgainstRelativeUrl.pkl @@ -0,0 +1,3 @@ +import "pkl:net" + +res = net.Url("a/foo")!!.resolve("b") diff --git a/pkl-core/src/test/files/LanguageSnippetTests/output/api/reflectedDeclaration.pcf b/pkl-core/src/test/files/LanguageSnippetTests/output/api/reflectedDeclaration.pcf index 73b2212ff..4cab83d1c 100644 --- a/pkl-core/src/test/files/LanguageSnippetTests/output/api/reflectedDeclaration.pcf +++ b/pkl-core/src/test/files/LanguageSnippetTests/output/api/reflectedDeclaration.pcf @@ -959,6 +959,36 @@ alias { allAnnotations = List(new { version = "0.29.0" }) + }, "isValidUrl", new { + location { + line = XXXX + column = 3 + displayUri = "https://github.com/apple/pkl/blob/$commitId/stdlib/base.pkl#LXXXX" + } + docComment = """ + Tells if this is a valid absolute URL according to + [RFC 3986](https://www.rfc-editor.org/rfc/rfc3986). + + Stricter than [net.Url()], which also parses a relative reference, and which percent-encodes + whatever cannot appear literally in a URL component. + + Facts: + ``` + "https://example.com/a?q=1#f".isValidUrl + "mailto:user@example.com".isValidUrl + !"./foo".isValidUrl + !"http://example.com/some path".isValidUrl + ``` + """ + annotations = List(new { + version = "0.33.0" + }) + modifiers = Set() + name = "isValidUrl" + allModifiers = Set() + allAnnotations = List(new { + version = "0.33.0" + }) }, "md5", new { location { line = XXXX @@ -1324,6 +1354,36 @@ alias { allAnnotations = List(new { version = "0.29.0" }) + }, "isValidUrl", new { + location { + line = XXXX + column = 3 + displayUri = "https://github.com/apple/pkl/blob/$commitId/stdlib/base.pkl#LXXXX" + } + docComment = """ + Tells if this is a valid absolute URL according to + [RFC 3986](https://www.rfc-editor.org/rfc/rfc3986). + + Stricter than [net.Url()], which also parses a relative reference, and which percent-encodes + whatever cannot appear literally in a URL component. + + Facts: + ``` + "https://example.com/a?q=1#f".isValidUrl + "mailto:user@example.com".isValidUrl + !"./foo".isValidUrl + !"http://example.com/some path".isValidUrl + ``` + """ + annotations = List(new { + version = "0.33.0" + }) + modifiers = Set() + name = "isValidUrl" + allModifiers = Set() + allAnnotations = List(new { + version = "0.33.0" + }) }, "md5", new { location { line = XXXX diff --git a/pkl-core/src/test/files/LanguageSnippetTests/output/api/string.pcf b/pkl-core/src/test/files/LanguageSnippetTests/output/api/string.pcf index bda6f39e8..3c02b73c3 100644 --- a/pkl-core/src/test/files/LanguageSnippetTests/output/api/string.pcf +++ b/pkl-core/src/test/files/LanguageSnippetTests/output/api/string.pcf @@ -24,6 +24,23 @@ facts { true true } + ["isValidUrl"] { + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + } ["lastIndex"] { true true diff --git a/pkl-core/src/test/files/LanguageSnippetTests/output/api/url.pcf b/pkl-core/src/test/files/LanguageSnippetTests/output/api/url.pcf new file mode 100644 index 000000000..d54af0dae --- /dev/null +++ b/pkl-core/src/test/files/LanguageSnippetTests/output/api/url.pcf @@ -0,0 +1,335 @@ +facts { + ["components"] { + true + true + true + true + true + true + true + true + } + ["parse(): keeps what is already encoded as it was written"] { + true + true + true + true + true + true + true + true + true + true + true + true + true + true + } + ["parse(): round-trips valid URIs"] { + true + true + true + true + true + true + true + true + true + true + true + true + true + true + } + ["parse(): encodes what it cannot keep"] { + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + } + ["decoded components"] { + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + } + ["parse(): failing"] { + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + } + ["parse(): RFC 6874 zone identifiers"] { + true + true + true + true + true + true + true + true + true + true + true + true + true + true + } + ["parse(): relative references"] { + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + } + ["resolve(): RFC 3986, section 5.4.1 normal examples"] { + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + } + ["resolve(): RFC 3986, section 5.4.2 abnormal examples"] { + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + } + ["resolve(): other cases"] { + true + true + true + true + true + true + true + true + true + true + true + } + ["new Url"] { + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + } + ["amending a Url"] { + true + true + true + true + true + true + true + true + true + } + ["hostKind and zoneId"] { + true + true + true + true + true + } + ["pathSegments"] { + true + true + true + true + true + true + true + true + true + } + ["queryParameters"] { + true + true + true + true + true + true + true + true + true + true + true + true + } + ["equality"] { + true + true + true + true + true + true + true + } + ["normalized"] { + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + } + ["equals()"] { + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + } +} diff --git a/pkl-core/src/test/files/LanguageSnippetTests/output/api/urlHelpers.pcf b/pkl-core/src/test/files/LanguageSnippetTests/output/api/urlHelpers.pcf new file mode 100644 index 000000000..145f6f7f1 --- /dev/null +++ b/pkl-core/src/test/files/LanguageSnippetTests/output/api/urlHelpers.pcf @@ -0,0 +1,55 @@ +facts { + ["encodeUrlComponent()"] { + true + true + true + true + true + true + true + } + ["decodeUrlComponent()"] { + true + true + true + true + true + true + true + true + } + ["buildQuery()"] { + true + true + true + true + true + true + true + true + true + true + true + true + true + true + } + ["parseQuery()"] { + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + true + } +} diff --git a/pkl-core/src/test/files/LanguageSnippetTests/output/errors/analyzeInvalidModuleUri.err b/pkl-core/src/test/files/LanguageSnippetTests/output/errors/analyzeInvalidModuleUri.err index c622c437d..9252d57e9 100644 --- a/pkl-core/src/test/files/LanguageSnippetTests/output/errors/analyzeInvalidModuleUri.err +++ b/pkl-core/src/test/files/LanguageSnippetTests/output/errors/analyzeInvalidModuleUri.err @@ -1,12 +1,19 @@ –– Pkl Error –– -Module URI `foo <>` has invalid syntax. +Type constraint `isValidUrl` violated. +Value: "foo <>" + + isValidUrl + │ + false + +xxxx | typealias Uri = String(isValidUrl) + ^^^^^^^^^^ +at pkl.analyze#importGraph (pkl:base) x | result = analyze.importGraph(Set("foo <>")) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ at analyzeInvalidModuleUri#result (file:///$snippetsDir/input/errors/analyzeInvalidModuleUri.pkl) -Illegal character in path at index 3: foo <> - xxx | renderer.renderDocument(value) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ at pkl.base#Module.output.text (pkl:base) diff --git a/pkl-core/src/test/files/LanguageSnippetTests/output/errors/analyzeRelativeModuleUri.err b/pkl-core/src/test/files/LanguageSnippetTests/output/errors/analyzeRelativeModuleUri.err index ddd46283d..5e7a6a677 100644 --- a/pkl-core/src/test/files/LanguageSnippetTests/output/errors/analyzeRelativeModuleUri.err +++ b/pkl-core/src/test/files/LanguageSnippetTests/output/errors/analyzeRelativeModuleUri.err @@ -1,5 +1,14 @@ –– Pkl Error –– -Cannot analyze relative module URI `foo.pkl`. +Type constraint `isValidUrl` violated. +Value: "foo.pkl" + + isValidUrl + │ + false + +xxxx | typealias Uri = String(isValidUrl) + ^^^^^^^^^^ +at pkl.analyze#importGraph (pkl:base) x | result = analyze.importGraph(Set("foo.pkl")) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/pkl-core/src/test/files/LanguageSnippetTests/output/errors/cannotDecodeUrlComponent.err b/pkl-core/src/test/files/LanguageSnippetTests/output/errors/cannotDecodeUrlComponent.err new file mode 100644 index 000000000..ddf067052 --- /dev/null +++ b/pkl-core/src/test/files/LanguageSnippetTests/output/errors/cannotDecodeUrlComponent.err @@ -0,0 +1,16 @@ +–– Pkl Error –– +Cannot percent-decode `100%`. + +x | res = net.decodeUrlComponent("100%") + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +at cannotDecodeUrlComponent#res (file:///$snippetsDir/input/errors/cannotDecodeUrlComponent.pkl) + +`%` is not a percent-encoded octet. + +xxx | renderer.renderDocument(value) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +at pkl.base#Module.output.text (pkl:base) + +xxx | if (renderer is BytesRenderer) renderer.renderDocument(value) else text.encodeToBytes("UTF-8") + ^^^^ +at pkl.base#Module.output.bytes (pkl:base) diff --git a/pkl-core/src/test/files/LanguageSnippetTests/output/errors/cannotDecodeUrlComponent2.err b/pkl-core/src/test/files/LanguageSnippetTests/output/errors/cannotDecodeUrlComponent2.err new file mode 100644 index 000000000..7c64a7f80 --- /dev/null +++ b/pkl-core/src/test/files/LanguageSnippetTests/output/errors/cannotDecodeUrlComponent2.err @@ -0,0 +1,16 @@ +–– Pkl Error –– +Cannot percent-decode `foo%zzbar`. + +x | res = net.decodeUrlComponent("foo%zzbar") + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +at cannotDecodeUrlComponent2#res (file:///$snippetsDir/input/errors/cannotDecodeUrlComponent2.pkl) + +`%zz` is not a percent-encoded octet. + +xxx | renderer.renderDocument(value) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +at pkl.base#Module.output.text (pkl:base) + +xxx | if (renderer is BytesRenderer) renderer.renderDocument(value) else text.encodeToBytes("UTF-8") + ^^^^ +at pkl.base#Module.output.bytes (pkl:base) diff --git a/pkl-core/src/test/files/LanguageSnippetTests/output/errors/cannotFindStdLibModule.err b/pkl-core/src/test/files/LanguageSnippetTests/output/errors/cannotFindStdLibModule.err index 8e835110d..2d46bb94f 100644 --- a/pkl-core/src/test/files/LanguageSnippetTests/output/errors/cannotFindStdLibModule.err +++ b/pkl-core/src/test/files/LanguageSnippetTests/output/errors/cannotFindStdLibModule.err @@ -16,6 +16,7 @@ pkl:EvaluatorSettings pkl:json pkl:jsonnet pkl:math +pkl:net pkl:pklbinary pkl:platform pkl:Project diff --git a/pkl-core/src/test/files/LanguageSnippetTests/output/errors/cannotParseQuery.err b/pkl-core/src/test/files/LanguageSnippetTests/output/errors/cannotParseQuery.err new file mode 100644 index 000000000..7322d5862 --- /dev/null +++ b/pkl-core/src/test/files/LanguageSnippetTests/output/errors/cannotParseQuery.err @@ -0,0 +1,16 @@ +–– Pkl Error –– +Cannot parse `a=1&b=100%` as a query. + +x | res = net.parseQuery("a=1&b=100%") + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +at cannotParseQuery#res (file:///$snippetsDir/input/errors/cannotParseQuery.pkl) + +`%` is not a percent-encoded octet. + +xxx | renderer.renderDocument(value) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +at pkl.base#Module.output.text (pkl:base) + +xxx | if (renderer is BytesRenderer) renderer.renderDocument(value) else text.encodeToBytes("UTF-8") + ^^^^ +at pkl.base#Module.output.bytes (pkl:base) diff --git a/pkl-core/src/test/files/LanguageSnippetTests/output/errors/cannotResolveAgainstRelativeUrl.err b/pkl-core/src/test/files/LanguageSnippetTests/output/errors/cannotResolveAgainstRelativeUrl.err new file mode 100644 index 000000000..86803a0b1 --- /dev/null +++ b/pkl-core/src/test/files/LanguageSnippetTests/output/errors/cannotResolveAgainstRelativeUrl.err @@ -0,0 +1,16 @@ +–– Pkl Error –– +Cannot resolve a reference against `a/foo`, which is a relative URL. + +x | res = net.Url("a/foo")!!.resolve("b") + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +at cannotResolveAgainstRelativeUrl#res (file:///$snippetsDir/input/errors/cannotResolveAgainstRelativeUrl.pkl) + +Only a URL that states a scheme can be resolved against. + +xxx | renderer.renderDocument(value) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +at pkl.base#Module.output.text (pkl:base) + +xxx | if (renderer is BytesRenderer) renderer.renderDocument(value) else text.encodeToBytes("UTF-8") + ^^^^ +at pkl.base#Module.output.bytes (pkl:base) diff --git a/pkl-core/src/test/files/LanguageSnippetTests/output/errors/intrinsifiedTypeAlias4.err b/pkl-core/src/test/files/LanguageSnippetTests/output/errors/intrinsifiedTypeAlias4.err index 1847cadba..64b194a5c 100644 --- a/pkl-core/src/test/files/LanguageSnippetTests/output/errors/intrinsifiedTypeAlias4.err +++ b/pkl-core/src/test/files/LanguageSnippetTests/output/errors/intrinsifiedTypeAlias4.err @@ -2,7 +2,7 @@ Expected value of type `String`, but got type `Int`. Value: 42 -xxxx | typealias Uri = String +xxxx | typealias Uri = String(isValidUrl) ^^^^^^ at intrinsifiedTypeAlias4#res1 (pkl:base) diff --git a/pkl-core/src/test/files/LanguageSnippetTests/output/projects/badPklProject4/bug.err b/pkl-core/src/test/files/LanguageSnippetTests/output/projects/badPklProject4/bug.err index 4cfca8b82..ed95eea35 100644 --- a/pkl-core/src/test/files/LanguageSnippetTests/output/projects/badPklProject4/bug.err +++ b/pkl-core/src/test/files/LanguageSnippetTests/output/projects/badPklProject4/bug.err @@ -1,6 +1,15 @@ –– Pkl Error –– -URI `file:///not a valid uri` has invalid syntax. +Type constraint `isValidUrl` violated. +Value: "file:///not a valid uri" -xxx | moduleCacheDir = resolvePath(enclosingUri, module.moduleCacheDir!!, forWindows) - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -at pkl.EvaluatorSettings#resolveForOs.moduleCacheDir (pkl:EvaluatorSettings) + isValidUrl + │ + false + +xxxx | typealias Uri = String(isValidUrl) + ^^^^^^^^^^ +at pkl.EvaluatorSettings#resolve (pkl:base) + +xxx | fixed resolvedEvaluatorSettings: EvaluatorSettingsModule = evaluatorSettings.resolve(projectFileUri) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +at pkl.Project#resolvedEvaluatorSettings (pkl:Project) diff --git a/pkl-gradle/src/test/kotlin/org/pkl/gradle/PkldocGeneratorsTest.kt b/pkl-gradle/src/test/kotlin/org/pkl/gradle/PkldocGeneratorsTest.kt index d48186b8a..bdc1a5dea 100644 --- a/pkl-gradle/src/test/kotlin/org/pkl/gradle/PkldocGeneratorsTest.kt +++ b/pkl-gradle/src/test/kotlin/org/pkl/gradle/PkldocGeneratorsTest.kt @@ -54,8 +54,8 @@ class PkldocGeneratorsTest : AbstractTest() { version = "1.0.0" importUri = "https://pkl-lang.org/" authors { "publisher@apple.com" } - sourceCode = "sources.apple.com/" - issueTracker = "issues.apple.com" + sourceCode = "https://sources.apple.com/" + issueTracker = "https://issues.apple.com" """ .trimIndent(), ) diff --git a/stdlib/base.pkl b/stdlib/base.pkl index 2796d0658..36736349b 100644 --- a/stdlib/base.pkl +++ b/stdlib/base.pkl @@ -23,6 +23,7 @@ module pkl.base import "pkl:json" import "pkl:jsonnet" import "pkl:math" +import "pkl:net" import "pkl:pklbinary" // noinspection Deprecated import "pkl:protobuf" @@ -1340,6 +1341,22 @@ external class String extends Any { @Since { version = "0.29.0" } external isBase64: Boolean + /// Tells if this is a valid absolute URL according to + /// [RFC 3986](https://www.rfc-editor.org/rfc/rfc3986). + /// + /// Stricter than [net.Url()], which also parses a relative reference, and which percent-encodes + /// whatever cannot appear literally in a URL component. + /// + /// Facts: + /// ``` + /// "https://example.com/a?q=1#f".isValidUrl + /// "mailto:user@example.com".isValidUrl + /// !"./foo".isValidUrl + /// !"http://example.com/some path".isValidUrl + /// ``` + @Since { version = "0.33.0" } + external isValidUrl: Boolean + /// The [MD5](https://en.wikipedia.org/wiki/MD5) /// hash of this string's UTF-8 byte sequence /// as hexadecimal string. @@ -1704,7 +1721,7 @@ external class String extends Any { typealias Charset = "UTF-8" | "UTF-16" | "ISO-8859-1" /// A string representing a [URI](https://en.wikipedia.org/wiki/Uniform_Resource_Identifier). -typealias Uri = String +typealias Uri = String(isValidUrl) /// Constructs a regular expression described by [pattern]. /// diff --git a/stdlib/net.pkl b/stdlib/net.pkl new file mode 100644 index 000000000..9ee18d9a4 --- /dev/null +++ b/stdlib/net.pkl @@ -0,0 +1,349 @@ +//===----------------------------------------------------------------------===// +// Copyright © 2026 Apple Inc. and the Pkl project authors. All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// https://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +//===----------------------------------------------------------------------===// + +/// Utilities for networking, like URL parsing. +@ModuleInfo { minPklVersion = "0.33.0" } +@Since { version = "0.33.0" } +module pkl.net + +/// Constructs a Url as described by [input], following +/// [RFC 3986](https://www.rfc-editor.org/rfc/rfc3986). +/// Accepts a URL reference (one without a scheme). +/// +/// This is lenient: whatever a component cannot hold literally is percent-encoded, and whatever is +/// already percent-encoded is kept as is. +/// +/// Throws if [input] is not a valid URL. +/// +/// To test if [input] is a valid URL, use [String.isValidUrl]. +/// +/// Facts: +/// ``` +/// net.Url("https://example.com/foo?a=1").host == "example.com" +/// net.Url("https://example.com/foo?a=1").query == "a=1" +/// net.Url("https://example.com/a b").rawPath == "/a%20b" +/// net.Url("/foo").scheme == null +/// ``` +external const function Url(input: String): Url + +/// A URL, as defined by [RFC 3986](https://www.rfc-editor.org/rfc/rfc3986). +/// [RFC 6874](https://www.rfc-editor.org/rfc/rfc6874) IPv6 zone literals are also supported. +/// +/// Every component is held percent-encoded, in its `raw` property, such as [rawPath]: +/// [Url()] encodes whatever a component cannot hold literally, and a component that is set by hand, +/// or amended, has to be encoded already. +/// The percent-decoded form of each component is available without the `raw` prefix, such as +/// [path]. +/// +/// Facts: +/// ``` +/// net.Url("https://example.com/a b").rawPath == "/a%20b" +/// net.Url("https://example.com/a b").path == "/a b" +/// net.Url("https://example.com/a%20b").rawPath == "/a%20b" +/// ``` +class Url { + /// The scheme, without the trailing `:`, or [null] if this is a relative reference. + scheme: String(isValidScheme(this))? + + /// The percent-encoded userinfo, without the trailing `@`, or [null] if this URL has none. + /// + /// Only a URL with a [rawHost] can have userinfo. + rawUserInfo: String(rawHost != null, isValidUserInfo(this))? + + /// The percent-decoded [rawUserInfo]. + hidden fixed userInfo: String? = + if (rawUserInfo != null) decodeUrlComponent(rawUserInfo) else null + + /// The percent-encoded host, or [null] if this URL has no authority. + /// + /// A [null] host is not the same as an empty one: `file:/tmp` has no host, `file:///tmp` has an + /// empty one. + /// + /// An IPv6 address must be enclosed in brackets, as in `[::1]`. + /// It may carry a zone identifier, as described in RFC 6874. + /// Anything else is a registered name, which cannot contain `[`, `]` or `:` literally. + rawHost: String(isValidHost(this))? + + /// The percent-decoded [rawHost]. + hidden fixed host: String? = if (rawHost != null) decodeUrlComponent(rawHost) else null + + /// The port, without the leading `:`, or [null] if this URL states none. + /// + /// Only a URL with a [rawHost] can have a port. + rawPort: String(rawHost != null, matches(Regex(#"\d*"#)))? + + /// The number that [rawPort] states. + /// + /// This is [null] if [rawPort] is [null], or if [rawPort] is not parseable as a [UInt16]. + /// + /// Facts: + /// ``` + /// net.Url("https://example.com:8080/").port == 8080 + /// net.Url("https://example.com:/").port == null + /// ``` + hidden fixed port: UInt16? = + let (parsed = rawPort?.toIntOrNull()) + if (parsed is UInt16) parsed else null + + /// The percent-encoded path, or the empty string if this URL has none. + /// + /// With a [rawHost], the path is empty or starts with `/`. + /// Without one, it may be *rootless*, as in `mailto:user@example.com`. + /// Without a [scheme], its first segment cannot contain `:`. + /// + /// To change the path without encoding it first, use [resolve()], which is as lenient as [Url()]. + /// Unlike amending, this drops [rawQuery] and [rawFragment]. + rawPath: String(isValidPath(this, scheme != null, rawHost != null)) = "" + + /// The percent-decoded [rawPath]. + /// + /// An encoded `/` is decoded like any other octet, so it cannot be distinguished from a `/` that + /// separates segments. + /// [pathSegments] disambiguates between separator and literal `/` characters. + /// + /// Facts: + /// ``` + /// net.Url("https://example.com/a%2Fb/c").path == "/a/b/c" + /// net.Url("https://example.com/a%2Fb/c").pathSegments == List("a/b", "c") + /// ``` + hidden fixed path: String = decodeUrlComponent(rawPath) + + /// The percent-encoded query, without the leading `?`, or [null] if this URL has none. + /// + /// The empty string is a URL ending in a bare `?`. + rawQuery: String(isValidQuery(this))? + + /// The percent-decoded [rawQuery]. + /// + /// Unlike [queryParameters], this does not read [rawQuery] as + /// `application/x-www-form-urlencoded`, so a `+` is kept as is. + hidden fixed query: String? = if (rawQuery != null) decodeUrlComponent(rawQuery) else null + + /// The percent-encoded fragment, without the leading `#`, or [null] if this URL has none. + /// + /// The empty string is a URL ending in a bare `#`. + rawFragment: String(isValidFragment(this))? + + /// The percent-decoded [rawFragment]. + hidden fixed fragment: String? = + if (rawFragment != null) decodeUrlComponent(rawFragment) else null + + /// The serialization of this URL. + fixed stringValue: String = toString() + + /// Tells if this URL is absolute, that is, has a non-null scheme. + hidden fixed isAbsolute: Boolean = scheme != null + + /// Tells if this URL's path is absolute, that is, starts with `/`. + hidden fixed hasAbsolutePath: Boolean = rawPath.startsWith("/") + + /// The authority, without the leading `//`, or [null] if this URL has none. + /// + /// A URL has an authority exactly when it has a [rawHost]. + /// This is [rawUserInfo], [rawHost] and [rawPort] joined back together. + external fixed authority: String? + + /// What [rawHost] is, or [null] if this URL has no authority. + /// + /// - `"ipv6"`: an IPv6 address, such as `[::1]` + /// - `"ipvFuture"`: an IP address of a version that RFC 3986 leaves for the future + /// - `"ipv4"`: an IPv4 address in dotted-decimal form, such as `192.0.2.1` + /// - `"name"`: any other host, a *registered name* + /// + /// Facts: + /// ``` + /// net.Url("https://example.com/").hostKind == "name" + /// net.Url("https://192.0.2.1/").hostKind == "ipv4" + /// net.Url("https://[::1]/").hostKind == "ipv6" + /// net.Url("/foo").hostKind == null + /// ``` + external fixed hostKind: ("name" | "ipv4" | "ipv6" | "ipvFuture")? + + /// The percent-decoded zone identifier of an IPv6 [rawHost], as described in RFC 6874, or [null] + /// if it states none. + /// + /// Facts: + /// ``` + /// net.Url("http://[fe80::1%25eth0]/").zoneId == "eth0" + /// net.Url("http://[fe80::1]/").zoneId == null + /// ``` + external fixed zoneId: String? + + /// The percent-decoded segments of [rawPath]. + /// + /// [pathSegments] does not tell if the path is absolute or not. + /// A relative URL reference `"a/b"` also has path segments `List("a", "b")`. + /// To disambiguate between the two, use [hasAbsolutePath]. + /// + /// Facts: + /// ``` + /// net.Url("https://example.com/a/b%20c").pathSegments == List("a", "b c") + /// net.Url("https://example.com/a/b%2Fc").pathSegments == List("a", "b/c") + /// ``` + external fixed pathSegments: List + + /// The percent-decoded parameters of [rawQuery], or [null] if this URL has no query. + /// + /// [rawQuery] is read as `application/x-www-form-urlencoded`. + /// A name that repeats holds the values of all its occurrences, in the order they are written. + /// A parameter that states no `=` holds no value at all, and is left out; this is not the same as + /// a bare `=`, which gives it the empty string. + /// + /// Facts: + /// ``` + /// net.Url("https://example.com/?q=a+b&q=c").queryParameters["q"] == new Listing { "a b"; "c" } + /// net.Url("https://example.com/?").queryParameters.isEmpty + /// net.Url("https://example.com/").queryParameters == null + /// ``` + external fixed queryParameters: Mapping>? + + /// Resolves [ref] against this URL. + /// + /// [ref] is either a relative reference, which is resolved against this URL, or an absolute URL, + /// which replaces it entirely. + /// Dot segments (`./` and `../`) are removed from the result's path, normalizing it. + /// + /// Throws if [ref] is a string that cannot be parsed, or if this URL is itself a relative + /// reference. + /// + /// Facts: + /// ``` + /// net.Url("https://example.com/a/b").resolve("../c").toString() == "https://example.com/c" + /// net.Url("https://example.com/a/b").resolve("#top").toString() == "https://example.com/a/b#top" + /// net.Url("foo:/a").resolve("..//x").toString() == "foo:/.//x" + /// ``` + external function resolve(ref: String | Url): Url + + /// The normalized version of this URL, following *syntax-based normalization* as described in + /// [section 6.2.2](https://www.rfc-editor.org/info/rfc3986/#section-6.2.2) of RFC 3986. + /// + /// [scheme] and [rawHost] are lowercased, as they are case-insensitive. + /// Percent-encoding is normalized: an octet standing for an unreserved character is decoded, and + /// any other octet is kept, with its hex digits uppercased. + /// + /// Dot segments (`./` and `../`) are only removed from an absolute [rawPath]. + /// A URL with a [rawHost] and no path is given the path `/`, which names the same resource. + /// + /// Does not perform normalization that requires knowing the scheme, such as removing a default + /// port (see [section 6.2.3](https://www.rfc-editor.org/info/rfc3986/#section-6.2.3)). + /// + /// Facts: + /// ``` + /// net.Url("HTTPS://Example.com/%7Euser").normalized == net.Url("https://example.com/~user") + /// net.Url("https://example.com/a/b/../c").normalized.rawPath == "/a/c" + /// net.Url("https://example.com").normalized.rawPath == "/" + /// ``` + external hidden fixed normalized: Url + + /// Tells if this URL is equal to [other] according to *syntax-based normalization* as + /// described in [section 6.2.2](https://www.rfc-editor.org/info/rfc3986/#section-6.2.2) of + /// RFC 3986. + /// + /// Returns [true] if the [normalized] versions of [this] and [other] are the + /// same, and [false] otherwise. + /// + /// Facts: + /// ``` + /// net.Url("HTTPS://Example.com/%7Euser").equals(net.Url("https://example.com/~user")) + /// !(net.Url("https://example.com/a%2Fb").equals(net.Url("https://example.com/a/b"))) + /// !(net.Url("https://example.com:443/").equals(net.Url("https://example.com/"))) + /// ``` + external function equals(other: Url): Boolean + + /// The serialization of this URL. + /// + /// Facts: + /// ``` + /// net.Url("https://example.com/a b").toString() == "https://example.com/a%20b" + /// net.Url("https://example.com/é").toString() == "https://example.com/%C3%A9" + /// net.Url("HTTPS://example.com").toString() == "HTTPS://example.com" + /// ``` + external function toString(): String + + /// Tells if [value] is a percent-encoded `userinfo`. + external function isValidUserInfo(value: String): Boolean + + /// Tells if [value] is a `scheme`. + external function isValidScheme(value: String): Boolean + + /// Tells if [value] is a percent-encoded `host`. + external function isValidHost(value: String): Boolean + + /// Tells if [value] is a percent-encoded path that can sit next to a scheme and an authority, or, + /// when [hasScheme] or [hasAuthority] is [false], next to no scheme or no authority. + external function isValidPath(value: String, hasScheme: Boolean, hasAuthority: Boolean): Boolean + + /// Tells if [value] is a percent-encoded `query`. + external function isValidQuery(value: String): Boolean + + /// Tells if [value] is a percent-encoded `fragment`. + external function isValidFragment(value: String): Boolean +} + +/// Percent-encodes [value] so that it can be interpolated into any URL component. +/// +/// Every character other than an ASCII alphanumeric or one of `-._~`, the *unreserved* characters +/// of RFC 3986, is percent-encoded as its UTF-8 bytes. +/// +/// Facts: +/// ``` +/// net.encodeUrlComponent("a b&c") == "a%20b%26c" +/// net.encodeUrlComponent("50%") == "50%25" +/// ``` +external const function encodeUrlComponent(value: String): String + +/// Percent-decodes [value], interpreting the decoded bytes as UTF-8. +/// +/// Throws if any `%` in [value] does not begin a percent-encoded octet. +/// To write a literal `%`, encode it as `%25`. +/// +/// Facts: +/// ``` +/// net.decodeUrlComponent("a%20b%26c") == "a b&c" +/// net.decodeUrlComponent("50%25") == "50%" +/// ``` +external const function decodeUrlComponent(value: String): String + +/// Builds a query from [parameters], for use as a [Url.rawQuery]. +/// +/// The result is `application/x-www-form-urlencoded`. +/// A name is written once per value it holds; one that holds no values is left out entirely. +/// +/// [Url.queryParameters] reads back exactly the parameters given here, less any name that holds no +/// values. +/// +/// Facts: +/// ``` +/// net.buildQuery(new Mapping { ["q"] = new Listing { "a b" } }) == "q=a+b" +/// net.buildQuery(new Mapping { ["a"] = new Listing { "1"; "2" } }) == "a=1&a=2" +/// net.buildQuery(new Mapping { ["a"] = new Listing { "1" }; ["b"] = new Listing {} }) == "a=1" +/// ``` +external const function buildQuery(parameters: Mapping>): String + +/// Reads [query] back into its parameters. +/// +/// [query] is read as `application/x-www-form-urlencoded`. +/// A name that repeats holds the values of all its occurrences, in the order they are written. +/// A parameter that states no `=` holds no value at all, and is left out. +/// +/// Throws if any `%` in [query] does not begin a percent-encoded octet. +/// +/// Facts: +/// ``` +/// net.parseQuery("a=1&a=2") == new Mapping { ["a"] = new Listing { "1"; "2" } } +/// net.parseQuery("q=a+b&bare") == new Mapping { ["q"] = new Listing { "a b" } } +/// ``` +external const function parseQuery(query: String): Mapping>