Implement SPICE-0009 External Readers (#660)

This adds a new feature, which allows Pkl to read resources and modules from external processes.

Follows the design laid out in SPICE-0009.

Also, this moves most of the messaging API into pkl-core

docs/modules/bindings-specification/pages/message-passing-api.adoc

@@ -9,8 +9,8 @@ The first element of the array is a code that designates the message's type, enc
 The second element of the array is the message body, encoded as a map.
 
 Messages are passed between the _client_ and the _server_.
-The _client_ is the host language (for example, the Swift application when using pkl-swift).
-The _server_ is the entity that provides controls for interacting with Pkl.
+When hosting Pkl (for example, the Swift application when using pkl-swift), the _client_ is the host program and the _server_ is the entity that provides controls for interacting with Pkl.
+When implementing an xref:language-reference:index.adoc#external-readers[external reader], the _client_ is the external reader process and the _server_ is the Pkl evaluator.
 
 For example, in JSON representation:
 
@@ -597,3 +597,96 @@ class PathElement {
   isDirectory: Boolean
 }
 ----
+
+[[initialize-module-reader-request]]
+=== Initialize Module Reader Request
+
+Code: `0x2e` +
+Type: <<server-message,Server>> <<request-message,Request>>
+
+Initialize an xref:language-reference:index.adoc#external-readers[External Module Reader].
+This message is sent to external reader processes the first time a module scheme it is registered for is read.
+
+[source,pkl]
+----
+/// A number identifying this request.
+requestId: Int
+
+/// The module scheme to initialize.
+scheme: String
+----
+
+[[initialize-module-reader-response]]
+=== Initialize Module Reader Response
+
+Code: `0x2f` +
+Type: <<client-message,Client>> <<response-message,Response>>
+
+Return the requested external module reader specification.
+The `spec` field should be set to `null` when the external process does not implement the requested module scheme.
+
+[source,pkl]
+----
+/// A number identifying this request.
+requestId: Int
+
+/// Client-side module reader spec.
+///
+/// Null when the external process does not implement the requested module scheme.
+spec: ClientModuleReader?
+----
+
+`ClientModuleReader` is defined above by <<create-evaluator-request,Create Evaluator Request>>.
+
+[[initialize-resource-reader-request]]
+=== Initialize Resource Reader Request
+
+Code: `0x30` +
+Type: <<server-message,Server>> <<request-message,Request>>
+
+Initialize an xref:language-reference:index.adoc#external-readers[External Resource Reader].
+This message is sent to external reader processes the first time a resource scheme it is registered for is read.
+
+[source,pkl]
+----
+/// A number identifying this request.
+requestId: Int
+
+/// The resource scheme to initialize.
+scheme: String
+----
+
+[[initialize-resource-reader-response]]
+=== Initialize Resource Reader Response
+
+Code: `0x31` +
+Type: <<client-message,Client>> <<response-message,Response>>
+
+Return the requested external resource reader specification.
+The `spec` field should be set to `null` when the external process does not implement the requested resource scheme.
+
+[source,pkl]
+----
+/// A number identifying this request.
+requestId: Int
+
+/// Client-side resource reader spec.
+///
+/// Null when the external process does not implement the requested resource scheme.
+spec: ClientResourceReader?
+----
+
+`ClientResourceReader` is defined above by <<create-evaluator-request,Create Evaluator Request>>. 
+
+[[close-external-process]]
+=== Close External Process
+
+Code: `0x32` +
+Type: <<server-message,Server>> <<one-way-message,One Way>> 
+
+Initiate graceful shutdown of the external reader process.
+
+[source,pkl]
+----
+/// This message has no properties.
+----

docs/modules/language-reference/pages/index.adoc

@@ -3199,7 +3199,8 @@ This section discusses language features that are generally more relevant to tem
 <<grammar-definition,Grammar Definition>> +
 <<reserved-keywords,Reserved Keywords>> +
 <<blank-identifiers,Blank Identifiers>> +
-<<projects,Projects>>
+<<projects,Projects>> +
+<<external-readers,External Readers>>
 
 [[meaning-of-new]]
 === Meaning of `new`
@@ -5465,3 +5466,68 @@ It can be imported using dependency notation, i.e. `import "@fruit/Pear.pkl"`.
 At runtime, it will resolve to relative path `../fruit/Pear.pkl`.
 
 When packaging projects with local dependencies, both the project and its dependent project must be passed to the xref:pkl-cli:index.adoc#command-project-package[`pkl project package`] command.
+
+[[external-readers]]
+=== External Readers
+
+External readers are a mechanism to extend the <<modules,module>> and <<resources,resource>> URI schemes that Pkl supports.
+Readers are implemented as ordinary executables and use Pkl's xref:bindings-specification:message-passing-api.adoc[message passing API] to communicate with the hosting Pkl evaluator.
+The xref:swift:ROOT:index.adoc[Swift] and xref:go:ROOT:index.adoc[Go] language binding libraries provide an `ExternalReaderRuntime` type to facilitate implementing external readers.
+
+External readers are configured separately for modules and resources.
+They are registered by mapping their URI scheme to the executable to run and additonal arguments to pass.
+This is done on the command line by passing `--external-resource-reader` and `--external-module-reader` flags, which may both be passed multiple times.
+
+[source,text]
+----
+$ pkl eval <module> --external-resource-reader <scheme>=<executable> --external-module-reader <scheme>='<executable> <argument> <argument>'
+----
+
+External readers may also be configured in a <<projects, Project's>> `PklProject` file.
+[source,{pkl}]
+----
+evaluatorSettings {
+  externalResourceReaders {
+    ["<scheme>"] {
+      executable = "<executable>"
+    }
+  }
+  externalModuleReaders {
+    ["<scheme>"] {
+      executable = "<executable>"
+      arguments { "<arg>"; "<arg>" }
+    }
+  }
+}
+----
+
+Registering an external reader for a scheme automatically adds that scheme to the default allowed modules/resources.
+As with Pkl's built-in module and resource schemes, setting explicit allowed modules or resources overrides this behavior and appropriate patterns must be specified to allow use of external readers.
+
+==== Example
+
+Consider this module:
+
+[source,{pkl}]
+----
+username = "pigeon"
+
+email = read("ldap://ds.example.com:389/dc=example,dc=com?mail?sub?(uid=\(username))").text
+----
+
+Pkl doesn't implement the `ldap:` resource URI scheme natively, but an external reader can provide it.
+Assuming a hypothetical `pkl-ldap` executable implementing the external reader protocol and the `ldap:` scheme is in the `$PATH`, this module can be evaluated as:
+
+[source,text]
+----
+$ pkl eval <module> --external-resource-reader ldap=pkl-ldap
+username = "pigeon"
+email = "pigeon@example.com"
+----
+
+In this example, the external reader may provide both `ldap:` and `ldaps:` schemes.
+To support both schemes during evaluation, both would need to be registered explicitly:
+[source,text]
+----
+$ pkl eval <module> --external-resource-reader ldap=pkl-ldap --external-resource-reader ldaps=pkl-ldap
+----