Custom ResourceReader implementation in Golang fails to decode read data #206

New Issue

Open

opened 2025-12-30 01:22:12 +01:00 by adam · 3 comments

adam commented 2025-12-30 01:22:12 +01:00

Owner

Copy Link

Copy Source

Originally created by @barahona42 on GitHub (Sep 22, 2024).

context

Hi all, first time submitting a bug report ever. Here's the info:

• What are you trying to achieve?
  • Implement a way for pkl to retrieve configuration values from AWS SSM Parameters when read("ssm:/foo/SomeParameterPath") is assigned to a pkl resource.
  • Working language: go1.23.0
• What's the impact of this bug?
  • Unmarshaling/resolving values from a module is failing.

outputs

• pkl --version: Pkl 0.26.3 (macOS 14.3.1, native)
• complete error message:

  java.lang.InterruptedException
  
  –– Pkl Error ––
  None (cause has no message)
  
  1 | SomeParameter = read("ssm:ParameterName")
      ^^^^^^^^^^^^^^^^
  at config#SomeParameter (file:///etc/server/config.pkl)
  
  Pkl 0.26.3 (Linux 5.15.0-1057-aws, native)
  
  java.lang.InterruptedException
          at java.base@17.0.10/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:386)
          at java.base@17.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
          at org.pkl.server.UtilsKt.getUnderlying(Utils.kt:64)
          at org.pkl.server.ClientResourceReader.doRead(ClientResourceReader.kt:104)
          at org.pkl.server.ClientResourceReader.read(ClientResourceReader.kt:46)
          at org.pkl.core.runtime.ResourceManager.lambda$read$1(ResourceManager.java:91)
          at java.base@17.0.10/java.util.HashMap.computeIfAbsent(HashMap.java:1220)
          at org.pkl.core.runtime.ResourceManager.read(ResourceManager.java:72)
          at org.pkl.core.ast.expression.unary.AbstractReadNode.doRead(AbstractReadNode.java:55)
          at org.pkl.core.ast.expression.unary.ReadNode.read(ReadNode.java:33)
          at org.pkl.core.ast.expression.unary.ReadNodeGen.executeAndSpecialize(ReadNodeGen.java:75)
          at org.pkl.core.ast.expression.unary.ReadNodeGen.executeGeneric(ReadNodeGen.java:66)
          at org.pkl.core.ast.PklRootNode.executeBody(PklRootNode.java:41)
          at org.pkl.core.ast.MemberNode.executeBody(MemberNode.java:53)
          at org.pkl.core.ast.member.TypeCheckedPropertyNode.evalTypedObjectCached(TypeCheckedPropertyNode.java:54)
          at org.pkl.core.ast.member.TypeCheckedPropertyNodeGen.executeAndSpecialize(TypeCheckedPropertyNodeGen.java:136)
          at org.pkl.core.ast.member.TypeCheckedPropertyNodeGen.execute(TypeCheckedPropertyNodeGen.java:101)
          at jdk.internal.vm.compiler/org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.executeRootNode(OptimizedCallTarget.java:718)
          at jdk.internal.vm.compiler/org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.profiledPERoot(OptimizedCallTarget.java:641)
          at jdk.internal.vm.compiler/org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callBoundary(OptimizedCallTarget.java:574)
          at org.graalvm.nativeimage.builder/com.oracle.svm.truffle.api.SubstrateOptimizedCallTarget.invokeCallBoundary(SubstrateOptimizedCallTarget.java:115)
          at com.oracle.svm.svm_enterprise/com.oracle.svm.enterprise.truffle.SubstrateEnterpriseOptimizedCallTarget.a(stripped:284)
          at com.oracle.svm.svm_enterprise/com.oracle.svm.enterprise.truffle.SubstrateEnterpriseOptimizedCallTarget.doInvoke(stripped:250)
          at jdk.internal.vm.compiler/org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callIndirect(OptimizedCallTarget.java:486)
          at jdk.internal.vm.compiler/org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.call(OptimizedCallTarget.java:467)
          at org.graalvm.truffle/com.oracle.truffle.api.nodes.IndirectCallNode$1.call(IndirectCallNode.java:91)
          at org.pkl.core.runtime.VmUtils.doReadMember(VmUtils.java:305)
          at org.pkl.core.runtime.VmUtils.doReadMember(VmUtils.java:218)
          at org.pkl.core.runtime.VmObject.force(VmObject.java:177)
          at org.pkl.core.runtime.VmObject.force(VmObject.java:197)
          at org.pkl.core.runtime.VmValue.force(VmValue.java:67)
          at org.pkl.server.BinaryEvaluator.evaluate$lambda-2(BinaryEvaluator.kt:62)
          at org.pkl.core.EvaluatorImpl.lambda$doEvaluate$13(EvaluatorImpl.java:353)
          at org.pkl.core.EvaluatorImpl.doEvaluate(EvaluatorImpl.java:301)
          at org.pkl.core.EvaluatorImpl.doEvaluate(EvaluatorImpl.java:349)
          at org.pkl.server.BinaryEvaluator.evaluate(BinaryEvaluator.kt:58)
          at org.pkl.server.Server.handleEvaluate$lambda-0(Server.kt:101)
          at java.base@17.0.10/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
          at java.base@17.0.10/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
          at java.base@17.0.10/java.lang.Thread.run(Thread.java:842)
          at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:807)
          at org.graalvm.nativeimage.builder/com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:210)
  
  
  goroutine 1 [running]:
  cdr-getter-api/configuration.AppConfig()
          /app/configuration/configuration.go:51 +0x1f6
  cdr-getter-api/controllers.init()
          /app/controllers/globals.go:24 +0x17
  bash-5.2# An unexpected error has occurred. Would you mind filing a bug report?
  
  java.lang.NullPointerException
          at org.pkl.server.ClientResourceReader$doRead$1$1$1.invoke(ClientResourceReader.kt:94)
          at org.pkl.server.ClientResourceReader$doRead$1$1$1.invoke(ClientResourceReader.kt:88)
          at org.pkl.server.MessageTransports$AbstractMessageTransport.accept(MessageTransports.kt:98)
          at org.pkl.server.MessageTransports$EncodingMessageTransport.doStart(MessageTransports.kt:50)
          at org.pkl.server.MessageTransports$AbstractMessageTransport.start(MessageTransports.kt:110)
          at org.pkl.server.Server.start(Server.kt:42)
          at org.pkl.cli.CliServer.doRun(CliServer.kt:29)
          at org.pkl.commons.cli.CliCommand.run(CliCommand.kt:44)
          at org.pkl.cli.commands.ServerCommand.run(ServerCommand.kt:30)
          at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:198)
          at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:211)
          at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:18)
          at com.github.ajalt.clikt.core.CliktCommand.parse(CliktCommand.kt:400)
          at com.github.ajalt.clikt.core.CliktCommand.parse$default(CliktCommand.kt:397)
          at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:415)
          at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:440)
          at org.pkl.cli.Main$main$1.invoke(Main.kt:39)
          at org.pkl.cli.Main$main$1.invoke(Main.kt:27)
          at org.pkl.commons.cli.CliMainKt.cliMain(CliMain.kt:33)
          at org.pkl.cli.Main.main(Main.kt:27)
  

• Reproducing:
  • contents of .pkl file:

    SomeParameterValue = read("ssm:/foo/SomeParameterPath")
    

  • implement a a custom evaluator whose Read method returns a byte array as shown below:

    // from the pkl.ResourceReader requirement of pkl.Reader interface
    func (r PklResourceReader) Read(url url.URL) ([]byte, error) {
        // v is type *string 
        v, err := GetDecryptedParameter(context.Background(), url.Path)
        if err != nil {
            return nil, err
        }
        if v == nil {
            return make([]byte, 0), nil
        }
        var b []byte
        // if err := msgpack.NewEncoder(bytes.NewBuffer(b)).EncodeString(*v); err != nil {
        //     return nil, err
        // }
        return b, nil
    }
    

  • load the file using the pkl module's .Load() method.
  • note that the contents of b do make a difference:
    • on an empty array, the error above is shown.
    • on a non-empty array like []byte(*v), []byte("foo"), the following error returns:

      msgpack: invalid code=94 decoding string/bytes length
      

Any pointers/insight on this would be appreciated. Thank you.

Originally created by @barahona42 on GitHub (Sep 22, 2024). # context Hi all, first time submitting a bug report ever. Here's the info: - What are you trying to achieve? - Implement a way for pkl to retrieve configuration values from AWS SSM Parameters when `read("ssm:/foo/SomeParameterPath")` is assigned to a pkl resource. - Working language: `go1.23.0` - What's the impact of this bug? - Unmarshaling/resolving values from a module is failing. ## outputs - `pkl --version: Pkl 0.26.3 (macOS 14.3.1, native)` - complete error message: ``` java.lang.InterruptedException –– Pkl Error –– None (cause has no message) 1 | SomeParameter = read("ssm:ParameterName") ^^^^^^^^^^^^^^^^ at config#SomeParameter (file:///etc/server/config.pkl) Pkl 0.26.3 (Linux 5.15.0-1057-aws, native) java.lang.InterruptedException at java.base@17.0.10/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:386) at java.base@17.0.10/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at org.pkl.server.UtilsKt.getUnderlying(Utils.kt:64) at org.pkl.server.ClientResourceReader.doRead(ClientResourceReader.kt:104) at org.pkl.server.ClientResourceReader.read(ClientResourceReader.kt:46) at org.pkl.core.runtime.ResourceManager.lambda$read$1(ResourceManager.java:91) at java.base@17.0.10/java.util.HashMap.computeIfAbsent(HashMap.java:1220) at org.pkl.core.runtime.ResourceManager.read(ResourceManager.java:72) at org.pkl.core.ast.expression.unary.AbstractReadNode.doRead(AbstractReadNode.java:55) at org.pkl.core.ast.expression.unary.ReadNode.read(ReadNode.java:33) at org.pkl.core.ast.expression.unary.ReadNodeGen.executeAndSpecialize(ReadNodeGen.java:75) at org.pkl.core.ast.expression.unary.ReadNodeGen.executeGeneric(ReadNodeGen.java:66) at org.pkl.core.ast.PklRootNode.executeBody(PklRootNode.java:41) at org.pkl.core.ast.MemberNode.executeBody(MemberNode.java:53) at org.pkl.core.ast.member.TypeCheckedPropertyNode.evalTypedObjectCached(TypeCheckedPropertyNode.java:54) at org.pkl.core.ast.member.TypeCheckedPropertyNodeGen.executeAndSpecialize(TypeCheckedPropertyNodeGen.java:136) at org.pkl.core.ast.member.TypeCheckedPropertyNodeGen.execute(TypeCheckedPropertyNodeGen.java:101) at jdk.internal.vm.compiler/org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.executeRootNode(OptimizedCallTarget.java:718) at jdk.internal.vm.compiler/org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.profiledPERoot(OptimizedCallTarget.java:641) at jdk.internal.vm.compiler/org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callBoundary(OptimizedCallTarget.java:574) at org.graalvm.nativeimage.builder/com.oracle.svm.truffle.api.SubstrateOptimizedCallTarget.invokeCallBoundary(SubstrateOptimizedCallTarget.java:115) at com.oracle.svm.svm_enterprise/com.oracle.svm.enterprise.truffle.SubstrateEnterpriseOptimizedCallTarget.a(stripped:284) at com.oracle.svm.svm_enterprise/com.oracle.svm.enterprise.truffle.SubstrateEnterpriseOptimizedCallTarget.doInvoke(stripped:250) at jdk.internal.vm.compiler/org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callIndirect(OptimizedCallTarget.java:486) at jdk.internal.vm.compiler/org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.call(OptimizedCallTarget.java:467) at org.graalvm.truffle/com.oracle.truffle.api.nodes.IndirectCallNode$1.call(IndirectCallNode.java:91) at org.pkl.core.runtime.VmUtils.doReadMember(VmUtils.java:305) at org.pkl.core.runtime.VmUtils.doReadMember(VmUtils.java:218) at org.pkl.core.runtime.VmObject.force(VmObject.java:177) at org.pkl.core.runtime.VmObject.force(VmObject.java:197) at org.pkl.core.runtime.VmValue.force(VmValue.java:67) at org.pkl.server.BinaryEvaluator.evaluate$lambda-2(BinaryEvaluator.kt:62) at org.pkl.core.EvaluatorImpl.lambda$doEvaluate$13(EvaluatorImpl.java:353) at org.pkl.core.EvaluatorImpl.doEvaluate(EvaluatorImpl.java:301) at org.pkl.core.EvaluatorImpl.doEvaluate(EvaluatorImpl.java:349) at org.pkl.server.BinaryEvaluator.evaluate(BinaryEvaluator.kt:58) at org.pkl.server.Server.handleEvaluate$lambda-0(Server.kt:101) at java.base@17.0.10/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base@17.0.10/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base@17.0.10/java.lang.Thread.run(Thread.java:842) at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:807) at org.graalvm.nativeimage.builder/com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:210) goroutine 1 [running]: cdr-getter-api/configuration.AppConfig() /app/configuration/configuration.go:51 +0x1f6 cdr-getter-api/controllers.init() /app/controllers/globals.go:24 +0x17 bash-5.2# An unexpected error has occurred. Would you mind filing a bug report? java.lang.NullPointerException at org.pkl.server.ClientResourceReader$doRead$1$1$1.invoke(ClientResourceReader.kt:94) at org.pkl.server.ClientResourceReader$doRead$1$1$1.invoke(ClientResourceReader.kt:88) at org.pkl.server.MessageTransports$AbstractMessageTransport.accept(MessageTransports.kt:98) at org.pkl.server.MessageTransports$EncodingMessageTransport.doStart(MessageTransports.kt:50) at org.pkl.server.MessageTransports$AbstractMessageTransport.start(MessageTransports.kt:110) at org.pkl.server.Server.start(Server.kt:42) at org.pkl.cli.CliServer.doRun(CliServer.kt:29) at org.pkl.commons.cli.CliCommand.run(CliCommand.kt:44) at org.pkl.cli.commands.ServerCommand.run(ServerCommand.kt:30) at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:198) at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:211) at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:18) at com.github.ajalt.clikt.core.CliktCommand.parse(CliktCommand.kt:400) at com.github.ajalt.clikt.core.CliktCommand.parse$default(CliktCommand.kt:397) at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:415) at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:440) at org.pkl.cli.Main$main$1.invoke(Main.kt:39) at org.pkl.cli.Main$main$1.invoke(Main.kt:27) at org.pkl.commons.cli.CliMainKt.cliMain(CliMain.kt:33) at org.pkl.cli.Main.main(Main.kt:27) ``` - Reproducing: - contents of `.pkl` file: ``` SomeParameterValue = read("ssm:/foo/SomeParameterPath") ``` - implement a a custom evaluator whose `Read` method returns a byte array as shown below: ```golang // from the pkl.ResourceReader requirement of pkl.Reader interface func (r PklResourceReader) Read(url url.URL) ([]byte, error) { // v is type *string v, err := GetDecryptedParameter(context.Background(), url.Path) if err != nil { return nil, err } if v == nil { return make([]byte, 0), nil } var b []byte // if err := msgpack.NewEncoder(bytes.NewBuffer(b)).EncodeString(*v); err != nil { // return nil, err // } return b, nil } ``` - load the file using the pkl module's `.Load()` method. - note that the contents of `b` do make a difference: - on an empty array, the error above is shown. - on a non-empty array like `[]byte(*v)`, `[]byte("foo")`, the following error returns: ``` msgpack: invalid code=94 decoding string/bytes length ``` Any pointers/insight on this would be appreciated. Thank you.

adam commented 2025-12-30 01:22:13 +01:00

Author

Owner

Copy Link

Copy Source

@HT154 commented on GitHub (Sep 22, 2024):

The issue with returning an empty/nil byte slice is definitely a bug! The read contents are assumed to be non-null even though an empty byte slice on the go side is encoded as nil. A fix similar to https://github.com/apple/pkl/pull/480 is needed for this, which should be pretty straightforward. I'll probably have a PR up for this later today or tomorrow.

@HT154 commented on GitHub (Sep 22, 2024): The issue with returning an empty/nil byte slice is definitely a bug! The read contents are assumed to be non-null even though an empty byte slice on the go side is encoded as nil. A fix similar to https://github.com/apple/pkl/pull/480 is needed for this, which should be pretty straightforward. I'll probably have a PR up for this later today or tomorrow.

adam commented 2025-12-30 01:22:13 +01:00

Author

Owner

Copy Link

Copy Source

@HT154 commented on GitHub (Sep 22, 2024):

The other issue is not as clear. Can you share some of the exact code that produced that error?

You definitely don't need to do the msgpack encoding yourself, just return the string produced by your resource directly, eg. return []byte("foo"), nil

PS: great bug report! ❤️

@HT154 commented on GitHub (Sep 22, 2024): The other issue is not as clear. Can you share some of the exact code that produced that error? You definitely don't need to do the msgpack encoding yourself, just return the string produced by your resource directly, eg. `return []byte("foo"), nil` PS: great bug report! ❤️

adam commented 2025-12-30 01:22:14 +01:00

Author

Owner

Copy Link

Copy Source

@barahona42 commented on GitHub (Sep 23, 2024):

Thanks for the response. Here's the implementation of the resource and the code that gets the ssm parameter value.

// wraps ssm call
func GetDecryptedParameter(ctx context.Context, name string) (*string, error) {
	// do some service initialization
	input := ssm.GetParameterInput{Name: aws.String(name), WithDecryption: aws.Bool(true)} 
        // call to aws sdk method
	output, err := svc.GetParameter(ctx, &input)
	if err != nil {
		return nil, err
	}
	return output.Parameter.Value, nil
} 


// pkl.ResourceReader implementation
type PklResourceReader struct{}

// Scheme: pkl.Reader implementation
func (r PklResourceReader) Scheme() string {
	return "ssm"
}

// IsGlobbable: pkl.Reader implementation
func (r PklResourceReader) IsGlobbable() bool {
	return false
}

// HasHierarchicalUris: pkl.Reader implementation
func (r PklResourceReader) HasHierarchicalUris() bool {
	return false
}

// ListElements: pkl.Reader implementation
func (r PklResourceReader) ListElements(url url.URL) ([]pkl.PathElement, error) {
	return nil, nil
}

// Read: pkl.Reader implementation
func (r PklResourceReader) Read(url url.URL) ([]byte, error) {
	v, err := GetDecryptedParameter(context.Background(), url.Path)
	if err != nil {
		return nil, err
	}
	if v == nil {
		return make([]byte, 0), nil
	}
	return []byte(*v), nil
}

@barahona42 commented on GitHub (Sep 23, 2024): Thanks for the response. Here's the implementation of the resource and the code that gets the ssm parameter value. ```golang // wraps ssm call func GetDecryptedParameter(ctx context.Context, name string) (*string, error) { // do some service initialization input := ssm.GetParameterInput{Name: aws.String(name), WithDecryption: aws.Bool(true)} // call to aws sdk method output, err := svc.GetParameter(ctx, &input) if err != nil { return nil, err } return output.Parameter.Value, nil } // pkl.ResourceReader implementation type PklResourceReader struct{} // Scheme: pkl.Reader implementation func (r PklResourceReader) Scheme() string { return "ssm" } // IsGlobbable: pkl.Reader implementation func (r PklResourceReader) IsGlobbable() bool { return false } // HasHierarchicalUris: pkl.Reader implementation func (r PklResourceReader) HasHierarchicalUris() bool { return false } // ListElements: pkl.Reader implementation func (r PklResourceReader) ListElements(url url.URL) ([]pkl.PathElement, error) { return nil, nil } // Read: pkl.Reader implementation func (r PklResourceReader) Read(url url.URL) ([]byte, error) { v, err := GetDecryptedParameter(context.Background(), url.Path) if err != nil { return nil, err } if v == nil { return make([]byte, 0), nil } return []byte(*v), nil } ```

adam referenced this issue 2025-12-30 01:24:33 +01:00

[PR #206] [MERGED] Delete stale files in LanguageSnippetTests/output #451

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

dependabot/github_actions/gradle/actions/dependency-submission-6.2.0

release/0.32

dependabot/gradle/com.squareup-kotlinpoet-2.3.0

truffle-object-model-clean

release/0.31

release/0.30

release/0.29

release/0.28

release/0.27

release/0.26

release/0.25

0.32.1

0.32.0

0.31.1

0.31.0

0.30.2

0.30.1

0.30.0

0.29.1

0.29.0

0.28.2

0.28.1

0.28.0

0.27.2

0.27.1

0.27.0

0.26.3

0.26.2

0.26.1

0.26.0

0.25.3

0.25.2

0.25.1

0.25.0

Labels

Clear labels

bug

duplicate

enhancement

good first issue

help wanted

pull-request

Mirrored from GitHub Pull Request

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

adam (Adam Melkus)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/pkl#206