binary serialization #185

New Issue

adam · 2025-12-30T01:21:50+01:00

adam commented

2025-12-30 01:21:50 +01:00

Originally created by @blacktop on GitHub (Jul 17, 2024).

It would be awesome if you could add binary serialization to compact the pkl files.

I started using it in a NEW feature for one of my projects and it is creating very large files ;)

Thanks!

Originally created by @blacktop on GitHub (Jul 17, 2024). It would be awesome if you could add binary serialization to compact the pkl files. I started using it in a NEW feature for one of my projects and it is creating very large files ;) Thanks!

adam closed this issue

2025-12-30 01:21:50 +01:00

adam commented

2025-12-30 01:21:51 +01:00

@bioball commented on GitHub (Jul 17, 2024):

WRT large files: are you referring to Pkl source files, or something else?

@bioball commented on GitHub (Jul 17, 2024): WRT large files: are you referring to Pkl source files, or something else?

adam commented

2025-12-30 01:21:51 +01:00

@blacktop commented on GitHub (Jul 17, 2024):

I might be using pkls not for their intended purpose, but I've enjoying the format so far.

Here is an example:

https://github.com/blacktop/symbolicator/blob/main/kernel/24/xnu.pkl

@blacktop commented on GitHub (Jul 17, 2024): I might be using `pkl`s not for their intended purpose, but I've enjoying the format so far. Here is an example: - https://github.com/blacktop/symbolicator/blob/main/kernel/24/xnu.pkl

adam commented

2025-12-30 01:21:52 +01:00

@blacktop commented on GitHub (Jul 17, 2024):

I prob can .tar.gz etc the files to make them smaller when sending around, but thought; since this is kinda the NEW plist, maybe there'd be a official way to create "bpkl"s similar to bplists to make them as small as possible etc

@blacktop commented on GitHub (Jul 17, 2024): I prob can .tar.gz etc the files to make them smaller when sending around, but thought; since this is kinda the NEW plist, maybe there'd be a official way to create "bpkl"s similar to bplists to make them as small as possible etc

adam commented

2025-12-30 01:21:52 +01:00

@bioball commented on GitHub (Jul 18, 2024):

Gotcha. If you're using the pkl-go bindings (seems like you are), you can "bake" Pkl config into their binary representation using EvaluateExpressionRaw.

These bytes can then saved to a file or whatnot, and then be unmarshall'd into the codegen'd structs via Unmarshal.

@bioball commented on GitHub (Jul 18, 2024): Gotcha. If you're using the pkl-go bindings (seems like you are), you can "bake" Pkl config into their binary representation using [EvaluateExpressionRaw](https://pkg.go.dev/github.com/apple/pkl-go/pkl#Evaluator.EvaluateExpressionRaw). These bytes can then saved to a file or whatnot, and then be unmarshall'd into the codegen'd structs via [Unmarshal](https://pkg.go.dev/github.com/apple/pkl-go/pkl#Unmarshal).

adam commented

2025-12-30 01:21:52 +01:00

@blacktop commented on GitHub (Jul 18, 2024):

Awesome! thank you

I don't really understand what the expr arg is for here?

4e6500ec73/cmd/internal/gen-fixtures/gen-fixtures.go (L41)

@blacktop commented on GitHub (Jul 18, 2024): Awesome! thank you I don't really understand what the `expr` arg is for here? https://github.com/apple/pkl-go/blob/4e6500ec73b5cd7665b7f697e4027760302af0c2/cmd/internal/gen-fixtures/gen-fixtures.go#L41

adam commented

2025-12-30 01:21:53 +01:00

@blacktop commented on GitHub (Jul 18, 2024):

I tried running this:

func BinarySerializeSignature(ctx context.Context, path string) ([]byte, error) {
	evaluator, err := pkl.NewEvaluator(ctx, pkl.PreconfiguredOptions)
	if err != nil {
		return nil, err
	}
	defer func() {
		cerr := evaluator.Close()
		if err == nil {
			err = cerr
		}
	}()
	return evaluator.EvaluateExpressionRaw(ctx, pkl.FileSource(path), "")
}

and this happened 😳

[ 160]  kernel/
├── [ 128]  23
│   ├── [2.7M]  xnu.pkl
│   └── [2.9M]  xnu.pkl.bin

@blacktop commented on GitHub (Jul 18, 2024): I tried running this: ```go func BinarySerializeSignature(ctx context.Context, path string) ([]byte, error) { evaluator, err := pkl.NewEvaluator(ctx, pkl.PreconfiguredOptions) if err != nil { return nil, err } defer func() { cerr := evaluator.Close() if err == nil { err = cerr } }() return evaluator.EvaluateExpressionRaw(ctx, pkl.FileSource(path), "") } ``` and this happened 😳 ```bash [ 160] kernel/ ├── [ 128] 23 │ ├── [2.7M] xnu.pkl │ └── [2.9M] xnu.pkl.bin ```

adam commented

2025-12-30 01:21:53 +01:00

@bioball commented on GitHub (Jul 18, 2024):

Hm, that makes sense I guess. We store information in our binary format that isn't present in source code. You'll probably see a lot of gains by compressing it with gzip or something similar, because there's a lot of repetition in there.

expr is the expression to evaluate within the module. If empty, it defaults to the whole module itself (equivalent to "module").

@bioball commented on GitHub (Jul 18, 2024): Hm, that makes sense I guess. We store information in our binary format that isn't present in source code. You'll probably see a lot of gains by compressing it with gzip or something similar, because there's a lot of repetition in there. `expr` is the expression to evaluate within the module. If empty, it defaults to the whole module itself (equivalent to `"module"`).

adam commented

2025-12-30 01:21:53 +01:00

@blacktop commented on GitHub (Jul 19, 2024):

thank you

@blacktop commented on GitHub (Jul 19, 2024): thank you

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/pkl#185