starred/pkl
1
0
Fork 0
mirror of https://github.com/apple/pkl.git synced 2026-08-28 14:37:18 +02:00
Code Issues 198 Packages Projects Releases 10 Wiki Activity

Slow performance on JVM? #21

New Issue
Closed
opened 2025-12-30 01:19:36 +01:00 by adam · 4 comments
adam commented 2025-12-30 01:19:36 +01:00
Owner
Copy Link
Copy Source

Originally created by @Omico on GitHub (Feb 4, 2024).

Just play around with pkl. It seems that parsing AST takes a very long time.

TLDR. In my PC, pkl takes 3089 ms, and kotlinx-serialization-properties takes 53 ms.

// benchmark.pkl
a {
  b {
    c {
      d = "Hello World!"
    }
  }
}

// benchmark.properties
a.b.c.d=Hello World!

I have created a very simple project for benchmarking. (I didn't use kotlinx benchmark or anything else)

https://github.com/Omico/pkl-benchmark

Originally created by @Omico on GitHub (Feb 4, 2024). Just play around with pkl. It seems that parsing AST takes a very long time. TLDR. In my PC, pkl takes 3089 ms, and kotlinx-serialization-properties takes 53 ms. ``` // benchmark.pkl a { b { c { d = "Hello World!" } } } // benchmark.properties a.b.c.d=Hello World! ``` I have created a very simple project for benchmarking. (I didn't use kotlinx benchmark or anything else) https://github.com/Omico/pkl-benchmark
adam closed this issue 2025-12-30 01:19:36 +01:00
adam commented 2025-12-30 01:19:36 +01:00
Author
Owner
Copy Link
Copy Source

@holzensp commented on GitHub (Feb 5, 2024):

This example case is somewhat fundamentally going to underperform. kotlinx-serialization-properties just parses this static data. Pkl brings a lot more plumbing out of the box (and will take ~0.5-1s just to load the stdlib). On the tiny static cases like this, Pkl could/should do better, but will never win.

That said, I'd love it if your pkl-benchmark grows out to significantly cover relevant cases!

@holzensp commented on GitHub (Feb 5, 2024): This example case is somewhat fundamentally going to underperform. `kotlinx-serialization-properties` just parses this static data. Pkl brings a _lot_ more plumbing out of the box (and will take ~0.5-1s just to load the stdlib). On the tiny static cases like this, Pkl could/should do better, but will _never_ win. That said, I'd _love_ it if your `pkl-benchmark` grows out to significantly cover relevant cases!
adam commented 2025-12-30 01:19:37 +01:00
Author
Owner
Copy Link
Copy Source

@bioball commented on GitHub (Feb 5, 2024):

Your setup here runs Pkl in the JVM, which means you're also incurring the time it takes for the JVM to bootstrap (e.g. it needs to load classes, fire up our parser, etc)

Performance-wise, Pkl is generally faster if you use the native executables. Here's the same test on my machine using the CLI:

time pkl eval .dan-scripts/test.pkl
a {
  b {
    c {
      d = "Hello World!"
    }
  }
}

________________________________________________________
Executed in  206.59 millis    fish           external
   usr time   73.38 millis   36.00 micros   73.35 millis
   sys time  127.20 millis  907.00 micros  126.29 millis

But, like Phil said, Pkl will never beat a static format, because it is a program that needs to be interpreted, not just parsed.

@bioball commented on GitHub (Feb 5, 2024): Your setup here runs Pkl in the JVM, which means you're also incurring the time it takes for the JVM to bootstrap (e.g. it needs to load classes, fire up our parser, etc) Performance-wise, Pkl is *generally* faster if you use the native executables. Here's the same test on my machine using the CLI: ``` time pkl eval .dan-scripts/test.pkl a { b { c { d = "Hello World!" } } } ________________________________________________________ Executed in 206.59 millis fish external usr time 73.38 millis 36.00 micros 73.35 millis sys time 127.20 millis 907.00 micros 126.29 millis ``` But, like Phil said, Pkl will never beat a static format, because it is a program that needs to be interpreted, not just parsed.
adam commented 2025-12-30 01:19:37 +01:00
Author
Owner
Copy Link
Copy Source

@Omico commented on GitHub (Feb 6, 2024):

Just updated my benchmark, seems not that horrible than before.

main: me.omico.pklbenchmark.JavaPropertiesBenchmark.test

Warm-up 1: 44600.802 ops/s
Warm-up 2: 48322.975 ops/s
Warm-up 3: 47936.852 ops/s
Warm-up 4: 48888.251 ops/s
Warm-up 5: 48884.637 ops/s
Iteration 1: 48651.885 ops/s
Iteration 2: 48480.269 ops/s
Iteration 3: 48167.602 ops/s
Iteration 4: 48239.609 ops/s
Iteration 5: 48942.755 ops/s

48496.424 ±(99.9%) 1214.203 ops/s [Average]
  (min, avg, max) = (48167.602, 48496.424, 48942.755), stdev = 315.325
  CI (99.9%): [47282.221, 49710.627] (assumes normal distribution)

main: me.omico.pklbenchmark.KotlinxSerializationPropertiesBenchmark.testOnDemandMap

Warm-up 1: 38855.693 ops/s
Warm-up 2: 43636.701 ops/s
Warm-up 3: 43427.721 ops/s
Warm-up 4: 42766.916 ops/s
Warm-up 5: 42838.192 ops/s
Iteration 1: 43648.165 ops/s
Iteration 2: 43467.138 ops/s
Iteration 3: 43313.115 ops/s
Iteration 4: 42778.440 ops/s
Iteration 5: 42654.035 ops/s

43172.179 ±(99.9%) 1675.063 ops/s [Average]
  (min, avg, max) = (42654.035, 43172.179, 43648.165), stdev = 435.009
  CI (99.9%): [41497.116, 44847.242] (assumes normal distribution)

main: me.omico.pklbenchmark.KotlinxSerializationPropertiesBenchmark.testWithPreParsedMap

Warm-up 1: 3058433.088 ops/s
Warm-up 2: 3247625.592 ops/s
Warm-up 3: 3234467.557 ops/s
Warm-up 4: 3229296.957 ops/s
Warm-up 5: 3247300.579 ops/s
Iteration 1: 3238172.409 ops/s
Iteration 2: 3256747.411 ops/s
Iteration 3: 3215626.194 ops/s
Iteration 4: 3247396.923 ops/s
Iteration 5: 3238568.379 ops/s

3239302.263 ±(99.9%) 58784.722 ops/s [Average]
  (min, avg, max) = (3215626.194, 3239302.263, 3256747.411), stdev = 15266.206
  CI (99.9%): [3180517.542, 3298086.985] (assumes normal distribution)

main: me.omico.pklbenchmark.PklBenchmark.testCreateEvaluatorOnDemand

Warm-up 1: 4612.738 ops/s
Warm-up 2: 8337.182 ops/s
Warm-up 3: 8322.103 ops/s
Warm-up 4: 8431.379 ops/s
Warm-up 5: 8398.744 ops/s
Iteration 1: 8394.138 ops/s
Iteration 2: 8364.193 ops/s
Iteration 3: 8402.756 ops/s
Iteration 4: 8382.008 ops/s
Iteration 5: 8339.333 ops/s

8376.486 ±(99.9%) 97.484 ops/s [Average]
  (min, avg, max) = (8339.333, 8376.486, 8402.756), stdev = 25.316
  CI (99.9%): [8279.002, 8473.970] (assumes normal distribution)

main: me.omico.pklbenchmark.PklBenchmark.testUsePreconfiguredEvaluator

Warm-up 1: 7454.698 ops/s
Warm-up 2: 11526.691 ops/s
Warm-up 3: 11606.931 ops/s
Warm-up 4: 11579.829 ops/s
Warm-up 5: 11595.770 ops/s
Iteration 1: 11544.253 ops/s
Iteration 2: 11502.271 ops/s
Iteration 3: 11538.845 ops/s
Iteration 4: 11554.954 ops/s
Iteration 5: 11506.545 ops/s

11529.374 ±(99.9%) 90.740 ops/s [Average]
  (min, avg, max) = (11502.271, 11529.374, 11554.954), stdev = 23.565
  CI (99.9%): [11438.634, 11620.114] (assumes normal distribution)
@Omico commented on GitHub (Feb 6, 2024): Just updated my benchmark, seems not that horrible than before. ``` main: me.omico.pklbenchmark.JavaPropertiesBenchmark.test Warm-up 1: 44600.802 ops/s Warm-up 2: 48322.975 ops/s Warm-up 3: 47936.852 ops/s Warm-up 4: 48888.251 ops/s Warm-up 5: 48884.637 ops/s Iteration 1: 48651.885 ops/s Iteration 2: 48480.269 ops/s Iteration 3: 48167.602 ops/s Iteration 4: 48239.609 ops/s Iteration 5: 48942.755 ops/s 48496.424 ±(99.9%) 1214.203 ops/s [Average] (min, avg, max) = (48167.602, 48496.424, 48942.755), stdev = 315.325 CI (99.9%): [47282.221, 49710.627] (assumes normal distribution) ``` ``` main: me.omico.pklbenchmark.KotlinxSerializationPropertiesBenchmark.testOnDemandMap Warm-up 1: 38855.693 ops/s Warm-up 2: 43636.701 ops/s Warm-up 3: 43427.721 ops/s Warm-up 4: 42766.916 ops/s Warm-up 5: 42838.192 ops/s Iteration 1: 43648.165 ops/s Iteration 2: 43467.138 ops/s Iteration 3: 43313.115 ops/s Iteration 4: 42778.440 ops/s Iteration 5: 42654.035 ops/s 43172.179 ±(99.9%) 1675.063 ops/s [Average] (min, avg, max) = (42654.035, 43172.179, 43648.165), stdev = 435.009 CI (99.9%): [41497.116, 44847.242] (assumes normal distribution) ``` ``` main: me.omico.pklbenchmark.KotlinxSerializationPropertiesBenchmark.testWithPreParsedMap Warm-up 1: 3058433.088 ops/s Warm-up 2: 3247625.592 ops/s Warm-up 3: 3234467.557 ops/s Warm-up 4: 3229296.957 ops/s Warm-up 5: 3247300.579 ops/s Iteration 1: 3238172.409 ops/s Iteration 2: 3256747.411 ops/s Iteration 3: 3215626.194 ops/s Iteration 4: 3247396.923 ops/s Iteration 5: 3238568.379 ops/s 3239302.263 ±(99.9%) 58784.722 ops/s [Average] (min, avg, max) = (3215626.194, 3239302.263, 3256747.411), stdev = 15266.206 CI (99.9%): [3180517.542, 3298086.985] (assumes normal distribution) ``` ``` main: me.omico.pklbenchmark.PklBenchmark.testCreateEvaluatorOnDemand Warm-up 1: 4612.738 ops/s Warm-up 2: 8337.182 ops/s Warm-up 3: 8322.103 ops/s Warm-up 4: 8431.379 ops/s Warm-up 5: 8398.744 ops/s Iteration 1: 8394.138 ops/s Iteration 2: 8364.193 ops/s Iteration 3: 8402.756 ops/s Iteration 4: 8382.008 ops/s Iteration 5: 8339.333 ops/s 8376.486 ±(99.9%) 97.484 ops/s [Average] (min, avg, max) = (8339.333, 8376.486, 8402.756), stdev = 25.316 CI (99.9%): [8279.002, 8473.970] (assumes normal distribution) ``` ``` main: me.omico.pklbenchmark.PklBenchmark.testUsePreconfiguredEvaluator Warm-up 1: 7454.698 ops/s Warm-up 2: 11526.691 ops/s Warm-up 3: 11606.931 ops/s Warm-up 4: 11579.829 ops/s Warm-up 5: 11595.770 ops/s Iteration 1: 11544.253 ops/s Iteration 2: 11502.271 ops/s Iteration 3: 11538.845 ops/s Iteration 4: 11554.954 ops/s Iteration 5: 11506.545 ops/s 11529.374 ±(99.9%) 90.740 ops/s [Average] (min, avg, max) = (11502.271, 11529.374, 11554.954), stdev = 23.565 CI (99.9%): [11438.634, 11620.114] (assumes normal distribution) ```
adam commented 2025-12-30 01:19:37 +01:00
Author
Owner
Copy Link
Copy Source

@bioball commented on GitHub (Feb 6, 2024):

By the way there's also an in-language benchmark suite:

// abcd.pkl

a {
  b {
    c {
      d = "Hello World!"
    }
  }
}

// benchmark.pkl
amends "pkl:Benchmark"

import "abcd.pkl"

outputBenchmarks {
  ["abcd"] {
    sourceModule = abcd 
  }
}

Then you can pkl eval benchmark.pkl (or do the equivalent in Java) and get benchmark results!

@bioball commented on GitHub (Feb 6, 2024): By the way there's also an in-language benchmark suite: ``` // abcd.pkl a { b { c { d = "Hello World!" } } } ``` ``` // benchmark.pkl amends "pkl:Benchmark" import "abcd.pkl" outputBenchmarks { ["abcd"] { sourceModule = abcd } } ``` Then you can `pkl eval benchmark.pkl` (or do the equivalent in Java) and get benchmark results!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
Mirrored from GitHub Pull Request
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
adam (Adam Melkus)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/pkl#21