New lines in a multi-line string not properly escaped in JSON output #273

New Issue

Closed

opened 2025-12-30 01:23:04 +01:00 by adam · 7 comments

adam commented 2025-12-30 01:23:04 +01:00

Owner

Copy Link

Originally created by @TheMadLeprechaun on GitHub (Feb 3, 2025).

When creating a multi-line string, the resulting JSON output does not properly escape the new line character.

Example file.pkl:

key = """
This is a
multi-line string
"""

When running pkl eval --format json file.pkl the output is

{
    "key" : "This is a\nmulti-line string"
}

The expected output would be the following since JSON requires \\n instead of \n

{
    "key": "This is a\\nmulti-line string"
}

I'm new to pkl so I might be missing something but I couldn't find anything about this.

Originally created by @TheMadLeprechaun on GitHub (Feb 3, 2025). When creating a multi-line string, the resulting JSON output does not properly escape the new line character. Example `file.pkl`: ``` key = """ This is a multi-line string """ ``` When running `pkl eval --format json file.pkl` the output is ``` { "key" : "This is a\nmulti-line string" } ``` The expected output would be the following since JSON requires `\\n` instead of `\n` ``` { "key": "This is a\\nmulti-line string" } ``` I'm new to pkl so I might be missing something but I couldn't find anything about this.

adam closed this issue 2025-12-30 01:23:04 +01:00

adam commented 2025-12-30 01:23:05 +01:00

Author

Owner

Copy Link

@HT154 commented on GitHub (Feb 3, 2025):

The expected output would be the following since JSON requires \\n instead of \n

This doesn't seem right to me. If this were the case, the string in the JSON data, once decoded would be

This is a\nmulti-line string

Can you share more about where this information is from or how you determined it?

@HT154 commented on GitHub (Feb 3, 2025): > The expected output would be the following since JSON requires `\\n` instead of `\n` This doesn't seem right to me. If this were the case, the string in the JSON data, once decoded would be ``` This is a\nmulti-line string ``` Can you share more about where this information is from or how you determined it?

adam commented 2025-12-30 01:23:05 +01:00

Author

Owner

Copy Link

@bioball commented on GitHub (Feb 3, 2025):

JSON uses \n to represent newlines (see https://datatracker.ietf.org/doc/html/rfc7159#section-7)

If you want to emit \\n in JSON (which means the literal \ and n characters), you need to represent \n as verbatim characters in Pkl too. To do that, you can either escape the slash, or use custom string delimiters.

Escaping the slash:

key = "This is a \\n multi-line string"

Custom string delimiters:

key = #"This is a \n multi-line string"#

@bioball commented on GitHub (Feb 3, 2025): JSON uses `\n` to represent newlines (see https://datatracker.ietf.org/doc/html/rfc7159#section-7) If you want to emit `\\n` in JSON (which means the literal `\` and `n` characters), you need to represent `\n` as verbatim characters in Pkl too. To do that, you can either escape the slash, or use custom string delimiters. Escaping the slash: ```pkl key = "This is a \\n multi-line string" ``` Custom string delimiters: ```pkl key = #"This is a \n multi-line string"# ```

adam commented 2025-12-30 01:23:05 +01:00

Author

Owner

Copy Link

@TheMadLeprechaun commented on GitHub (Feb 3, 2025):

So, the main reason I'm using pkl is for readability. The software that the JSON goes into allows for embedded scripts. Since JSON doesn't support multi-lines in the actual JSON, I started using pkl. Otherwise I'd have to put it all in a very unreadable single line. Using the built in multi-line support of pkl it looks like this:

Script = #"""
#!C:\WINDOWS\system32\WindowsPowerShell\v1.0\powershell.exe  -File
$date = get-date -format 'MMddyyy_HHmmss'
move-item \\UNC\path\to\file
"""#

The built in multi-line automatically normalizes the line breaks to \n so it looks like

"Script": "#!C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\powershell.exe  -File\n$date = get-date -format 'MMddyyy_HHmmss'\nmove-item \\\\UNC\\path\\to\\file"

If I do

Script = #"""
#!C:\WINDOWS\system32\WindowsPowerShell\v1.0\powershell.exe  -File\n
$date = get-date -format 'MMddyyy_HHmmss'\n
move-item \\UNC\path\to\file
"""#

I get

{
  "Script": "#!C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\powershell.exe  -File\\n\n$date = get-date -format 'MMddyyy_HHmmss'\\n\nmove-item \\\\UNC\\path\\to\\file"
}

Doing

Script = """
#!C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\powershell.exe  -File\\n
$date = get-date -format 'MMddyyy_HHmmss'\\n
move-item \\\\UNC\\path\\to\\file
"""

Gets the same output

{
  "Script": "#!C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\powershell.exe  -File\\n\n$date = get-date -format 'MMddyyy_HHmmss'\\n\nmove-item \\\\UNC\\path\\to\\file"
}

This is where I found the \\n information outside of my own experiences: https://stackoverflow.com/questions/42068/how-do-i-handle-newlines-in-json
In my use case, it's a different error but it's for the same reason. The error being along the lines of "there is an invalid control character in the string."
If it helps, the software the JSON goes to is BMC Control-M. I probably should have made clear that the issue is with the built in multi-line support as opposed to adding a new line to a string. My bad. I was using this as reference: https://pkl-lang.org/main/current/language-reference/index.html#multiline-strings

Clear as mud I hope. 😛

@TheMadLeprechaun commented on GitHub (Feb 3, 2025): So, the main reason I'm using pkl is for readability. The software that the JSON goes into allows for embedded scripts. Since JSON doesn't support multi-lines in the actual JSON, I started using pkl. Otherwise I'd have to put it all in a very unreadable single line. Using the built in multi-line support of pkl it looks like this: ``` Script = #""" #!C:\WINDOWS\system32\WindowsPowerShell\v1.0\powershell.exe -File $date = get-date -format 'MMddyyy_HHmmss' move-item \\UNC\path\to\file """# ``` The built in multi-line automatically normalizes the line breaks to `\n` so it looks like ``` "Script": "#!C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\powershell.exe -File\n$date = get-date -format 'MMddyyy_HHmmss'\nmove-item \\\\UNC\\path\\to\\file" ``` If I do ``` Script = #""" #!C:\WINDOWS\system32\WindowsPowerShell\v1.0\powershell.exe -File\n $date = get-date -format 'MMddyyy_HHmmss'\n move-item \\UNC\path\to\file """# ``` I get ``` { "Script": "#!C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\powershell.exe -File\\n\n$date = get-date -format 'MMddyyy_HHmmss'\\n\nmove-item \\\\UNC\\path\\to\\file" } ``` Doing ``` Script = """ #!C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\powershell.exe -File\\n $date = get-date -format 'MMddyyy_HHmmss'\\n move-item \\\\UNC\\path\\to\\file """ ``` Gets the same output ``` { "Script": "#!C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\powershell.exe -File\\n\n$date = get-date -format 'MMddyyy_HHmmss'\\n\nmove-item \\\\UNC\\path\\to\\file" } ``` This is where I found the `\\n` information outside of my own experiences: https://stackoverflow.com/questions/42068/how-do-i-handle-newlines-in-json In my use case, it's a different error but it's for the same reason. The error being along the lines of "there is an invalid control character in the string." If it helps, the software the JSON goes to is BMC Control-M. I probably should have made clear that the issue is with the built in multi-line support as opposed to adding a new line to a string. My bad. I was using this as reference: https://pkl-lang.org/main/current/language-reference/index.html#multiline-strings Clear as mud I hope. 😛

adam commented 2025-12-30 01:23:05 +01:00

Author

Owner

Copy Link

@HT154 commented on GitHub (Feb 3, 2025):

This is where I found the \n information outside of my own experiences: https://stackoverflow.com/questions/42068/how-do-i-handle-newlines-in-json

Ah, I see what happened here! The example in that SO post is in js and shows a JSON object serialized inside a string. Given this js:

var data = '{"count" : 1, "stack" : "sometext\\n\\n"}';

The escaped backslash \\ tells the js parser to produce a literal \ in the resulting string. Thus, the variable data has value

{"count" : 1, "stack" : "sometext\n\n"}

This is syntactically valid JSON

It's not necessary to add this extra level of escaping in Pkl! Pkl knows how to encode any string value as valid JSON. The equivalent Pkl code here is

count = 1
stack = """
  sometext
  
  
  """

And when rendered to JSON, this produces the exact same output:

{
  "count": 1,
  "stack": "sometext\n\n"
}

The difference here is that in the SO post the js code must contend with an extra layer of string escaping that is not required when working in Pkl.

If you're finding you need to do this to get the JSON working with your target system, it may be expecting string values to be escaped an extra time. This would be a quirk of that system and not related to Pkl or its JSON implementation.

Given this, you might consider doing something like this instead:

Script = #"""
#!C:\WINDOWS\system32\WindowsPowerShell\v1.0\powershell.exe  -File\n
$date = get-date -format 'MMddyyy_HHmmss'\n
move-item \\UNC\path\to\file
"""#.replace(#"\"#, #"\\"#)

Or maybe even better, set up an output converter in your module in the name of reusability:

Script = #"""
#!C:\WINDOWS\system32\WindowsPowerShell\v1.0\powershell.exe  -File\n
$date = get-date -format 'MMddyyy_HHmmss'\n
move-item \\UNC\path\to\file
"""#

output {
  renderer = new JsonRenderer {
    converters {
      ["Script"] = (it) -> it.replace(#"\"#, #"\\"#)
    }
  }
}

@HT154 commented on GitHub (Feb 3, 2025): > This is where I found the \\n information outside of my own experiences: https://stackoverflow.com/questions/42068/how-do-i-handle-newlines-in-json Ah, I see what happened here! The example in that SO post is in js and shows a JSON object serialized inside a string. Given this js: ```js var data = '{"count" : 1, "stack" : "sometext\\n\\n"}'; ``` The escaped backslash `\\` tells the js parser to produce a literal `\` in the resulting string. Thus, the variable `data` has value ```json {"count" : 1, "stack" : "sometext\n\n"} ``` This is syntactically valid JSON It's not necessary to add this extra level of escaping in Pkl! Pkl knows how to encode any string value as valid JSON. The equivalent Pkl code here is ```pkl count = 1 stack = """ sometext """ ``` And when rendered to JSON, this produces the exact same output: ```json { "count": 1, "stack": "sometext\n\n" } ``` The difference here is that in the SO post the js code must contend with an extra layer of string escaping that is not required when working in Pkl. If you're finding you need to do this to get the JSON working with your target system, it may be expecting string values to be escaped an extra time. This would be a quirk of that system and not related to Pkl or its JSON implementation. Given this, you might consider doing something like this instead: ```pkl Script = #""" #!C:\WINDOWS\system32\WindowsPowerShell\v1.0\powershell.exe -File\n $date = get-date -format 'MMddyyy_HHmmss'\n move-item \\UNC\path\to\file """#.replace(#"\"#, #"\\"#) ``` Or maybe even better, set up an output converter in your module in the name of reusability: ```pkl Script = #""" #!C:\WINDOWS\system32\WindowsPowerShell\v1.0\powershell.exe -File\n $date = get-date -format 'MMddyyy_HHmmss'\n move-item \\UNC\path\to\file """# output { renderer = new JsonRenderer { converters { ["Script"] = (it) -> it.replace(#"\"#, #"\\"#) } } } ```

adam commented 2025-12-30 01:23:06 +01:00

Author

Owner

Copy Link

@TheMadLeprechaun commented on GitHub (Feb 3, 2025):

In the original output, all of the backslashes are properly escaped in the Windows paths in the JSON output it's the literal \n that pkl puts out that's the issue.

I used replaceAll since I got an error just trying replace

Script = #"""
#!C:\windows\system32\windowspowershell\v1.0\powershell.exe  -file\n
$date = get-date -format 'mmddyyy_hhmmss'\n
move-item \\unc\path\to\file
"""#.replaceAll(#"\"#, #"\\"#)

and

Script = #"""
#!c:\windows\system32\windowspowershell\v1.0\powershell.exe  -file\n
$date = get-date -format 'mmddyyy_hhmmss'\n
move-item \\unc\path\to\file
"""#

output {
  renderer = new JsonRenderer {
    converters {
      ["Script"] = (it) -> it.replaceAll(#"\"#, #"\\"#)
    }
  }
}

These both give this output

{
  "Script": "#!c:\\\\windows\\\\system32\\\\windowspowershell\\\\v1.0\\\\powershell.exe  -file\\\\n\n$date = get-date -format 'mmddyyy_hhmmss'\\\\n\nmove-item \\\\\\\\unc\\\\path\\\\to\\\\file"
}

So, on top of the excessive number of \, it gives \\\\n\n. If I do either of the above and remove the literal \n then the output has all of the \ but still has the \n that pkl puts in.
From https://pkl-lang.org/main/current/language-reference/index.html#multiline-strings

The content of a multiline string starts on the first line after the opening quotes and ends on the last line before the closing quotes. Line breaks are included in the string and normalized to \n.

This normalization is the issue. It sounds like there isn't a way to manipulate that normalized line break. It's not a major problem. Just something I was curious about if it was something specific to the JSON implementation or pkl. Writing these in pkl is still significantly more readable.

@TheMadLeprechaun commented on GitHub (Feb 3, 2025): In the original output, all of the backslashes are properly escaped in the Windows paths in the JSON output it's the literal `\n` that pkl puts out that's the issue. I used `replaceAll` since I got an error just trying `replace` ``` Script = #""" #!C:\windows\system32\windowspowershell\v1.0\powershell.exe -file\n $date = get-date -format 'mmddyyy_hhmmss'\n move-item \\unc\path\to\file """#.replaceAll(#"\"#, #"\\"#) ``` and ``` Script = #""" #!c:\windows\system32\windowspowershell\v1.0\powershell.exe -file\n $date = get-date -format 'mmddyyy_hhmmss'\n move-item \\unc\path\to\file """# output { renderer = new JsonRenderer { converters { ["Script"] = (it) -> it.replaceAll(#"\"#, #"\\"#) } } } ``` These both give this output ``` { "Script": "#!c:\\\\windows\\\\system32\\\\windowspowershell\\\\v1.0\\\\powershell.exe -file\\\\n\n$date = get-date -format 'mmddyyy_hhmmss'\\\\n\nmove-item \\\\\\\\unc\\\\path\\\\to\\\\file" } ``` So, on top of the excessive number of `\`, it gives `\\\\n\n`. If I do either of the above and remove the literal `\n` then the output has all of the `\` but still has the `\n` that pkl puts in. From https://pkl-lang.org/main/current/language-reference/index.html#multiline-strings > The content of a multiline string starts on the first line after the opening quotes and ends on the last line before the closing quotes. Line breaks are included in the string and normalized to \n. This normalization is the issue. It sounds like there isn't a way to manipulate that normalized line break. It's not a major problem. Just something I was curious about if it was something specific to the JSON implementation or pkl. Writing these in pkl is still _significantly_ more readable.

adam commented 2025-12-30 01:23:06 +01:00

Author

Owner

Copy Link

@bioball commented on GitHub (Feb 3, 2025):

If you want a newline in Pkl to turn into \\n in the JSON output, you can actually just do replaceAll("\n", "\\n").

For example:

Script = #"""
  #!c:\windows\system32\windowspowershell\v1.0\powershell.exe  -file
  $date = get-date -format 'mmddyyy_hhmmss'
  move-item \\unc\path\to\file
  """#.replaceAll("\n", "\\n")

And here's the output:

{
  "Script": "#!c:\\windows\\system32\\windowspowershell\\v1.0\\powershell.exe  -file\\n$date = get-date -format 'mmddyyy_hhmmss'\\nmove-item \\\\unc\\path\\to\\file"
}

@bioball commented on GitHub (Feb 3, 2025): If you want a newline in Pkl to turn into `\\n` in the JSON output, you can actually just do `replaceAll("\n", "\\n")`. For example: ```pkl Script = #""" #!c:\windows\system32\windowspowershell\v1.0\powershell.exe -file $date = get-date -format 'mmddyyy_hhmmss' move-item \\unc\path\to\file """#.replaceAll("\n", "\\n") ``` And here's the output: ```json { "Script": "#!c:\\windows\\system32\\windowspowershell\\v1.0\\powershell.exe -file\\n$date = get-date -format 'mmddyyy_hhmmss'\\nmove-item \\\\unc\\path\\to\\file" } ```

adam commented 2025-12-30 01:23:06 +01:00

Author

Owner

Copy Link

@TheMadLeprechaun commented on GitHub (Feb 3, 2025):

@bioball That's exactly what I was looking for. Thanks everyone.

@TheMadLeprechaun commented on GitHub (Feb 3, 2025): @bioball That's exactly what I was looking for. Thanks everyone.

Sign in to join this conversation.

No Branch/Tag Specified

Branches Tags

main

release/0.30

srueg/run-format-apply

srueg/format-multiple

srueg/formatter-test-cases

release/0.29

release/0.28

release/0.27

release/0.26

release/0.25

0.30.2

0.30.1

0.30.0

0.29.1

0.29.0

0.28.2

0.28.1

0.28.0

0.27.2

0.27.1

0.27.0

0.26.3

0.26.2

0.26.1

0.26.0

0.25.3

0.25.2

0.25.1

0.25.0

Labels

Clear labels

bug

duplicate

enhancement

good first issue

help wanted

pull-request

Mirrored from GitHub Pull Request

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

adam

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: starred/pkl#273