starred/aryx
1
0
Fork 0
mirror of https://github.com/davidkaya/aryx.git synced 2026-08-06 11:48:46 +02:00
Code Issues 3 Packages Projects Releases 10 Wiki Activity

fix: remove streaming text boundary heuristics that injected spurious spaces

Browse Source 
The StreamingTextMerger (C#) and mergeStreamingText (TypeScript) contained
custom word-boundary heuristics that inserted spaces between streaming deltas
based on incorrect assumptions. Two bugs caused mid-word spaces:

1. Unconditional space insertion when incoming delta started with uppercase or
   digit characters, breaking acronyms (MDA -> M DA, UAG -> UA G)
2. LooksLikeWordBoundary using global token counts that fired on nearly every
   mid-word token split (writable -> wr itable, fragile -> frag ile)

Both upstream projects (Copilot SDK and Agent Framework) use simple string
concatenation for delta accumulation. LLM streaming tokens natively include
whitespace when needed, so no boundary detection is required.

Removed all boundary heuristic methods and replaced with plain concatenation.
Kept overlap detection and snapshot handling for event redelivery scenarios.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
David Kaya
2026-04-15 18:55:34 +02:00
co-authored by Copilot
parent 418946b854
commit c7910e6e4b
4 changed files with 79 additions and 166 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/shared/streamingText.test.ts
+22 -5
View File
@@ -26,17 +26,34 @@ describe('streaming text merge', () => {
expect(mergeStreamingText(current, incoming)).toBe(incoming);
});
test('inserts whitespace when snapshot-like updates would otherwise glue words together', () => {
expect(mergeStreamingText('How about', 'The **Ashen Crown** feels')).toBe(
test.each([
['requires all wr', 'itable fields', 'requires all writable fields'],
['becomes frag', 'ile for clients', 'becomes fragile for clients'],
['Endpoint (domain) uniqu', 'eness across tenants', 'Endpoint (domain) uniqueness across tenants'],
['The doc says "wildc', 'ards are allowed"', 'The doc says "wildcards are allowed"'],
[
'What wildcard syntax supported (*.cont',
'oso.com? contoso.* ?)',
'What wildcard syntax supported (*.contoso.com? contoso.* ?)',
],
['How does Pur', 'view match traffic', 'How does Purview match traffic'],
['more M', 'DA properties', 'more MDA properties'],
['does UA', 'G normalize them?', 'does UAG normalize them?'],
])('does not inject spaces into split words: %s + %s', (current, incoming, expected) => {
expect(mergeStreamingText(current, incoming)).toBe(expected);
});
test('preserves whitespace already present in delta', () => {
expect(mergeStreamingText('How about', ' The **Ashen Crown** feels')).toBe(
'How about The **Ashen Crown** feels',
);
expect(mergeStreamingText('The **Ashen Crown** feels', 'classic and timeless.')).toBe(
expect(mergeStreamingText('The **Ashen Crown** feels', ' classic and timeless.')).toBe(
'The **Ashen Crown** feels classic and timeless.',
);
});
test('inserts a newline before streamed markdown block markers', () => {
expect(mergeStreamingText('If you want, I can also give you', '- darker titles')).toBe(
test('preserves newline already present in delta', () => {
expect(mergeStreamingText('If you want, I can also give you', '\n- darker titles')).toBe(
'If you want, I can also give you\n- darker titles',
);
});