mirror of
https://github.com/davidkaya/aryx.git
synced 2026-08-05 03:18:44 +02:00
docs: refresh roadmap completion status
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
David Kaya
+167
-165
@@ -19,16 +19,16 @@ Modern AI chat apps set a clear baseline for user expectations: projects, memory
|
||||
|
||||
Based on the current codebase, Kopaya already has:
|
||||
|
||||
- persistent local workspace state for projects, patterns, and sessions in `src/main/persistence/workspaceRepository.ts`
|
||||
- built-in orchestration modes in `src/shared/domain/pattern.ts`: `single`, `sequential`, `concurrent`, `handoff`, `group-chat`, with `magentic` reserved for future support
|
||||
- a dynamic model catalog with provider metadata and reasoning-effort support in `src/shared/domain/models.ts`
|
||||
- scratchpad-specific in-chat model overrides in `src/main/KopayaAppService.ts`
|
||||
- real-time turn streaming and agent activity events in `src/shared/contracts/sidecar.ts`, `src/shared/domain/event.ts`, and `sidecar/src/Kopaya.AgentHost/Services/CopilotWorkflowRunner.cs`
|
||||
- a right-side activity panel that already surfaces per-agent state, model, and effort in `src/renderer/components/ActivityPanel.tsx`
|
||||
- a pattern editor and settings flow in `src/renderer/components/SettingsPanel.tsx`
|
||||
- Copilot CLI-backed runtime access via the system-installed `copilot` command, with Kopaya sanitizing inherited runtime env vars before spawning the sidecar
|
||||
- refreshable Copilot connection diagnostics and settings UI for `ready`, `copilot-cli-missing`, `copilot-auth-required`, and `copilot-error` states in `src/renderer/components/CopilotStatusCard.tsx`, `src/renderer/components/SettingsPanel.tsx`, `src/main/KopayaAppService.ts`, and `sidecar/src/Kopaya.AgentHost/Services/SidecarProtocolHost.cs`
|
||||
- an OS secret store wrapper in `src/main/secrets/secretStore.ts` that can support future non-Copilot secrets and integrations
|
||||
- [x] persistent local workspace state for projects, patterns, and sessions in `src/main/persistence/workspaceRepository.ts`
|
||||
- [x] built-in orchestration modes in `src/shared/domain/pattern.ts`: `single`, `sequential`, `concurrent`, `handoff`, `group-chat`, with `magentic` reserved for future support
|
||||
- [x] a dynamic model catalog with provider metadata and reasoning-effort support in `src/shared/domain/models.ts`
|
||||
- [x] scratchpad-specific in-chat model overrides in `src/main/KopayaAppService.ts`
|
||||
- [x] real-time turn streaming and agent activity events in `src/shared/contracts/sidecar.ts`, `src/shared/domain/event.ts`, and `sidecar/src/Kopaya.AgentHost/Services/CopilotWorkflowRunner.cs`
|
||||
- [x] a right-side activity panel that already surfaces per-agent state, model, and effort in `src/renderer/components/ActivityPanel.tsx`
|
||||
- [x] a pattern editor and settings flow in `src/renderer/components/SettingsPanel.tsx`
|
||||
- [x] Copilot CLI-backed runtime access via the system-installed `copilot` command, with Kopaya sanitizing inherited runtime env vars before spawning the sidecar
|
||||
- [x] refreshable Copilot connection diagnostics and settings UI for `ready`, `copilot-cli-missing`, `copilot-auth-required`, and `copilot-error` states in `src/renderer/components/CopilotStatusCard.tsx`, `src/renderer/components/SettingsPanel.tsx`, `src/main/KopayaAppService.ts`, and `sidecar/src/Kopaya.AgentHost/Services/SidecarProtocolHost.cs`
|
||||
- [x] an OS secret store wrapper in `src/main/secrets/secretStore.ts` that can support future non-Copilot secrets and integrations
|
||||
|
||||
That is a strong base. The biggest gaps are not around "can it run agents?" but around:
|
||||
|
||||
@@ -61,14 +61,14 @@ That is a strong base. The biggest gaps are not around "can it run agents?" but
|
||||
|
||||
These are the improvements users will expect from any serious AI desktop app.
|
||||
|
||||
| Priority | Initiative | Why users need it | Likely layers |
|
||||
| --- | --- | --- | --- |
|
||||
| Highest | Copilot connection and account status management | Users need a clear way to see whether Copilot is installed, authenticated, healthy, and able to serve the expected models. | Renderer, main, sidecar |
|
||||
| Highest | Conversation organization and search | Users need to find old work quickly, pin important threads, archive noise, and search by project, title, agent, and content. | Renderer, persistence |
|
||||
| Highest | Session export and sharing | Users will want to export runs to Markdown/JSON/PDF, share patterns, and preserve outcomes outside the app. | Renderer, main, persistence |
|
||||
| High | Attachments and artifact handling | Modern chat apps let users drop files into a thread and keep generated artifacts nearby. This is table stakes for research and coding workflows. | Renderer, main, sidecar |
|
||||
| High | Chat branching and session forking | Users need to explore alternate solution paths without losing the original conversation. | Renderer, persistence |
|
||||
| High | Better error and diagnostics UX | Sidecar/runtime issues need clear explanation, retry actions, and debug details instead of vague failure states. | Renderer, main, sidecar |
|
||||
| Priority | Initiative | Why users need it | Likely layers |
|
||||
| -------- | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------- |
|
||||
| Highest | Copilot connection and account status management | Users need a clear way to see whether Copilot is installed, authenticated, healthy, and able to serve the expected models. | Renderer, main, sidecar |
|
||||
| Highest | Conversation organization and search | Users need to find old work quickly, pin important threads, archive noise, and search by project, title, agent, and content. | Renderer, persistence |
|
||||
| Highest | Session export and sharing | Users will want to export runs to Markdown/JSON/PDF, share patterns, and preserve outcomes outside the app. | Renderer, main, persistence |
|
||||
| High | Attachments and artifact handling | Modern chat apps let users drop files into a thread and keep generated artifacts nearby. This is table stakes for research and coding workflows. | Renderer, main, sidecar |
|
||||
| High | Chat branching and session forking | Users need to explore alternate solution paths without losing the original conversation. | Renderer, persistence |
|
||||
| High | Better error and diagnostics UX | Sidecar/runtime issues need clear explanation, retry actions, and debug details instead of vague failure states. | Renderer, main, sidecar |
|
||||
|
||||
### What this should look like
|
||||
|
||||
@@ -97,59 +97,61 @@ If Kopaya later adds direct OpenAI, Anthropic, Google, MCP, or team-managed secr
|
||||
|
||||
- [x] global search across sessions, messages, projects, and patterns
|
||||
- [x] pinned sessions, archived sessions
|
||||
- [x] filters like "running", "errored", "scratchpad", "project X", "pattern Y"
|
||||
- [ ] filters like "running", "errored", "scratchpad", "project X", "pattern Y"
|
||||
- [x] duplicate session, rename session, and favorite pattern
|
||||
- [ ] lightweight tags
|
||||
- [ ] recent activity views and "resume where I left off"
|
||||
|
||||
The shared/backend query layer supports these filters, but the dedicated filter UI is currently deferred.
|
||||
|
||||
#### Session export and sharing
|
||||
|
||||
- export a full run to Markdown
|
||||
- export machine-readable JSON for debugging and replay
|
||||
- copy/share a clean transcript without internal activity noise
|
||||
- export a pattern with its agents, instructions, and model selections
|
||||
- [ ] export a full run to Markdown
|
||||
- [ ] export machine-readable JSON for debugging and replay
|
||||
- [ ] copy/share a clean transcript without internal activity noise
|
||||
- [ ] export a pattern with its agents, instructions, and model selections
|
||||
|
||||
#### Attachments and artifacts
|
||||
|
||||
- drag-and-drop files into chat
|
||||
- inline preview for code, Markdown, images, and PDFs
|
||||
- artifact shelf for generated outputs
|
||||
- "open in project", "save as", and "promote to workspace artifact"
|
||||
- [ ] drag-and-drop files into chat
|
||||
- [ ] inline preview for code, Markdown, images, and PDFs
|
||||
- [ ] artifact shelf for generated outputs
|
||||
- [ ] "open in project", "save as", and "promote to workspace artifact"
|
||||
|
||||
#### Chat branching and session forking
|
||||
|
||||
- fork from any message
|
||||
- compare branch A vs branch B
|
||||
- keep separate titles and summaries for each branch
|
||||
- optionally turn a fork into a new pattern experiment
|
||||
- [ ] fork from any message
|
||||
- [ ] compare branch A vs branch B
|
||||
- [ ] keep separate titles and summaries for each branch
|
||||
- [ ] optionally turn a fork into a new pattern experiment
|
||||
|
||||
#### Better error and diagnostics UX
|
||||
|
||||
- collapsible run diagnostics panel
|
||||
- sidecar logs per session
|
||||
- retry failed turn
|
||||
- "why did this fail?" summaries
|
||||
- copyable debug bundle for issue reports
|
||||
- [ ] collapsible run diagnostics panel
|
||||
- [ ] sidecar logs per session
|
||||
- [ ] retry failed turn
|
||||
- [ ] "why did this fail?" summaries
|
||||
- [ ] copyable debug bundle for issue reports
|
||||
|
||||
## 2. Project-aware coding improvements
|
||||
|
||||
Kopaya should feel much smarter about the project it is attached to.
|
||||
|
||||
| Priority | Initiative | Why it matters | Likely layers |
|
||||
| --- | --- | --- | --- |
|
||||
| Highest | Project context controls | Users need to know what the agents can see and what is excluded. | Renderer, main, sidecar |
|
||||
| High | Git-aware context | Branch, diff, dirty state, and commit context are essential for coding workflows. | Main, renderer |
|
||||
| High | Workspace indexing and summaries | Large projects need a fast, understandable overview before orchestration starts. | Main, sidecar, persistence |
|
||||
| High | File pinning and working sets | Users need to constrain attention to a selected set of files or folders. | Renderer, persistence, sidecar |
|
||||
| Medium | Project presets | Teams will want reusable project-level context, rules, and exclusion templates. | Persistence, renderer |
|
||||
| Priority | Initiative | Why it matters | Likely layers |
|
||||
| -------- | -------------------------------- | --------------------------------------------------------------------------------- | ------------------------------ |
|
||||
| Highest | Project context controls | Users need to know what the agents can see and what is excluded. | Renderer, main, sidecar |
|
||||
| High | Git-aware context | Branch, diff, dirty state, and commit context are essential for coding workflows. | Main, renderer |
|
||||
| High | Workspace indexing and summaries | Large projects need a fast, understandable overview before orchestration starts. | Main, sidecar, persistence |
|
||||
| High | File pinning and working sets | Users need to constrain attention to a selected set of files or folders. | Renderer, persistence, sidecar |
|
||||
| Medium | Project presets | Teams will want reusable project-level context, rules, and exclusion templates. | Persistence, renderer |
|
||||
|
||||
### Recommended features
|
||||
|
||||
- show project metadata: repo name, branch, dirty state, languages, package managers, solution files
|
||||
- explicit include/exclude controls using gitignore-aware defaults plus manual overrides
|
||||
- working set support: "only reason about these files/folders"
|
||||
- project summary card: architecture snapshot, detected stack, test commands, important entry points
|
||||
- saved session context packs, such as "frontend only", "API layer", or "build pipeline"
|
||||
- [ ] show project metadata: repo name, branch, dirty state, languages, package managers, solution files
|
||||
- [ ] explicit include/exclude controls using gitignore-aware defaults plus manual overrides
|
||||
- [ ] working set support: "only reason about these files/folders"
|
||||
- [ ] project summary card: architecture snapshot, detected stack, test commands, important entry points
|
||||
- [ ] saved session context packs, such as "frontend only", "API layer", or "build pipeline"
|
||||
|
||||
This is where Kopaya can beat generic chat apps: not just talking about a project, but acting like a focused control surface for that project.
|
||||
|
||||
@@ -157,14 +159,14 @@ This is where Kopaya can beat generic chat apps: not just talking about a projec
|
||||
|
||||
This is the highest-leverage product area. If done well, it becomes Kopaya's signature advantage.
|
||||
|
||||
| Priority | Initiative | Why users need it | Likely layers |
|
||||
| --- | --- | --- | --- |
|
||||
| Highest | Rich run timeline | Users need to see the exact sequence of thinking, handoffs, tool calls, and outputs. | Renderer, sidecar |
|
||||
| Highest | Replayable run traces | Users need to inspect how a result was produced, not just read the final answer. | Sidecar, persistence, renderer |
|
||||
| High | Pattern versioning | Users need to know which pattern version produced which session. | Persistence, renderer |
|
||||
| High | Run comparison lab | Users need to compare models, patterns, prompts, or reasoning settings side by side. | Renderer, persistence, sidecar |
|
||||
| High | Guardrails and policy controls | Users need caps for cost, tools, runtime, file access, and escalation behavior. | Renderer, sidecar, persistence |
|
||||
| High | Approval checkpoints | Users need pause-and-approve steps before risky tool use, file writes, or final actions. | Renderer, sidecar |
|
||||
| Priority | Initiative | Why users need it | Likely layers |
|
||||
| -------- | ------------------------------ | ---------------------------------------------------------------------------------------- | ------------------------------ |
|
||||
| Highest | Rich run timeline | Users need to see the exact sequence of thinking, handoffs, tool calls, and outputs. | Renderer, sidecar |
|
||||
| Highest | Replayable run traces | Users need to inspect how a result was produced, not just read the final answer. | Sidecar, persistence, renderer |
|
||||
| High | Pattern versioning | Users need to know which pattern version produced which session. | Persistence, renderer |
|
||||
| High | Run comparison lab | Users need to compare models, patterns, prompts, or reasoning settings side by side. | Renderer, persistence, sidecar |
|
||||
| High | Guardrails and policy controls | Users need caps for cost, tools, runtime, file access, and escalation behavior. | Renderer, sidecar, persistence |
|
||||
| High | Approval checkpoints | Users need pause-and-approve steps before risky tool use, file writes, or final actions. | Renderer, sidecar |
|
||||
|
||||
### Rich run timeline
|
||||
|
||||
@@ -172,75 +174,75 @@ The current activity model already exposes `thinking`, `tool-calling`, `handoff`
|
||||
|
||||
Build on that with:
|
||||
|
||||
- a vertical run timeline in the side panel
|
||||
- event cards with timestamps, agent identity, tool name, and result status
|
||||
- per-agent run lanes
|
||||
- grouping of streaming deltas into one coherent answer step
|
||||
- jump-to-message and jump-to-agent actions
|
||||
- [ ] a vertical run timeline in the side panel
|
||||
- [ ] event cards with timestamps, agent identity, tool name, and result status
|
||||
- [ ] per-agent run lanes
|
||||
- [ ] grouping of streaming deltas into one coherent answer step
|
||||
- [ ] jump-to-message and jump-to-agent actions
|
||||
|
||||
### Replayable run traces
|
||||
|
||||
- store a structured event log per run
|
||||
- replay a completed run step by step
|
||||
- scrub through the run like a debugger timeline
|
||||
- inspect the exact sequence of agent activations and tool invocations
|
||||
- preserve environment metadata such as model, effort, project path, and pattern version
|
||||
- [ ] store a structured event log per run
|
||||
- [ ] replay a completed run step by step
|
||||
- [ ] scrub through the run like a debugger timeline
|
||||
- [ ] inspect the exact sequence of agent activations and tool invocations
|
||||
- [ ] preserve environment metadata such as model, effort, project path, and pattern version
|
||||
|
||||
This would be a major differentiator. Most chat apps show outputs; very few make multi-agent execution truly inspectable.
|
||||
|
||||
### Pattern versioning
|
||||
|
||||
- immutable versions for saved patterns
|
||||
- "session used pattern v7"
|
||||
- diff view for instructions, models, agents, and iteration counts
|
||||
- rollback and duplicate-from-version
|
||||
- changelog notes for team-facing patterns
|
||||
- [ ] immutable versions for saved patterns
|
||||
- [ ] "session used pattern v7"
|
||||
- [ ] diff view for instructions, models, agents, and iteration counts
|
||||
- [ ] rollback and duplicate-from-version
|
||||
- [ ] changelog notes for team-facing patterns
|
||||
|
||||
### Run comparison lab
|
||||
|
||||
- run the same user prompt against multiple patterns or model mixes
|
||||
- compare output quality, latency, handoff structure, and cost
|
||||
- save a winner as the new default
|
||||
- benchmark patterns against a reusable prompt set
|
||||
- [ ] run the same user prompt against multiple patterns or model mixes
|
||||
- [ ] compare output quality, latency, handoff structure, and cost
|
||||
- [ ] save a winner as the new default
|
||||
- [ ] benchmark patterns against a reusable prompt set
|
||||
|
||||
### Guardrails and policy controls
|
||||
|
||||
- max iterations per run
|
||||
- max tool calls per agent
|
||||
- time budget and cost budget
|
||||
- allowed tools per pattern
|
||||
- allowed paths per project
|
||||
- "tool use requires approval" mode
|
||||
- [ ] max iterations per run
|
||||
- [ ] max tool calls per agent
|
||||
- [ ] time budget and cost budget
|
||||
- [ ] allowed tools per pattern
|
||||
- [ ] allowed paths per project
|
||||
- [ ] "tool use requires approval" mode
|
||||
|
||||
### Approval checkpoints
|
||||
|
||||
- pause before a tool call
|
||||
- pause before handing off outside the original working set
|
||||
- pause before final answer publication
|
||||
- assign specific checkpoints to specific agents or pattern modes
|
||||
- [ ] pause before a tool call
|
||||
- [ ] pause before handing off outside the original working set
|
||||
- [ ] pause before final answer publication
|
||||
- [ ] assign specific checkpoints to specific agents or pattern modes
|
||||
|
||||
## 4. Advanced orchestration capabilities
|
||||
|
||||
Once the control plane is solid, Kopaya should move from multi-agent chat to true workflow orchestration.
|
||||
|
||||
| Priority | Initiative | Why it matters | Likely layers |
|
||||
| --- | --- | --- | --- |
|
||||
| High | Planner-executor-evaluator loops | Strong default pattern for quality, validation, and self-correction. | Sidecar, pattern system, renderer |
|
||||
| High | Conditional routing and DAG flows | Real workflows need branching, retries, and conditional steps beyond today's fixed modes. | Pattern system, sidecar, renderer |
|
||||
| High | Background and long-running jobs | Users need runs that continue while they browse or switch sessions. | Main, sidecar, renderer |
|
||||
| Medium | Cross-project campaigns | Some workflows should coordinate across multiple repositories or workspaces. | Persistence, sidecar, renderer |
|
||||
| Medium | Memory layers | Users need structured memory beyond raw chat history. | Persistence, sidecar, renderer |
|
||||
| Medium | Autonomy levels | Some runs should be advisory, some supervised, some semi-autonomous. | Renderer, sidecar, persistence |
|
||||
| Backlog | Magentic mode support | Already reserved in the domain model and should activate when the runtime supports it. | Sidecar, shared domain, renderer |
|
||||
| Priority | Initiative | Why it matters | Likely layers |
|
||||
| -------- | --------------------------------- | ----------------------------------------------------------------------------------------- | --------------------------------- |
|
||||
| High | Planner-executor-evaluator loops | Strong default pattern for quality, validation, and self-correction. | Sidecar, pattern system, renderer |
|
||||
| High | Conditional routing and DAG flows | Real workflows need branching, retries, and conditional steps beyond today's fixed modes. | Pattern system, sidecar, renderer |
|
||||
| High | Background and long-running jobs | Users need runs that continue while they browse or switch sessions. | Main, sidecar, renderer |
|
||||
| Medium | Cross-project campaigns | Some workflows should coordinate across multiple repositories or workspaces. | Persistence, sidecar, renderer |
|
||||
| Medium | Memory layers | Users need structured memory beyond raw chat history. | Persistence, sidecar, renderer |
|
||||
| Medium | Autonomy levels | Some runs should be advisory, some supervised, some semi-autonomous. | Renderer, sidecar, persistence |
|
||||
| Backlog | Magentic mode support | Already reserved in the domain model and should activate when the runtime supports it. | Sidecar, shared domain, renderer |
|
||||
|
||||
### Planner-executor-evaluator loops
|
||||
|
||||
Add first-class support for patterns like:
|
||||
|
||||
- planner -> implementer -> reviewer
|
||||
- researcher -> synthesizer -> critic
|
||||
- triage -> specialist -> verifier
|
||||
- generator -> judge -> repair loop until threshold
|
||||
- [ ] planner -> implementer -> reviewer
|
||||
- [ ] researcher -> synthesizer -> critic
|
||||
- [ ] triage -> specialist -> verifier
|
||||
- [ ] generator -> judge -> repair loop until threshold
|
||||
|
||||
This should be more than custom instructions. It should be a product concept with templates, visibility, and metrics.
|
||||
|
||||
@@ -248,39 +250,39 @@ This should be more than custom instructions. It should be a product concept wit
|
||||
|
||||
Move beyond a fixed list of orchestration modes and introduce:
|
||||
|
||||
- conditional edges
|
||||
- retries on low confidence
|
||||
- fallback agent paths
|
||||
- multi-branch flows that rejoin
|
||||
- "if tool X returns Y, route to specialist Z"
|
||||
- [ ] conditional edges
|
||||
- [ ] retries on low confidence
|
||||
- [ ] fallback agent paths
|
||||
- [ ] multi-branch flows that rejoin
|
||||
- [ ] "if tool X returns Y, route to specialist Z"
|
||||
|
||||
A node-and-edge designer would make this much easier to understand than a purely form-based editor.
|
||||
|
||||
### Background and long-running jobs
|
||||
|
||||
- queue runs for later
|
||||
- continue in the background while the user works elsewhere
|
||||
- desktop notifications when a run reaches a checkpoint or fails
|
||||
- background research, repo audits, doc generation, or code review workflows
|
||||
- [ ] queue runs for later
|
||||
- [ ] continue in the background while the user works elsewhere
|
||||
- [ ] desktop notifications when a run reaches a checkpoint or fails
|
||||
- [ ] background research, repo audits, doc generation, or code review workflows
|
||||
|
||||
### Cross-project campaigns
|
||||
|
||||
Examples:
|
||||
|
||||
- audit the same policy across multiple repositories
|
||||
- generate migration plans across a workspace portfolio
|
||||
- compare implementation patterns across projects
|
||||
- run one planner over many project-specific executor sessions
|
||||
- [ ] audit the same policy across multiple repositories
|
||||
- [ ] generate migration plans across a workspace portfolio
|
||||
- [ ] compare implementation patterns across projects
|
||||
- [ ] run one planner over many project-specific executor sessions
|
||||
|
||||
### Memory layers
|
||||
|
||||
Kopaya should eventually distinguish between:
|
||||
|
||||
- session memory: just this thread
|
||||
- project memory: facts about a specific repository
|
||||
- pattern memory: lessons or defaults attached to a workflow
|
||||
- user preferences: tone, depth, risk tolerance, approval defaults
|
||||
- team memory: shared conventions and approved instructions
|
||||
- [ ] session memory: just this thread
|
||||
- [ ] project memory: facts about a specific repository
|
||||
- [ ] pattern memory: lessons or defaults attached to a workflow
|
||||
- [ ] user preferences: tone, depth, risk tolerance, approval defaults
|
||||
- [ ] team memory: shared conventions and approved instructions
|
||||
|
||||
Users should be able to inspect, edit, clear, and scope each memory layer.
|
||||
|
||||
@@ -288,10 +290,10 @@ Users should be able to inspect, edit, clear, and scope each memory layer.
|
||||
|
||||
Introduce explicit run modes such as:
|
||||
|
||||
- advisory only
|
||||
- supervised execution
|
||||
- auto-run within guardrails
|
||||
- background delegated task
|
||||
- [ ] advisory only
|
||||
- [ ] supervised execution
|
||||
- [ ] auto-run within guardrails
|
||||
- [ ] background delegated task
|
||||
|
||||
This gives users a clearer mental model than burying autonomy inside pattern instructions.
|
||||
|
||||
@@ -299,34 +301,34 @@ This gives users a clearer mental model than burying autonomy inside pattern ins
|
||||
|
||||
Single-user desktop value is important, but long-term adoption will benefit from team workflows.
|
||||
|
||||
| Priority | Initiative | Why users need it | Likely layers |
|
||||
| --- | --- | --- | --- |
|
||||
| High | Pattern import/export and registry | Teams need to share proven workflows. | Persistence, renderer |
|
||||
| High | Shared run reports | Users need a clean way to send outcomes to teammates. | Renderer, persistence |
|
||||
| Medium | Team workspaces | Shared projects, pattern libraries, and session visibility. | Persistence, backend services |
|
||||
| Medium | Comments and annotations | Humans need to discuss runs and approve or reject outputs. | Renderer, persistence |
|
||||
| Medium | Audit logs and secret governance | Important for enterprise or regulated use. | Main, persistence, backend services |
|
||||
| Medium | Roles and permissions | Useful once teams share patterns, credentials, and automation. | Backend services, renderer |
|
||||
| Priority | Initiative | Why users need it | Likely layers |
|
||||
| -------- | ---------------------------------- | -------------------------------------------------------------- | ----------------------------------- |
|
||||
| High | Pattern import/export and registry | Teams need to share proven workflows. | Persistence, renderer |
|
||||
| High | Shared run reports | Users need a clean way to send outcomes to teammates. | Renderer, persistence |
|
||||
| Medium | Team workspaces | Shared projects, pattern libraries, and session visibility. | Persistence, backend services |
|
||||
| Medium | Comments and annotations | Humans need to discuss runs and approve or reject outputs. | Renderer, persistence |
|
||||
| Medium | Audit logs and secret governance | Important for enterprise or regulated use. | Main, persistence, backend services |
|
||||
| Medium | Roles and permissions | Useful once teams share patterns, credentials, and automation. | Backend services, renderer |
|
||||
|
||||
### Practical collaboration features
|
||||
|
||||
- comment on a pattern version
|
||||
- share a run summary instead of a raw transcript
|
||||
- mark a pattern as approved, experimental, or deprecated
|
||||
- create a pattern library with tags like "coding", "docs", "triage", "research"
|
||||
- import/export pattern bundles
|
||||
- [ ] comment on a pattern version
|
||||
- [ ] share a run summary instead of a raw transcript
|
||||
- [ ] mark a pattern as approved, experimental, or deprecated
|
||||
- [ ] create a pattern library with tags like "coding", "docs", "triage", "research"
|
||||
- [ ] import/export pattern bundles
|
||||
|
||||
## 6. Evaluation and continuous improvement
|
||||
|
||||
If Kopaya is going to orchestrate important work, it needs a way to measure quality.
|
||||
|
||||
| Priority | Initiative | Why it matters | Likely layers |
|
||||
| --- | --- | --- | --- |
|
||||
| High | Prompt and pattern eval suites | Users need a repeatable way to see if a pattern got better or worse. | Persistence, sidecar, renderer |
|
||||
| High | Regression testing for workflows | Teams need confidence before updating a widely used pattern. | Sidecar, persistence |
|
||||
| Medium | Run quality scoring | Helpful for ranking candidate outputs and routing retries. | Sidecar, renderer |
|
||||
| Medium | Cost/latency analytics | Users need to understand trade-offs between quality and speed. | Sidecar, persistence, renderer |
|
||||
| Medium | Golden datasets for coding tasks | Useful for tuning workflows for a repository or team. | Persistence, tooling |
|
||||
| Priority | Initiative | Why it matters | Likely layers |
|
||||
| -------- | -------------------------------- | -------------------------------------------------------------------- | ------------------------------ |
|
||||
| High | Prompt and pattern eval suites | Users need a repeatable way to see if a pattern got better or worse. | Persistence, sidecar, renderer |
|
||||
| High | Regression testing for workflows | Teams need confidence before updating a widely used pattern. | Sidecar, persistence |
|
||||
| Medium | Run quality scoring | Helpful for ranking candidate outputs and routing retries. | Sidecar, renderer |
|
||||
| Medium | Cost/latency analytics | Users need to understand trade-offs between quality and speed. | Sidecar, persistence, renderer |
|
||||
| Medium | Golden datasets for coding tasks | Useful for tuning workflows for a repository or team. | Persistence, tooling |
|
||||
|
||||
### What this could unlock
|
||||
|
||||
@@ -345,12 +347,12 @@ These are the ideas with the best chance of making Kopaya feel distinct rather t
|
||||
|
||||
A proper debugger for AI runs:
|
||||
|
||||
- event timeline
|
||||
- step replay
|
||||
- handoff graph
|
||||
- tool call trace
|
||||
- per-agent output inspection
|
||||
- final answer provenance
|
||||
- [ ] event timeline
|
||||
- [ ] step replay
|
||||
- [ ] handoff graph
|
||||
- [ ] tool call trace
|
||||
- [ ] per-agent output inspection
|
||||
- [ ] final answer provenance
|
||||
|
||||
This would make complex agent runs understandable in a way most products do not.
|
||||
|
||||
@@ -358,10 +360,10 @@ This would make complex agent runs understandable in a way most products do not.
|
||||
|
||||
Let a user turn an ad hoc scratchpad conversation into a reusable orchestration pattern:
|
||||
|
||||
- detect the roles that emerged
|
||||
- suggest agent breakdowns
|
||||
- convert a successful chat into a draft workflow
|
||||
- save the resulting pattern back into the library
|
||||
- [ ] detect the roles that emerged
|
||||
- [ ] suggest agent breakdowns
|
||||
- [ ] convert a successful chat into a draft workflow
|
||||
- [ ] save the resulting pattern back into the library
|
||||
|
||||
This would connect casual use and power-user workflow design.
|
||||
|
||||
@@ -369,11 +371,11 @@ This would connect casual use and power-user workflow design.
|
||||
|
||||
Instead of just comparing model outputs, compare:
|
||||
|
||||
- one agent vs multi-agent
|
||||
- sequential vs concurrent
|
||||
- GPT-heavy vs Claude-heavy
|
||||
- high-effort vs medium-effort
|
||||
- guarded vs unguarded runs
|
||||
- [ ] one agent vs multi-agent
|
||||
- [ ] sequential vs concurrent
|
||||
- [ ] GPT-heavy vs Claude-heavy
|
||||
- [ ] high-effort vs medium-effort
|
||||
- [ ] guarded vs unguarded runs
|
||||
|
||||
Kopaya should become the easiest place to answer, "which setup is actually better for this task?"
|
||||
|
||||
@@ -381,21 +383,21 @@ Kopaya should become the easiest place to answer, "which setup is actually bette
|
||||
|
||||
Most agent tools either automate too much or stop at chat. Kopaya can own the middle ground:
|
||||
|
||||
- route to human review at key points
|
||||
- require approval before tool execution or publishing
|
||||
- allow humans to override, edit, or redirect handoffs
|
||||
- [ ] route to human review at key points
|
||||
- [ ] require approval before tool execution or publishing
|
||||
- [ ] allow humans to override, edit, or redirect handoffs
|
||||
|
||||
### 5. Reproducible run snapshots
|
||||
|
||||
Every important run should be reproducible with a snapshot of:
|
||||
|
||||
- project path or repo revision
|
||||
- pattern version
|
||||
- agent list
|
||||
- model selection
|
||||
- reasoning effort
|
||||
- tool permissions
|
||||
- event trace
|
||||
- [ ] project path or repo revision
|
||||
- [ ] pattern version
|
||||
- [ ] agent list
|
||||
- [ ] model selection
|
||||
- [ ] reasoning effort
|
||||
- [ ] tool permissions
|
||||
- [ ] event trace
|
||||
|
||||
This is especially valuable for engineering and enterprise use cases.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user