Improving AI Agent Accuracy Through Structured Prompting
What initially appeared to be an AI accuracy issue in a financial AI analysis tool was ultimately an interaction design problem. By introducing structured object-based referencing within chat, I reduced ambiguity, decreased reference-related errors by 54%, and significantly improved report reliability and analysis speed.
My Role
Lead Product Designer
Status
Shipped to production · Scaled platform-wide
Scope
Core AI chat interaction
Alfa is Boosted.ai’s agentic AI platform designed for investment professionals.
It combines conversational AI with autonomous financial agents to help users research markets, analyze portfolios, and generate reports through natural language workflows. By connecting directly to structured financial data and automating complex analysis, Alfa delivers real-time insights that support faster, more informed investment decisions.
PROBLEM
After Alfa’s initial rollout, reports frequently failed due to incomplete results and inconsistent outputs, reducing user trust and slowing product adoption.
These issues were especially common in more complex analyses that involved longer, more detailed prompts.
SOLUTION
Structured Prompting with Object-Based Referencing
To address ambiguity and inconsistent outputs, I led the end-to-end design of a structured, object-based referencing system within chat.
This solution is implemented through what we call a chip system - interactive, structured elements embedded directly inside the message input. Users can insert specific portfolios, securities, or documents while composing a prompt, and each chip links to a unique system ID rather than plain text.
This makes user intent explicit, eliminates guesswork for the agent, and significantly improves report reliability while preserving the flexibility of natural language interaction.
Design Thinking Process
01 Discovery
To understand the issue better, I did:
1. Session Replay Analysis: Reviewed FullStory session recordings to identify user friction points.
2. Internal Interviews: Interviewed customer success team to figure out the user pain points
3. User Interviews: Conducted sessions with portfolio managers and analysts to uncover how they use Alfa
KEY FINDINGS
When analyzing failed reports, I saw a pattern - most errors appeared in longer and more complex prompts that referenced multiple objects.
To provide the necessary context for analysis, users had to manually locate and reference multiple structured objects in chat. This required them to:
-
Search for relavent documents
-
Drag in or copy past several documents (Earnings, research, etc. )
-
Search and enter portfolio and watchlists names to reference
-
Enter stock tickers by hand
Complex prompts
failed more often
Users often searched
3–5 documents
Some prompt writing took
30-60 seconds
KEY OBSERVATIONS
Example of a Real User Prompt - Complex and Fragile
This is the kind of long prompt users often type into the chat. In a single message, they may reference 13 structured objects - including a portfolio, several stocks, and multiple documents.
To the user, the request feels detailed and clear. But for the system, it becomes a complex prompt with many references that are harder to interpret accurately.
Each item must be matched exactly. If a stock is misspelled, a document name is slightly different, or one file is missing, the system can become confused. Because everything is written in plain text, even small mistakes can lead to errors, incomplete results, or failed reports.
02 Define
After analyzing failed reports, prompt patterns, and system behavior, I organized the insights to understand what was really happening. The issue wasn’t that users didn’t know how to prompt, the issue was that structured data was being referenced through unstructured text.
This created ambiguity for the agent and increased failure risk.
The system already had structure.
The chat input did not.
We needed to redesign how structured data integrates into conversational workflows without restricting the user experience.
03 Design
To address ambiguity and interpretation errors, I led the design of a Smart Chip System - a structured interaction layer embedded directly within chat.
Rather than relying purely on free-text prompts, users can insert explicit object references while composing their message:
-
Type @ to trigger object selection
-
Choose an object type (Document, Portfolio, Stock, etc.)
-
Search and select the relevant item
Once selected, the object appears as a structured chip inside the input field.
Each chip:
-
Represents a real system object
-
Is linked to its unique system ID rather than plain text
-
Provides the AI with explicit intent signals instead of inferred context
This redesign shifted the interaction from ambiguous language to structured intent, significantly improving how the system interprets prompts.
Chips are also editable during drafting. Users can click a chip to quickly replace it with another object of the same type, allowing precise adjustments without rewriting the entire prompt. This increases both efficiency and confidence in high-stakes analytical workflows.
The same system was later used in our Prompt Library, allowing users to swap structured variables, such as replacing one portfolio, stock, or document with another - without rewriting the entire prompt.
INTERACTION EXPLORATIONS
I made deliberate design decisions to stabilize a complex interaction model. I defined how chips reference system objects, established a clear structure and visual language, and designed distinct styles for different chip types (Portfolio, Watchlist, Stock, Document, etc.). I also defined consistent interaction patterns for insertion, editing, and removal to ensure Alfa remains reliable, intuitive, and scalable.
Chip Anatomy & Constraints
Chip Editing Experience
Usage and Behaviour
04 Deliver
After validating the interaction model internally I:
-
Created high-fidelity designs in Figma
-
Defined interaction states (hover, selected, invalid, empty search, loading)
-
Documented behavior logic for engineering (ID binding, deletion rules, fallback states)
-
Partnered closely with engineering to ensure technical feasibility
I presented the final solution to stakeholders across Product, Engineering, and Customer Success, securing cross-functional alignment and approval for implementation.
BEFORE
Manual inputs and free-text prompting created ambiguity and frequent errors.
AFTER
Structured, machine-readable referencing allowed user to easily submit requests
05 Impact
The Chip System significantly improved report reliability and prompt efficiency. By turning free-text references into structured, system-recognizable objects, we reduced errors, increased report completion rates, and made complex analysis more stable.This shift moved the experience from fragile and time-consuming to structured and efficient, improving both system performance and user confidence.
Reflection
What initially appeared to be a gap in accuacury was ultimately a gap in how input was structured and information was referenced. By reframing the problem at the interaction layer, we were able to improve system performance without changing the underlying model.
Designing the Chip System deepened my understanding of AI product design. It’s not just about crafting interfaces - it’s about shaping the inputs, constraints, and flows that influence user and system behavior. Thoughtful interaction design can meaningfully increase accuracy, trust, and user confidence.
This experience strengthened how I approach ambiguous AI challenges: start at the user level, not just the surface or system level.