Users need to understand what context, tools, and retrieval steps shaped an AI response.
These prototypes are about making memory, tool calls, retrieved sources, and execution paths visible enough that people can question the system, correct it, and build trust in the collaboration.
The interesting part is the loop: the model asks for tools, fetches scores, asks again for a box score, and returns data the interface can render as game cards.
After wrapping up my HCI class, I came back to this human-AI prototype and rebuilt the chat loop around local tool calling. The stack is Next.js with Ollama running gpt-oss:20b on an M4 Max.
I am less interested in chat as a transcript and more interested in chat as a coordination surface. In this version, the model can request tools, fetch NBA scores, call again for a box score, and hand structured data back to the UI.
The client renders that response as NBA game cards and box score views instead of leaving everything as text. I also show the intermediate steps, partly because the tool orchestration is the point of the prototype and partly because I want to see where the model is making decisions.
If context is going to shape a response, I want the interface to show what context was used and give people a way to correct it.
A recurring question for me is how software stays understandable, especially when an LLM is involved. If a system gives a personalized answer, it is reasonable to wonder why it said that and what it thinks it knows about you.
So I made the user context passed along with each request visible, editable, and deletable item by item. The goal is simple: show the facts and guesses that may shape the response, then let the user correct them.
This became especially timely when OpenAI introduced memory sources for ChatGPT. That update gave people more visibility into the context behind a personalized response, along with controls to delete or correct stale information.
Even after a few interactions, the prototype had collected a mix of facts and guesses. Separating those categories changes the feel of the system: it becomes easier to inspect, challenge, and trust.
Selective memory is partly about better responses, but it is also about showing which pieces of context actually mattered.
After making memory visible and editable, the next question was how much of that memory should be sent with each request.
For this version, I experimented with retrieving only the memories that seem relevant to the current prompt instead of passing the user's entire history every time.
That matters for quality, but it also matters for trust. If an answer feels personal, the interface should make it possible to see which pieces of personal context were actually used.
The prototype uses embedding-based retrieval to find context-relevant memories and inject them into the request. The retrieved memories are shown in the UI and can be removed one at a time.
The prototype improves in two practical ways: the responses use more relevant context, and the system is easier to inspect.
The implementation is fairly standard. The more interesting question is how much of the retrieval pipeline should be visible in the interface.
I added RAG to the LLM prototype. The core implementation is the familiar version: chunk documents, generate embeddings, store vectors, retrieve close matches, and inject those chunks into the LLM context before generating a response.
For this pass, I used gpt-oss, embeddinggemma, and PostgreSQL.
The retrieval itself is not the most interesting part to me. What I care about is making the pipeline visible: generating the prompt embedding, searching memories and reference sources, scoring chunks, and injecting the final context into the LLM.
The UI also shows which chunks matched and how strong each match was. Seeing that context makes the system feel less opaque, because the answer has a visible trail back to the material that shaped it.
Once the system can work with real files, the interaction starts to feel less like prompting a chatbot and more like collaborating with a software environment.
I added an Electron wrapper so the prototype can expose filesystem capabilities to the LLM through the MCP layer.
This changes the interaction in a concrete way. The system is no longer only responding on screen; it can create and manipulate persistent local artifacts.
In one example, I asked the system to create a Hello World ASP page like it was 1996 and save it to a folder. The prototype shows the whole path: the prompt, the available tools, the model's reasoning and tool calls, and the final folder contents card.
Clicking default.asp opens another card with the generated file contents. From there, the file can be opened in the default editor, downloaded, or zipped together with the rest of the folder.
Navigation is still restricted to the agent-files directory, but even with that constraint the prototype is doing work a standard chat interface usually leaves outside the system.
Instead of treating code as something to paste in and out of chat, the prototype can work inside an actual project directory.
I added workspace switching so the prototype can point itself at a real project directory, including this Next.js portfolio site, and work from inside the codebase. It can inspect files, follow the project structure, create features, modify existing code, and coordinate changes across multiple files.
That changes the texture of the interaction. The agent is no longer only describing what I might do next or handing me code to copy. It can act on the same artifacts I am working on, which makes the experience feel much closer to collaborating inside a software environment.
GraphRAG-style relationship extraction turns documents into lightweight knowledge graphs for richer LLM context.
RAG is excellent at retrieving semantically similar chunks of text, but semantic similarity alone isn't always enough. Many questions depend on understanding how people, products, documents, APIs, and other concepts relate to one another.
To address this, I extract entities and relationships from documents to build a lightweight knowledge graph. Those nodes and edges provide additional context that complements vector search, helping the LLM reason over connected concepts instead of isolated text passages.
This work is inspired by Microsoft's 2024 paper, From Local to Global: A Graph RAG Approach to Query-Focused Summarization, which demonstrated how extracting relationships into a graph can improve retrieval and reasoning over large collections of information. My implementation uses those relationships to provide richer context for LLM responses in interactive AI applications.
A playful voice that makes the prototype more fun and interactive.
In this prototype we use the Web Speech API to read model output aloud in real‑time, letting users instantly verify what the AI says and enhancing transparency. As soon as the response appears, SpeechSynthesis starts speaking the content immediately – no button press required.
We select a single voice from the available options—Zarvox—to give the AI a distinctive, robot‑like timbre. The system builds a SpeechSynthesisUtterance, configures it with the chosen voice, and attaches listeners to indicate start, end, or errors so users get visual cues during playback.
The playful voice and immediate feedback make the experience feel more engaging and interactive for users.
A local-first coding assistant where the model's context, actions, and results stay visible and connected to the developer's workspace.
Coding App is a local-first desktop prototype for AI-assisted software development, built with Electron and React. It separates the interface from system-level operations through a secure preload bridge and IPC layer: the Electron main process manages project files, local persistence, model providers, tool execution, and the agent loop, while the React renderer handles the interactive workspace and conversation UI.
Everything stays on the machine. Projects, conversations, and messages are stored locally in a lightweight SQLite database, and runtime settings live on the user's machine. The app supports local MLX and Ollama models through interchangeable provider adapters, so the same agent loop can run against whichever local model fits the hardware.
The agent can inspect files, search code, make targeted edits, inspect Git changes, and search the public web when additional context is needed. Conversations are rendered as structured Markdown chats with visible tool activity, collapsible results, and controls for replaying or editing prompts.
The model doing the work is Qwen 3.8. The Qwen3.8-27B release is a native multimodal dense model tuned for coding, agentic workflows, and office automation on local hardware, and it is the first Qwen-Max-class model released with open weights. That combination is what makes this prototype feel different: a local model that sustains multi-step tool loops, holds a long task together, and can be run, inspected, and fine-tuned on your own machine.
The prototype explores a more inspectable approach to coding assistants—one where the model's context, actions, and results remain visible and connected to the developer's workspace, instead of hidden behind a chat transcript.
Visual understanding lets a coding agent work from the design itself instead of a textual description of it.
Qwen 3.8 is a natively multimodal model, which means it can work with images and text together in the same context. A screenshot of a UI becomes another kind of input the model can reason about alongside code, prompts, and tool results.
I added image understanding to Coding App to take advantage of that. You can now attach a visual reference—a Figma export, a screenshot of a mockup, or a photo of an interface—and the model analyzes the layout, spacing, typography, and UI structure before writing any code.
This fills an important gap in front-end work. Describing a design in words is lossy: someone has to translate the visual into a prompt, and every translation step drops detail. When the agent can see the design itself, it can work from the visual structure and styling instead of reconstructing them from a description, giving it a much richer source of truth for reproducing the interface.
The flow follows the same inspectable pattern as the rest of Coding App: the image is visible as part of the conversation context, the model's observations about the design appear in its response, and the resulting code edits go through the same visible workspace tools as any other change.
Once an agent can use a browser, another AI system becomes a source of feedback it can consult directly—not just a separate tab the human has to manage.
I added an integrated browser to Coding App, along with tools the agent can call to navigate and inspect web pages. The browser lives beside the project workspace, so the agent can move between local files, its own tool results, and the outside world without losing the thread of the task.
That makes it possible for the agent to consult another AI system as part of its workflow. In this example, the local Qwen model reads the existing /hai content, opens ChatGPT in the embedded browser, asks for feedback on the most recent post, and brings the response back into its own context. It can then use that critique to decide what to change.
The interesting shift is that the browser is not only a way to fetch information. It becomes a visible collaboration surface between AI systems: one model can ask another to critique, explain, compare, or pressure-test an idea, while the human can see the request, the navigation, and the returned page in the same interface.
This also extends the inspectability theme of the series. Once the agent can act outside the local workspace, it becomes important to show where it is going, what it is doing there, and what information it brings back. The browser is more useful when it is connected to the workspace, but that connection should remain visible enough for the human to question, interrupt, and control.
The OpenAI-backed version felt fast right away, but the bigger point is architectural: the same loop, the same tools, and the same interaction pattern carried over without reworking the system around a new model.
I added the OpenAI API to Coding App as another model provider. The app was already set up around interchangeable providers, so this was less about inventing a new path through the system and more about plugging a new model into the path that was already there.
What I liked about this change is that the agent loop and tool system did not need to change. Conversation state, tool execution, streaming, and the general orchestration were already in place. The thing that changed was the provider adapter underneath that loop, not the loop itself.
That made the integration feel almost suspiciously clean. I did not have to rewrite the tools for OpenAI. File inspection, search, targeted edits, Git inspection, and browser actions all kept working through the same tool definitions and execution layer because the contract between the model and the app stayed basically the same.
The speed difference was obvious right away. Responses came back quickly, tool decisions happened quickly, and the whole interaction felt more responsive. In an agentic coding workflow that matters because latency stacks across every inspection step, tool call, and follow-up decision.
A big part of why this works is that the tools are sent to the model as structured definitions. The app sends the available tools with the request, and the model determines what to call, when to call it, and whether it needs another tool result before it can answer.
That is the part I find most interesting. Coding App is starting to feel less like a demo built around one model and more like a harness I could adapt to different purposes and connect to a local or cloud-based LLM. Keep the workspace, keep the tools, keep the inspectable loop, and swap the model underneath to see how it behaves.