Human-AI Interaction & AI Product Engineering

After wrapping up my HCI class, I came back to this human-AI prototype and rebuilt the chat loop around local tool calling. The stack is Next.js with Ollama running gpt-oss:20b on an M4 Max.

I am less interested in chat as a transcript and more interested in chat as a coordination surface. In this version, the model can request tools, fetch NBA scores, call again for a box score, and hand structured data back to the UI.

The client renders that response as NBA game cards and box score views instead of leaving everything as text. I also show the intermediate steps, partly because the tool orchestration is the point of the prototype and partly because I want to see where the model is making decisions.

A recurring question for me is how software stays understandable, especially when an LLM is involved. If a system gives a personalized answer, it is reasonable to wonder why it said that and what it thinks it knows about you.

So I made the user context passed along with each request visible, editable, and deletable item by item. The goal is simple: show the facts and guesses that may shape the response, then let the user correct them.

This became especially timely when OpenAI introduced memory sources for ChatGPT. That update gave people more visibility into the context behind a personalized response, along with controls to delete or correct stale information.

Even after a few interactions, the prototype had collected a mix of facts and guesses. Separating those categories changes the feel of the system: it becomes easier to inspect, challenge, and trust.

After making memory visible and editable, the next question was how much of that memory should be sent with each request.

For this version, I experimented with retrieving only the memories that seem relevant to the current prompt instead of passing the user's entire history every time.

That matters for quality, but it also matters for trust. If an answer feels personal, the interface should make it possible to see which pieces of personal context were actually used.

The prototype uses embedding-based retrieval to find context-relevant memories and inject them into the request. The retrieved memories are shown in the UI and can be removed one at a time.

The prototype improves in two practical ways: the responses use more relevant context, and the system is easier to inspect.

I added RAG to the LLM prototype. The core implementation is the familiar version: chunk documents, generate embeddings, store vectors, retrieve close matches, and inject those chunks into the LLM context before generating a response.

For this pass, I used gpt-oss, embeddinggemma, and PostgreSQL.

The retrieval itself is not the most interesting part to me. What I care about is making the pipeline visible: generating the prompt embedding, searching memories and reference sources, scoring chunks, and injecting the final context into the LLM.

The UI also shows which chunks matched and how strong each match was. Seeing that context makes the system feel less opaque, because the answer has a visible trail back to the material that shaped it.

I added an Electron wrapper so the prototype can expose filesystem capabilities to the LLM through the MCP layer.

This changes the interaction in a concrete way. The system is no longer only responding on screen; it can create and manipulate persistent local artifacts.

In one example, I asked the system to create a Hello World ASP page like it was 1996 and save it to a folder. The prototype shows the whole path: the prompt, the available tools, the model's reasoning and tool calls, and the final folder contents card.

Clicking default.asp opens another card with the generated file contents. From there, the file can be opened in the default editor, downloaded, or zipped together with the rest of the folder.

Navigation is still restricted to the agent-files directory, but even with that constraint the prototype is doing work a standard chat interface usually leaves outside the system.

I added workspace switching so the prototype can point itself at a real project directory, including this Next.js portfolio site, and work from inside the codebase. It can inspect files, follow the project structure, create features, modify existing code, and coordinate changes across multiple files.

That changes the texture of the interaction. The agent is no longer only describing what I might do next or handing me code to copy. It can act on the same artifacts I am working on, which makes the experience feel much closer to collaborating inside a software environment.

RAG is excellent at retrieving semantically similar chunks of text, but semantic similarity alone isn't always enough. Many questions depend on understanding how people, products, documents, APIs, and other concepts relate to one another.

To address this, I extract entities and relationships from documents to build a lightweight knowledge graph. Those nodes and edges provide additional context that complements vector search, helping the LLM reason over connected concepts instead of isolated text passages.

This work is inspired by Microsoft's 2024 paper, From Local to Global: A Graph RAG Approach to Query-Focused Summarization, which demonstrated how extracting relationships into a graph can improve retrieval and reasoning over large collections of information. My implementation uses those relationships to provide richer context for LLM responses in interactive AI applications.