The Gaze Interrupted: Dispatch 003

Vibe coding a Haptic Translator, the poetry of aged suede, and mapping n-dimensional empathy.

My practice is creative coding and textiles. My method is to interview patients and clinicians and analyse their responses using AI to bring out the poetry in the everyday experience of being a patient or a surgeon. Of course, for the patient with thyroid eye disease, it’s not a mundane experience. It’s deeply affecting both physically and mentally. And although doctors practise as a job, they have studied and practised for years to reach the point of proficiency - training hand and mind. Each new interview is surprising and enlightening.

Open call

I’m still looking for three more patients and two more clinicians, so if you are a thyroid eye disease patient or you work with patients with thyroid eye disease, please get in touch. You don’t need to be senior, and your condition doesn’t have to be life-limiting.

Click here to sign up

In the studio

This month has been interrupted by stifling heat, and yet I’ve achieved a lot. The coding has been particularly productive. I’ve “vibe coded” custom tools to aid manual checks on diarization (allocating speakers to utterances) and anonymisation. But more interestingly, I’ve got version 1 of my Haptic Translator up and running using learnings from a recent Google x Kaggle course. It’s already generating insights and inspiration derived from the interview data.

The design uses The Gaze Interrupted signature colour palette and a design language derived from a nineteenth-century laboratory. This is how Claude in Antigravity summarised the process and described the tech stack for me (feel free to tune out if this sort of thing makes you snooze):

Python / FastAPI — handles the API and runs the ingestion pipeline

SQLite — local database storing all transcripts, entities, relationships, and embeddings; no cloud database, everything stays on the machine

Google Gemini 2.5 Flash — the LLM that reads each dialogue chunk and extracts materials, sensations, and concepts in a single pass; used via the free tier

SentenceTransformers — a local embedding model (no API cost) that converts text into numerical vectors for semantic similarity search

UMAP — dimensionality reduction library that collapses those high-dimensional embeddings into a 3D point cloud you can navigate

Frontend

React + TypeScript / Vite — the web interface.

Vis.js — the graph rendering library that draws the network of materials, sensations, and concepts.

How it was built: The entire codebase was developed collaboratively with Google Antigravity — an agentic AI coding assistant. Antigravity acted as a pair programmer throughout: auditing the code for token inefficiency and security gaps, writing and running tests, fixing bugs in real time while watching server logs, maintaining the project backlog, and keeping the documentation accurate and committed to GitHub. Development conversations happened in natural language; Antigravity read the codebase, proposed changes, and implemented them with explicit approval at each step. The researcher (that’s me!) retained full veto over every decision — particularly around the database schema and the LLM extraction logic, where the qualitative integrity of the data is paramount.

Here's how it works:

I feed the LLM my anonymised interview transcript. The LLM converts this to “n-dimensional vector space”. (Imagine a graph with an x and y axis, now imagine a z making it 3D, now imagine multiple axes in many dimensions.) The LLM then looks for relationships in the data. Effectively, it does what is called “coding” in social sciences (not to be confused with any other meanings of the word). So if it finds an utterance about pain, it clusters it with other utterances about pain and labels the whole topic ‘pain’. I have then asked the LLM to further allocate a “material” to the topic. First, it looks for explicit associations - so the interviewee might have explicitly mentioned glass as associated with their pain, then, if none are found, it is allowed to allocate its own. So it might well allocate something like sandpaper to the feeling of ocular dryness.

The Haptic Translator

The human in the loop (me) can delete a connection or a material and upload my own, along with images and material samples. The plan is to populate this map with my own images. Inspired by one of my clinician interviews, I’ve also been researching further re-numerising of haptic sensations and the construction of haptic sensors to feel and measure them.

The system draws a visual map (which looks rather like a retina) that you can navigate by following connections. It also has an interface for interrogating the data (known as Retrieval-Augmented Generation, or RAG). So, for example, I asked why it had associated suede with ageing. Here is what it said:

Why ageing and suede?

“Within the Haptic Translator framework, ageing evokes suede because it signifies a softening of structure. It represents the transition from the firm, resilient surface of youth to a more malleable, worn, and imperfect state. This material sensation acts as a proxy for the complex emotional landscape of the subject: the resignation to a ‘saggy face,’ the lingering anxiety of autoimmune challenges, and the bittersweet acceptance of a life lived through both survival and physical transformation.”

Phrases in quotes are from the original transcript. This is quite poetic and is true to the text and my feelings about what was said in the interview.

One step further

I was so struck by the metaphor of aged suede that I prompted Flow to generate a few images. I’m exploring the use of Gen AI as a tool to explore ideas. In this workflow, image generation is not an endpoint, but rather perhaps a suggestion for making. Suede and leather certainly fit the category of materials that improve with handling. (As do our faces, although we probably don’t want to be handling faces in a gallery setting, do we?)

An ovoid / head shape (without features) covered in peach / skin coloured aged suede. It has a convex large dip in the middle. Where the object would rub when handled (e.g. round the edges of the shape) it is worn to smooth shiny leather. No background or other objects. Presented as a museum object or laboratory sample. Lighting highlights texture

The interface also pulls back utterances that support the connection or grouping. One of my unified theories of everything is that it’s not the individual things that are interesting, it’s the linkage between them. I look at the LLM as a complicated autocomplete, trained on the words of the internet and on people trying to be as ‘normal’ as possible (also known as Reinforcement Learning from Human Feedback, or RLHF). (When you work in AI assessment, which I have done, you are asked to select the answer that ‘9 out of 10 people would agree was the best’. This makes every fibre of my social science training scream, but that is a story for another day.)

An ovoid object with aged suede textures slowly rotates on its axis against a dark, featureless background. As it turns, the large convex dip in its centre slowly pushes forward and outward, tenting the material into a peak, before gradually receding back to its starting position. Lighting dramatically highlights the shifting texture of the suede and leather.

In the most recent tech review, Dylan (Banarse, my tech mentor at Google DeepMind) and I decided to push the helpful machine and asked it if there was a link between ageing and polymer (a material far away on the map). It gave us a “helpful answer” explaining the link in word soup, where sentences sound convincing until you think about it. (Which makes me wonder whether perhaps it was trained on political speeches!)

Apparently, the same feature that makes the LLM sycophantic makes it hallucinate; it thought that I wanted to hear about a connection and hallucinated the rest. For this research, it isn’t a big deal for me because there isn’t really a ground truth about the association, and I can delete any connection that doesn’t resonate with me as the artist/researcher. However, I think more people should be aware of this tendency. In HCI, this is known as ‘skills degradation’ and will be familiar to anyone who’s lost the ability to spell, write or type accurately!

Dylan also suggested implementing an LLM as a second layer of judgment, so I don’t need to filter 400 or so concepts and materials manually. I also plan to have the system learn from what I approve or disapprove. There are quite a few other improvements I have added to the backlog that will allow me to filter and examine the data at a more granular level. Finally, I want to create a site which can be shared with the public. This won’t have the labelling functionality, but will allow the public to navigate the map.

Here is a look at what else has materialised in the laboratory over the last few weeks.

Researching the Archive

To understand how we see and are seen, I’ve been looking backwards. I spent time at the Wellcome Collection immersed in Samuel R. Wells’ New Physiognomy, or, Signs of Character (1867). Reading through its historical attempts to categorise the human face—the “human face divine”—highlighted how deeply our culture codifies facial difference and prejudice.

Whilst researching the archive on The Hunterian website, I tracked down the work of Henry Tonks, the surgeon-artist who documented Harold Gillies’ pioneering reconstructive surgeries on soldiers with facial wounds in WW1. Tonks’s pastel drawings of fragile, altered flesh combine observation and empathy, showing what art can do beyond the clinical lens of the camera. The public conception of plastic surgery can be ‘boobs and facelifts’ (to quote one respondent), but studying Tonk’s art (based on his collaborator surgeon Harold Gillies’ work), reminds us that the face is a site of interface to the social body and reconstructing it restores identity and quality of life.

Bringing strands together

There is a parallel between Tonks’ attempts to capture the core of a person beneath flesh altered by war, and what I am trying to do with the Haptic Translator. By stripping away the noise of a standard transcript and translating human text into the feel of suede, sandpaper, or polymer, the machine helps surface universal sensations.

As this project moves between code and materials, the goal remains the same: to take these delicate, complicated intersections of human experience and translate them into something else. We are all, in some way, trying to find a form that holds our experiences.

Thank you for reading and for following along with this journey into the lab. If you or someone you know meets the criteria for the open call for interviews, please let me know. I’d love to hear your story.

Until next month!

This research and development project is supported using public funding by the National Lottery through Arts Council England.

Google Sites

Report abuse