Academic Jobs - Home of Higher Ed Logo

Research Spotlight: Kai Xu on PrompTHis and the Prompt History AI Artists Lose

10 min read0 Opinião

At a glance

A new system called PrompTHis, co-authored by University of Nottingham associate professor Kai Xu, turns an artist's prompt history into a navigable graph showing which edits moved an AI-generated image. The study reveals both how much people can read from that map and where they still struggle.

A person in silhouette against a glowing grid display reading "Progress is never done."
Photo by Michael Shtern on Unsplash

Eleven people sat down with an AI image generator and were then asked to explain what they had just done. They could name the distinct visual themes they had explored 82.21% of the time, and they could describe how one cluster of generated images differed from another 96.92% of the time. When the question narrowed to which particular word in a prompt drove a particular change, accuracy fell to 78.12%.

Those figures come from the evaluation of a system called PrompTHis, built by researchers at Peking University and the University of Nottingham. The paper, by Yuhan Guo, Hanning Shao, Can Liu, Kai Xu and Xiaoru Yuan, appears in IEEE Transactions on Visualization and Computer Graphics and was presented at IEEE VIS 2024. Its premise is that the most useful artifact in generative art isn't the final image. It's the record of what the artist tried, changed and threw away.

That record usually evaporates. Most text-to-image tools keep the last prompt that worked and little else, so the deleted words, reweighted phrases, abandoned branches and half-finished side experiments behind a finished picture are gone. The artist is left guessing at their own process, and often re-running an experiment they already ran two hours earlier.

The trail of edits that nobody keeps

Kai Xu is an associate professor in the School of Computer Science at the University of Nottingham and co-director of the school's visualization research group. He took his undergraduate degree in computer engineering at Shanghai Jiaotong University and a PhD in computer science at the University of Queensland, and much of his career has gone into data-driven computer graphics and 3D vision. PrompTHis is a turn toward the human side of the generative boom. "Following this idea, we created the PrompTHis to help artists better understand the capability of generative AI models and how to control them," he wrote on his research blog when the work was published.

The design started with two working artists, not with a dataset. One built images through Disco Diffusion in a Google Colab notebook; the other used Stable Diffusion and Midjourney. The researchers watched them work and interviewed them, then coded the transcripts thematically. What emerged was a workflow with no clean ending: artists often begin without a fixed idea and adjust as images arrive, and the loop stops when they're satisfied, when they no longer know how to improve the prompt, or when time runs out. The authors note that the last two happen far more often than the first.

The misalignment between what an artist means and what the model renders is where the anecdotes get specific. One artist's prompt contained the words "head" and "shoulder" and the model produced a bottle of shampoo. Another ended up with robotic animals nobody asked for, liked them, and folded the term into the next prompt. Both directions matter: the failures are as informative as the lucky accidents, and neither survives in a tool that only stores the final text.

A third finding pushed the project's scope. Artists wanted the model to make suggestions tuned to their own taste, covering everything from automatic parameter exploration to judging an output against their preferences. The authors decided that sort of personalization requires knowing how a person actually explores, so they started with the exploration record first and left recommendations for later.

A family tree where every branch carries its own edit

The system's centrepiece is the Image Variant Graph. Each generated image becomes a node. Each line between two nodes represents a difference between their prompts, so the graph is built from edits rather than from images alone. Edits are colour-coded: blue for an added word or an increased weight, red for a removal or a reduced weight, green for a reordering. Every line carries a short annotation naming the change, something like removing "1boy" and adding "1girl". Distance on the canvas reflects how far apart the images sit, and clusters of similar outputs collect into bubbles.

The easiest way to picture it is a family tree for images in which every line is labelled with the exact edit that produced the child, and the length of the line tells you how far the child moved from its parent. Some edits barely nudge the picture; others send it somewhere else entirely, and the graph makes that difference visible at a glance.

Getting there takes two passes. On the text side, the system computes Jaccard similarity between every pair of prompts to build a distance matrix, and only compares pairs that are close enough, with a default threshold of 0.6 that the user can loosen or tighten. Word alignment uses the Myers algorithm, which identifies insertions and deletions, treats a deletion followed by an insertion as a reorder, and also catches the numeric weights that diffusion models let users attach to individual terms.

On the image side, each picture is passed through both the text and image encoders of CLIP, the contrastive model that maps text and images into a shared space, producing 512-dimensional vectors. Those vectors are squashed to two dimensions with t-SNE using cosine distance, which yields two separate embeddings, one shaped by text and one by pixels. Procrustes analysis aligns the two spaces, and by default they're averaged, with a slider letting users shift the balance. Images are then grouped by hierarchical agglomerative clustering, and the connecting lines are bundled by cluster before their weights are redistributed to reflect how much each edit actually moved the output.

Around the graph sit three supporting panels: a creation panel for entering prompts and opening sessions, a history box that shows each prompt with its changes highlighted against the previous attempt, and a navigation mini-map that summarises the whole session. A full description of the method and the study is available in the preprint on arXiv.

Glitching document with blue and yellow text

Photo by Egor Komarov on Unsplash

What six interviewees did with the graph

The qualitative work involved six people. Four were university professors who teach and practise visual art, each with more than twenty years of professional experience, and two of them had taken part in the original design interviews. The fifth was a designer with seven years in interaction design. The sixth was a computer science postgraduate with no art training. Each session began with fifteen minutes of training, included a short creative run on a stripped-back version of the tool containing only the history box and mini-map, then roughly twenty minutes of iterative image making, ten minutes of recounting the session, and a semi-structured interview.

Reviewing came first. Everyone leaned on the history box to find earlier images worth refining. The graph entered later, as an overview. "The graph enables me to ignore the prompts and look at the images on their own merits," one participant said. Another pointed at the bubbles that gather images made from an identical prompt: "The images generated by one prompt can be quite different. I believe the visualization helps to identify more stable prompts."

The most concrete result involved a phrase collision. One participant had spent a baseline session mixing stylistic and descriptive terms and could not work out any rule behind the outputs. Reconstructing the session on the graph, he saw that "light red" was pulling against "Chinese painting", introducing modern elements into pictures meant to stay traditional. In a later session he got stuck, traced the run of disappointing images back to a single insertion, the phrase "album cover", and removed it. The outputs improved.

Memory turned out to be part of the payload. Another participant had once tried an apple in a painter's style and disliked the result. Reviewing a different session, a cartoon pig where less familiar style references had been swapped for "Monet", he recalled that the model handled Monet well and applied the lesson to the apple. A third participant described how the graph changed the shape of exploration: "When creating with the baseline system, I often focused on the most recent steps and was unwilling to branch out." With the graph in view, he said, "I could imagine where the desired results are in the embedding space and fine-tune prompts accordingly."

The separate quantitative study with eleven amateur participants tested whether people could read the visualisation correctly rather than whether they liked it. Identifying the themes of a session and the differences between clusters proved straightforward. Attributing influence to individual words did not, and the authors explain why: participants tended to fix on the most obvious variation in an image and miss the same word's effect in a different context.

The honest limits

That 78.12% is the number to hold onto. A word in a prompt does not act in isolation, and a graph that summarises an edit with one label can only ever tell part of the story. The authors also acknowledge that the system was not designed for live use: the current version mainly supports reviewing and recounting a session after the fact, not steering one in real time.

Sample size is the other constraint. Eleven people in the quantitative study and six in the interviews is normal for visualization research, where deep observation of a few expert users carries more weight than a broad survey. It still means the findings describe what a small group did in a controlled, remote setting, not what a studio of forty artists would do over a month. Two of the six interviewees had helped shape the design, which makes their enthusiasm worth reading with some care.

The authors describe PrompTHis as "an initial step towards understanding and utilizing individual creation history". Their own list of what comes next is unglamorous and specific: tighten the graph's layout and encoding so it can support people mid-session rather than afterwards; capture provenance completely, including the context that prompts and images alone leave out; and extend the system beyond the professional artists it was built for. The artists, for their part, asked for features that don't exist yet. They wanted to rate and pin images, annotate attempts, arrange a session along a story line when the goal is a film, and use their own sketches as seeds for generation.

Why researchers outside visualization are paying attention

The paper's framing reaches past art. The authors treat a prompt history as a case of analytic provenance, a term from visual analytics for the full trail of an analysis: the inputs, the interactions, the operations performed, the intermediate and final results, and the reasoning behind each decision. Provenance is what makes an analysis reproducible and auditable, and generative models have made it scarce. A published image rarely arrives with the sequence of edits that produced it.

That gap has consequences beyond studios. Anyone teaching with these tools has to reconstruct process from memory. Journals and galleries fielding AI-assisted work have few ways to verify how a picture was made. And the record of an artist's exploration is exactly the kind of data that could train better assistance, which is where the authors' longer ambition sits. They argue the system can also serve researchers who want to understand how artists actually interact with these models, rather than how vendors say they do.

The published version of the work is available through the paper's DOI at IEEE. The research was supported by the National Natural Science Foundation of China under grant 62272012, with additional support acknowledged from the Wuhan East Lake High-Tech Development Zone national experimental base for governance of intelligent society. Some of the prompts and images used in the study were supplied by Richard Ramchurn, and the authors thank Rebecca Ruige Xu and the other interview participants.

What happens next is a harder problem than plotting history. The authors' long-term goal is to infer creative intention itself, which can only be guessed at from the prompts and images a person leaves behind. If that inference works, even narrowly in text-to-image generation, it would unlock recommendation and adaptive behaviour that respond to a specific artist's taste instead of a generic average. Suggestion engines that currently ask what you want would instead read what you have been circling for the past hour.

The practical version is already visible. A tool that shows an artist which word broke a promising image is useful on a Tuesday afternoon. A decade of such records, kept across projects and studios, is something else: a searchable account of how a visual idea was formed, by whom, and with what model at what version. That account doesn't exist yet. PrompTHis is a first attempt at keeping it.

  • PrompTHis
  • Kai Xu
  • text-to-image generation
  • information visualization
  • prompt engineering
Retrato do Jarrod Kanizay
Sobre o autor

Jarrod KanizayVeja o autor

Academic Jobs In House Author

Visionary leader transforming academic recruitment with 20+ years in higher education.

Discussão

De sorte em:

Seja o primeiro a comentar este artigo!

Você

Você será solicitado a entrar antes que seu comentário seja postado.

Browse por Faculdade

Browse por assunto

Frequently Asked Questions

🎨What is PrompTHis?

PrompTHis is a visual analysis system for reviewing and understanding the history of prompts used to generate images with models such as Stable Diffusion, DALL-E and Midjourney. It was developed by Yuhan Guo, Hanning Shao, Can Liu, Kai Xu and Xiaoru Yuan at Peking University and the University of Nottingham, and published in IEEE Transactions on Visualization and Computer Graphics. Instead of storing only the final prompt, it records every attempt and shows how each text edit changed the resulting picture.

📊What is the Image Variant Graph?

It is the paper's core contribution: a representation in which each generated image is a node, and each connecting line stands for a difference between two prompts. Lines are coloured by the kind of edit — blue for an added word or increased weight, red for a removal or reduced weight, green for reordering — and annotated with the specific change. Because the layout places similar images near each other, a long line signals an edit that had a large effect and a short one signals an edit that barely mattered.

⚙️How does the system work out which words mattered?

Text is compared prompt by prompt using Jaccard similarity, with a default threshold of 0.6 so that only reasonably related prompts are compared. The Myers algorithm aligns words between two prompts to catch insertions and deletions, and treats a removal followed by an addition as a reorder. Numeric weights attached to words are parsed too. Edges are then weighted so that the modifications best matching the shared differences between two clusters of images receive more of the credit for the visual change.

🖼️How were the images grouped into clusters?

Each image passes through the text encoder and the image encoder of CLIP, producing two 512-dimensional vectors. Both are reduced to two dimensions using t-SNE with cosine distance, giving one embedding shaped by language and one shaped by pixels. Procrustes analysis aligns the spaces and they are averaged by default, with a user-controlled slider to change the balance. Hierarchical agglomerative clustering then groups the images into the bubbles shown on the graph.

🧪How was the system evaluated?

Two ways. A quantitative study with eleven amateur participants tested whether people could read the visualisation correctly, measuring accuracy at identifying image themes, distinguishing between clusters and attributing changes to individual words. A qualitative study interviewed six users: four university professors with more than twenty years of art practice, a designer with seven years in interaction design, and a computer science postgraduate. Each interview included a baseline session on a reduced version of the tool for comparison.

🔍What did the user study find?

Participants identified image themes with 82.21% accuracy and correctly described differences between clusters 96.92% of the time. Accuracy dropped to 78.12% when they were asked to identify the influence of particular words, because people tended to focus on the most obvious visual variation and miss a word's effect in other contexts. Qualitatively, users reported that the graph helped them review sessions, compare prompt-image pairs, reason about model behaviour and plan further exploration.

⚖️What limitations do the authors acknowledge?

PrompTHis is currently built for reviewing and recounting a session rather than supporting live generation, and the graph's readability needs work before it can help in real time. The study samples are small, as is typical in visualization research. The authors also note that complete provenance capture, support for a wider range of users and testing in more realistic settings remain open, and they describe the work as an initial step rather than a finished product.

📝What did the participating artists ask for that isn't in the system?

They wanted to rate and pin generated images, annotate attempts, and arrange a session along a story line when the goal is a film rather than a single picture. Some asked for more textual information on screen, such as the phrases shared across a group of images. Others wanted to use their own sketches or artwork as seeds for generation, a capability the paper's authors list as part of fuller provenance capture.

📚Why does prompt history matter outside art studios?

The authors frame prompt history as a form of analytic provenance: the full record of inputs, operations, intermediate results and reasoning behind an analysis. That record is what makes work reproducible and auditable. As AI-generated images spread through teaching, publishing and exhibition, the ability to reconstruct how a picture was made matters to instructors, editors and anyone assessing a submission. Prompt logs are becoming institutional documents rather than private clutter.

💡What is the long-term goal of the research?

The authors want to infer an artist's creative intention from the record of their exploration. Intention is far harder to capture than prompts or images and has to be deduced from them, but if that inference works even narrowly in text-to-image generation, it would enable personalised recommendations and adaptive systems that respond to an individual's taste rather than an average. The current system collects and visualises the record; the reasoning layer is future work.

🎯Who funded the research, and where was it published?

The work was supported by the National Natural Science Foundation of China under grant 62272012, with additional support acknowledged from the Wuhan East Lake High-Tech Development Zone national experimental base for governance of intelligent society. It appeared in IEEE Transactions on Visualization and Computer Graphics, volume 31, issue 9, pages 4547 to 4559, and was presented at IEEE VIS 2024. The full text is available through the DOI record.

🧭Who is Kai Xu?

Kai Xu is an associate professor in the School of Computer Science at the University of Nottingham in the United Kingdom, where he co-directs the school's visualization research group. He received a BEng in computer engineering from Shanghai Jiaotong University in 1999 and a PhD in computer science from the University of Queensland in 2004. His research spans data-driven computer graphics, 3D vision and robotic perception, and he has served on the editorial boards of several journals in graphics and visualization.