📝 Research summary: How product designers respond to AI-powered personas

AI representations of real people seem to be showing small glimmers of promise as a research tool... with some important caveats

This week’s research summary requires a bit of background and framing. If you’re not a tech/product person, what you need to know is this: the people who design your favorite apps often use “user personas” to help them imagine the people they’re building for. Most times, these personas are grounded in research and data about the app’s user base, and they distill that data into archetypes that designers then use to ground their work in empathy and an understanding of their audience. A persona might look something like this, often with a photo attached.

A sample user persona from the paper. Source

Personas are often bullshit. But that’s not what’s interesting right now.

What’s interesting right now is a new paper that explores how using AI personas might affect designers across dimensions like empathy for the persona, understanding the needs of the persona, and their ideation outcomes. The authors turned personas like the one outlined in the image above (minus a photo, to remove potential demographic bias) into interactive characters powered by GPT-3 and had participants conduct design tasks using both the interactive, AI-powered persona and a more traditional, static persona.

The results were mixed. Measures of understanding and empathy weren’t statistically different between the traditional persona use and the AI persona use, but designers did come up with more varied understandings of the persona.

While designers showed comparable diversity in the ideas that emerged from both conditions, we find in the Synthetic User condition a greater variation in how designers perceive the persona’s attributes. We also find that the Synthetic User allows novel interactions such as seeking feedback and testing assumptions.

So, basically, a tossup with maybe a slight advantage given to the AI persona.

Hilariously, the authors point out that part of what held designers back in using the AI-powered personas was treating them as research subjects and employing proper qualitative research techniques to their interrogations of them:

Designers are trained to conduct interviews with users… and to be careful to not influence the interviewee’s mental processes; they are encouraged to employ “neutral prompts” in order to be non-leading and non-directive, such as “Tell me about …” and “How do you feel about …” (Wilson, 2013). This creates a dichotomy in thinking modes– the intuitive, emergent thinking of designers versus the deliberate, structured prompting needed to effectively engage with LLMs (Cross, 2023). Our study results suggest that this clash may limit the effectiveness of Synthetic Users in iterative, constructive dialogues, essential in the design process.

This nugget in itself, while a conjecture of the authors and not a finding per se, is interesting. It suggests that the observed effects may have been greater had the participants been more fluent or advanced in their use of LLM-powered chatbots to begin with. So add that to the standard requirements in your next product designer job description :)

While I don’t plan on delving too deep into meta discussion of user research as a field in this newsletter, I’ll use this paper as an excuse to stray from that rule for a bit, because it explores one expression of a really important and broader idea: the possibility of doing research to understand real people by interviewing artificial ones. This concept has caused a mini-freakout among user researchers in the last couple of years as a startup called, aptly, Synthetic Users popped up on the scene.

Although the idea may seem absurd on the face of it, and while there is certainly no binary yes-or-no answer as to whether it’s a good idea for any and all research questions, there has been some evidence that synthetic users hold promise as a research tool. An amazing paper from last year took a rigorous look at how it might work to a certain degree of effectiveness.

The researchers interviewed 1,000 people about their lives and fed the transcripts of those interviews into an LLM, creating 1,000 synthetic users. Those synthetic users were then given a barrage of standard attitudinal tests like the General Social Survey and the Big Five Personality test. The humans took the tests, too, but twice. This let the authors compare the synthetic users’ ability to predict “accurate” survey responses (which vary even among the same person over time) to the actual humans’ own recall of their own responses. In the General Social Survey, the synthetic users did that with 85% accuracy 🤯

Source

While the study about designers using AI personas is interesting and thought-provoking, the design of this latter study seems to make a more compelling case for how synthetic users may impact social science research of all types in the future. Of course, the questions in the GSS aren’t reflective of every type of research question, and the accuracy of an approach like this will be highly dependent on the context of a particular study and what a researcher hopes to learn. But a wide-ranging, two-hour, in-depth interview like the one powering the synthetic users in this second paper is obviously far more data-rich and therefore more powerful than what’s included in a user persona— a tool that’s reductive by design. These synthetic users almost surely would have fared better in the first study than the ones the researchers used.

The implications here are pretty profound, I think. It could mean that researchers can use one sample of many long interview transcripts to conduct hundreds of studies rather than just one. It could be a massive efficiency, unlocking a new speed of doing social science or user research.

But when the users of AI are researchers or designers, all of the same dimensions of trust, understanding, and influence that affect consumer uses are still at play, of course. Researchers, too, are prone to falling back on wobbly heuristics of trustworthiness, and the ease of doing new research without suffering the pain of recruiting real life participants will be a siren song to many of them. It will be easy for them to jump headfirst into this world and throw caution to the wind. But being vigilant about sample and participant quality will still be critical in a world of doing research on synthetic users. Doing so will just look very, very different.