The Noematograph and Interpreting Complex Ideas

AI generated subjective interpretations, art and complex ideas

Maybe a noematograph?

In an essay on AI as a unique artistic medium, Ken Liu describes his experience interrogating the meaning behind a poem with the help of ChatGPT:

What’s interesting here is that I was playing, alone, with a Chinese poem that had relatively few human-composed English translations. Nonetheless, ChatGPT conjured up a bunch of translations done by excavated/recombined/synthesized subjectivities, a (ghostly) sense of “someone stands behind these thoughts.” The question that matters here isn’t “Which translation is most accurate?” but “Taken together, do these translations make the poem more interesting?” And the answer to that is a resounding yes.

The nature of translation is that it is a lossy process dependent on subjective choices about what information to include and exclude. There are no right choices, only choices that are more appropriate for a given context. The Chinese phrase 不好意思 could translate to both "Excuse me" and "Sorry" and the choice of one or the other is a choice that acknowledges the context that the speaker is in. The former makes more sense on a crowded subway and the latter in a work email.

So, it's not surprising that given a fixed prompt, ChatGPT will inevitably give a translation for a poem that feels uninteresting to a reader that understands both languages. The reader has their own context that influences how they would translate the poem and as long as ChatGPT's output is different it will feel unsatisfying. If they are in an existentialist mood, the translation will lack a sense of isolation and vulnerability. If they have an anti-authoritarian mindset, the translation won't feel critical enough of leaders.

This is not an AI specific problem. The same problem exists for human translators too! But, what is unique about AI translation is that if you want to see the poem from a different angle you are just a prompt away. If you want a stoic reflection lens on the poem you can prompt for it. A translation from the point of view of a contemporary Chinese reader? Just specify that in the prompt. The only limit is the number of interpretations you are wondering about.

This ability to take a static piece of content and generate an unlimited number of subjective interpretations of it with distinct points of view leads Liu to describe AI as a new type of machine:

The implied translators behind these divergent translations restored the kind of textural richness inherent in the original composition, which is lost in any single translation. I conversed with the subjectivities behind the words, probed them, interrogated them, and found myself with a new understanding: AI is a machine that captures and projects subjectivities, a “noematograph.”

After reading the essay, it struck me that while the noematograph was presented as an artistic medium, it more generally can be thought of as a tool to interrogate any complex idea that is too high dimensional to flatten into a single quick summary. Art happens to be a domain with many ideas that can't be encapsulated in simple verbal language and can only be expressed indirectly in aesthetic forms. In other domains, it might just be impractical to exhaustively express every facet of an idea in a way that is relatable to each unique individual.

But, what if you could interrogate a list of policy proposals for a political candidate by prompting an AI to interpret them from the point of view of personas with different political affiliations and socioeconomic backgrounds?

Or, what if you could interrogate an account of a historical event by prompting an AI to interpret it from the point of view of a historian, economist, politician, etc.?

Or, what if you could interrogate a textbook description of quantum phenomena by prompting an AI to interpret it as a dialogue between two characters or an interactive visualization?

In all cases, I see the value of using AI as a noematograph not to generate a single authoritative output, but to generate many outputs that encompass a diverse set of interpretations to get a richer, more complete understanding of a complex idea.