Notes on the iconography of algorithmic fog (working paper)
With AI-generated images, can you develop something like an iconography of themes that have no iconography? For example, I gave Midjourney as a prompt a varied version of the allegory of cosmography from Cesare Ripa.
Aside from the supernumerary fingers, there is an interesting detail in one of the images — that is, what I called the ‘celestial sphere’ is depicted as a kind of globe traversed by fluid currents — which looks like a combination of a marble object and a gaseous sphere, like Jupiter.
How do I define this object Iconographically? is certainly a fusion of similar images, but, as in the beings and writing of Serafini’s Codex, it is also a more or less creative combination of different elements.
I will say more: the language invented by Serafini traces a kind of writing, but it is apparently asignificant, just as these images generated, resemble iconographic themes, but they are not.
Since they generate images from a description, and from further images containing metadata, the question of their ‘textual sources’ is similar to the iconographic one. Only, within the description is not only my ‘celestial sphere,’ but also all the descriptions and images it used. This can for all intents and purposes generate insoluble visual puzzles, if you will, that is, symbols that refer to nebulae of content that are virtually invisible to the person who entered the prompt, unless he or she has control over the images provided.
Is it therefore possible to think of an iconography of themes or symbols with no referent, or uncertain referent?
I ran a test with DAll-E 2 to see what resulted. The theme is imaginary herbaria. I told him to merge the drawing of a papaver somniferum with that of a platypus in an Arabic manuscript.
Comparing this image with the Voynich manuscript (which I believe has been deciphered) and with Serafini’s phytomorph chimeras one can make a series of observations.
Gregorio Magini proposed the category of ‘fake symbol’ to define objects that seem symbolic but are not. A similar definition comes from Sperber’s cognitive anthropology. Now, if I think of the definition of symbol given in interpretive semiotics by Eco, it is, in Hjemlslevian terms, the expression of a nebula of content. Nebula, or imperfection (Greimas) in this sense is the average of the traits that the algorithm proposes from the prompt and the images in the database. The writing is unintelligible, and the style is copied from other manuscripts. This image (1) does not mean much, while the second, taken from the Voynich did not mean much before decipherment, the third (Seraphim) is a kind of surrealist or Magrittian exercise applied to the style of encyclopedias. What can be said about the iconography of 1? That it means nothing, simply on an iconic level it resembles things we have already seen. However, this does not prevent us from going to an iconographic level, because we always try to interpret semantically what we see, and so, like in front of a cryptic manuscript, I try to understand what it means. Only in this case, what it means is self-evident: it is an average of similar features. Here we come back to the evocative aspect: by creating an average of traits, the AI-generated image confuses the contents (or referents), creating precisely that fog effect that Eco mentioned about Nerval’s Aurelia. It is a symbol because it evokes more than one content, according to ‘family resemblances,’ but is not referable to any specific image.
The idea that the form of AI-generated writing is similar to that of dreams is interesting (Francesco D’Isa). Indeed, when we sleep, some areas of the brain, which are deputed to computation, language processing and rational thinking in general, are narcotized. Other areas, such as those related to emotion and fast reasoning, on the other hand, are more active. This generates so-called dream hyperconnectivity: a condition in which ideas are associated and confused rhapsodically. For this reason, calculations and writing, in dreams, are vague. Similarly, DALL E 2 does not recognize writing as an OCR algorithm or as GPT, but interprets it as images, and in particular, breaks down the source image into a series of features.
The image I have provided comes from the page of a memory art, in which alphabetical letters are associated with similar figures. When DALL E takes my image as his cue, it interprets the letters as iconic content, and reproduces other letters, or mixes them. This is not very far from the way we go from letters to images in the arts of memory-that is, through an iconic connection that prescinds from linguistic content
The idea that writing is not processed as such, but as an image, leads us to a further consideration, namely the fact that the ‘dream-like’ process of producing new images can be interpreted according to the categories of analytic iconography (Didi-Huberman, Arasse), and in particular with respect to the link between painting and the arts of memory. One particular case, Magritte’s La clef des sognes, shows how the association between text and image can be arbitrary. The idea is to mess up the cards of a spelling book-the result is a kind of ineffective memory art because there is no iconic relationship between words and images.
In the case of the images generated by DALL-E 2 from Magritte, however, we see a process of iconic displacement and association taking place — the candle is replaced by a lamp and the glass by a bottle. Now, this process does not differ much from what happens in dreams, surrealist painting and the arts of memory — it is an association between similar iconic features: the writing disappears while the images are associated by family resemblance.
All this brings us back to the original question: What is an empty symbol? In general, it is possible to say that every symbol, from the perspective of cognitive anthropology, is empty. Certainly there are esoteric and esoteric levels-often symbols imported into ritological and mythological life are interpreted in all hermeneutic ramifications only by a priestly caste.
And here we come back to the question: if a symbol is interesting, it is not because there is something like an archetype that it embodies, but because, seeing it, we are driven to make associations.
The way DALL E proceeds is thus based on the way we draw iconic associations. We can therefore provide a starting image — drawn from Luca Pacioli’s Divina Proportione — depicts a solid with a ribbon that binds the figure to its name. In Pythagorean and Platonic philosophy, geometric shapes represent reality. In practice, training in vision and theory serves to show, beyond the blanket of phenomena, a series of mathematical structures.
What if these structures-abstract shapes-are nothing more than the lowest common denominator of a great many sensory inputs? Image recognition algorithms often get fooled by abstract shapes that they connect to objects. Between objects and shapes there seem to be no iconic connections.
Even our alphabet starts from images with a form that can be traced back to a natural referent and then becomes pure abstract schematizations that are connected according to syntactic and grammatical rules.
It becomes clear that, once abstracted, these symbols no longer function as semiotic references to the natural figure, but are simple combinatorial units. Such a process of abstraction had been hypothesized by Leibniz in his writings on combinatorics and linguistics — starting from complex forms such as chemical elements he had come to the conclusion that the most effective system for conveying information was the use of a purely syntactic mechanism — such as the binary code.
In short, AIs also dream, abstract, or more appropriately they delude us, through the generation of empty symbols, variations from millions of similar instances, creating a kind of family air, but also a perturbing feeling, like something that looks like but is not what it is supposed to represent.
A nebula of content is a kind of smokescreen that obscures and mediates the features of similarity and dissimilarity that the algorithm used to create some ‘new’. And so, although the geometric figures generated by DALL-E 2 from the Proportione divina are ramshackle and incorrect, they represent its way of generating variations and counterfeits-which we interpret as if there is something deeper than what they are.
A final consideration and direct comparison with Freud’s page. In The Interpretation of Dreams one finds comic plates depicting dream scenes in relation to physiological stimuli that caused them. As can be seen, a boat becomes a gondola and then an ocean liner.
In a not dissimilar way by trying to give DALL-E 2 the image of an etching by Tiepolo, taken from the Capricci (a work that in itself already has a vague and dreamlike content, as Calasso shows in Tiepolo Pink , i.e., symbols and allegories taken from emblematic visual culture remixed), one can see how by moving the generating process forward one immediately has a change in style, from the sharp line of the etching to a kind of charcoal that is reminiscent of Nolde’s woodcuts or some of Hugo’s depictions.
Again, once the first image is reinterpreted the subjects change and merge. It goes from a tomb pyramid to a statue on a pedestal, then changes again to a pair of figures-a woman and a jumble of shadows, and finally becomes a scene with a woman sitting in front of some kind of bear-shaped animal.
AI transformed the images metaphorically, taking cues from the iconic resemblance, but also from the way it interpreted the iconographic components (the subjects) and how he decided to change a more precise style (etching) into a style much more akin to Expressionism.
