Podcast episode 54: Neil Cohn on Speaking in Pictures

In this interview, Neil Cohn tells us about his theory that visual art and spoken language draw on the same underlying cognitive abilities.

Download | Spotify | Apple Podcasts | YouTube

References for Episode 54

Cohn, Neil, and Joost Schilperoord. 2025. A Multimodal Language Faculty: A cognitive framework for communication. London: Bloomsbury Academic.

Cohn, Neil. 2026. Speaking in Pictures: A vision of language. London: Bloomsbury Academic. https://www.bloomsbury.com/us/speaking-in-pictures-9781350402164/

Lerdahl, Fred, and Ray S. Jackendoff. 1985. A Generative Theory of Tonal Music. Cambridge, Mass.: MIT Publishing.

Transcript by Luca Dinu

JMc: Hi, I’m James McElvenny, and you’re listening to the History and Philosophy of the Language Sciences podcast, online at hiphilangsci.net. [00:09] There you can find links and references to all the literature we discuss. [00:13] Today we’re talking to Neil Cohn, who’s a Professor of Cognitive Science at the University of Tilburg in the Netherlands. [00:20] Neil’s developed a fascinating theory that visual art draws on the same cognitive faculties as human language. [00:28] He’s set out this theory in the book A Multimodal Language Faculty, co-authored with Joost Schilperood, published in 2024 with Bloomsbury Academic. [00:38] He’s going to present these same ideas in graphic novel form in the book Speaking in Pictures: A Vision of Language, which will appear in February 2026, also with Bloomsbury Academic. [00:50] So, Neil, could you outline your theory for us? [00:53] What are the links between visual art and language? [00:57]

NC: Well, first, thank you for having me. [00:59] I’m a big fan of the podcast, so it’s a pleasure to be on. [01:02] The way I often like to start is by kind of at a basis of human communication cognition, which is that human beings as a species have only three ways in which we can express our concepts: [01:14] We can create sounds (which is what I’m doing right now), we can move our bodies (which is also what I’m doing right now, but people won’t be able to see that), and we can draw things. [01:26] And that’s really all humans do in terms of our ability to express concepts overtly. [01:33] We don’t emit smells to each other. [01:35] We don’t secrete things and lick each other to communicate. [01:40] We don’t emit, say, light pulses or something. [01:43] We do these three things. [01:45] Given that, these three communicative faculties, why would we believe that they should be different from each other is, I guess, the basic foundational idea. [01:57] So given that they come from the same brain, why would we assume that they have different cognitive building blocks? [02:03] Certainly in the last, well, over a century, people have been interested in how the body conveys meaning. [02:11] As you know, I’m sure, people in the 1800s debated whether or not to actually include sign languages as part of the definitions of language and ultimately said, “Now we’re going to ignore those for now,” and then, of course, there was a resurgence in the structure of studying signed languages, to which we now believe that they share the same structural properties as spoken languages. [02:33] And I would argue that we should be adding one more modality to that, which is graphic representations. [02:39] Given that it comes from the same brain, they should have the same cognitive building blocks and structural properties. [02:46] So this follows from what I call generally the principle of equivalence, in a very lofty-sounding thing, which is that given that we have one brain, we should expect the mind and brain to treat all modalities in a similar way, given modality-specific constraints. [03:03] So the places where they should be different is motivated by the nature of the type of modality that they’re addressing, because sound and light are different things, and so we should imagine that that will yield different types of structure down the road, but it’s all still coming from the same brain. [03:20] So this is the path that I pursued. [03:22] I think to some degree, when people hear what I’m working on, they think I’m making an analogy, which is to say, graphics or pictures or drawing is like language, and that language is like a source domain for the metaphor of this mapping. [03:43] And that’s, I believe, you know, I’m not the first person who’s made these sorts of comparisons. [03:47] I believe that sort of analogic mapping is what people have done throughout the, say, 20th century in movements of structural semiotics, applying, you know, principles of linguistic structuralism to graphics. [04:02] You know, they would be searching for the whatever minimal units of emes there are, graphemes, logemes, whatever it is, and it would largely be taken as kind of the linguistic structuralism metaphor of mapping, “OK, here’s what’s happening in what we know as language to these other domains.” [04:21] And that’s not ultimately my argument, is, my argument is kind of to say that the human brain — and/or mind, however you want to think about the granularity — uses similar principles and structures and mechanisms that operate across modalities, no matter what those are. [04:43] So that puts spoken languages and, say, drawn visual languages on equal footing as both involving something more general, rather than it being an analogic comparison. [04:57] And to that extent, I actually don’t think that they are in competition with each other in any sort of sense, because ultimately, those three modalities that we have are complementary parts of one larger whole system, and ultimately, the language faculty or communicative faculty, or however you want to describe it, is multimodal across these three parts, [05:23] out of which there are emergent properties of different behaviors that share the same building blocks, but it’s ultimately a multimodal system, not a unimodal or amodal system. [05:34]

JMc: OK, and why do you think that there are these three specific modalities? [05:39] I mean, what about the case of, you know, like a famous case like, you know, Helen Keller with, who doesn’t have physical access to any of those modalities? [05:47]

NC: Sure. So I would first say that I think modality, the notion of a modality is important here, and the way we define the notion of a modality is, it’s an interaction between a sensory system and a cognitive correlate, abstract representation. [06:02] So in the canonical case of speech, we have sound is the sensory system, which we then have dedicated receptors to receiving through our ears and crucially producing through our mouths. [06:17] We then, in addition to that, have a corresponding cognitive correlate of a phonological system, which is… abstracts across the tokens of the individual sounds, right? [06:31] And this interplay is crucial, which is to say that a modality is not merely a sensory system—it’s an interaction between sensory systems that involve both input and output and abstractable cognitive correlates. [06:48] So in the case of, say, the body, that would be to say we have sensory systems of touch and vision involved in receiving those signals and also a corresponding, let’s call it phonology for the sake of ease for right now, and that abstracts across the individual tokens of those expressions. [07:11] And again, we’re not even talking about anything about meaning right now. [07:13] We’re just talking about individual expressions. [07:16] I should… And correspondingly, I would say that the graphics are the same. [07:19] So you have an experience in light that is then abstractable into, you know, basic, I don’t know if we’d want to call it phonological anymore, graphological cognitive correlates. [07:34] Now, part of that is also that all of our modalities are pretty, are multisensory, so none of these modalities are guided by the singular sensory system alone, which is why the McGurk effect exists, which is where you might be producing sounds with the mouth, but then seeing different articulations than the actual sounds, and you, your brain instead perceives it as being produced a different sound than what you’re actually hearing because of what you see. [08:10] So speech alone is not unisensory; it’s multisensory, and vision is actually involved in all of them. [08:19] So that makes sensory systems and vision in particular a poor characterization of modalities. [08:27] The production system is much more, a much stronger characterization of the nature of these things. [08:35] So you don’t produce speech sounds through anything but your mouth, right? [08:40] Our bodies move around to… with different articulation points to be expressive, primarily hands and faces. [08:47] You can also use body orientation and things like that as well, and graphic systems use various types of traces left in various types of materials to correspond to meaning, so it’s not just a sensory system. [09:03] And on the case of, say, Helen Keller, what you’re left with is a, the absence of receptive systems — that is, through the eyes or the ear —, and so you fall back on a different type of sensory aspect of the bodily production system, which is less used, but at least provides a point of access for that production system of the body. [09:30] If you look at children who are developing naturally, or typically developing, we should say, those typically developing children will naturally produce in these three ways. [09:43] They produce sounds, they produce bodily motions, and they will produce graphics without prompting, just through exposure and practice, and the onset of those expressions largely occurs around the same time in childhood, roughly eight months, plus or minus, given that children are different, right, and are exposed to different things. [10:09] So they generally will start doing things like pointing at the same time as they start babbling, maybe manually babbling if they’ve been exposed to signed languages, and they will produce graphic representations around that time. [10:26] By graphics, I mean smearing milk to make lines, or… in ways that they recognize that they are making a representation out of something that’s not merely just, “I’m making a mess,” but maybe, you know, “I’m going to create some sort of trace that is recognized as a trace.” [10:45]

JMc: Yeah. [10:47]

NC: So those are why I would say those are the three primary things. [10:50] And if you follow the developmental trajectories, children will develop, maybe to varying degrees of complexity, given the exposure and practice across all those three forms, [11:02] but not something like writing, right, where that requires explicit instruction to understand, “Oh, this graphic representation corresponds to this sound,” [11:16] in which case you’re creating a cross-modal representation across the existing modalities that are already there, [11:23] but with a new mapping that isn’t part of our innate capacity to have that correspondence, [11:31] which is why it’s an invention, because… [11:33] But it’s an invention that makes use of existing pathways that are purposefully there for drawing, and purposefully there for speaking. [11:41] It’s just that we fuse them together to make a very culturally useful tool, but it’s not part of our biological, you know, endowment, let’s say. [11:50]

JMc: Yeah. But what about something like music, you know, which is also auditory, but is also structured? [11:58]

NC: That’s right. So music is also seemingly an innate capacity. [12:03] I should say I am experiencing developmental things quite a lot. [12:06] I have a two-and-a-half-year-old who does draw, and he also loves singing, and making what one could construe as music, [laughs] if you are generous, has the development of tonality, at least, in singing some things. [12:26] So there definitely is an innate capacity of music, but music does not necessarily correspond to conceptual representations in the same way that drawing, and speaking, and gesticulating does, but I would argue music does share the same suite of cognitive capacities. [12:47] As you said, it’s structured and organized, but in this case, those structures do not necessarily correspond to meaning in a compositional way. [12:56] So they, you can certainly have correspondences between music and meaning, like that the Jaws theme we’ve associated with some sort of impending doom, right? Or, you know, movie references… [13:10] The Star Wars Imperial March gives you a sense of authoritarianism. [13:14] So we’ve clearly built up some correspondences between music and meaning, but we don’t do, “This set of notes and this set of notes each have independent meanings, and then we will compositionally, combinatorially create higher-order meaning out of them,” but we do do that with the other modalities. [13:36] So we have different classes. [13:38] What we describe is one of the challenges that I think the language sciences has faced over the last centuries is, you know, one, defining what language is, and usually looks to things like various features of… lists of features that are there or not there. [13:59] If language is a cognitive phenomena, I would say you need to define it by its cognitive building blocks, and part of the problem that you have when making these sorts of comparisons, say, when I say that there’s visual languages that are structured of graphics that are the same as other sorts of languages, then people come back and say, “Well, that can’t be language, because it’s graphic, and by principle, I can’t allow it to be language.” [14:26] But the structural properties are still there, and so are the cognitive principles that are seemingly the same across them. [14:33] So what do you do? [14:34] And mostly the challenge has been something like that we’re only allowed a binary, that, “Well, there are languages, and there are not-languages, and this is the options categorically that we have.” [14:50] So what do you do with things that maybe have the same properties as languages, but not fully? [14:55] So let’s say music is an example. [14:57] Like, for 40 years now, since Lerdahl and Jackendoff’s book, the Generative Theory of Tonal Music came out, [15:05] they’ve argued that there’s these hierarchical combinatorial structures in music, and corresponding cognitive research has shown that you mostly elicit the same sorts of brain responses as in sentence structure from manipulating sequences of music. [15:22] Now, it’s an ongoing debate to the degree to which the same patch of neurons might be used in both, but there at least seems to be similar mechanisms at the least, and this is work that my lab is also involved in. [15:36] Now, what do you do where you have… [15:37] Well, this thing is clearly language-like, let’s call it, but there’s clearly also facets that are not, like compositionality of meaning, right? [15:46] So how do you then want to class music? [15:48] Do you want to say it’s the language of music? [15:50] What do you do with, say, emoji, which my research and others have shown, well, there’s clearly systematic lexicons, but doesn’t really seem to have a grammar to sequencing of emoji. [16:01] OK, well, now what do you do with that, right? [16:04] Or what do you do with co-speech gesture, which clearly is meaningful, has a tight relationship with spoken languages, but do you want to say that it’s part of language, separate from language? [16:17] How do you ontologically classify these things? [16:20] So our approach has been to say that the existing categories structures are not helpful, and using a word like ‘language’ as a scientific term is actually problematic, because it’s a social term also that is also value-laden. [16:41] So we need a different set of categories, and the ones we arrive at are based on the component parts of linguistic structure, of a modality, a grammar, and a meaning, or in traditional terms… phonology, syntax, and semantics, or whatever set of terms you want to use. [17:00] And so based on the relative allocation of those sorts of components, you can derive a classification system, so we call an omnia is a system that has all of them. [17:10] So spoken languages, sign languages, and what I call visual languages, are all omnia, but there’s also what we call semia. [17:18] Semia use, let’s say, phonology and semantics, but not necessarily a robust syntax, or a robust grammatical system. [17:31] So gestures are a semia, because they don’t use a grammar of their own. [17:36] Sign languages do, which is why they are an omnia. [17:39] And emoji do not; they don’t have a grammar, but they are meaningful, and expressive, and in the graphic form. [17:44] So those are also semia, which is why emoji have been compared to gestures, because they are structurally similar in that way. [17:52] Music are on the flip side, so music are the inverted side, so they have a modality and a grammar, but not necessarily meaning in a compositional way, and so those are what we call sequentia. [18:05] And that might also include something like the combinatorics that are involved in athletic things, like, say, martial art forms, or synchronized swimming, or these other things that are very clearly pattern-structured sequences, but are not, they don’t have some additional referent. [18:25] They are sequenced for themselves, their own sequencing, right? [18:28] It’s not like a punch in a martial art form is then a referent for some other concept. [18:35] It’s just a punch, right? [18:37] But it is combinatoric. [18:38]

JMc: OK. [18:39] So it seems, it seems that the key thing is ‘meaning’, in inverted commas, and the idea that meaning can be compositional and referential. [18:47] Is that, have I understood that correctly? [18:49]

NC: And has a combinatorial system. [18:51]

JMc: OK. [18:51]

NC: So it has to have all three.

JMc: Yeah. [18:53]

NC: For the way I use the term ‘language,’ usually, it’s the equivalent of what I would call an omnia, which is then, it has to have some modality of expression, has to have a complex combinatorial system that is capable of hierarchic embedding, center embedding, anaphoric relationships, structural ambiguities, etc., and there needs to be some sort of compositional conceptual structure or meaning that is associated with that behavior. [19:21] Semia lack the complex combinatorics. [19:26] However, within our architecture, it’s still one architecture that produces all of these things, so there’s no relative value statement about these classifications. [19:38] They are purely descriptive, as is the spirit of linguistics. [19:42] They… [19:44] And they have different complexity, structurally, but that doesn’t imbue them necessarily with any sort of relative value. [19:53] And because it’s a multimodal system, it resolves ontological things like, “Are co-speech gestures part of language or separate from language?” because that means that language isn’t the supercategory to which something would or would not belong. [20:07] There’s some bigger multimodal whole in which both speech and co-speech gesture both co-emerge, meaning that they’re on equal footing structurally and manifest together as part of a common system. [20:22] But… [20:24] So language isn’t the supercategory anymore, and it’s not the comparative case, because you’ve broken everything down into its Lego blocks, and are rebuilding everything in different ways, because they share common building blocks. [20:37] So it kind of escapes a little bit of the myope of, say, thinking that, [20:43] “Well, language is everything, and now we have to base everything around that.” [20:47] Instead, you say, “Well, what if we just expand to look at everything comparatively?” [20:53] And we can see the place of what we’ve traditionally called language alongside and in comparison with everything else, without thinking one needs to be overtly the root or the source domain. [21:08] The additional contrast, though, is, we have studied language — by which I mean, in this case, spoken languages — with a lot more rigor than we have the other types, and… which has developed historically (so to return to the theme of the podcast, right?) [21:25] There’s been a methodology that is very clearly defined with a lot more rigor for, say, the study of spoken languages, by which… spoken languages, we really mean written languages, in the case of the history of linguistics. [laughs] [21:41] Also, it’s not even speech, it’s really actually written versions of speech. [21:46] But there’s methodologies, right? [21:48] Like complementary distribution as a fundamental principle, or things like that, which are not typically what people have done when they have made the comparisons to other domains. [21:58] They haven’t followed the linguistic methods, so much as they’ve theorized possible, you know, connections. [22:05] So the… [22:07] And that’s, again, where my approach differs is — I like to believe, at least — all of it’s grounded in the linguistic methodologies, to then use those methodologies to yield whatever the structure happens to be. [22:22] And so in the case of spoken language and traditional linguistic thinking, there is a lot more rigor, and that rigor has not been applied to everything equally, and that’s where we really need to look to is the methodologies that we’ve been built up, which are hopefully successful for revealing the sorts of structures in cognition that we want to be talking about. [22:51]

JMc: OK, so just to come back to this question of meaning, aren’t there things that language, that spoken language can do that visual language can’t? [23:01] Like, any spoken language can represent a proposition, right? [23:07] But a visual representation can’t necessarily do that, can it? [23:11] I mean, even in your graphic novel, there are lots of words, lots of written words, you know, which are a representation of spoken language. [23:19] Like, the whole thing is not just a picture. [23:23]

NC: Right. So there’s a couple parts there that I would speak to. [23:27] Again, our modalities are not in competition, and they are subparts of a broader whole. [23:35] The broader whole is what the system is made to be doing, rather than partitioning it into unimodal types of expressions. [23:43] So in that regard, we wouldn’t expect each modality to be exactly the same in its affordances as every other one, because then they’d be fully redundant, and there’d be no purpose to a species evolving to have different modalities. [23:58] So having speech allow for some types of semiotic capacities that is… has strengths that other modalities don’t have is an evolutionary advantage, and the same would be true of graphics being able to do things that speech is not able to do. [24:17] So for example… Oftentimes, I think our speech-centricism makes us forget how much is not done with spoken languages or written languages. [24:28] For example, it’d be very hard to design a house without drawing, or design fashion or technology without any sort of pictures whatsoever. [24:41] And I bring up these cases, because, you know, as we sit in my office here, we have bookshelves, walls, we’re in a building, we have computers, there’s desks, we’re both wearing, I don’t know if we want to call the fashionable clothes or not. [24:56] But all of these things were first drawings before they were produced as products. [25:03] Now, we no longer see those drawings, so we forget that they were drawings, but that’s how all of these things were first designed, not because someone was writing a paragraph about what a sweater should look like. [25:17] Right? [25:18] So in that regard, there are certainly distinct advantages and uses for the drawn modality, more so than spoken modalities. [25:30] They allow for different things. [25:32] So that said, with things like propositional content, you know, there’s a long list of things that I’ve heard, “Well, pictures can’t do this. They can’t do argumentation, they can’t do persuasion, they can’t do negation, they can’t do these things.” [25:48] In all of those cases, they absolutely can do those things. [25:53] They just do it in a different way. [25:55] So they might do it, for example… [25:58] So usually negation, people say, “Well, they don’t have, you know, specific, you know, units as morphemes.” [26:04] Well, they do, you know, in the case of, say, think of a “no smoking” sign has the red circle with a line through it. [26:11] That’s a productive affix that takes whatever fills in the slot for its base that is being negated. [26:19] But that’s not… [26:20] That’s a single-unit utterance, let’s call it, rather than part of a grammatical system. [26:26] In comics, which is where much of my research is, they have a variety of negation devices. [26:32] One frequent one is absence. [26:35] So that is your show something and then you don’t show it anymore, or you show something and then you, in another frame, you show where that thing once was, but with a dotted outline, which now indicates that that thing is absent, which there have been papers written about this as being a type of visual negation. [26:55] It does it in a different way than spoken languages do. [26:58] That’s part of the affordances that different modalities have for how they accomplish different types of meaning making. [27:06] Because again, it’s about accomplishing the whole of those together to create a richer signal because they are able to do different things. [27:14]

JMc: Yeah. But are these different kinds of images really comparable? [27:18] Because like a “no smoking” sign is fairly arbitrary, right? [27:22] You have to learn that connection of the circle with the line through it, that that means “this is not permitted.” [27:28] However, showing something in one frame and then not showing it in another is arguably more iconic. [27:35] You know, it looks more like the thing that it represents from a semiotic perspective, whereas in spoken language, you know, it’s a sort of, it’s a commonplace that spoken language is radically arbitrary. [27:48] So can you really compare the sort of highly conventionalized arbitrary symbols that have a graphic realization with techniques that you use in comic books that are more iconic in character? [28:02]

NC: So there, I think, it’s a great place to return to history, because the idea that linguistic signs are arbitrary, at least was codified in Saussurean linguistics, if not, you know, you know, it predated it, but it was codified by the idea of a Saussurean sign. [28:21] And it persists, and it carries certain assumptions about the nature of what linguistic signs should be. [28:27] Unfortunately, it’s bo- [28:29] The Saussurean sign is both not reflective of actual human behaviors, and the semiotics that are involved. [28:38] So the first case is, there’s a very rich literature now on iconicity and language that’s doing careful examinations of the nature of how, to what degree are spoken languages, say, iconic versus not. [28:52] And most of the time, we talk about the word forms. [28:55] But there are also… [28:56] There’s also iconicity in various other places, for example, the order in which you, you know, state events is more preferred to reflect the order in which those events happened than a non-iconic event order, or direct quotes, which are essentially a type of iconicity to tell you, “Now I’m telling you iconically what this person said.” [29:18] So there’s a variety of iconicity that’s embedded within spoken languages, beyond word forms, which is also a type of study. [29:27] And we know that sign languages have a large amount of iconicity as well. [29:34] So I would say that iconicity, so all modalities allow for both… for all of our types of semiotic reference, I would say. [29:42] The other part where I say it’s not a semiotically sound idea is that the Saussurean sign conflates two properties, which are both things that you said in your statement, which is, it conflates conventionality with arbitrariness, which are not the same thing. [30:00] And if you look to say, Peircean semiotics, there’s an alternative treatment of this. [30:05] Unfortunately, through much of the linguistic sciences, Peircean semiotics has kind of been shoehorned into a Saussurean sign to not really let the Peircean flower bloom so much. [30:20] Now, Peircean semiotics can get very complicated very quickly, which is why people usually rein it in, but I think there’s a few crucial things in Peircean semiotics that are necessary in order to just understand basic semiosis. [30:34] And the first is that what Peirce would call a representamen (or in Saussurean linguistics would be a signifier, right?), that signifier, representamen, sign vehicle, whatever you want to call it, the sensory signal has properties to it. [30:51] Those properties can be idiosyncratic, that is, let’s say, a signal that is novel and different every time, versus a regularized signal that is repeatable and systematic. [31:04] Now, if you have a repeatable signal that’s systematic that has a idiosyncratic manifestation, that would be what Peirce would call a replica. [31:14] That’s essentially type-token distinction, right? [31:17] So the regularization is a type. [31:20] Its manifestation ends up being a token, but sometimes tokens might have no regularization whatsoever. [31:25] They’re just idiosyncratic. [31:29] If two people share a regularization, then it’s conventionalized, right? [31:35] So now we both share some sort of regularized signal that we recognize as being regularized across our systems. [31:43] That has nothing to do with meaning at this point. [31:45] It’s just about regularization. [31:47] And we have things in languages that are purely regularized sensory signals that have no overt correspondence to meaning. [31:55] Think of things like ‘be-bop-a-loo-bop’ or ‘shalalalala,’ which are sung vocables, which don’t necessarily have a referent other than “listen to the sounds I’m making.” [32:06] So those are systematic, regularized phonological signals that don’t have a corresponding referent. [32:13] Now, in the Peircean sense, the referent or the object is then some type of meaning, and the mapping between that signal and the meaning is what he would call the ground. [32:28] And the ground is what… is an interface, [32:31] and that interface is what’s characterized by things like iconicity, indexicality, and symbolicity. [32:36] Symbolicity just happens to make use of a conventionalized signal alone in order to make that mapping. [32:44] But it’s just another type of mapping, and conventionalization and symbolicity are not the same thing, because you can have conventionalized mappings for iconicity as well and indexicality, which is what, say, onomatopoeia end up being (or at least the thought of them being iconic is… to what degree we can debate over how iconic they actually are in that mapping, but they are certainly conventionalized and regularized), and then have a particular mapping to meaning that is meant to be iconic, at the very least, charitably, let’s say. [33:21] And the form itself, that regularization or that signal, isn’t itself semiotically imbued with any of those properties. [33:36] And the example that I like to give is the, say, the typical bathroom sign that has a highly schematic male- and female-looking person on them, or looking people on them. [33:47] That is a highly regularized graphic representation, so it is definitely conventionalized. [33:53] Now, it has multiple types of mappings to meaning. [33:57] It’s iconic in that the people look like stereotypical male and female people, right? [34:05] So that’s an iconic mapping. [34:07] It is symbolic of bathrooms, because certainly it doesn’t look like a bathroom in any sort of way. [34:14] You simply make an arbitrary association with that meaning a bathroom, and it also is indexical for where it is placed to signal to you “here is the location of a bathroom.” [34:26] So that one signal is conventionalized and regularized but has multiple different types of semiotic reference that it invokes, depending on what aspect of it you’re engaging with, possibly all at once. [34:40] And this is built into Peircean semiotics, as he would very readily say, “Oh, yeah, individual signals are multi-semiotic, or they might have multiple types of signification.” [34:51] What that also does is, it says there isn’t such a thing as an icon or a symbol, because that’s a property by which they make reference. [35:00] That’s signification. And the signal itself is not arbitrary, right? [35:07] Because that’s a characteristic of the mapping. [35:09] So to some degree, I reject the Saussurean framing based on this interplay. [35:16] And again, that would be to say, so to return to our question of things like negation, or something like that, they are absolutely conventionalized, but they use a different sort of mapping, potentially, than the types of mappings that would be involved in spoken languages, mostly. [35:32] But you also have, you know, spoken lang- [35:34] Another example is reduplication: to give a property of more, you know, of increased quantity or plurality, you then repeat the word. [35:42] Well, that’s an iconic type of signaling, or iconic type of mapping based on the properties of the signal, right? [35:54] So, you know, and plurality in the visual form would be probably reduplication is another good form of doing plurality in the visual form as well, as opposed to, say, pure affixation, which is less of a optimal form for plurality in the visual form than, say, the auditory form, or the spoken form. [36:17] So it’s about the nature of the mappings, and they all allow for conventionalization. [36:22] All of our modalities allow for conventionalization. [36:25] They all allow for all three types of signification: They all do iconicity, they all do indexicality, they all do symbolicity. [36:32] They just do it in different ways, and, I would argue, they have relative strengths for them. [36:38] So the spoken language, I believe, is particularly good at symbolicity, which is why it’s been keyed in on as, for arbitrariness, as its kind of fundamental property. [36:48] It’s really good at that. [36:49] The graphic form is really good at iconicity, which is why that’s keyed in on. [36:53] And I think the body is particularly good at indexicality, locating things relative to one’s body in space through indexical relationships. [37:03] They all do all of them. [37:04] But the advantage of having different modalities that have affordances to have strengths in different types of semiotic reference, again, gives you a richer overall signal, because now you can combine the strengths across those modalities for a richer multimodal signal than having, say, everything being relying on one modality, which just isn’t the case of what we do anyways, right? [37:27]

JMc: OK. Yeah. Well, I mean, I came here with all of my sceptical arguments, but seem fairly convinced now. [37:33]

NC: Oh, well, that’s very kind of you to say. [37:35]

JMc: Yeah. So thank you very much for answering those questions. [37:38]

NC: Thank you very much for having me.