You asked me what symbol manipulation is and how it is different from tool development. This is my reply.
My personal fear in all this is that interaction will continue to be suppressed by the notion of ease of use and that simplicity propaganda will be more powerful than individual responsibility and thought. My personal dream in all this is simply to develop beautiful tools, like a beautiful mont blanc pen, to work on magical papers where information and interactions are as free as possible.
In both cases, we need more of a McLuhan study of the media of interactive text, of symbol manipulation to really make progress I think. With a goal of what I call Deeper Literacy (www.deep-literacy.com) we need a separate study of symbol manipulation in digital media in the same was as semiotics and literature study in general. We need to look at the inherent potential of symbols much in the way we can study poetry without studying typography though of course they are related.
Symbol Manipulation as a separate topic
Why it deserves to be a separate topic is a VERY hard question to answer and I believe it fits under Doug’s notion of C level development. The very shapes of text and how to interact with them should not be simply looked at as a programming issue but a deeply human issue, since what matters is not simply what characters there are on the screen but what the author’s intention was and the readers ability to critically comprehend. It is about what we see and how we can engage the occipital lobe to augment the prefrontal cortex, it’s not about simple layouts, it’s about what can happen when you select text and when you look at text spaces. This is an area for art, linguistics, philosophy, psychology and more to come together and for the technical implementations to help shine lights into the forest of opportunity. This is why I have been hosting The Future of Text Symposium every year, with a wide range of participants, from VCs, to artists, developers, typographers, anthropologists and many more types of people. It’s hard to specify what symbol manipulation is but it’s clear it’s a richly interdisciplinary question.
Other Media
Tool/implementation issues are tied to issues of symbol manipulation but it’s not the same since there are many issues with tool building which does not address the manipulation of symbols, such as knowledge graphs, AI, video, 3D environments and so on. These are also important topics but I feel that the magical Harry Potter like world of fantastically powerfully interactive text deserves focused attention.
Other media can and should be integrated with text, but the amount of research and attention video, audio and images and so on are getting far outpaces the efforts for text. Just compare Word now with Word 10 years ago and then look at the change for video editing and photo editing and computer games in the same period.
Teasing out the underlying principles
There is indeed a relationship between symbol manipulation and tools, we need to both develop tools to better understand what symbol manipulation can offer us and to realise this potential.
I spend my time implementing in order to find out what possibilities there are. For this ‘science’ practical exploration is as crucial as for any other but I don’t look for what the most useful tool and do and leave it at that, I look for what the underlying principles appear to be, chief among them the issue of linking/connections and visual exploration.
The basic question is simple: What is it possible to with text and the next question is what is useful to do with text and this needs to be judged within a framework of what we are trying to support the user–the author and the reader–to accomplish.
Visual
The visual aspect is of utmost importance since this is the way the text reaches our brains. The basic typographic considerations remain important in the digital medium and it is further augmented by the potential for the user to rapidly and easily change the visual presentation of the text and other symbols, such as changing colour while scrolling.
Non linear representations of the data, such as the Liquid Space Chris and I are developing, are further areas of investigation where we are learning what looks cool vs what is actually useful for real work.
Connections
The issues around symbol manipulation all revolves around the way information is connected. With text, we have mechanisms for writing text to say that the text refers to a specific location or other identifying attribute of something. Doug had rich links with link types and other attributes. Today we have what we call web-links which are not really links, they are simply addresses to intermediary commuters which may or may not be operating when clicked on to send the web browser to fetch a whole document.
Doug referred to implicit links and Vint talks of binding, where, for example, we are bound by having worked on this project. The opportunities, the vectors, the dimensions of movement and views are profoundly rich and worth investigation.
I have looked at this with Liquid | Flow and Liquid | Author, host hosted on www.liquid.info providing new ways to follow implicit links and explicit links, which is helping map out the space of possible useful connection creation and following mechanisms.
Addressability
You can’t manipulate something you cannot point to, cannot address, and the means of pointing is important too. As has been discussed often, you cannot usually point to inside a web page and never inside a digital book. There are many philosophical opportunities for pointing at something explicit but there are also issues for pointing to something which is implicit.
I have written on the symbol space and addressability aspect here: symbolspace.info and I have a series of articles/posts on why I try to do my work here, of which many try to address this issue: wordpress.liquid.info/category/why/
FoT
Someone asked about my previous presentations for the Future of Text. The symposium has had small and larger venues and has differed in support, but here are my presentations, for reference:
2017 https://youtu.be/XDrowxW2u1Y?t=3459
2016 https://youtu.be/iZvCeYOh1zY?t=5209
2015 https://www.youtube.com/watch?v=rQg5ekt_iw0&t=0s&list=PLYx4DnFWaXV8tT5j3yVzh3XXunyQqXUfP&index=15
2014 was not well recorded: http://www.thefutureoftext.org/futureoftext2014.html
2013 is here: https://www.youtube.com/watch?v=TeyJSLuFp7A&list=PLhy1DoMLdoIYDjFdlEAID46A3AlotyDZC
2102 was not recorded but my slides are available: http://www.thefutureoftext.org/frode-fot.pdf
2011 was recorded in one long stream: https://vimeo.com/30992192
I don’t have much more to add at the moment, it’s a very difficult topic and I am writing this from quite a defensive posture. I hope however to have this further clarified for my own sake at some point.
The key to communicate is that we all have some degree of responsibility in developing how we can view, interact with, connect and follow connections of symbols and this involves issues of visual design, linking and the social networks the work will take place inside, to be implemented as rough tests and polished products, always trying to tease out the power of richly intuitive text.
10 Comments
vint cerf
Now, start thinking about people who are blind. How shall they interact with text? We need to find a way to make symbolic manipulation and interaction feasible across the full spectrum of disabilities.
Frode Hegland
Indeed. The full spectrum of interactions will be important to research.
skreutzer
It’s not clear who asked the question. I’m not convinced that symbols can be manipulated, because they’re atomic entities. They symbolically stand for the real thing, so how can you manipulate them without damaging the reference to the real thing? If it’s about fiddling with the reference, it’s usually cheap and easy to just create another symbol instead of trying to manipulating existing ones, or how would one go about manipulating existing ones? To me, it sounds that you actually mean text manipulation, which encodes/serializes symbols, and by manipulating text, you can do a lot of things, which rarely manipulate the symbols themselves. Text features can be agnostic in regard to the encoded symbols, but can as well be augmented for providing access to symbols and work with them, not on them (how would you?).
The link to symbolspace.info is confusing as well. Sure, individual, abstract characters are symbols themselves, but in dealing with symbols, you well know that we rarely read on the character level, but on the word level to identify symbolic meaning, so text manipulation on the character level as opposed to text manipulation at the word level is rarely about symbols, because doing things with individual characters most of the time switches to a different word symbol and doesn’t do anything to the character as a symbol itself. Characters are the atomic encoding format/convention to construct textual symbols. Characters are atomic in the information encoding scope, but the symbol scope is one level above the mere encoding concerns. The atomic entity for textually encoding a symbol is a word.
I think most of this is pretty obvious to software developers, “technologists”, tool systems people, who have to deal with questions like this all the time. I could be wrong, but it shouldn’t be too difficult to give a practical example from daily life how we manipulate a symbol if that’s what we actually do, as opposed to the notion that symbols are atomic and we only manipulate their composition.
Frode Hegland
You don’t agree that the word ‘insight’ is a symbol? Of course it is, but there is nothing physical in the world it relates to, it’s abstracted. I think http://www.dougengelbart.org/content/view/348/102/ outlines it quite well. It’s not just characters which are symbols, words and sentences are also symbolically represented I would say.
skreutzer
Of course I agree, that’s my point I guess. We don’t care about the letter ‘s’ in “insight” as a symbol, because the symbols ‘s’ and “insight” are different from each other, so I think words are most of the time the atomic elements when it comes to dealing with symbols and individual characters (when not used symbolically, but as building blocks to construct words from character combination that then can be proper, better symbols than what the limited alphabet provides). Now, if they’re atomic, how to manipulate them? If we start to change characters, we likely create new words/symbols, or switch to totally different symbols like “insignia” or “insider”. That’s for the part that “symbol manipulation” probably doesn’t refer to writing or text manipulation.
As we have established that “insight” is a symbol, what can we do with it or manipulate it? There’s the option to re-define it or fiddle with its meaning, which can be considered a bad, confusing action or a very creative activity as well, depending on context. The “insight” symbol doesn’t reference a tangible physical object of course, but an abstract concept, which isn’t a big difference, it’s just giving names/identifiers to whatever we might want to talk about, as a shorthand or “link”/”address” to the real thing. The actual meaning of the symbol has a scope (can be more vague or more strict), which includes a sudden realization or gained understanding about something non-obvious; the long and deep study of something that lead to better, more correct understanding than what others learn from a quick look on the surface; to look into oneself as the practice of self-reflection or -analysis; that’s what our language standard says, what the dictionaries as lookup tables for resolving and dereferencing say. But then I could start to call my product “insight”, we both could agree to use the term/symbol to mean the exact opposite as some kind of secret code or in ironic context, I could “abuse” the term/symbol by using it to describe/name the event that a physical object comes into view/sight of an observer (as in “the ship came into insight distance”), or similar. Notice that the symbol hasn’t changed and hasn’t been manipulated (itself), I instead manipulated what the symbol is pointing to, what it can and can’t point to. Symbol manipulation in terms of changing and overloading it’s meaning is somewhat dangerous because it becomes less useful if we do it too much.
What is symbol manipulation then? If I come up with the word “outsight” to refer to things about which insight never can be obtained; sudden, surprising findings about something else than I was looking for; looking from inside outwards; general dumbness or whatever else (similarities in meaning scope is just because I followed a similar character construction rule that allows the deduction of a negated meaning, but the actual referenced concepts/meanings are different and distinct, they may or may not be opposed even, and I could have picked a different selection of meanings or a different combination of characters to refer to some or all of the mentioned concepts), it barely affects the original “insight” symbol and its meanings, only by mere accident/coincidence. One could claim that this is a symbol manipulation example because I relied on the original symbol to construct/derive the new one, so there is a relation, but I could make the point that the symbol itself is rather arbitrary, it’s perfectly fine to come up with new words that don’t have any resemblance to existing words/symbols (although it’s considered bad design) and define their meaning or meaning scope. I could just define that “anpecatrm” refers to the activity of looking out of the window (to specify the scope, specifically and only used when there is a window of an implied house, not to be used looking out of the window/windshield of a car).
How else could symbols be manipulated? We could consider the usual manipulations of typography, typesetting, rendering, visualization, but if “insight” in red has a distinct different meaning than in green, changing the color changes what meaning is referenced, the two symbols stay the same two symbols and can’t be manipulated interchangeably. Such operations can be a way to trigger/hint different connotations however, to indicate a slight difference in meaning scope, I’m not too clear how to think about those activities.
I can imagine that symbol manipulation referred to a specific historical meaning for Engelbart: if you’re an electrical engineer and encounter computers with their binary information encoding, the realization is (see Turing) that the bit patterns are arbitrary symbols that can represent other symbols like numbers (most prominent back in the day), text, images, abstract concepts and whatever else, and just as we manipulate binary and numeric symbols, we can as well manipulate text, image, audio symbols (if we can find reasonable methods to do so, that is). For binary and numbers, arithmetic is a useful manipulation method (in contrast to useless manipulation, like picking a random bit or digit of a large number and make all other bits/digits that very bit/digit). What is it for text? Converting upper-case characters to lower-case? Make a word/symbol italic (but what would that change, do we enter pictorial/visual symbolism and would it still remain to be the same symbol)? I have some trouble of listing useful methods that manipulate pure text symbols, which is in essence what I try to say in these comments. It may be much easier to list useful symbol manipulation methods for numbers, audio, images, but that too changes the symbol so it refers to something else (most dramatically with numbers). I’m not sure that Engelbart isn’t misleading in “Improving our Ability to Improve: A call for investment in a new future” in regard to this topic as whatever we do to symbols themselves, usually has to follow pretty narrow constraints in order to remain useful and correct.
So what is it what we really care about? It could be moving symbols around, combining, separating and rearranging them, “enacting” them (to attach effects to symbols and trigger them), and indeed augment their use (“writing” them or picking them from a list of symbols, insert them into other contexts as, for example, formal constructs, or whatever else). Those activities rarely change the symbol itself as it’s supposed to retain the reference/meaning.
Does any of this make sense? I would like to investigate the question further and learn how we could look at this. I certainly don’t follow a particular academic theory, but for practical application need to at least have some theoretical framework, which may have its influences of course or confusion on my side. Just would like to know.
Frode Hegland
I’m glad you are thinking about this and asking questions. I am too, so let’s see where we end go 🙂
skreutzer
See, I didn’t study Engelbart’s writings very closely (how would I), but in “Improving our Ability to Improve: A call for investment in a new future”, the occasion and context of the 3x “manipulate symbols” is somewhat interesting. There are several other notions that refer to symbols and symbolic, often it’s about their use, rendering, augmentation, which are distinct from manipulation. Where manipulation is explicitly referred to, the second occasion talks about the general power of the digital computer as a symbol manipulation machine, which likely is more about numeric operations and code execution (!) as the general/universal purpose, Turing-complete machine (which is where the power comes from), and immediately after that doesn’t fail to claim that symbol portrayal and interaction is even more important. The third occasion refers to pretty much the same digital Turing computer theme, and the first one combines symbol and language manipulation, not without setting it/both into contrast of number manipulation (on this semantic level of general data types, what do we have: numbers, text, data, at least there’s the historic notion of data processing vs. text/word processing). How would we manipulate language, if that’s similar enough to symbol manipulation, if not equivalent/synonymous? Or are symbols atomic entities and “language” the rules where to put them? Is it about us changing vocabulary and/or grammar? Potentially to some extend, but I think it’s really about dealing with text, as we encounter it as a large collection of symbols, being composed in a specific language (in which our knowledge is encoded, remember: DKR, complex+urgent world problems, what all the symbol manipulation technology was intended to do/support, to augment reports and research results, which are traditional texts, just as NLS primarily used and produced text and not some new forms of symbolism). There might be a hint towards topic mapping in there, but in what way did NLS manipulate language? There’s little reason to do so, it just went with the existing conventions of language as encountered in text. “Manipulate symbols and language” might simply refer to using the digital computer as a tool not only for calculations, but for all use cases that are related to what humans do with symbols/language anyway, to use this processing/augmentation tool for interacting and working with the human systems that are in place for a long time now already and can’t be changed easily.
skreutzer
“Manipulate symbols” is always in plural, which could be an indication that we rarely care about a single one, specific symbol like the knowledge/topic/identity/AI people do, but always a whole bunch of symbols who might have nothing in common in terms of meaning, but are of a certain type/category/group that rendering can act upon. Would you add individual symbols/words/strings to a single, specific ViewSpec to have the symbols/words visualized in the same way based on their identity, or define an abstract grouping type that can be assigned to individual symbols/words, to then apply different ViewSpecs onto that type? If a single symbol/word is hard-wired to a specific ViewSpec, what about contextual uses where the particular ViewSpec shouldn’t be applied? I can only speculate, but would find it hard to imagine that such considerations didn’t play a role while designing and building the system.
Gyuri Lajos
Thanks Frode for the link to http://www.dougengelbart.org/content/view/348/102/
As Doug stated, he had been patient for 50 years (in 2002)
but clearly we must face up to the fact that indeed there is a “systematic bias against “Collective IQ”” at work.
https://hypothes.is/a/2zxe_uXEEeifWgs73UAvJQ
skreutzer
Thank you very much for the link to the Future of Text 2013 recordings, I was not aware of them, and for some strange reason, they’re all unlisted.