You asked me what symbol manipulation is and how it is different from tool development. This is my reply.
My personal fear in all this is that interaction will continue to be suppressed by the notion of ease of use and that simplicity propaganda will be more powerful than individual responsibility and thought. My personal dream in all this is simply to develop beautiful tools, like a beautiful mont blanc pen, to work on magical papers where information and interactions are as free as possible.
In both cases, we need more of a McLuhan study of the media of interactive text, of symbol manipulation to really make progress I think. With a goal of what I call Deeper Literacy (www.deep-literacy.com) we need a separate study of symbol manipulation in digital media in the same was as semiotics and literature study in general. We need to look at the inherent potential of symbols much in the way we can study poetry without studying typography though of course they are related.
Symbol Manipulation as a separate topic
Why it deserves to be a separate topic is a VERY hard question to answer and I believe it fits under Doug’s notion of C level development. The very shapes of text and how to interact with them should not be simply looked at as a programming issue but a deeply human issue, since what matters is not simply what characters there are on the screen but what the author’s intention was and the readers ability to critically comprehend. It is about what we see and how we can engage the occipital lobe to augment the prefrontal cortex, it’s not about simple layouts, it’s about what can happen when you select text and when you look at text spaces. This is an area for art, linguistics, philosophy, psychology and more to come together and for the technical implementations to help shine lights into the forest of opportunity. This is why I have been hosting The Future of Text Symposium every year, with a wide range of participants, from VCs, to artists, developers, typographers, anthropologists and many more types of people. It’s hard to specify what symbol manipulation is but it’s clear it’s a richly interdisciplinary question.
Tool/implementation issues are tied to issues of symbol manipulation but it’s not the same since there are many issues with tool building which does not address the manipulation of symbols, such as knowledge graphs, AI, video, 3D environments and so on. These are also important topics but I feel that the magical Harry Potter like world of fantastically powerfully interactive text deserves focused attention.
Other media can and should be integrated with text, but the amount of research and attention video, audio and images and so on are getting far outpaces the efforts for text. Just compare Word now with Word 10 years ago and then look at the change for video editing and photo editing and computer games in the same period.
Teasing out the underlying principles
There is indeed a relationship between symbol manipulation and tools, we need to both develop tools to better understand what symbol manipulation can offer us and to realise this potential.
I spend my time implementing in order to find out what possibilities there are. For this ‘science’ practical exploration is as crucial as for any other but I don’t look for what the most useful tool and do and leave it at that, I look for what the underlying principles appear to be, chief among them the issue of linking/connections and visual exploration.
The basic question is simple: What is it possible to with text and the next question is what is useful to do with text and this needs to be judged within a framework of what we are trying to support the user–the author and the reader–to accomplish.
The visual aspect is of utmost importance since this is the way the text reaches our brains. The basic typographic considerations remain important in the digital medium and it is further augmented by the potential for the user to rapidly and easily change the visual presentation of the text and other symbols, such as changing colour while scrolling.
Non linear representations of the data, such as the Liquid Space Chris and I are developing, are further areas of investigation where we are learning what looks cool vs what is actually useful for real work.
The issues around symbol manipulation all revolves around the way information is connected. With text, we have mechanisms for writing text to say that the text refers to a specific location or other identifying attribute of something. Doug had rich links with link types and other attributes. Today we have what we call web-links which are not really links, they are simply addresses to intermediary commuters which may or may not be operating when clicked on to send the web browser to fetch a whole document.
Doug referred to implicit links and Vint talks of binding, where, for example, we are bound by having worked on this project. The opportunities, the vectors, the dimensions of movement and views are profoundly rich and worth investigation.
I have looked at this with Liquid | Flow and Liquid | Author, host hosted on www.liquid.info providing new ways to follow implicit links and explicit links, which is helping map out the space of possible useful connection creation and following mechanisms.
You can’t manipulate something you cannot point to, cannot address, and the means of pointing is important too. As has been discussed often, you cannot usually point to inside a web page and never inside a digital book. There are many philosophical opportunities for pointing at something explicit but there are also issues for pointing to something which is implicit.
I have written on the symbol space and addressability aspect here: symbolspace.info and I have a series of articles/posts on why I try to do my work here, of which many try to address this issue: wordpress.liquid.info/category/why/
Someone asked about my previous presentations for the Future of Text. The symposium has had small and larger venues and has differed in support, but here are my presentations, for reference:
2014 was not well recorded: http://www.thefutureoftext.org/futureoftext2014.html
2013 is here: https://www.youtube.com/watch?v=TeyJSLuFp7A&list=PLhy1DoMLdoIYDjFdlEAID46A3AlotyDZC
2102 was not recorded but my slides are available: http://www.thefutureoftext.org/frode-fot.pdf
2011 was recorded in one long stream: https://vimeo.com/30992192
I don’t have much more to add at the moment, it’s a very difficult topic and I am writing this from quite a defensive posture. I hope however to have this further clarified for my own sake at some point.
The key to communicate is that we all have some degree of responsibility in developing how we can view, interact with, connect and follow connections of symbols and this involves issues of visual design, linking and the social networks the work will take place inside, to be implemented as rough tests and polished products, always trying to tease out the power of richly intuitive text.