Abraham Coetzee (MSc - Computer Science).

Visual Thinking in Computer Programming

Humans, not machines, should be central to man-machine interaction as machines are tools that serve us, not the other way around. To make interaction as natural as possible, computers should work with the entire range of physical and mental faculties employed by humans. For this reason, I am exploring the possibilities around bringing visual thinking to the act of computer programming.

There are a number of reasons why I would like to focus on the field of computer programming specifically, with the main reason being the power of human expression. As the price of computers continues to drop and they become ubiquitous, the ability to use the computer as a powerful tool, to make it do and make all sorts of different and creative things, becomes a powerful asset if not eventual need, which more and more people can use to their own advantage — from children to the elderly, accountants to artists. Programming being the most general and powerful way of making computers “produce stuff”, means that advances here could have the largest impact — “The programmers of tomorrow are the wizards of the future.”

However, programming is currently a difficult mathematical exercise, as it requires the human to encode situations in a strictly analytical and logical fashion. As we discover how to make computers communicate with us in ways we find most natural, interaction with them becomes less time-consuming, less frustrating, more enjoyable and results in an increase in creative expression by all members of society. This raises a question — what does ‘natural’ mean for humans? There are two sides to consider, the physical and the mental.

Physically, considering our different senses, I believe the greatest gains in computer programming will be made when focusing on our visual faculties. Though our sense of vision includes both text and visuals as we consume both with our eyes, programming is currently mostly focused on text, so I am focussing on bringing non-textual visuals into programming.

To understand what is mentally natural for us, consider how a common thread runs through a number of theories in the field of psychology, especially Jerome Bruner’s three distinct cognitive modes of representation, Allan Paivio’s Dual Coding Theory and Howard Gardner’s Theory of Multiple Intelligences. They state that we obtain the best comprehension and recall of information when multiple cognitive abilities work in parallel, especially language and visuals. Language, numbers and visuals are each better at different ways of communication, and we have long recognised the importance of oracy, literacy and numeracy. Now, especially in our current age of big data, we are realising the importance of graphicacy, “the intellectual skill necessary for the communication of relationships which cannot be successfully communicated by words or mathematical notation alone”. From an extensive survey by the mathematician Jacques Hadamard in the early 1900s, we know that many of the world’s greatest thinkers, such as Albert Einstein, also thought by making use of visuals. Hadamard concluded that “mathematicians born or resident in America, whom I asked, … practically all … avoid not only the use of mental words, but also, just as I do, the mental use of algebraic or any other precise signs; also as in my case, they use vague images”. Visual ways of thinking appear to be indispensable to thought, which means that bringing visuals into the field of predominantly language-based computer programming, holds much promise.

There is also an ongoing debate, based on ongoing research, about the degree to which language influences thought. This debate carries over to programming languages where many well-known people, such as Edsger Dijkstra, have a strong opinion: “Nearly all computing scientists I know well will agree without hesitation … The tools we use have a profound (and devious!) influence on our thinking habits, and, therefore, on our thinking abilities”.

This raises an interesting question: How would our thinking be affected if we were to incorporate visuals into our programming environments and programming languages?

My contact details: abrahamcoetzee@gmail.com