Diagram Project

Nick Davis

Individual Research Objectives

Research Proposals

Let's face it, artificial intelligence is a hard thing to achieve. Some theorists say that the idea is fundamentally flawed because computers can only process things in a serial manner rather than in parrallel, like our brains. However, what if we frame the artificial intelligence problem in a different way. What if we are not trying to replicate our consciousness, but create a new consciousness that utlizes and builds off of human intelligence. To create a system that works in harmony with the human cognitive architecture, but expands its abilities.

The idea is this: create an intelligent environment within which we can act and explain concepts that are mapped out in a visual manner. A kind of diagramming system that takes the content of our speech and actions and realizes it in a visual domain. As we know, many people are visual thinkers, it helps people to see concrete representations of abstract knowledge. If there was a system that is able to do this is a dynamic way as someone was presenting a concept, information would be transmitted more efficiently. The kind of research that is going on with the Center for Culture and Cognition and the Cognitive Science Department, if synthesized, may be able to bring about a system such as this.

Consider the different labs in the Department: Gesture, Robotics, Neuroscience, Speech Synthesis, Diagrammatics, Semiotics. If each of these labs was working in harmony for an overall goal, the collaborative efforts may be able to achieve the groundbreaking invention mentioned above.

Here is an outline of what each component would contribute:

Cognitive Diagrammatics: Steve Dee is working on a project that will allow a user to intuitively create diagramms. In essence, the software will understand what the user is trying to portray and will give the user applicable tools that will allow for ease of knowledge representation. If this software is linked with Stemmatic Syntax by telling the system that certain knowledge domains, or schemas, relate to a particular node, the system may be able to reverse engineer a stemmatized phrase into a diagram. This would essentially be parsing diagrams. It would be nice if the software was able to read a diagram and parse each component. It may be able to go from a prose-> diagram and vice versa.

Speech Synthesis: Patrizia Bonaventura is working with the Case Speech Production Lab to analyze speech in relation to stemmatic syntax. If this system was able to transcribe speech, Matt's parsing program could stemmatize the text, which would in turn be analyzed by the cognitive diagramming software.

Gesture: Per Aage mentioned looking into simulating gesture. Analyzing which gesture corresponds to what emotion. We could also team up Fay Parrill, directory of the Gesture and Cognition Lab, in order to examine their data and receive guidance. If we are able to accurately model gestures that relate to concepts and domains, we may be able to reverse engineer that process in real time to infer what concepts someone is trying to convey with gestures. Use this data in parallel with the stemmatic system which would feed into the diagramming software to create a more robust dynamic diagram of the informational content of what someone is trying to convey.

Semiotics: The study of symbol meaning goes into the diagram research in what each element in the diagram is actually trying to convey. This is important in creating a system that understands that content of diagrams and how to appropriately draw concepts.

Robotics: In essence, the software that will be creating this diagram would be best suited for an agent that knows the goal of what it is trying to do. If this is the case, we could train the software, provide it with feedback on how it is performing. It would essentially be following the dual-pane model of perception, the sensory input would be the human interacting in the space and the representation space would be the diagram. Essentially, the task would be to get the agent to understand what the human is trying to say. Given the diagramming tools at the agent's disposal, how well could it construct a representation of what the human is trying to convey. Using reinforced learning, we may be able to train the agent to draw the content, as well as search for relevant information on wikipedia or some other source for further resources at the disposal of the human. It would be a dynamic relationship betwen human and machine, creating a new kind of learning environment that utilizes technology to increase the efficiency of knowledge transmission.