Nick Davis    

     The objective of this work is to examine the neural and cognitive correlates of categorization and conceptualization and the implications this has for linguistics and perception. The basic hypothesis is that upon seeing a word or an object, the category is activated and subsequently singularized based on the goal state of the individual. Examining this hypothesis requires analyzing the relationship between top-down and bottom-up cognition. To investigate this issue, the categorization method proposed in Barsalou’s (1999) theory of perceptual symbols is combined with the affordance-based conceptualization posited by Glenberg’s (1997) theory of meshing.

     Barsalou hypothesizes that the multi-modal neural correlates of perception and sensory motor activity are grouped into systems, termed simulators in his theory, equated with concepts, which are then activated upon viewing an item of similar caliber in the future. These simulators serve to facilitate an embodied understanding of objects in the environment by activating all relevant neural correlates in an unconscious ‘simulation’ for the specific object of inquiry. So, the simulator is the type while the simulation is the token. This implies that comprehending sensory input, linguistic or otherwise, is first a function of categorization and secondarily singular. The token follows the type.

      Meshing proposes a system of conceptualization that relies strongly on the notion of object affordances, which is viewing an object in terms of its function. Glenberg calls these affordances activity patterns because they relate what kinds of activities the body can do to the world. A distinction between automatically attributed activity patterns and manually attributed activity patterns elucidates the different roles of intentions and sensory activations by labeling goal-oriented activity patterns as non-projectable properties of the environment and innate bodily potentials (Gibsonian affordances) as projectable properties of the environment. In Glenberg's view, conceptualization is a mesh between these projectable and non-projectable features of the environment. Memory’s role in this situation is to recall the non-projectable features of the environment based on previous encounters.

     These are complimentary theories, and in fact, by bringing them together, we may be able to come to a more thorough understanding of perception.

     The crucial addition to Barsalou’s theory is looking at the influence online mental simulation could potentially have for perception. First off, it is important to state that these perceptual symbols are not necessarily symbols in the semiotic sense, but rather neuronal groupings that schematically resemble a perceptual state. He recognizes that these neuronal groupings can be activated through two distinct routes: First, through sensory stimulation, and next, through introspection.  The unconscious simulation happens when one encounters an object. The simulator is activated, and if the simulator creates an adequate simulation, then that object is within the category of the simulator and the vast information the simulator contains can be attributed to the object in question. The conscious simulations are for introspection after the fact, or for predicting future states of affairs. He doesn’t consider mentally simulating in order to solve problems online, or how mental simulation could activate perceptual symbols and alter conceptualization in both linguistics and perception.

Glenberg considers the phenomenological aspects of conceptualization, acknowledging that objects inherently suggest to the human perceiver what kinds of activities one can do with them, which is why he creatively labels these activity patterns. Additionally, he notes that this happens automatically, projectable properties of the environment, and manually, non-projectable properties of the environment. The online relation between these components may inform Barsalou’s theory. 

     Using Glenberg’s path example, distinguishing a path and knowing that one can traverse it is automatic, previous encounters with paths provides recognition of this feature. However, being ‘the path home’ is a subjective judgment of the path, a goal driven process that is not an inherent property of the path. So, the path is conceptualized both in terms of the way the body can interact with it (projectable properties, traversability) and the desired outcomes of bodily interaction (non-projectable properties), i.e. becoming the path home, making this instance of path special.

     He elaborates this by explaining that we conceptualize the path by meshing the projectable and non-projectable aspects. But he only defines a mesh as combining activity patterns whose affordances agree with each other. For example, a table can be a seat because they both afford the act of sitting. Here, I am going to consider this mesh from the point of view of coactive simulators constrained by common proprioceptive neural correlates.

     If we compare the two perceptual symbol levels, namely those activated through sensory information and those activated through mental simulation, with the automatic activity patterns and manual activity patterns Glenberg proposes, we may push the discussion of perception further along. In this thought experiment, we would have the sensory information automatically activating simulators with exhibiting similar features as those perceived, which inherently contain proprioceptive potentials, affordances, the simulator with the greatest affordance alignment is chosen, thereby accounting for how objects automatically suggest affordances.

With respect to the manual activity pattern attribution, reasoning about ones goals is sometimes benefitted by mentally simulating a situation. For example, visualizing the outcome of taking one path versus another, i.e. will this path lead me home? The influence of mental simulation on conceptualization is not explicitly developed in Glenberg’s theory, so let’s look at the implications this mental simulation could have.

     First off, recent cognitive research on visual imagery by Kosslyn et.al. (1995), and motor simulation by Jeannerod (1994) has shown that these simulations utilize similar neural resources as actually doing that action. From this, we can conclude that these conscious mental simulations are activating the perceptual symbols they relate to, the simulators that contain them. When conducting these mental simulations in circumstances in which one is trying to reason about objects or solve some goal, as in the path example, the consciously driven perceptual symbols mesh with the automatic perceptual symbols by means of common proprioceptive qualities. If the actions for the desire match up with the possible actions of a given object, then the object is perceived in a new way, i.e. the path becomes the ‘path home.’ This is saying that two (or more) distinct simulators are contributing to the conceptualization of an object, the aspects that are co-active are those whose proprioceptive qualities align, whose affordances are common between the two. To be clear, there is a simulator activated from bottom up processes based on feature detection, and also a simulator based on the goal state, each of which have associated affordances. If these affordances align, as normally happens, nothing else result, but if there is some problem that needs to be solved, then the goal state is concentrated on, usually by way of mental simulation.

     This mental simulation is touched upon in Glenberg’s theory with respect to clamping and suppression. Glenberg states that “to keep the system reality-oriented, it is necessary to ensure that patterns based on projectable properties of the environment are primary... I will I will refer to this as “clamping” projectable properties of the environment.”(Glenberg, 1997, pg. 6)  While suppressing is just the opposite, as Glenberg states, “in the service of prediction, we have developed the ability to, if not ignore, at least to suppress the overriding contribution of the current environment to conceptualization.” (Glenberg, 1997, pg. 7). 

Clamping, seen in the context of perceptual symbols, would unconsciously hold the simulator with the most relevant projectable properties stable, allowing for efficient cognition of linguistic and perceptual items. While suppression serves to loosen the contribution from the simulator activated from projectable properties to serve the needs expressed by mental simulation, namely the non-projectable properties. The detail of mental simulation is determined by the extent it is made conscious. Here, I am trying to make clear that there are unconscious simulations, as in Barsalou’s theory, but also conscious simulations, in which visualization is an important component. This state of awareness only results when one concentrates, that is, when the temporal aspects of an object are mentally simulated. The past is extracted in traces of activity patterns and the future as potential activity patterns. In this concentrated state, perception becomes semiotically inclined. This semiosis enables mono-modal aspects of the environment, entities one experiences through vision, to transform into multi-modal conceptualizations based on previous experience. This cognitive trait would enable signification in general.

     The alignment of affordances coming from the similar proprioceptive information contained in simulators, serves to frame the conceptualization and constitutes what Glenberg refers to as a mesh. This happens by suppressing, to varying degrees, the simulator based on environmental input. For example, let us take up a creative scenario in which a piece of paper is conceptualized as a shim used to level a table. The goal state is held in mind, mentally simulated to some degree, consciously activating simulators whose proprioceptive information contains something do with inserting an object in order to bring about stability. Additionally, the sensory information automatically activates the simulator for paper by means of object recognition, bringing with it those automatic affordances it offers, for example writing on, crumpling, folding, etc. Then, these two distinct simulators mesh to bring about an end state of folding the piece of paper and using it as a shim. By holding in mind the goal activity, the potential affordances that are automatically attributed, align themselves by means of co-activation and mentally elaborating activity possibilities. The more fine grained the simulation, the more specific the meshing becomes. For example, at first, the piece of paper could be conceived of as a shim without any modifications. But when this activity unfolds, the problem is not solved because the paper is not thick enough, therefore the mental simulation highlights the fact that it needs a thicker shim, which then aligns with the automatic affordance which makes the paper thicker, namely folding.  And this continues until the problem is solved. This process is happening on-line with feedback from the environment informing the mental simulation and vice versa.

     Furthermore, the degree that simulation is made conscious dictates the amount of suppression, i.e. concentration controls suppression. For example, once I mentally simulate the goal state of shimming the table, the projectable properties of the environment are suppressed, paper is not considered as a tool for writing, but as an object, which can be folded and used as a shim. Suppression loosens the constraints on which simulators can run a satisfactory simulation, so many different simulators, with varying proprioceptive alignment, can be activated. It is as if this suppression lowers the threshold for activities that are relevant to be considered.  This allows for the possibility of an object with an already established simulator, with a fully functioning category, to run in a foreign simulator, bringing to mind the combined associations from each that mesh based on affordance weights. 

     The intentional state holds the goal outcome of the current action in mind. It compares the sensory data coming in and runs an unconscious simulation to gauge the outcome if current route is kept. If all is well, continue on with same plan. For example with garden path sentences, a meaning space is constructed on the fly, but as soon as the ambiguous word comes up, the target no longer matches the trajectory. In this situation, one mentally simulates in order to construct a new goal state that takes into consideration the current change in the sensory input. During this mental simulation, projectable properties of the environment are suppressed, the usual affordances associated with objects are overlook and other simulators are activated to see if they have the activity patterns to help over come this problem.

     There will always at least be an unconscious Barsalou simulator simulation, which requires activating the neural correlates and seeing if the proprioceptive information is robust enough to deal with the current object. If all is well and the outcome of the simulation matches the ideal goal state, the things mesh and projectable properties stay clamped. The bottom up simulator stays dominant. However, if the affordances don’t align, a problem arises, a narrative structure is built or elaborated upon by a mental simulation which then decides a new plan of action or meaning assignment of the current state. This suppression changes conceptualization to be somewhat mental and by doing this draws upon other simulators, memory is used to in order to activate other simulators that may be able to solve the current issue. Suppression looks past current sensory input for other narrative scenario’s that would fulfill the current situation. When it finds one, it gauges whether proprioceptive affordances align, then it meshes, it runs a simulation of the current sensory input in the new simulator, bringing with it new results.

    Looking at the integrated dynamic system, we see two major loops. First, the sensory loop in the top right, and next, the intentional loop in the bottom left. There is only one source of information coming in from the environment and that is the sensory arrow coming from conceptualization. This activates neurons and then the simulator, which in turn activates the subset of proprioceptive information in the simulator in order to determine if the entity under examination can be a part of the current simulator. Next, the information either does one of two things, there is a bifurcation in the diagram, namely to higher order processes or to conceptualization. The sensory data will either be clamped, and reality will be the surface level qualities, like color and form, or one concentrates and elaborates on certain elements of reality, making it more meaningful by attributing a narrative history or future. The sensory loop will be more active when a lot of sensory information is considered in a coarse grained way, while the intention loop will be active when analyzing a small subset of the sensory data with respect to planning a situation, creating some narrative structure to be executed in reality. 

     Thinking about this in terms of Dynamical systems, the simulators, groups of neuronal activations, would correspond to attractors, are experienced based groupings that are activated based on current interaction. The intentional state creates an attractor that makes the bottom up simulators more or less relevant. At first many simulators are activated and whichever contains proprioceptive information that relates to the goal state held in mind will have a deeper ‘well’ as stated in dynamical terms. In other words, those simulators that relate to proprioceptive information provided by the mind will be more active, allowing multiple simulators to contribute to the current conceptualization, while still highlighting some over others.

     The two loops, sensory and mental, have a certain speed as mentioned earlier. This aspect of the theory is underdeveloped on the biological side, but is required for the eventual computational application of the theory. By this, I am referring to the rate at which the environment is scanned, or the mental representation is scanned or attended to. Normally sensory re-fresh rate is high, but low grained and mental rate is low with schematic qualities. However, in order to plan something, memory has to be consulted and those non-projectable qualities have to be attributed. This requires concentration and suppression, forcing the mental to go more detailed on a smaller subset of sensory activation in order to make a plan, whether that be an action plan or a meaning plan, as we have seen with some word groupings tending toward different meaning construals, it changes perception momentarily to set up a relevant meaning mechanisms, a model with which to interpret current sensory input based on the non-projectable affordances. The time domain is extracted by simulating past and future- traces, narrative histories, or potential, narrative structures. Next, with this template in place as a stable attractor, mass sensory scanning starts again. This continues as long as the current schema remains relevant, that is the system performs at optimum efficiency by simply holding the goal state in mind, effectively compressing the entire simulation into the resulting affordances necessary in order to realize it. It discriminates sensory input based on affordance constraints set by the non projectable aspect of the mesh. This dichotomy of environmental input vs. mental input is seen in figure 2 as two types of perception: epistemic (scanning) or epimonic (analyzing/semiotic cognition). This is of course a continuum between the two, but for display purposes I have only drawn out two states. 

    To summarize, conceptualization, with respect to both linguistics and perception, is a process dealing with two levels of perceptual symbols. The bottom up level automatically activates a simulator based on feature detection. This simulator runs an unconscious simulation to ensure category membership. It does this by verifying the affordances based on the proprioceptive information contained in that simulator are applicable to the object or word in question. The higher order perceptual symbol level is activated based on the goal state of the individual. Normally, this level only contributes minimally to optimize cognition and sustain a reality-based conceptualization: projectable properties are clamped. However, if a problem arises where one has to interpret current sensory input in a novel way, one conducts a conscious mental simulation, activating relevant neural correlates, which subsequently activate a simulator. This simulator contains different affordances that are meshed with the bottom up level to the degree that the bottom up level is suppressed by concentration.

References:

Barsalou, Lawrence W. "Perceptual symbol systems ." Behavior and Brain Sciences,

1999.

Glenberg, Arthur M. "What memory is for." Behavioral and Brain Sciences, 1997: 1-

55.

Kosslyn, S.M., Thompson, W.L., Kim, I.J., & Alpert, N.M. (1995). Topographical

representations of mental images in primary visual cortex. Nature, 378, 496-

498.

Jeannerod, M. (1994). The representing brain: Neural correlates of motor intention

and imagery. Behavioral and Brain Sciences, 17, 187-245.