3.3 Informational Integration in the Visual System

In the fi rst chapter, I argued that the mere fact information is exchanged

between and among the various processes and systems of an organism does

not fully capture the nature of an organism as a hierarchically organized

system. The hierarchical nature of an organism suggests that the more

complex levels exhibit an amount of information control over less complex

levels. There seem to be advisory mechanisms that emerge from these

complex operations, and this would make sense, since the more complex

a process or system becomes, the more there is a need for mechanisms of

control so that these processes or systems can operate effi ciently (cf. Poggio

& Hurlbert, 1994; Johnson-Laird, 1988; Cziko, 1992, 1995). However, I

posited that this control is more than merely a selection of data for its

usefulness, or a segregating of useful data (information) from nonuseful

data. The more complex processes and systems in these hierarchies

not only select relevant data for their usefulness as information but also coherently integrate information in manifesting a control of that information

so as to respond effi ciently in some environment.

In the last section, I noted that the complex processing of information

in the nervous system seems to require the fourfold steps of (1) detecting

data in some environment, (2) discriminating between relevant and irrelevant

data, (3) integrating information, and (4) initiating some kind of

response. The third step in this process, namely, integrating information,

will be the focus of this section, as I show that integration is a key feature

of the visual system, especially when considering the relationship the

animal has to its external environment. True, the visual system detects and

then selects or segregates information; however, since selection alone

cannot account for how this information is organized for some purpose,

neural networks possess an ability to integrate the information so as to aid

the animal in optimally negotiating some environment. Once the information

has been selected, it must be organized in a coherent manner so that

an animal can go about the business of feeding, fi ghting, fl eeing, reproducing,

and the like in the most optimal and effi cient manner possible.

For example, through the visual unimodal association area, such integration

is made evident in the visual system’s ability to align shape and color

in the what system with distance and position in the where system so as

to visually process an approaching predator (Goodale et al., 1994; Goodale

& Murphy, 2000; Kosslyn & Koenig, 1995). Another example of this integration

is the ability of the higher areas of the visual system to extract a

coherent three-dimensional picture of a visual scene from two-dimensional

images on the retina (Zigmond et al., 1999). Other examples include the

integration of information specifying relations of depth among objects, as

well as the integration of information specifying the distance between a

perceiver and an object (Bruno & Cutting, 1988).

It would seem that any organized hierarchical system—including the

visual system—must come together part by part, with the separate parts,

at fi rst, functioning so as to solve a certain distinct problem. This is how

computer networks are built up from the fundamental ifs and thens or the

1s and 0s to the more complexly functioning Big Blues or World Wide

Webs (see Sperber, 1994; Johnson-Laird, 1988; Barto, 1985; Chellapilla &

Fogel, 2001; Copeland, 1993; Arp, 2007a, 2007b). This is also the way

natural/historical processes appear to work in evolutionary advances (see

Deacon, 1997; Berra, 1990; Gould, 1977; Dawkins, 1996). Thus, it is understood

by neurologists, philosophers, psychologists, and other thinkers that

the mammalian visual system is made up of parts, or brain-process modules,

that have been selected for in an evolutionary history to process color,

The Visual System 71

shape, depth, motion, the edges of objects, and the like (Goodale &

Murphy, 2000; Casagrande & Kaas, 1994; Edelman & Tononi, 2000; Shallice,

1997; Marr, 1983; Sereno et al., 1995). At the same time, such an

organized hierarchical system seems to have evolved advisory mechanisms

that can both segregate or select certain parts as relevant, as well as integrate

or bind relevant parts together, so as to adapt to an environment.

Thus, van Essen et al. (1994, p. 271) maintain that the “need for highly

fl exible linkages between a large number of physically separate modules”

requires a mechanism that controls and integrates the information gathered

from such modules.

It is important for an animal’s survival that it be able to select relevant

visual information about color, shape, distance, and the like from the

environment and then integrate that information so as to know whether

to fi ght, fl ee, eat, mate, and so forth. In other words, the recognition and

discrimination of objects is key to an animal’s survival. A question now

arises: How is it that the disparate pieces of selected data that have been

carried by separate pathways at the various levels of the visual hierarchy

are organized into a coherent visual perception, enabling object recognition

and/or discrimination? This is actually a kind of binding problem question,

of which there are probably many at the various neurobiological and

psychological levels of the visual system (Roskies, 1999; Gray, 1999).

Another way to frame the question is this: How is it that the parallel processing

of lines, shapes, forms, colors, motion, distance, depth, and the

like are combined in such a way as to yield the image of a particular object

in one’s visual fi eld, not of something else entirely? How is this information

coherently integrated or bound together so as to become informative

for the perceiver?

I suggest that this is possible through the phenomenon of visual modularity

and the mechanism of visual integration. When relevant visual areas are

bound together so as to make coherent sense out of some external stimuli

in terms of object recognition or discrimination, this bundle comprises the

integration of visual modules.

Visual modularity refers to the fact that the visual system is made up of

distinctly functioning and interacting modules, parts, or areas, having

evolved to respond to certain features of an object in typical environments.

A module, in this sense, is simply a brain process or brain system devoted

to some specifi ed task concerning object recognition and/or discrimination.

The concept of the module is nothing new and has been utilized by

neuroscientists, biologists, evolutionary psychologists, and other thinkers

for years (see Fodor, 1983, 1985; Bruno & Cutting, 1988; van Essen et al., 1994; Mithen, 1996; Gardner, 1993; Bear et al., 2001; Kandel et al., 2000;

Zigmond et al., 1999; Kosslyn & Koenig, 1995).

For example, we have noted already that the visual cortex and related

pathways are split up into many areas, each processing a different aspect

of the visual fi eld; V1 is responsible for initial visual processing, V2 for

stereo vision, V3 for distance, V4 for color, V5 for motion, and V6 for object

position. Each of these processes can be viewed as a module as Marr (1983)

makes clear in his famous work on vision. DeYoe et al. (1994, p. 151) have

shown that the blobs and interblobs of V1 and V2 in macaque monkeys

contain neurons with distinctive visual response properties suggesting, as

they call it, “modularity” and “multistream processing.” Also, Broca’s area

and Wernicke’s area would be considered as other examples of brainprocess

modules, since grammar–usage and language comprehension

appear to be localized in these areas, respectively (see Lueders et al., 1991;

Patterson & Wilson, 1987; cf. the new research of Petrides, Cadoret, &

Mackey, 2005). Further, the face-recognition area in IT cortex already

mentioned is another example of a brain-process module (Tovee &

Cohen-Tovee, 1993; Tovee, 1998; cf. the new research in Sinha, Balas,

Ostrovsky, & Russell, 2006).

The parallel processing associated with the visual system would be considered

as a suite of coordinated physiological or brain-process modules—

the information processed about color is one brain-process module, the

information about distance is another module, the information about form

still another module, and so forth. In fact, the very idea of parallel processing

entails modularity, since the processes are made up of components that

operate independently in completing tasks. The brain can be represented

as a host of modules, some of which are located in a single spot (like Broca’s

area for grammar–usage) and others of which are dispersed over the entire

cortex. Finally, these brain-process modules are viewed as nested within

hierarchies, and one can envision larger modules coordinating input from

smaller modules, which themselves collate neural processes from still

smaller neural bundles, and so forth.

The phenomenon of visual modularity works with the mechanism of

visual integration to produce a coherent visual perception. Visual integration

refers to a neurobiological process or set of processes that bind together

the relevant information gleaned from visual modules into a coherent,

cognitive representation of some object, enabling an organism to function

in typical environments. As Zigmond et al. (1999, p. 822) note, in the

visual system “high-level neurons classify visual stimuli by integrating

information that is present in the earlier stages of processing.”

What areas of the brain would be likely candidates for visual integration?

We know that the what and where visual unimodal systems are laid out

along trajectories from V1 in the occipital lobe to the temporal and parietal

regions, respectively. And we know that different aspects of an object—

color, form, distance, and the like—are processed along each one of these

trajectories. There appears to be some kind of integrating mechanism that

allows the primate to determine either what an object is or where an object

is that is present in each of these systems. Information about an object

from V1, V2, and V4 must be integrated somehow along the trajectory that

forms the what system; likewise, information about an object from V1, V2,

V3, V5, and V6 must be integrated somehow along the trajectory that

forms the where system. We can infer that integration of information is

taking place from the fact that if the what system is nonfunctioning, a

primate still may be able to distinguish where an object is; conversely, if

the where system is nonfunctioning, a primate still may be able to distinguish

what an object is (Goodale et al., 1994; Goodale & Murphy, 2000;

Desimone & Ungerleider, 1989; Ungerleider & Haxby, 1994). How would

an animal be able to determine, coherently, the what or the where of an

object independent of one another if the information from these areas was

not somehow integrated along the individual trajectories?

Further, the very concept of an association area implies an integrating

mechanism. Thus, it is likely that the visual unimodal association area of

the occiptotemporal cortex acts as the integrative mechanism for the information

processed from the what and where visual unimodal systems. This

area is involved in processing the information received from the parietal

and temporal unimodal areas concerning color, motion, depth, form, distance,

and the like. We know that there is a division of labor concerning

a primate’s abilities to distinguish what an object is from where an object

is. However, there are times when a primate must perform both of these

tasks and, given the neuronal projections from the parietal and temporal

areas to this common site in the occiptotemporal cortex, it makes sense

that a primate be able to integrate visual information about what and

where an object is in its visual fi eld at the same time. Kandel et al. (2000)

claim that these areas integrate information about form, color, and motion,

noting that their evidence comes directly from studies of humans who

have suffered brain injuries, experimental studies on monkeys, and radiological

imaging techniques of humans. Beason-Held et al. (1998) have

shown through PET scans that the occiptotemporal lobes are active in elementary

form perception in humans. Also, Honda et al. (1998) noted the

activation of these areas in PET scans when humans performed visuomotor

tasks in a matching-to-sample test where both the what and the where

systems were utilized.

Further, it is plausible to posit that the multimodal areas act as the

neuronal integrating mechanism for the information that is processed

through the highest level of sensory systems and those systems associated

with memory, attention, planning, judging, emotions, and motor

control. Kandel et al. (2000) name the prefrontal, parietotemporal, and

limbic cortices as the most likely neural candidates. Roberts, Robbins,

& Weiskrantz (1998), Uylings & van Eden (1990), and Rees & Lavine

(2001) point to these areas as primary integrating mechanisms for higher

level functions, including conscious awareness. Through PET scans,

Macaluso, Frith, & Driver (2000) have shown that the unimodal and

multimodal areas are active in tasks involving the utilization of both

the visual and somatosensory systems (also see Eimer & van Velzen,

2002; Calvert, 2001).

Consider a possible exchange between two chimps: chimp A has food,

and chimp B wants chimp A’s food. If chimp A is being approached by

chimp B, it must be able to visually judge space and shape (What is this

thing coming at me?), along with distance and size (Where is this thing

in relation to me?), as well as interpret the facial expressions of its

approacher. As we have seen already, the what and the where systems

follow trajectories from the visual cortex to the temporal/ventral and

parietal/dorsal areas respectively, and facial recognition has neural correlates

found in the IT cortex. All of this modular processing occurs in a

parallel fashion, by separate modular processes, as neuroscientists indicate

(e.g., Felleman & van Essen, 1991; Desimone et al., 1984; Crick, 1994).

When facial recognition, body position, and proximity are brought

to cognition—as when chimp A communicates to chimp B something

like “this is my food; don’t touch it or I’ll bite you”—there must be

an integration of this modular information so that the chimp can form

a coherent perception. Further, there are various sorts of stimuli coming

in through the other sensory modalities that must be integrated with

the visual system so that chimp A ultimately can initiate a response

in terms of either fi ghting, fl eeing, making friends, or making some

other response. The brains of our chimps bind together the various

modules of the visual system, as well as binding together the visual

system with other systems, while negotiating this exchange. The phenomenon

of visual modularity and the mechanism of visual integration

work together to explain how this exchange between these two chimps

is possible.

In the fi rst chapter, I argued that the mere fact information is exchanged

between and among the various processes and systems of an organism does

not fully capture the nature of an organism as a hierarchically organized

system. The hierarchical nature of an organism suggests that the more

complex levels exhibit an amount of information control over less complex

levels. There seem to be advisory mechanisms that emerge from these

complex operations, and this would make sense, since the more complex

a process or system becomes, the more there is a need for mechanisms of

control so that these processes or systems can operate effi ciently (cf. Poggio

& Hurlbert, 1994; Johnson-Laird, 1988; Cziko, 1992, 1995). However, I

posited that this control is more than merely a selection of data for its

usefulness, or a segregating of useful data (information) from nonuseful

data. The more complex processes and systems in these hierarchies

not only select relevant data for their usefulness as information but also coherently integrate information in manifesting a control of that information

so as to respond effi ciently in some environment.

In the last section, I noted that the complex processing of information

in the nervous system seems to require the fourfold steps of (1) detecting

data in some environment, (2) discriminating between relevant and irrelevant

data, (3) integrating information, and (4) initiating some kind of

response. The third step in this process, namely, integrating information,

will be the focus of this section, as I show that integration is a key feature

of the visual system, especially when considering the relationship the

animal has to its external environment. True, the visual system detects and

then selects or segregates information; however, since selection alone

cannot account for how this information is organized for some purpose,

neural networks possess an ability to integrate the information so as to aid

the animal in optimally negotiating some environment. Once the information

has been selected, it must be organized in a coherent manner so that

an animal can go about the business of feeding, fi ghting, fl eeing, reproducing,

and the like in the most optimal and effi cient manner possible.

For example, through the visual unimodal association area, such integration

is made evident in the visual system’s ability to align shape and color

in the what system with distance and position in the where system so as

to visually process an approaching predator (Goodale et al., 1994; Goodale

& Murphy, 2000; Kosslyn & Koenig, 1995). Another example of this integration

is the ability of the higher areas of the visual system to extract a

coherent three-dimensional picture of a visual scene from two-dimensional

images on the retina (Zigmond et al., 1999). Other examples include the

integration of information specifying relations of depth among objects, as

well as the integration of information specifying the distance between a

perceiver and an object (Bruno & Cutting, 1988).

It would seem that any organized hierarchical system—including the

visual system—must come together part by part, with the separate parts,

at fi rst, functioning so as to solve a certain distinct problem. This is how

computer networks are built up from the fundamental ifs and thens or the

1s and 0s to the more complexly functioning Big Blues or World Wide

Webs (see Sperber, 1994; Johnson-Laird, 1988; Barto, 1985; Chellapilla &

Fogel, 2001; Copeland, 1993; Arp, 2007a, 2007b). This is also the way

natural/historical processes appear to work in evolutionary advances (see

Deacon, 1997; Berra, 1990; Gould, 1977; Dawkins, 1996). Thus, it is understood

by neurologists, philosophers, psychologists, and other thinkers that

the mammalian visual system is made up of parts, or brain-process modules,

that have been selected for in an evolutionary history to process color,

The Visual System 71

shape, depth, motion, the edges of objects, and the like (Goodale &

Murphy, 2000; Casagrande & Kaas, 1994; Edelman & Tononi, 2000; Shallice,

1997; Marr, 1983; Sereno et al., 1995). At the same time, such an

organized hierarchical system seems to have evolved advisory mechanisms

that can both segregate or select certain parts as relevant, as well as integrate

or bind relevant parts together, so as to adapt to an environment.

Thus, van Essen et al. (1994, p. 271) maintain that the “need for highly

fl exible linkages between a large number of physically separate modules”

requires a mechanism that controls and integrates the information gathered

from such modules.

It is important for an animal’s survival that it be able to select relevant

visual information about color, shape, distance, and the like from the

environment and then integrate that information so as to know whether

to fi ght, fl ee, eat, mate, and so forth. In other words, the recognition and

discrimination of objects is key to an animal’s survival. A question now

arises: How is it that the disparate pieces of selected data that have been

carried by separate pathways at the various levels of the visual hierarchy

are organized into a coherent visual perception, enabling object recognition

and/or discrimination? This is actually a kind of binding problem question,

of which there are probably many at the various neurobiological and

psychological levels of the visual system (Roskies, 1999; Gray, 1999).

Another way to frame the question is this: How is it that the parallel processing

of lines, shapes, forms, colors, motion, distance, depth, and the

like are combined in such a way as to yield the image of a particular object

in one’s visual fi eld, not of something else entirely? How is this information

coherently integrated or bound together so as to become informative

for the perceiver?

I suggest that this is possible through the phenomenon of visual modularity

and the mechanism of visual integration. When relevant visual areas are

bound together so as to make coherent sense out of some external stimuli

in terms of object recognition or discrimination, this bundle comprises the

integration of visual modules.

Visual modularity refers to the fact that the visual system is made up of

distinctly functioning and interacting modules, parts, or areas, having

evolved to respond to certain features of an object in typical environments.

A module, in this sense, is simply a brain process or brain system devoted

to some specifi ed task concerning object recognition and/or discrimination.

The concept of the module is nothing new and has been utilized by

neuroscientists, biologists, evolutionary psychologists, and other thinkers

for years (see Fodor, 1983, 1985; Bruno & Cutting, 1988; van Essen et al., 1994; Mithen, 1996; Gardner, 1993; Bear et al., 2001; Kandel et al., 2000;

Zigmond et al., 1999; Kosslyn & Koenig, 1995).

For example, we have noted already that the visual cortex and related

pathways are split up into many areas, each processing a different aspect

of the visual fi eld; V1 is responsible for initial visual processing, V2 for

stereo vision, V3 for distance, V4 for color, V5 for motion, and V6 for object

position. Each of these processes can be viewed as a module as Marr (1983)

makes clear in his famous work on vision. DeYoe et al. (1994, p. 151) have

shown that the blobs and interblobs of V1 and V2 in macaque monkeys

contain neurons with distinctive visual response properties suggesting, as

they call it, “modularity” and “multistream processing.” Also, Broca’s area

and Wernicke’s area would be considered as other examples of brainprocess

modules, since grammar–usage and language comprehension

appear to be localized in these areas, respectively (see Lueders et al., 1991;

Patterson & Wilson, 1987; cf. the new research of Petrides, Cadoret, &

Mackey, 2005). Further, the face-recognition area in IT cortex already

mentioned is another example of a brain-process module (Tovee &

Cohen-Tovee, 1993; Tovee, 1998; cf. the new research in Sinha, Balas,

Ostrovsky, & Russell, 2006).

The parallel processing associated with the visual system would be considered

as a suite of coordinated physiological or brain-process modules—

the information processed about color is one brain-process module, the

information about distance is another module, the information about form

still another module, and so forth. In fact, the very idea of parallel processing

entails modularity, since the processes are made up of components that

operate independently in completing tasks. The brain can be represented

as a host of modules, some of which are located in a single spot (like Broca’s

area for grammar–usage) and others of which are dispersed over the entire

cortex. Finally, these brain-process modules are viewed as nested within

hierarchies, and one can envision larger modules coordinating input from

smaller modules, which themselves collate neural processes from still

smaller neural bundles, and so forth.

The phenomenon of visual modularity works with the mechanism of

visual integration to produce a coherent visual perception. Visual integration

refers to a neurobiological process or set of processes that bind together

the relevant information gleaned from visual modules into a coherent,

cognitive representation of some object, enabling an organism to function

in typical environments. As Zigmond et al. (1999, p. 822) note, in the

visual system “high-level neurons classify visual stimuli by integrating

information that is present in the earlier stages of processing.”

What areas of the brain would be likely candidates for visual integration?

We know that the what and where visual unimodal systems are laid out

along trajectories from V1 in the occipital lobe to the temporal and parietal

regions, respectively. And we know that different aspects of an object—

color, form, distance, and the like—are processed along each one of these

trajectories. There appears to be some kind of integrating mechanism that

allows the primate to determine either what an object is or where an object

is that is present in each of these systems. Information about an object

from V1, V2, and V4 must be integrated somehow along the trajectory that

forms the what system; likewise, information about an object from V1, V2,

V3, V5, and V6 must be integrated somehow along the trajectory that

forms the where system. We can infer that integration of information is

taking place from the fact that if the what system is nonfunctioning, a

primate still may be able to distinguish where an object is; conversely, if

the where system is nonfunctioning, a primate still may be able to distinguish

what an object is (Goodale et al., 1994; Goodale & Murphy, 2000;

Desimone & Ungerleider, 1989; Ungerleider & Haxby, 1994). How would

an animal be able to determine, coherently, the what or the where of an

object independent of one another if the information from these areas was

not somehow integrated along the individual trajectories?

Further, the very concept of an association area implies an integrating

mechanism. Thus, it is likely that the visual unimodal association area of

the occiptotemporal cortex acts as the integrative mechanism for the information

processed from the what and where visual unimodal systems. This

area is involved in processing the information received from the parietal

and temporal unimodal areas concerning color, motion, depth, form, distance,

and the like. We know that there is a division of labor concerning

a primate’s abilities to distinguish what an object is from where an object

is. However, there are times when a primate must perform both of these

tasks and, given the neuronal projections from the parietal and temporal

areas to this common site in the occiptotemporal cortex, it makes sense

that a primate be able to integrate visual information about what and

where an object is in its visual fi eld at the same time. Kandel et al. (2000)

claim that these areas integrate information about form, color, and motion,

noting that their evidence comes directly from studies of humans who

have suffered brain injuries, experimental studies on monkeys, and radiological

imaging techniques of humans. Beason-Held et al. (1998) have

shown through PET scans that the occiptotemporal lobes are active in elementary

form perception in humans. Also, Honda et al. (1998) noted the

activation of these areas in PET scans when humans performed visuomotor

tasks in a matching-to-sample test where both the what and the where

systems were utilized.

Further, it is plausible to posit that the multimodal areas act as the

neuronal integrating mechanism for the information that is processed

through the highest level of sensory systems and those systems associated

with memory, attention, planning, judging, emotions, and motor

control. Kandel et al. (2000) name the prefrontal, parietotemporal, and

limbic cortices as the most likely neural candidates. Roberts, Robbins,

& Weiskrantz (1998), Uylings & van Eden (1990), and Rees & Lavine

(2001) point to these areas as primary integrating mechanisms for higher

level functions, including conscious awareness. Through PET scans,

Macaluso, Frith, & Driver (2000) have shown that the unimodal and

multimodal areas are active in tasks involving the utilization of both

the visual and somatosensory systems (also see Eimer & van Velzen,

2002; Calvert, 2001).

Consider a possible exchange between two chimps: chimp A has food,

and chimp B wants chimp A’s food. If chimp A is being approached by

chimp B, it must be able to visually judge space and shape (What is this

thing coming at me?), along with distance and size (Where is this thing

in relation to me?), as well as interpret the facial expressions of its

approacher. As we have seen already, the what and the where systems

follow trajectories from the visual cortex to the temporal/ventral and

parietal/dorsal areas respectively, and facial recognition has neural correlates

found in the IT cortex. All of this modular processing occurs in a

parallel fashion, by separate modular processes, as neuroscientists indicate

(e.g., Felleman & van Essen, 1991; Desimone et al., 1984; Crick, 1994).

When facial recognition, body position, and proximity are brought

to cognition—as when chimp A communicates to chimp B something

like “this is my food; don’t touch it or I’ll bite you”—there must be

an integration of this modular information so that the chimp can form

a coherent perception. Further, there are various sorts of stimuli coming

in through the other sensory modalities that must be integrated with

the visual system so that chimp A ultimately can initiate a response

in terms of either fi ghting, fl eeing, making friends, or making some

other response. The brains of our chimps bind together the various

modules of the visual system, as well as binding together the visual

system with other systems, while negotiating this exchange. The phenomenon

of visual modularity and the mechanism of visual integration

work together to explain how this exchange between these two chimps

is possible.