2

Neural Networks

When I started at UC Berkeley in January 1986, the first thing I did was compile a
history of theories of intelligence and brain function. I read hundreds of papers by
anatomists, physiologists, philosophers, linguists, computer scientists, and
psychologists. Numerous people from many fields had written extensively about
thinking and intelligence. Each field had its own set of journals and each used its
own terminology. I found their descriptions inconsistent and incomplete. Linguists
talked of intelligence in terms such as "syntax" and "semantics." To them, the
brain and intelligence was all about language. Vision scientists referred to 2D,
2½D, and 3D sketches. To them, the brain and intelligence was all about visual
pattern recognition. Computer scientists talked of schemas and frames, new terms
they made up to represent knowledge. None of these people talked about the
structure of the brain and how it would implement any of their theories. On the
other hand, anatomists and neurophysiologists wrote extensively about the
structure of the brain and how neurons behave, but they mostly avoided any
attempt at large-scale theory. It was difficult and frustrating trying to make sense
of these various approaches and the mountain of experimental data that
accompanied them.

Around this time, a new and promising approach to thinking about intelligent
machines burst onto the scene. Neural networks had been around since the late
1960s in one form or another, but neural networks and the AI movement were
competitors, for both the dollars and the mind share of the agencies that fund
research. AI, the 800-pound gorilla in those days, actively squelched neural
network research. Neural network researchers were essentially blacklisted from
getting funding for several years. A few people continued to think about them
though, and in the mid-1980s their day in the sun had finally arrived. It is hard to
know exactly why there was a sudden interest in neural networks, but undoubtedly
one contributing factor was the continuing failure of artificial intelligence. People
were casting about for alternatives to AI and found one in artificial neural
networks.

Neural networks were a genuine improvement over the AI approach because their
architecture is based, though very loosely, on real nervous systems. Instead of
programming computers, neural network researchers, also known as
connectionists, were interested in learning what kinds of behaviors could be
exhibited by hooking a bunch of neurons together. Brains are made of neurons;
therefore, the brain is a neural network. That is a fact. The hope of connectionists
was that the elusive properties of intelligence would become clear by studying how
neurons interact, and that some of the problems that were unsolvable with AI
could be solved by replicating the correct connections between populations of
neurons. A neural network is unlike a computer in that it has no CPU and doesn't

18
store information in a centralized memory. The network's knowledge and
memories are distributed throughout its connectivity- just like real brains.

On the surface, neural networks seemed to be a great fit with my own interests.
But I quickly became disillusioned with the field. By this time I had formed an
opinion that three things were essential to understanding the brain. My first
criterion was the inclusion of time in brain function. Real brains process rapidly
changing streams of information. There is nothing static about the flow of
information into and out of the brain.

The second criterion was the importance of feedback. Neuroanatomists have
known for a long time that the brain is saturated with feedback connections. For
example, in the circuit between the neocortex and a lower structure called the
thalamus, connections going backward (toward the input) exceed the connections
going forward by almost a factor of ten! That is, for every fiber feeding information
forward into the neocortex, there are ten fibers feeding information back toward