You may have misunderstood what is meant by “stand-alone.” Read the section titled “The Cerf experiment” in my paper.

Asim

Asim –

Distributed representation does not “encompass any possible localist representation.” Distributed representation implies a pattern consisting of at least two units and none of them have any meaning on an individual basis. Here’s from Plate (2002) again:

“Researchers generally accept that a neural representation with the following two properties is a distributed representation (e.g., Hinton et al, 1986):
• Each concept (e.g., an entity, token, or value) is represented by more than one neuron (i.e., by a pattern of neural activity in which more than one neuron is active.)
• Each neuron participates in the representation of more than one concept. Another equivalent property is that in a distributed representation one cannot interpret the meaning of activity on a single neuron in isolation: the meaning of activity on any particular neuron is dependent on the activity in other neurons (Thorpe 1995).”

So distributed representation involves “more than one neuron” and localist representation involves just one neuron. Please take a look at the definition carefully.

References:

Plate T. (2002). Distributed representations. In: Encyclopedia of cognitive science. Nadel L, editor. Macmillan, London.

2. Shankar-K – “So, when a single neuron active in multiple tasks/contexts, I would consider it as a negative-evidence for that neuron denoting a localist representation, but is not a negative-evidence for distributed representation.”

Asim –

On the one hand, you admit in the beginning that conjunctive coding does not disprove localist representation: “I can agree with Asim that conjunctive coding is not necessarily evidence against localist representation.” On the other hand, now you say: “when a single neuron active in multiple tasks/contexts, I would consider it as a negative-evidence for that neuron denoting a localist representation.” This is going around in circles and is confusing. Conjunctive coding could involve multiple tasks (e.g. can indicate place, odor, time) and still be localist as long as its meaning does not depend on reading other cells.

3. Shankar-K – “The looseness of the definition of distributed representation makes it impossible to pick a counterexample to disprove it.”

Asim –

You keep on harping on the “looseness” of the definition of distributed representation. There is no “looseness” in the definition and there are plenty of counter examples in neurophysiology to disprove it. Here’s from Plate (2002) again on the definition:

“In distributed representations concepts are represented by patterns of activity over a collection of neurons. This contrasts with local representations, in which each neuron represents a single concept, and each concept is represented by a single neuron. Researchers generally accept that a neural representation with the following two properties is a distributed representation (e.g., Hinton et al, 1986):
• Each concept (e.g., an entity, token, or value) is represented by more than one neuron (i.e., by a pattern of neural activity in which more than one neuron is active.)
• Each neuron participates in the representation of more than one concept. Another equivalent property is that in a distributed representation one cannot interpret the meaning of activity on a single neuron in isolation: the meaning of activity on any particular neuron is dependent on the activity in other neurons (Thorpe 1995).”

Which part of this definition is “loose?” On counterexamples to disprove distributed representation, I have given you plenty last time.

References:

Plate T. (2002). Distributed representations. In: Encyclopedia of cognitive science. Nadel L, editor. Macmillan, London.

4. Shankar-K – “But that does not mean that the hypothesis of distributed representation in the entire brain is a good hypothesis, because a hypothesis that does not yield itself to clear testing amounts to not much utility.”

Asim –

I am assuming that this statement is based on your “looseness” argument. The “looseness” argument doesn’t hold. You have to cite literature to argue for “looseness.” You are trying to make distributed representation a vague concept. It is not. I can cite plenty of papers where the meaning of distributed representation is well-defined and concrete. So the above statement doesn’t make sense.

5. Shankar-K – “On the other hand, the definition of localist representation is much more stringent, and much more amenable for AI implementation, but it is much easier to find negative-evidence for it in the brain.”

Asim –

The definition of distributed representation is as stringent as the localist one and there is plenty of neurophysiological evidence for localist representation as cited in the paper.

6. Shankar-K – “Note that I’m not saying that there cannot exist localist representations in the brain, I’m just saying that it if you pinpoint a localist representation in the brain– a neuron with a standalone meaning–then there is a possibility of disproving that by getting that neuron activated in a different context that cannot be attributed the meaning you started with.”

Asim –

As a start, why don’t you take a look at retinal ganglion and visual cortex cells. They have been shown to be tuned for visual characteristics such as orientation, color, motion, shape and so on. And there are billions of those cells. Why don’t you try to show that they can have a different meaning in a different context.

7. Shankar-K – “I can agree with Asim that conjunctive coding is not necessarily evidence against localist representation. There is no reason why neurons from a localist representation should not be able learn to conjunctively code to enhance performance in specific tasks (Manns and Eichenbaum, 2009 is a good example Asim points out).”

Asim – Thanks.

8. Shankar-K – “But I’m very confused by Asim’s point about neurons having a “standalone meaning”. I don’t think anybody disagrees that there is significant evidence for neurons with highly selective receptive fields. But in order to attribute a standalone meaning to all those neurons, I think we need to make sure that they do not show activity in completely different contexts—that is, when the animals perform categorically different tasks. It is very possible that some neurons (like the orientation/color detecting cells) have this property and can be legitimately characterized to be a localist representation, but the activity of many others may be context dependent and should not be characterized as a local representation.”

Asim –

It is fine for a localist cell to be context dependent. There is nothing wrong with that. Place cells, for example, get remapped, but they still can have meaning and interpretation on a stand-alone basis because you don’t have to read other cells to interpret their meaning. Ekstrom et al. (2003) had epilepsy patients play a taxi driver computer game. They found cells in the hippocampus that responded to specific spatial locations, in the parahippocampal region that responded to views of specific landmarks (e.g. shops) and in the frontal and temporal lobes that responded to navigational goals. Note that these place, view and goal cells were created almost instantaneously as the patients learned how to play the game.

As long as a cell has meaning and interpretation on a stand-alone basis and its meaning does not depend on reading other cells, it is a localist cell. It doesn’t matter if its meaning changes from one context to another. There is nothing in the definition of localist representation that says that the meaning of a cell can’t change. There is substantial evidence for reuse of neural circuits, remapping of place cells being an example (see Anderson 2010). Reuse also means mapping additional functionality onto the same circuit.

References:

Anderson, M. L. (2010). Neural reuse: A fundamental organizational principle of the brain. Behavioral and Brain Sciences, 33(4), 245.

9. Shankar-K – “Asim clearly accepts remapping of place cells in different environments but insists that these cells have a meaning in a standalone basis, and that confuses me the most. How can that be? There has to be other cells that disambiguate the two environments, and the meaning of the place cells has to be conditional on the activity of those other cells… right?”

Asim –

I just explained this in Response 8 above. Look at the literature on place cells. Just ask the question: Are they reading other cells in order to interpret the meaning of a particular place cell?

10. Shankar-K – “Here I would like to point out that place-cells do more than just remap in different spatial environments; recent evidence shows that the place-cells can also be interpreted as time-cells (see for eg, Pastalkova Etal 2008, MacDonald Etal 2011) when the task set-up is changed. The same neurons that code for specific places while the rat performs a spatial navigation task, respond very differently when the task set up is changed. When the rat stays in a particular location (say running on a treadmill, or waiting to perform a memory task), those cells respond at different points in time—as though coding for time since the beginning of the trial. The information conveyed by a particular place cell is conditional to the specific environment and the specific task. I can certainly accept that the neuron has a meaning, but I can’t accept that it has a meaning is in a standalone basis. Even if we ascribe a meaning to a neuron in a standalone basis w.r.t a particular task, all we have to do to discard that meaning is to find a completely different task in which that neuron still responds, but for which that meaning will be invalid—like how it is invalid to interpret the neuron as a place cell when the animal performs a task from a fixed location (it makes more sense to interpret it as time-cells in those tasks).”

Asim –

There is substantial evidence for reuse of neural circuits, remapping of place cells being an example (see Anderson 2010). Reuse also means mapping additional functionality onto the same circuit. What you cite is something similar.

However, there is no conflict between neural reuse, in all its various manifestations, and localist representation. They still have meaning, as you described – “recent evidence shows that the place-cells can also be interpreted as time-cells..…The same neurons that code for specific places while the rat performs a spatial navigation task, respond very differently when the task set up is changed. When the rat stays in a particular location (say running on a treadmill, or waiting to perform a memory task), those cells respond at different points in time—as though coding for time since the beginning of the trial.” Did the interpretation “those cells respond at different points in time—as though coding for time since the beginning of the trial” depend on reading the activity of other cells? Of course not. And that’s what’s meant by “on a stand-alone” basis.

References:

Anderson, M. L. (2010). Neural reuse: A fundamental organizational principle of the brain. Behavioral and Brain Sciences, 33(4), 245.

But I’m very confused by Asim’s point about neurons having a “standalone meaning”. I don’t think anybody disagrees that there is significant evidence for neurons with highly selective receptive fields. But in order to attribute a standalone meaning to all those neurons, I think we need to make sure that they do not show activity in completely different contexts—that is, when the animals perform categorically different tasks. It is very possible that some neurons (like the orientation/color detecting cells) have this property and can be legitimately characterized to be a localist representation, but the activity of many others may be context dependent and should not be characterized as a local representation. And in this aspect, I agree with Hameroff that the brain most likely has both local and distributed representations.

Asim clearly accepts remapping of place cells in different environments but insists that these cells have a meaning in a standalone basis, and that confuses me the most. How can that be? There has to be other cells that disambiguate the two environments, and the meaning of the place cells has to be conditional on the activity of those other cells… right? Here I would like to point out that place-cells do more than just remap in different spatial environments; recent evidence shows that the place-cells can also be interpreted as time-cells (see for eg, Pastalkova Etal 2008, MacDonald Etal 2011) when the task set-up is changed. The same neurons that code for specific places while the rat performs a spatial navigation task, respond very differently when the task set up is changed. When the rat stays in a particular location (say running on a treadmill, or waiting to perform a memory task), those cells respond at different points in time—as though coding for time since the beginning of the trial. The information conveyed by a particular place cell is conditional to the specific environment and the specific task. I can certainly accept that the neuron has a meaning, but I can’t accept that it has a meaning is in a standalone basis. Even if we ascribe a meaning to a neuron in a standalone basis w.r.t a particular task, all we have to do to discard that meaning is to find a completely different task in which that neuron still responds, but for which that meaning will be invalid—like how it is invalid to interpret the neuron as a place cell when the animal performs a task from a fixed location (it makes more sense to interpret it as time-cells in those tasks).

When I previously said that the definition of distributed representation is loose when compared to the definition of localist representation, this is what I meant. In a distributed representation any task-relevant concept will be represented by a pattern distributed across a set of neurons—but there is no constraint on what that pattern should look like—some of the concepts can even be represented by single neurons. In some sense, the distributed representation encompasses any possible localist representation. So, when a single neuron active in multiple tasks/contexts, I would consider it as a negative-evidence for that neuron denoting a localist representation, but is not a negative-evidence for distributed representation. The looseness of the definition of distributed representation makes it impossible to pick a counterexample to disprove it. But that does not mean that the hypothesis of distributed representation in the entire brain is a good hypothesis, because a hypothesis that does not yield itself to clear testing amounts to not much utility. On the other hand, the definition of localist representation is much more stringent, and much more amenable for AI implementation, but it is much easier to find negative-evidence for it in the brain. Note that I’m not saying that there cannot exist localist representations in the brain, I’m just saying that it if you pinpoint a localist representation in the brain– a neuron with a standalone meaning–then there is a possibility of disproving that by getting that neuron activated in a different context that cannot be attributed the meaning you started with. So, as McClelland previously said, at least place-cells should not be considered as a localist representation.

To wrap up, it appears to me that my (and probably McClelland’s) disagreement with Asim might be purely at a semantic level, on what he means by “standalone”.

References:

1. Pastalkova, E., Itskov, V., Amarasingham, A., & Buzsaki, G. (2008). Internally gen- erated cell assembly sequences in the rat hippocampus. Science, 321(5894), 1322– 1327.

2. MacDonald, C. J., Lepage, K. O., Eden, U. T., & Eichenbaum, H. (2011). Hippocampal time cells bridge the gap in memory for discontiguous events. Neuron, 71(4), 737–749.

Regarding the interesting discussion between Asim Roy and Shankar-k
on distributed vs localist representation, let me say again I believe this
is a false dichotomy.

Both types of representation occur in the brain, implying a scale-invariant, 1/f, fractal or holographic-like representation, for which ample evidence exists.
Shankar-K on December 6, 2012 11:15 am
This conversation thread is about localist-vs-distributed representation of information at the level of neurons. We should probably not hijack this conversation thread by further discussing the utility and/or possibility of Hameroff’s sub-neural (possibly quantum) accounts of cognition here. These are crucial issues, but they need a separate thread. So I am going to privately respond to Dr.Hameroff in an email and he can start a separate thread of conversation if he finds it worth following.
Stuart
Sub-neural, possibly quantum processes are very much relevant to the debate about localist-vs-distributed representations (false dichotomy it may be). But I accept the terms of your surrender. I think I’ve made my point that A.I. approaches (those intended to reproduce essential brain functions) based exclusively on neuronal membrane activities are bogus.

Regarding the overall relevance of this thread to the Lifeboat Foundation mission, if everything goes to hell we may need to resort to downloading consciousness into some artificial medium, and for that we need to understand the mechanism by which consciousness occurs. Current A.I. approaches won’t work.

Asim –
Given your criterion for proving a theory, I guess the same should hold the other way around. There’s probably more than one example to disprove distributed representation. Hameroff provides two good examples of single cell creatures. And there probably are billions of such single cell creatures. And there’s more than four decades of research on receptive fields being tuned for particular functions like detection of line orientation, motion, color and so on and two guys won the Nobel Prize for discovering this “secret code.” And there probably are trillions of such receptive fields in all kinds of living things. That’s more than one example to disprove distributed representation.

2. Shankar-K – “The distributed representation is a very loose hypothesis, so it is very difficult to disprove it.”

Asim –
In what way is it a loose hypothesis that it’s difficult to disprove? Here’s a definition from Plate (2002):

“In distributed representations concepts are represented by patterns of activity over a collection of neurons. This contrasts with local representations, in which each neuron represents a single concept, and each concept is represented by a single neuron. Researchers generally accept that a neural representation with the following two properties is a distributed representation (e.g., Hinton et al, 1986):
• Each concept (e.g., an entity, token, or value) is represented by more than one neuron (i.e., by a pattern of neural activity in which more than one neuron is active.)
• Each neuron participates in the representation of more than one concept. Another equivalent property is that in a distributed representation one cannot interpret the meaning of activity on a single neuron in isolation: the meaning of activity on any particular neuron is dependent on the activity in other neurons (Thorpe 1995).”

Which part is “loose” in the distributed representation theory?

References:

Plate T. (2002). Distributed representations. In: Encyclopedia of cognitive science. Nadel L, editor. Macmillan, London.

Thorpe S. (1995). Localized versus distributed representations. In The Handbook of brain theory and neural networks, ed Arbib. MIT Press, Cambridge.

3. Shankar-K – “But Asim has given a rather tight definition for localist representation, and hence is more vulnerable to disproof.”

Asim –
I have not invented a new definition for localist representation. It’s an existing and well-understood definition from cognitive science. Here are a few more citations on the difference between the two representation schemes:

• Thorpe (1995, p. 550): “With a local representation, activity in individual units can be interpreted directly … with distributed coding individual units cannot be interpreted without knowing the state of other units in the network.”

• Elman (1995, p. 210): “These representations are distributed, which typically has the consequence that interpretable information cannot be obtained by examining activity of single hidden units.”

References:

Elman, J. (1995). Language as a dynamical system. In R. Port & T. van Gelder (Eds.), Mind as motion: Explorations in the dynamics of cognition (195–223). MIT Press.

4. Shankar-K – “Theoretically, it appears easy to disprove the localist interpretation, because there exists cells conjunctively coding for stimuli across radically different categories (like place and odor).”

Asim –
Conjunctive coding doesn’t disprove localist theory. Conjunctively coded cells are not devoid of “meaning and interpretation” on a stand-alone basis. They actually add to the evidence for localist theory.

For example, Quian Quiroga et al. (2009) found a neuron in the entorhinal cortex of a subject that responded (p. 1308) “selectively to pictures of Saddam Hussein as well as to the text ‘Saddam Hussein’ and his name pronounced by the computer….. There were no responses to other pictures, texts, or sounds.” They call these neurons “triple invariant” ones — they were those that had the visual invariance property and also had significant response to spoken and written names of the same person. But it’s an example of conjunctively coded cell that has “meaning and interpretation” on a stand-alone basis.

Here’s from a recent paper by Barry and Doeller (2010):

“While position is the most distinct correlate of place cell activity, a parallel body of work suggests how other factors might be encoded in addition to, or possibly in place of, the spatial code. Two examples are provided by Wood et al. (1999) and more recently Manns and Eichenbaum (2009). In the former case, rats moved around an open enclosure to perform a delayed-non-match-to-sample task, and in the latter, animals circled an annular maze encountering objects placed onto the track. Under these conditions, many hippocampal pyramidal cells encode nonspatial cues (e.g., presence of a specific odor) in addition to a primary spatial correlate (Manns and Eichenbaum, 2009). These cells exhibit conjunctive properties, for example, responding optimally to a particular combination of position and odor. Similar results have been observed when auditory fear conditioning was conducted while rats freely perambulated: place cells retained their spatial firing but at the same time firing became synchronized to the audible conditioned stimulus (Moita et al., 2003). Also, hippocampal recordings made from humans (patients with pharmacologically intractable epilepsy were asked to navigate in virtual reality) revealed that approximately a quarter of the cells characterized as place cells had conjunctive representations and were modulated by the subject’s destination (Ekstrom et al., 2003).”

Conjunctive cells have meaning and interpretation too and on a stand-alone basis. The meaning and interpretation is not dependent on reading the activity of other cells.

References:

Quian Quiroga, R., Kraskov, A., Koch, C., & Fried, I. (2009). Explicit Encoding of Multimodal Percepts by Single Neurons in the Human Brain. Current Biology, 19, 1308–1313.

Barry, C., & Doeller, C. F. (2010). Conjunctive representations in the hippocampus: what and where?. The Journal of Neuroscience, 30(3), 799–801.

5. Shankar-K – “May be we do not have to think of across-category-conjunctive coding as evidence against localist representation. However for that, the entire modeling perspective has to be changed—does conjunctive coding learned to enhance output performance or does it happen randomly for no reason? The former would support the localist representation and the latter would support distributed representation.”

Asim –
I am not sure what the question or the issue is. As far as I know, conjunctive coding doesn’t “happen randomly for no reason.” You have to cite the literature where there is evidence for the “no reason” case. And conjunctive coding, whatever the reason for its occurrence, does not support distributed representation. See Response 4 above.

6. Shankar-K – “What would happen if for example, the “t”-node in the letter-layer also gets activated when you smell an orange. Does the node have the meaning of the letter “t” or the smell of orange. We need extra information to infer its meaning. Remember that most place cells also responds to particular odors, and there are many papers that classify cells as conjunctive place-odor cells.”

Asim –
I have answered the conjunctive coding issue in Reponses 4 and 5 above.

7. Shankar-K – “If a cell “C” is a place cell that fires at a position x1 in an environment E1 and fires at a position x2 in an environment E2, how can C have a standalone meaning?”

Asim –
There is remapping of place cells. Perhaps you are referring to that phenomenon. Here’s again a citation from Barry and Doeller (2010):

“In the open environments that many experimenters prefer, and in the absence of particular task demands, activity is independent of the animal’s orientation, stable between visits to the same position even across days, and robust to the removal of individual spatial cues (O’Keefe, 1976). Transportation of the animal to a different and sufficiently distinct enclosure typically results in a new representation being established: place fields change position relative to one another and radically alter firing rates. Remapping, as this effect is known, has been understood as a process by which the hippocampus generates independent codes to represent distinct spatial contexts (Wills et al., 2005). Far from being a uniquely rodent curiosity, place cells seem to be important to the wider function of the hippocampus, because cells with similar properties have been identified in a range of animals as disparate as birds, monkeys, and humans (Ekstrom et al., 2003).”

Place cells still have “meaning and interpretation” on a stand-alone basis when in a different location. Its meaning is not dependent on reading other cells.

• Shankar-K on December 6, 2012 6:10 am
Hameroff makes an important point that there are many levels of representations lower than neural spikes that could be responsible for cognition, and researchers are not paying attention there. True.

Stuart
Actually I am making two points. 1) There is a lower level responsible for cognition inside neurons in microtubules, and 2) dendritic integration is far more relevant to cognition (and consciousness) than spikes.

Shankar-K
But we have not yet sufficiently explored the possibility of explaining cognition at the level of neurons, so going to deeper levels might result in heavy/unwarranted theoretical speculations as in String theory.

Stuart
I am no fan of string theory, but using it as a bogey-man to fend off consideration of intra-neuronal processes is absurd. The deeper levels to which I am specifically referring are microtubules (including possible quantum computing in microtubules).

Shankar-k
We have to be extremely careful here because the hypothesis space would be so enormous that we will not be able to test out any theory.

Stuart
The only hypothesis space I am suggesting involves quantum and classical information processing in microtubules. I have published 20 testable predictions of the Penrose-Hameroff Orch OR theory, a number of which have been validated. I am still waiting for any testable prediction of cognition at the level, exclusively, of neuronal membranes (you say at the level of neurons, but neurons are full of microtubules). You can’t explain synaptic membrane plasticity without microtubules (because synaptic membrane proteins are short-lived and memories last lifetimes).

Shankar-k
For amoeba and paramecium, we don’t have a choice but to resort to the molecular level of theory, because the hypothesis set is naturally bounded above at the single-cell level.

Stuart
Not just molecular (which implies soluble diffusion mechanisms). Microtubules are solid-state devices with information processing capabilities. Do you really think evolution would abandon intra-cellular cognition?

Shankar-k
But the important point is that the biophysicists and chemists are modeling these only at the molecular level, and not attempting to go lower to the quarks-level. It is not a good idea to abandon hypotheses from a higher/simpler level when it is not completely explored.

Stuart
You don’t have to abandon anything. But face reality, after 60 years of neuronal membrane-level investigations, cognition and consciousness remain mysterious. Do you even have a single testable prediction for neuronal membrane-based cognition without internal microtubules?

For amoeba and paramecium, we don’t have a choice but to resort to the molecular level of theory, because the hypothesis set is naturally bounded above at the single-cell level. But the important point is that the biophysicists and chemists are modeling these only at the molecular level, and not attempting to go lower to the quarks-level. It is not a good idea to abandon hypotheses from a higher/simpler level when it is not completely explored.

I think Asim used a good example of Maclelland’s IA model in the paper to illustrate the similarities of the localist and distributed representation. Let me try to use the same example to point out the main distinction between the two representations. The node corresponding to the letter “t” in the letter-layer will be active for many different nodes in the word-layer (say “trip” or “time” or …). Asim points out that each node in the letter-layer has a precise meaning—the letter itself. But this is possible only when we have the a priori information that we are looking at the letter-category. What would happen if for example, the “t”-node in the letter-layer also gets activated when you smell an orange. Does the node have the meaning of the letter “t” or the smell of orange. We need extra information to infer its meaning. Remember that most place cells also responds to particular odors, and there are many papers that classify cells as conjunctive place-odor cells.

We need to remember that to prove a theory we need infinite examples but to disprove it, we just need one. The distributed representation is a very loose hypothesis, so it is very difficult to disprove it. But Asim has given a rather tight definition for localist representation, and hence is more vulnerable to disproof. Theoretically, it appears easy to disprove the localist interpretation, because there exists cells conjunctively coding for stimuli across radically different categories (like place and odor).

May be we do not have to think of across-category-conjunctive coding as evidence against localist representation. However for that, the entire modeling perspective has to be changed—does conjunctive coding learned to enhance output performance or does it happen randomly for no reason? The former would support the localist representation and the latter would support distributed representation.