Berkeley Initiative in Soft Computing (BISC)
Subject: Some fundamental questions about learning
Mon, 09 Jul 2001 12:24:15 -0400
Paul Werbos <email@example.com>
"masoud nikravesh" <nikraves@EECS.Berkeley.EDU>
firstname.lastname@example.org, LDOLM@erols.com, email@example.com
In soft computing, intelligent control theory, and in "data mining,"
is a "simple" basic question which has been revisited again and again by
How can any system (brain or software...) learn to approximate a
mapping from a vector of inputs X to a vector of outputs Y, when given a
database (fixed or real-time) of examples of X and Y?
(One example: Shankar Shastri of Berkeley and his student Claire Tomlin
have done excellent work in "hybrid control" -- which ends up requiring
nonlinear function approximator in the insides of the design. In fact,
is all quite close
to what we have done with approximate dynamic programming or
different words, different spins, but the same underlying mathemnatics.)
In neural networks, we call this the "supervised learning task." In
logic, Jim Bezdek
has called it "system identification." But in any case... one cannot
systems capable of
brain-like decision capability without a subsystem that can perform that
task (among others).
Thus I would argue that no model of learning in the brain could capture
of the brain, UNLESS it had enough richness to be able to handle this
Here is my concern: in the last few years, there has been a certain
of drifting apart
between the computational neuroscience world and the world of
Many people believe this is just fine... but what if the consequence is
that the neuroscience side
is dominated by models which cannot possibly explain the basic learning
capabilities of the system they
are studying? Perhaps there is a great need for some new mathematical
results which would explain
more clearly what the problem is, and encourage more interest in the
of model which can solve it.
(By the way, if anyone is interested... research on these lines would
well as one of the many topics of
great interest to what we fund in computational intelligence...)
Most people on this list probably know already that many types of ANN
fuzzy system are "universal approximators,"
that they can learn any smooth mapping from X to Y.
Some of you may know about the very important results of Andrew Barron
(statistician at Yale), related to some results
of Sontag of Rutgers: he proved, in effect, that some universal
approximators are a lot better than others.
There are lots of simple "smoothed lookup table" approximators which
fine for low dimensions...
but the number of parameters or hidden units required grows
with the number of inputs. But for multilayer perceptrons (MLPs)
the growth in complexity is only polynomial. This is an incredibly
important result. It says that MLPs may be viable
for large (brain-like) induction problems, while those others are not.
usual theorems for fuzzy logic approximation
and RBF approximation(and fuzzy ARTmap) are all based on some kind of
linear basis function argument, or someting very close to it,
which would imply an exponential growth in terms.
Now: DO THOSE RESULTS show that MLPS trained with backprop can perform
basic task of approximating at least smooth
nonlinear functions, and scaling up, while fuzzy and Hebbian systems
cannot? (If virtually all models now used in
computational neureoscience are of the Hebbian variety, continuous
or discrete (spiking), this is serious...)
For example... when I think about Elastic Fuzzy Logic (ELF, first
by me in the IIzuka 1992 proceedings, pretty much equivalent to
parts of some of the later designs of Yager and Fukuda)....
it is clear that feedback to redefine the membership functions and so on
can achieve a lot better, more parsimonious
approximation ability than mere preset lookup tables! I would conjecture
could also achieve Barron-like capability. And Cooper (of Nestor) has
played similar games
of tuning hidden units... long ago...
BUT: to achieve all this, one needs a FEEDBACK to train/select those
One may define a general concept of H-locality (Hebb-locality), similar
some of the rules Grossberg has
discussed, which prohibit both backpropagation and other similar types
feedback. The conjecture
is that ANY system made of simple units, whose learning must obey
H-locality, can never acheive
Barron-like polynomial scaling in approximating smooth functions.
To prove or disprove this would be of ENORMOUS scientific importance.
Proving it... eliminates the main reason for not using GENERALIZED
backpropagation more in neuroscience models...
more capable neuroscience models would of course make it much more
plausible for engineers to seriously consider trying to use/miomic such
in addressing difficult computational tasks.
Best of luck.
If you ever want to remove yourself from this mailing list,
you can send mail to <Majordomo@EECS.Berkeley.EDU> with the following
command in the body of your email message:
or from another account,
unsubscribe bisc-group <your_email_adress>
This message was posted through the fuzzy mailing list.
(1) To subscribe to this mailing list, send a message body of
"SUB FUZZY-MAIL myFirstName mySurname" to firstname.lastname@example.org
(2) To unsubscribe from this mailing list, send a message body of
"UNSUB FUZZY-MAIL" or "UNSUB FUZZY-MAIL email@example.com"
(3) To reach the human who maintains the list, send mail to
(4) WWW access and other information on Fuzzy Sets and Logic see
(5) WWW archive: http://www.dbai.tuwien.ac.at/marchives/fuzzy-mail/index.html
This archive was generated by hypermail 2b30 : Tue Jul 10 2001 - 23:05:24 MET DST