**Previous message:**masoud nikravesh: "BISC: EUNITE Workshop "Intelligent Systems in Patient Care""**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ]

*********************************************************************

Berkeley Initiative in Soft Computing (BISC)

*********************************************************************

Subject: Some fundamental questions about learning

Date:

Mon, 09 Jul 2001 12:24:15 -0400

From:

Paul Werbos <pwerbos@nsf.gov>

To:

"masoud nikravesh" <nikraves@EECS.Berkeley.EDU>

CC:

dwunsch@ece.umr.edu, LDOLM@erols.com, mdesai@nsf.gov

In soft computing, intelligent control theory, and in "data mining,"

there

is a "simple" basic question which has been revisited again and again by

many people:

How can any system (brain or software...) learn to approximate a

nonlinear

mapping from a vector of inputs X to a vector of outputs Y, when given a

database (fixed or real-time) of examples of X and Y?

(One example: Shankar Shastri of Berkeley and his student Claire Tomlin

of

Stanford

have done excellent work in "hybrid control" -- which ends up requiring

a

general-purpose

nonlinear function approximator in the insides of the design. In fact,

it

is all quite close

to what we have done with approximate dynamic programming or

reinforcement

learning...

different words, different spins, but the same underlying mathemnatics.)

In neural networks, we call this the "supervised learning task." In

fuzzy

logic, Jim Bezdek

has called it "system identification." But in any case... one cannot

build

systems capable of

brain-like decision capability without a subsystem that can perform that

task (among others).

Thus I would argue that no model of learning in the brain could capture

the

higher abilities

of the brain, UNLESS it had enough richness to be able to handle this

simple task.

--------

Here is my concern: in the last few years, there has been a certain

amount

of drifting apart

between the computational neuroscience world and the world of

computational

intelligence.

Many people believe this is just fine... but what if the consequence is

that the neuroscience side

is dominated by models which cannot possibly explain the basic learning

capabilities of the system they

are studying? Perhaps there is a great need for some new mathematical

results which would explain

more clearly what the problem is, and encourage more interest in the

types

of model which can solve it.

(By the way, if anyone is interested... research on these lines would

fit

well as one of the many topics of

great interest to what we fund in computational intelligence...)

Moe precisely:

Most people on this list probably know already that many types of ANN

and

fuzzy system are "universal approximators,"

that they can learn any smooth mapping from X to Y.

Some of you may know about the very important results of Andrew Barron

(statistician at Yale), related to some results

of Sontag of Rutgers: he proved, in effect, that some universal

approximators are a lot better than others.

There are lots of simple "smoothed lookup table" approximators which

work

fine for low dimensions...

but the number of parameters or hidden units required grows

exponentially

with the number of inputs. But for multilayer perceptrons (MLPs)

the growth in complexity is only polynomial. This is an incredibly

important result. It says that MLPs may be viable

for large (brain-like) induction problems, while those others are not.

The

usual theorems for fuzzy logic approximation

and RBF approximation(and fuzzy ARTmap) are all based on some kind of

linear basis function argument, or someting very close to it,

which would imply an exponential growth in terms.

Now: DO THOSE RESULTS show that MLPS trained with backprop can perform

the

basic task of approximating at least smooth

nonlinear functions, and scaling up, while fuzzy and Hebbian systems

cannot? (If virtually all models now used in

computational neureoscience are of the Hebbian variety, continuous

(graded)

or discrete (spiking), this is serious...)

Not quite.

For example... when I think about Elastic Fuzzy Logic (ELF, first

published

by me in the IIzuka 1992 proceedings, pretty much equivalent to

parts of some of the later designs of Yager and Fukuda)....

it is clear that feedback to redefine the membership functions and so on

can achieve a lot better, more parsimonious

approximation ability than mere preset lookup tables! I would conjecture

that ELF

could also achieve Barron-like capability. And Cooper (of Nestor) has

played similar games

of tuning hidden units... long ago...

BUT: to achieve all this, one needs a FEEDBACK to train/select those

hidden

units!

Conjecture:

One may define a general concept of H-locality (Hebb-locality), similar

to

some of the rules Grossberg has

discussed, which prohibit both backpropagation and other similar types

of

feedback. The conjecture

is that ANY system made of simple units, whose learning must obey

H-locality, can never acheive

Barron-like polynomial scaling in approximating smooth functions.

To prove or disprove this would be of ENORMOUS scientific importance.

Proving it... eliminates the main reason for not using GENERALIZED

backpropagation more in neuroscience models...

more capable neuroscience models would of course make it much more

plausible for engineers to seriously consider trying to use/miomic such

models

in addressing difficult computational tasks.

Best of luck.

Paul W.

--------------------------------------------------------------------

If you ever want to remove yourself from this mailing list,

you can send mail to <Majordomo@EECS.Berkeley.EDU> with the following

command in the body of your email message:

unsubscribe bisc-group

or from another account,

unsubscribe bisc-group <your_email_adress>

############################################################################

This message was posted through the fuzzy mailing list.

(1) To subscribe to this mailing list, send a message body of

"SUB FUZZY-MAIL myFirstName mySurname" to listproc@dbai.tuwien.ac.at

(2) To unsubscribe from this mailing list, send a message body of

"UNSUB FUZZY-MAIL" or "UNSUB FUZZY-MAIL yoursubscription@email.address.com"

to listproc@dbai.tuwien.ac.at

(3) To reach the human who maintains the list, send mail to

fuzzy-owner@dbai.tuwien.ac.at

(4) WWW access and other information on Fuzzy Sets and Logic see

http://www.dbai.tuwien.ac.at/ftp/mlowner/fuzzy-mail.info

(5) WWW archive: http://www.dbai.tuwien.ac.at/marchives/fuzzy-mail/index.html

**Next message:**NESI: "IEEE Conf.on Software Maintenance Italy,Florence, ICSM2001"**Previous message:**masoud nikravesh: "BISC: EUNITE Workshop "Intelligent Systems in Patient Care""**Messages sorted by:**[ date ] [ thread ] [ subject ] [ author ]

*
This archive was generated by hypermail 2b30
: Tue Jul 10 2001 - 23:05:24 MET DST
*