BISC: BISC View Point Corner

From: masoud nikravesh (nikraves@eecs.berkeley.edu)
Date: Tue Jul 10 2001 - 22:44:03 MET DST

  • Next message: NESI: "IEEE Conf.on Software Maintenance Italy,Florence, ICSM2001"

    *********************************************************************
    Berkeley Initiative in Soft Computing (BISC)
    *********************************************************************

    Subject: Some fundamental questions about learning
       Date:
            Mon, 09 Jul 2001 12:24:15 -0400
       From:
            Paul Werbos <pwerbos@nsf.gov>
         To:
            "masoud nikravesh" <nikraves@EECS.Berkeley.EDU>
        CC:
            dwunsch@ece.umr.edu, LDOLM@erols.com, mdesai@nsf.gov

    In soft computing, intelligent control theory, and in "data mining,"
    there
    is a "simple" basic question which has been revisited again and again by

    many people:

    How can any system (brain or software...) learn to approximate a
    nonlinear
    mapping from a vector of inputs X to a vector of outputs Y, when given a

    database (fixed or real-time) of examples of X and Y?

    (One example: Shankar Shastri of Berkeley and his student Claire Tomlin
    of
    Stanford
    have done excellent work in "hybrid control" -- which ends up requiring
    a
    general-purpose
    nonlinear function approximator in the insides of the design. In fact,
    it
    is all quite close
    to what we have done with approximate dynamic programming or
    reinforcement
    learning...
    different words, different spins, but the same underlying mathemnatics.)

    In neural networks, we call this the "supervised learning task." In
    fuzzy
    logic, Jim Bezdek
    has called it "system identification." But in any case... one cannot
    build
    systems capable of
    brain-like decision capability without a subsystem that can perform that

    task (among others).
    Thus I would argue that no model of learning in the brain could capture
    the
    higher abilities
    of the brain, UNLESS it had enough richness to be able to handle this
    simple task.

    --------

    Here is my concern: in the last few years, there has been a certain
    amount
    of drifting apart
    between the computational neuroscience world and the world of
    computational
    intelligence.
    Many people believe this is just fine... but what if the consequence is
    that the neuroscience side
    is dominated by models which cannot possibly explain the basic learning
    capabilities of the system they
    are studying? Perhaps there is a great need for some new mathematical
    results which would explain
    more clearly what the problem is, and encourage more interest in the
    types
    of model which can solve it.

    (By the way, if anyone is interested... research on these lines would
    fit
    well as one of the many topics of
    great interest to what we fund in computational intelligence...)

    Moe precisely:

    Most people on this list probably know already that many types of ANN
    and
    fuzzy system are "universal approximators,"
    that they can learn any smooth mapping from X to Y.

    Some of you may know about the very important results of Andrew Barron
    (statistician at Yale), related to some results
    of Sontag of Rutgers: he proved, in effect, that some universal
    approximators are a lot better than others.
    There are lots of simple "smoothed lookup table" approximators which
    work
    fine for low dimensions...
    but the number of parameters or hidden units required grows
    exponentially
    with the number of inputs. But for multilayer perceptrons (MLPs)
    the growth in complexity is only polynomial. This is an incredibly
    important result. It says that MLPs may be viable
    for large (brain-like) induction problems, while those others are not.
    The
    usual theorems for fuzzy logic approximation
    and RBF approximation(and fuzzy ARTmap) are all based on some kind of
    linear basis function argument, or someting very close to it,
    which would imply an exponential growth in terms.

    Now: DO THOSE RESULTS show that MLPS trained with backprop can perform
    the
    basic task of approximating at least smooth
    nonlinear functions, and scaling up, while fuzzy and Hebbian systems
    cannot? (If virtually all models now used in
    computational neureoscience are of the Hebbian variety, continuous
    (graded)
    or discrete (spiking), this is serious...)

    Not quite.

    For example... when I think about Elastic Fuzzy Logic (ELF, first
    published
    by me in the IIzuka 1992 proceedings, pretty much equivalent to
    parts of some of the later designs of Yager and Fukuda)....

    it is clear that feedback to redefine the membership functions and so on

    can achieve a lot better, more parsimonious
    approximation ability than mere preset lookup tables! I would conjecture

    that ELF
    could also achieve Barron-like capability. And Cooper (of Nestor) has
    played similar games
    of tuning hidden units... long ago...

    BUT: to achieve all this, one needs a FEEDBACK to train/select those
    hidden
    units!

    Conjecture:

    One may define a general concept of H-locality (Hebb-locality), similar
    to
    some of the rules Grossberg has
    discussed, which prohibit both backpropagation and other similar types
    of
    feedback. The conjecture
    is that ANY system made of simple units, whose learning must obey
    H-locality, can never acheive
    Barron-like polynomial scaling in approximating smooth functions.

    To prove or disprove this would be of ENORMOUS scientific importance.

    Proving it... eliminates the main reason for not using GENERALIZED
    backpropagation more in neuroscience models...

    more capable neuroscience models would of course make it much more
    plausible for engineers to seriously consider trying to use/miomic such
    models
    in addressing difficult computational tasks.

    Best of luck.
      Paul W.

    --------------------------------------------------------------------
    If you ever want to remove yourself from this mailing list,
    you can send mail to <Majordomo@EECS.Berkeley.EDU> with the following
    command in the body of your email message:
    unsubscribe bisc-group
    or from another account,
    unsubscribe bisc-group <your_email_adress>

    ############################################################################
    This message was posted through the fuzzy mailing list.
    (1) To subscribe to this mailing list, send a message body of
    "SUB FUZZY-MAIL myFirstName mySurname" to listproc@dbai.tuwien.ac.at
    (2) To unsubscribe from this mailing list, send a message body of
    "UNSUB FUZZY-MAIL" or "UNSUB FUZZY-MAIL yoursubscription@email.address.com"
    to listproc@dbai.tuwien.ac.at
    (3) To reach the human who maintains the list, send mail to
    fuzzy-owner@dbai.tuwien.ac.at
    (4) WWW access and other information on Fuzzy Sets and Logic see
    http://www.dbai.tuwien.ac.at/ftp/mlowner/fuzzy-mail.info
    (5) WWW archive: http://www.dbai.tuwien.ac.at/marchives/fuzzy-mail/index.html



    This archive was generated by hypermail 2b30 : Tue Jul 10 2001 - 23:05:24 MET DST