Da Vinci, Fukushima, the aerial screw and the neocognitron
I saw a post by @juergen schmidhuber recently (link) that gave me pause. About convolutional neural networks and the neocognitron, a 1979-80 neural network architecture by the Japanese scientist Kunihiko Fukushima (link). It turns out I know a little bit about the Neocognitron and found it important enough conceptually to teach a course about it in 1991, in Stephen Grossberg’s department if anyone is keeping scores.
First, let me say that I think Jürgen is a brilliant scientist and creative who has been ahead of the curve many times, from LSTM networks to theories of creativity, discovery, subjective aesthetics and open-endedness. Whether he is the father (or “a” father) of generative AI is a paternity claim I don’t want to touch with a ten-foot pole, but it is true that recognition in science works through self-reinforcing social networks as much as (sometimes more than) merit. In his vocal campaign for the recognition of the “true” pioneers (including himself), he points to the seminal work of others, such as the brilliant Shun'ichi Amari or the quirky Fukushima, incidentally both born in 1936 and still kicking (see his [opinionated] annotated history of Deep Learning: link ).
And indeed, the neocognitron introduced core architectural principles such as layered feature detection, weight sharing, local receptive fields, downsampling and spatial pooling, that were directly inspired by the hierarchical organization of the visual cortex as elucidated by Hubel and Wiesel. Fukushima had also introduced ReLUs in 1969. That’s a lot of modern features of Deep Learning that were introduced almost 50 years ago.
So far so good, but my personal practical experience of the neocognitron is that it was quasi impossible to train. The winner-take-all rule and specific heuristics to determine feature detectors needed to be tuned to perfection... manually. Cell masks and training patterns also required careful manual selection and tweaking to achieve optimal performance, or any performance at all. But mostly, the absence of a well-defined training algorithm forced researchers to create various ad-hoc methods for different parts of the network or specific tasks, such as the "add-if-silent" rule or the "interpolating-vector" method, which added to the implementation complexity.
As a concept, the neocognitron reminds me of Leonardo Da Vinci’s “helicopter” (or aerial screw, pictured): a great architectural achievement (itself inspired by a Chinese toy) lacking the materials, power sources and steering and control mechanisms to ever fly. No one would dispute that Igor Sikorsky’s 1939 helicopter is really the first flying helicopter (pictured). Similarly, modern convolutional networks “fly”, thanks to increases in compute, data and algorithmic advances. Combining all these things together is what makes them work.
Perhaps being compared to Leonardo is not such a bad thing.
https://www.facebook.com/photo.php?fbid=2027767234262689&set=p.2027767234262689&type=3
https://nicofranz.art/en/leonardo-da-vinci/inventions
Annotated history of Deep Learning: link