IT Assorti
blog by Vasyl Mylko
Monthly Archives: July 2017
AI Quintessence
It’s Deep
Today, we are trying Deep Neural Networks on many [previously] unsolved problems. Image and language recognition with CNNs and LSTMs has become a standard. Machines can classify images/speech/text faster, better and much longer than humans.
There is breakthrough in computer vision in real-time, capable to identify objects and object segments. That’s very impressive, it enables self-driving cars, and in-doors positioning without radio beacons or other infrastructure. The machine sees more than human, because the machine sees it all in 360 degrees. And the machine sees more details simultaneously; while human overlooks majority of them.
We created some new kind of intelligence, that is similar to human, but is very different from human. Let’s call this AI as Another Intelligence. The program is able to recognize and identify more than one billion human faces. This is not equivalent what humans are capable to do. How many people could you recognize/remember? Few thousands? Maybe several thousands? Less than ten thousands for sure (it’s the size of small town); so 1,000,000,000 vs. 10,000 is impressive, and definitely is another type of intelligence.
DNNs are loved and applied almost to any problem, even previously solved via different tools. In many cases DNNs outperform the previous tools. DNNs started to be a hammer, and the problems started to be the nails. In my opinion, there is overconfidence in the new tool, and it’s pretty deep. Maybe it slows us down on the way of reverse engineering the common sense, consciousness…
The Man Without Brain
DNNs were inspired by neuroscience, and we were confident that we were digitally recreating the brain. Here is cold shower – a man with a tiny brain – 10% size of the normal human brain. The man was considered normal by his relative and friends. He lived normal life. The issue was discovered accidentally, and it shocked medical professionals and scientists. There are hypothesis how to explain what we don’t understand.
There are other brain-related observations, that threaten the modern theory of brain understanding. Birds – some birds are pretty intelligent. Parrots, with tiny brains, could challenge dolphins, with human-sized brains, and some chimps. Bird’s brain is structured differently from the mammalian brain. Does size matter? Elephants have huge brain, with 3x more neurons than humans. Though the vast majority of those neurons are within different block of the brain, in comparison to humans.
All right, the structure of the brain matters more than the size of the brain. So are we using/modeling correct brain structure with DNNs?
The Structure And The Function
Numenta is working on reverse engineering the neocortex for a decade. Numenta’s machine intelligence technology is built on the own computational theory of the neocortex. It deals with hierarchical temporal memory (HTM), sparse distributed memory (SDM), sparse distributed representations (SDR), self-organizing maps (SOM). The network topologies are different from the mainstream deep perceptrons.
It’s fresh stuff from the scientific paper published in free frontiers magazine, check it out for the missing link between structure and function. “… remarkably intricate and previously unseen topology of synaptic connectivity. The synaptic network contains an abundance of cliques of neurons bound into cavities that guide the emergence of correlated activity. In response to stimuli, correlated activity binds synaptically connected neurons into functional cliques and cavities that evolve in a stereotypical sequence toward peak complexity. We propose that the brain processes stimuli by forming increasingly complex functional cliques and cavities.”
One-Shot Learning
When human is shown a new symbol, from previously unseen alphabet, it is usually enough to recognize the other such symbols, when shown again later. Even in the mix with other known and unknown symbols. When human is shown a new object, for the first time, like segway or hoverboard, it is enough to recognize all other future segways and hoverboards. It is called one-shot learning. You are given only one shot at something new, you understand that it is new for you, you remember it, you recognize it during all future shots. The training set consists of only one sample. One sample. One.
Check out this scientific paper on human concept learning with segway and zarc symbol. DNNs require millions and billions of training samples, while the learning is possible from the only one sample. Do we model our brain differently? Or are we building different intelligence, on the way of reverse-engineering our brain?
God Doesn’t Build In Straight Lines
These are two models of the same kart, created differently. On the left is human-designed model. On the right is machine-designed model within given restrictions and desired parameters (gathered via telemetry from the real kart from the track). It is paradigm shift, from constructed to grown. Many things in nature do grow, they have lifecycle. It’s true for the artificial things too. Grown model tends to be more efficient (lighter, stiffer, even visually more elegant), than constructed ones.
Take a look at those DNNs, including GoogLeNet and MS ResNet. On the left we have human-designed [constructed] models. Imagine what could be machine-generated [grown] on the right…
How to generate? Good start would be to use evolutionary programming, with known primitives for cells and layers. Though it is not easy to get it right. By evolving an imaginable creature, that moves to the left, right, ahead, back, it is easy to get asymmetrical blocks, handling the left and right. Even by running long evolutions, it could be hardly possible to achieve the desired symmetry, observed in the real world, and considered as common sense. E.g. the creature has very similar or identical ears, hands, feet. What to do to fix the evolution? To bring in the domain knowledge. When we know that left and right must be symmetrical, we could enforce this during the evolution.
The takeaway from this section – we are already using three approaches to AI programming simultaneously: domain rules, evolution and deep learning via backpropagation. Altogether. No one of them is not enough for the best possible end result. Actually, we even don’t know what the best result is possible. We are just building a piece of technology, for specific purposes.
The Master Algorithm
The above approach of using domain rules, evolution and deep learning via backpropagation altogether might not be capable to solve the one-shot learning problem. How that kind of problems could be solved? Maybe via Bayesian learning. Here is another paper on Bayesian Framework, that allows to learn something new from few samples. Together with Bayes we have four AI approaches. There is a work on AI, identifying five [tribes] of them.
The essense is in how to learn to learn. Without moving the design of AI to the level when AI learns to learn, we are designing throw-away pieces, like we did with Perl programming, like we do with Excel spreadsheets. Yes, we construct and train the networks, and then throw them away. They are not reusable, even if they are potentially reusable (like substituting the final layers for custom classification). Just observe what people are doing, they all are training from the very beginning. It is the level of learning, not the learning to learn – i.e. it’s throw-away level. People are reusable, they could train again; while networks are not reusable.
The Master Algorithm is the work, that appeals to the AI creators, who are open-minded to try to break through the next level of abstraction. To use multiple AI paradigms, in different combinations. It is design of design – you design how you will design the next AI thing, then apply that design to actually build it. Most probably good AI must be built with special combination of those approaches and tools within each of them. Listen to Pedro Domingos for his story, please. Grasp the AI quintessence.
IT Assorti 