June brought record-breaking temperatures, perfectly highlighting the global challenge of climate change. Is that AI-related news? Check and see in the latest AI Monthly Digest.

A common misconception about machine learning projects is that they are by definition big.  However, any number of AI-powered micro-tweaks and improvements are applied in everyday work. A good example of both micro and macro tweaks that can fix a major problem can be found in the paper described below.

AI tackling climate change

The world witnessed an extraordinarily hot June, with average temperatures 2 degrees celsius above normal in Europe. According to the World Meteorological Organization, the heatwave is consistent with predictions based on greenhouse gas concentrations and human-induced climate change.

Tackling this challenge will not be easy: according to World Bank Data, fossil fuel energy consumption still stacks to 79% of total. Furthermore, greenhouse gasses, particularly methane, are emitted by cattle, with livestock being responsible of 14.5% of total human-induced greenhouse emissions.

The most prominent figures in AI today, including DeepMind CEO Demis Hassabis, Turing award winner Yoshua Bengio, and Google Brain co-founder Andrew Ng, have authored a comprehensive paper on ways that AI can tackle the changing climate.

Their call for collaboration is meant to inspire practitioners, engineers and investors to deliver short- and long-term solutions for measures within our reach. Those include producing low-carbon electricity through better forecasting, scheduling, and control for variable sources of energy, mitigating the damage produced by high-carbon economies through, for example, better predictive maintenance as well as help minimize energy use in transportation, smart buildings and cities. The applications can vary from designing grid-wide control systems or optimizing scheduling with more accurate demand forecasting.

Why does it matter

Climate change is one of the greatest challenges mankind faces today, with truly cataclysmic scenarios approaching. Further temperature increases may lead to a variety of disasters, from flooding coastal regions due to melting ice caps, agricultural crises and conflicts over access to water.

Green energy promises solutions, yet these are not without their challenges, many of which could be solved with machine learning, deep learning or reinforcement learning. Responsibility is among deepsense.ai’s most important AI trends, and being responsible for the planet would be an excellent example of just why we chose to focus on that trend.

We will provide more in-depth content on climate change and AI-powered ways of tackling it. So stay tuned!

Giants racing to produce the best image recognition

If machine learning is today’s equivalent of the steam engine revolution, data and hardware are the coal and engine that power the machines. Facebook and Google are like the coal mines of yesteryear, having access to large amounts of fuel and power to build new models and experiment.

It should come as no surprise that breakthroughs are usually powered by the tech giants. Google’s state of the art in image recognition, EfficientNet, has been a recent giant step forward. The model was delivered by automated searching procedure uniformly scaling each dimension of the network in order to find the best combination.

EfficientNet stands for something.

The result is state-of-the-art in Image recognition. At least when it comes to combining efficiency and accuracy. But not when it comes to accuracy alone.

Not even a month later Facebook delivered a model that outperformed Google’s. The key lay in scaling the enormous dataset it was trained on. The social media mogul has access to Instagram’s database, which holds no less than billions of user-tagged images, a dataset ready to be chewed over by a hungry deep learning model.

The neural network was released to the public using a recently launched Pytorch Hub platform for sharing cutting edge models.

Why does it matter

Both advances show how important machine learning is for the tech giants and how much effort they invest in pushing their research forward. Every advancement in image recognition brings new breakthroughs closer. For example, models are becoming more accurate in detecting diabetic retinopathy using images of the eye. Every further development delivers new ways to solve problems that would be unsolvable without ML (Machine learning) – manufacturing for visual quality control is among the best examples.

XLNet outperforms BERT

As we noted in a past AI Monthly Digest, Google has released Bidirectional Encoder Representation from Transformations (BERT). BERT was, until recently, the state-of-the-art when it comes to Natural Language Processing benchmarks. The newly announced XLNet is an autoregressive pretraining method (as opposed to an autoencoder-like BERT) which learns a language model by predicting the next word in a sequence using the permutation of all the surrounding words. An intuitive explanation can be found (here).

The XLNet model proved more effective than BERT in beating all 20 benchmark tasks.

Why does it matter

Understanding a natural language was considered a benchmark for intelligence, with Alan Turing’s test being among the best examples. Every push forward delivers new possibilities in building new products and solving problems, be they business ones or something more uncommon, like the example below.

AI-powered archeology? Bring it on!

Deep learning-based models are getting even better at understanding natural language. But what about language that is natural, but has never been deciphered due to lack of knowledge or a frustratingly small amount of extant text?

Recent research from MIT and Google shows that a machine learning approach can deliver major improvements in deciphering ancient texts. In the basics of modern natural language processing techniques, all of the words in a given text are assumed to be related to each other. The machine itself doesn’t “understand” text it in a human way, but rather forms its own assumptions based on the relations and connotations of each word in a sentence.

Disc of Phaistos, one of the most famous mysteries of archaeology

In this approach, the translation process is not built on understanding the world, but rather finding similarly connotated words that transfer the same message. This is entirely different than humans’ approach to language.

By making the algorithm less data-hungry, the researchers deliver a model that translates texts from rare and long-lost languages. The approach is described in this paper.

Why does it matter

While there are countless examples of machine learning in business, there are also new horizons to discover in the humanities. Deciphering the secrets of the past is every bit as exciting as building defenses against the challenges of the future.

The more sophisticated approach to and possible brute-force breaking of unknown languages provides a way to uncover more language-related secrets.

A Disc of Phaistos? Or a Voynich manuscript maybe?

February came with the groundbreaking event of building a new state-of-the-art natural language processing neural network. And then the unthinkable happened.

Machine learning has already shown that it is a world-transforming technology used in various appliances, including drug discovery, saving endangered species and designing software for sophisticated prosthetic legs. Therefore it is natural that this should spark debates, both ethical and business-related. What reaction has there been to this in February?

1. Open-AI designed a new gold standard for natural language processing…

While in image processing artificial intelligence models outperform humans in most tasks, natural language processing (NLP) is still a challenge for machines to master. Although AI-based services are already usable with Google Translate being the pinnacle of today’s achievements, the texts produced by machines is easy to recognize by humans. But this could change soon with the latest state-of-the-art from OpenAI.

They designed a model called GPT-2 and prepared a dataset containing 8 million web pages for training. The network objective is to predict the next word given all the words to that point, which is the simplest way of doing unsupervised learning in NLP. The main improvements, in this case, are scaling up the model size (1.5B parameters) and training it on the gargantuan database of Internet text (40GB) at an unprecedented scale (32 x TPUv3).

The resulting network analyzes the starting word of a sentence and then it adds the next words to create a text based on “the most probable output”. The effect is surprisingly accurate. The model is able to recognize the type of the starting text. If the first sentence is a press title it will produce a legitimate-sounding short news article. It is also good at adopting various styles – something unimaginable for previously sturdy models which could produce problematic sentences.

Example:

Human written:

In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.

Model completion (machine-written, 10 TRIES)

The scientist named the population, after their distinctive horn, Ovid’s Unicorn. These four-horned, silver-white unicorns were previously unknown to science.

Now, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved.

Dr. Jorge Pérez, an evolutionary biologist from the University of La Paz, and several companions, were exploring the Andes Mountains when they found a small valley, with no other animals or humans. Pérez noticed that the valley had what appeared to be a natural fountain, surrounded by two peaks of rock and silver snow.

Pérez and the others then ventured further into the valley. “By the time we reached the top of one peak, the water looked blue, with some crystals on top,” said Pérez.

Pérez and his friends were astonished to see the unicorn herd. These creatures could be seen from the air without having to move too much to see them – they were so close they could touch their horns.

While examining these bizarre creatures the scientists discovered that the creatures also spoke some fairly regular English. Pérez stated, “We can see, for example, that they have a common ‘language,’ something like a dialect or dialectic.”

More examples of AI-generated texts may be found at Open-AI blog.

Moreover – the network is capable of transfer learning – it is easier to perform a slight additional training to enable the model to perform more sophisticated tasks. Transfer learning is a gateway technology for modern image recognition appliances, including art recognition and visual quality control.

Why does it matter

Alan Turing, the godfather of modern information technology, was heavily convinced that the ability to understand language is a key indicator of intelligence. Natural language, the emotions hidden beneath the words and all the cultural and societal contexts behind sentences are truly unique for human beings.

Making this part of the world approachable for machines is undoubtedly the great breakthrough and passes countless possibilities, from battling hate crime to various business appliances.

2. … and they didn’t release it to the public

With great power comes great responsibility and the events from February 2019 show it clearly. Despite best practices and established custom, Open-AI decided NOT TO release the state-of-the-art neural network to the public as an open source software.

The organization did this to prevent the neural network from being used for malicious purposes, with perpetuating fake news as a top concern. The decision sparked a debate on Reddit, Twitter and other various media, where participants argued on the safety of releasing this “potentially dangerous technology” and tackling the established model of open-sourcing the effect of research. Researchers are afraid of building the doorway for “deepfakes for texts”.

Why does it matter

AI ethics and responsibility was highlighted as one of the major AI trends 2019. The situation where researchers choose safety over progress is unusual. On the other hand, fake news is considered one of the major threats of the future and was recognized one of the most dangerous online activities in 2018 by the World Economic Forum.

With the rise of autonomous cars and IoT revolution building a frame for sophisticated AI-powered solutions to work, such dilemmas may get increasingly common.

3. China’s AI strategy at a glance (or even more)

In the last AI monthly digest, we covered the AI strategy forged in Finland, where the national pursuit for the AI appliances grew from a grassroots movement. In contrast, China has a centralized strategy to make the country AI-giant.

The Center for a New American Security (CNAS) has published a report in which they provide deeper insights into understanding China’s AI strategy. The report is a long read written by Gregory C. Allen who had a chance to meet a few times with high-ranking Chinese officials on conferences focusing on Artificial Intelligence.

Topics covered in the text:

• Chinese views on importance and security of AI
• Strengths and weaknesses of China’s AI Ecosystem
• China’s short-term goals in AI
• The role of semiconductors

The author concludes that gaining expertise in and understanding of AI developments in China should help U.S. policymakers sort out their priorities. Instead of influencing China’s competitiveness, they should focus on boosting the technological and economic competitiveness of the United States.

Why does it matter

Early adopters are usually those who earn knowledge with first failures. Thus, for now, it is wise to observe and learn.

4. Introducing Paperswithcode

Staying updated on the latest developments in machine learning is not an easy task, especially considering the fact that not all the researchers decide to make their code available. Releasing the code is a good practice, but providing only the paper that “describes the matter enough” to reproduce the effect is also correct.

Using the paperswithcode makes the process of paper-and-code matching much easier, effectively saving a lot of time for people willing to broaden their knowledge.

Why does it matter

The portal itself is a handy tool to gain knowledge and, a bit unwillingly promotes the constituted approach of making the research code public. If it gains notoriety, it may be a tool of social pressure to publish code with the research.

5. Popularity of thispersondoesnotexist.com

The neural networks’ ability to create convincing human faces is one of the benchmarks for modern AI appliances. To make the images available for the public, the StyleGan-based model publishes random face on thispersondoesnotexist.com website.

As in every aspect of life, the devil is in the details. The website is a great tool to spot the weakest parts of generated faces.

• Hair – it is common to place some hair in the air, with no attachment to other or floating in an unnatural way. Sometimes the skin around hair is “sticking” to them, creating some kind of scars that are disturbing when seen. The model sometimes messes up hairstyles, mixing dreadlocks with straight and curly hair, but it is more challenging to spot
• Teeth – algorithms tend to mess up with teeth, placing them unnaturally, sometimes merged or blurred.
• Glasses – it is common that the eyes inside the glasses do not fit the rest of the depicted person. Sometimes it means placing eyes of an old woman, surrounded with wrinkles, on a child’s face.

Why does it matter

Making these images publicly available with commentary, which all have been made by neural networks, is an interesting way to spread knowledge about AI-related topics. The concerns about deep fakes and other illicit content created with the support of artificial intelligence are mostly based on lack of knowledge – the AI-based techniques are seen as omnipotent. But they aren’t, as can be easily seen by the example of teeth and hair.

Moreover, the initiative is another way of encouraging people to verify news sources and distrusting information and tackling the challenge of fake news and deep fakes, mentioned above.

And a bonus one:

deepsense.ai in cooperation with Google Brain has finished research on enabling reinforcement learning (RL) agents to make predictions about their upcoming actions to reduce the number of actions needed to train skills.

RL agents typically need incomparably more actions to train skill than humans and the ability to build the skill “in a head” may be a part of the answer. Starting from childhood, people repeat their actions in their minds when they seek perfection or just recall an enjoyable or pleasant time.

Following this idea, our researchers have designed neural networks that possess the imagination that enables the model to simulate the action before executing it. A more detailed story about these projects and their outcome may be found in our blogpost about artificial imagination and Arxiv paper.

Why does it matter?

Artificial imagination is basically the idea of building the world simulated only by the mind’s power. Our researchers were able not only to reduce training time, but also to replace the simulated environment with another neural network’s imagination.

Maintaining and providing the simulated environment is one of the highest costs in building reinforcement learning agents, so tackling this challenge may be a great step to making this technique more popular.

Fake images of hamburgers, the autonomous trolley problem, Google’s BERT for NLP and more stories from October, curated by deepsense.ai’s team, right here in our AI Monthly Digest.

October brought important developments in machine learning and sparks for interesting discussions about machine morality. deepsense.ai’s Arkadiusz Nowaczyński and Konrad Budek chose the five stories below.

1. Enter the fakeburger – DeepMind managed to produce convincing images of hamburgers, animals and landscapes

Renowned AI company DeepMind produced synthetic photos of hamburgers, landscapes and animals out of the ImageNet dataset. In most cases, as a team, it was difficult for us to determine if the pictures depicted a real or fake burger.

This was not the first time a neural network was used to create a convincing fake photo – take a look at NVIDIA’s one hour of imaginary celebrities.

We now have the ability to produce realistic images after training on the ImageNet dataset famous for advancing state-of-the-art image classification. DeepMind researcher Andrew Brock achieved his breakthrough with highly tuned Generative Adversarial Networks (GAN). The GAN uses a generator that produces artificial samples of images, and a discriminator, which distinguishes between fake and a real-world examples. The GANs here are scaled up, leading to the impressive results. Larger networks and training batch sizes (=2048) vastly improve the quality over previous work. Google’s Tensor Processing Unit (TPU) made the training feasible on such a large scale.

Animals, nature and food look convincing, but scenes involving humans aren’t there yet.

Even though the model isn’t perfect, this is a remarkable step towards generating realistic looking photos with neural networks. To read more about the BigGAN and creating images of fakeburgers, check out the Arxiv paper.

2. AI-generated portrait sold for $432,500 at auction

Contrary to the fakeburgers and fakedogs built by DeepMind, the de Bellamy family consists of people with AI-generated eerie faces that are clearly non-human. The entire de Bellamy family images was generated by Obvious, a group of French AI engineers and artists.
Images were produced using GAN (Generative Adversarial Network) in the same manner DeepMind used it to produce an image of a hamburger. The model was fed with 15,000 portraits from last 600 years and attempted to build new ones using the data.
It is easy to see that images have no match to the master’s paintings, even considering the variety of styles represented by the artificial artist. Nevertheless, the portrait of Edmond de Belamy was sold for nearly half million dollar.

3. The machine morality – biases and automated discrimination

The transparency of machine learning models and Artificial Intelligence again in October came to the fore, igniting discussion over the Internet. The first important issue was gender bias in an Amazon-developed Artificial Intelligence model for preprocessing resumes delivered to the company. Trained with 10 years’ worth of resumes, the system was unintentionally trained to choose male candidates, as tech is dominated by men. The company changed its model but as the MIT Technology Review states, the company is no longer certain about the system’s neutrality in other areas.
Given the towering AI adoption rate, discussion about machine ethics and transparency is necessary. According to data from Deloitte, the number of machine learning project implementations was twice higher in 2018 than in 2017, and is expected to quadruple from there by 2020. As machines themselves are incapable of being racist, misogynist or biased, creating proper datasets and designing the evaluation process to spot hidden bias is crucial to building models that best and most fairly serve companies and society alike.
The data science community has not left the problem unaddressed – Google AI launched a Kaggle competition aiming to design image recognition models that could perform well on photos taken in different geographical regions than those used for training. The competition is being held as a competition track at the NIPS 2018 Conference, which is better-attended than even ComicCon.
This all goes to show that experts have been right from the start — AI is yet another tool that needs to be constantly evaluated and developed if it is to achieve its goals.

4. And the machine morality again – autonomous trolley problem

The topic of machine bias is even more important considering the rise of autonomous cars. A global study shows that people from various social and cultural backgrounds differ in their perception of “a lesser evil” when it comes to the “trolley problem” in autonomous cars. In a nutshell, if a car has to choose whether to hit an elderly person or a child, whom should it choose? A group of people or a single person? A pregnant woman or a medical doctor?
The study shows that choices vary by country, and the differences are more than significant. When facing the extreme situation of a car accident, human drivers make their mind autonomously and are solely responsible for their choices. On the other hand, every autonomous car will carry the AI model its manufacturer has provided. Are the legal system and society ready to transfer the responsibility for driving and the choices it necessitates from the driver to a non-human machine? Should the AI model fit the culture it operates in or follow some other code? These questions have yet to be answered.

5. Google’s BERT for NLP – new state-of-the-art in language modeling

Natural Language Processing may enter a new era with Google’s Bidirectional Encoder Representation from Transformations (BERT).
For now, NLP practitioners continue to use pre-trained word embeddings as initialization or input features in custom architectures for specific tasks. BERT, a model that can be pre-trained on a large text corpus and then fine-tuned for various NLP downstream tasks, may change that. It might be similar to what we have seen in Computer Vision in the last couple of years, where fine-tuning models pre-trained on ImageNet has proved a great success.
BERT is a multi-layer bidirectional encoder taken from Transformer architecture, which was introduced in Attention Is All You Need. The pre-training procedure is entirely unsupervised and includes two objectives: filling random gaps in the input sequence and classifying whether two input sentences are actually two consecutive sentences cut out from the larger text. During the fine-tuning, predictions can be made for entire sequences or each input token separately.
The study shows that fine-tuning a pre-trained BERT model has set a new gold standard in 11 benchmark tasks.
Google has released an official implementation of BERT for NLP available on github.

And now for some bonus information:

Paul Romer, winner of this year’s Nobel Prize in Economics, is a 62-year old ex-World Bank chief economist, writer and Python programming language user. He is also a firm supporter of making research available and clear, so he shares his findings via Jupyter notebooks and makes data available for everyone to process and interpret.
His example perhaps shows that combining knowledge about economics and science with a proper toolset for one’s daily work can help lead to a rewarding career.