deepsense.ai
  • Clients’ stories
  • Industries
    • Retail
    • Manufacturing
    • Financial Services
    • IT Operations
    • TMT and Other
  • Train your team
  • R&D Hub
  • Blog
  • About us
    • Our story
    • Management
    • Advisory Board
    • Press center
    • Careers
    • Internship program
  • Contact
  • Menu Menu
AI Monthly digest #4 - artificial intelligence and music, a new GAN standard and fighting depression

AI Monthly digest #4 – artificial intelligence and music, a new GAN standard and fighting depression

January 7, 2019/in Data science, Machine learning, AI Monthly Digest /by Konrad Budek and Arkadiusz Nowaczynski

December brought both year-end summaries and impressive research work pushing the boundaries of AI.

Contrary to many hi-tech trends, artificial intelligence is a popular term in society and there are many (not always accurate) views on the topic. The December edition of AI Monthly digest both shares the news about the latest developments and addresses doubts about this area of technology.
Previous editions of AI Monthly digest:

  • AI Monthly digest #3: Artificial Intelligence in science getting big
  • AI Monthly digest #2 – the fakeburger, BERT for NLP and machine morality
  • AI Monthly digest #1 – AI stock trading & Kaggle record

Tuning up artificial intelligence and music

Machines are getting better at image recognition and natural language processing. But several fields remain unpolished, leaving considerable room for improvement. One of these fields is music theory and analytics.
Music has different timescales, as some parts repeat at scale of seconds while others extend throughout an entire composition and sometimes beyond. Moreover, music composition employs a great deal of repetition.

Google’s Magenta-designed model leverages the relative attention to spot how far two tokens (motifs) are, and produced convincing and quite relaxing pieces of piano music. The music it generated generally evokes Bach more than Bowie, though.
Researchers provided samples of both great and flawed performances. Although AI-composed samples still refer to classical music, there are more jazz-styled improvisations.

Related:  Don’t waste the power. AI supply chain management

Further information about studies on artificial music can be found on Magenta’s Music Transformer website.
Why does it matter?
Music and sound is another type of data machine learning can analyze. Pushing research on music further will deepen our knowledge of music and styles in a way that AI made Kurt Vonnegut’s dream of analyzing literature a reality.
Furthermore, the music industry may be the next to leverage data and the knowledge and talents of computer scientists. Apart from tuning the recommendation engines for streaming services, they may contribute more to the creation of music. A keyboard is after all a musical instrument.

2. The machine style manipulator

Generating fake images from real ones is a thing. Generative Adversarial Networks enhance their training abilities by analyzing real images, generating fake ones and then training to be as good as possible in determining which is real and what is shown in the images.

The challenge neural networks and those who design them fact is in producing convincing images of people, cars, or anything else that is to be recognized by the networks. In a recent research paper, the group behind the “one hour of fake celebrity faces” project introduced a new neural network architecture that separates high-level attributes and stochastic variations. In the case of human images, the high-level attribute may be a pose while freckles or the hairdo are stochastic variations.
In a recent video, researchers show the results of applying a style-based generator and manipulating styles later to produce different types of images.

The results are impressive – researchers were able to produce convincing images of people from various ethnic backgrounds. By controlling different levels of styles, researchers were able to tune up everything on image – from gender and ethnicity to the shape of glasses worn.
Why does it matter?
That’s basically the next state-of-the-art in GAN networks, the best-performing image recognition technology used. Producing fancy-looking fake images of faces and houses can significantly improve the performance of image recognition models. Ultimately, however, this technology may be a life saver, especially when applied in medical diagnosis, for example in diabetic retinopathy.

3. AI failures – errare (not only) humanum est

2018 saw significant improvements in machine learning techniques and artificial intelligence proved even further how useful it is. However, using it in day-to-day human life will not be without its challenges.

In his 1896 novel “An Outcast of the Islands”, Joseph Conrad wrote “It’s only those who do nothing that make no mistakes”. This idea can also be applied to theoretically mistake-proof machines. Apart from inspiring successes, 2018 also witnessed some significant machine learning failures:

  • Amazon’s gender-biased AI recruiter – the machine learning model designed to pre-process the resumes sent to the tech giant overlooked female engineers due to bias in the dataset. The reason was obvious – the tech industry is male-dominated. As algorithms have neither common sense nor social skills, it assumed that women are just not a good match for the tech positions the company was trying to fill. Amazon ditched the flawed recruiting tool, yet the questions about hidden bias in datasets remain.
  • Uber’s fatal autonomous car crash – the story of fatal crash is a bitter lesson for all autonomous car manufacturers. Uber’s system not only detected the pedestrian it hit while driving, but also autonomously decided to proceed and ignore warnings, killing 49-year old Elaine Herzberg.
  • World Cup predictions gone wrong – The World Cup gave us another bitter lesson, this time for predictive analytics. While the model built to predict brackets may have been sophisticated, it failed entirely. According to its predictions, Germany should have met Brazil in the finals. Instead, the German team didn’t manage to get out of its group while Brazil bent the knee before South Korea. The final came down to France versus Croatia, an unthinkable combination, both for machine learning and football enthusiasts around the world. The case was further described in our blogpost about failure in predictive analytics.

More examples of AI failures can be found in the Synced Review Medium blogpost.
Why does it matter?
Nobody’s perfect. Including machines. That’s why users and designers need to be conscious of the need to make machine learning models transparent. What’s more, it is the next voice to ensure that machine learning model results are validated – a step that is tempting to overlook for early adopters.

4. Smartphone-embedded AI may detect the first signs of depression

A group of researchers from Stanford University has trained a model with pictures and videos of people who are depressed and people who are not. The model analyzed all the signals the subjects sent, including tone of voice, facial expressions and general behaviour. These were observed during interviews conducted by an avatar controlled by a real physician. The model proved effective in detecting depression more than 80% of the time. The machine was able to recognize slight differences between people suffering from depression and people who were not.
Why does it matter?
According to WHO, depression is the leading cause of disability worldwide. If not cured, it can lead to suicide, the second most common cause of death among 15-29-year-olds.

Related:  Powering up demand forecasting with machine learning

One barrier to helping people suffering from depression is inaccurate assessment. There are regions in the world where less than 10% of people have access to proper treatment. What’s more, mental illness is often stigmatized, and treatment is both costly and hard to access. These factors, and the fact that early symptoms are easily overlooked, lead many patients to avoid looking for care and medical support.
The experiment is a step toward building an automated and affordable system for spotting signs of depression early on, when the chance for a cure is highest.

5. So just how can AI hurt us?

Machine learning is one of the most exciting technologies of the twenty-first century. But science fiction and common belief have provided no lack of doomsday scenarios of AI harming people or even taking over the world. Dispelling the myths and disinformation and providing knowledge should be a mission all AI-developing companies. If you’re new to the discussion, here’s an essay addressing the threat of AI.
Why does it matter?
Leaving the doubts unaddressed may result in bias and prejudice when making decisions, both business and private ones. The key to making the right decisions is to be informed on all aspects of the issue. Pretending that Stephen Hawking’s and Elon Musk’s warnings about the cataclysmic risks AI poses were pointless would indeed be unwise.
On the other hand, the essay addresses less radical fears about AI, like hidden bias in datasets leading to machine-powered discrimination or allowing AI to go unregulated.
That’s why the focus on machine morality and the transparency of machine learning models is so important and comes up so frequently in AI Monthly digest.

Summary

December is the time to go over the successes and failures of the past year, a fact that applies equally to the machine learning community. Facing both the failures and challenges provides an opportunity to address common issues and make the upcoming work more future-proof.

https://deepsense.ai/wp-content/uploads/2019/02/AI-Monthly-digest-4-artificial-intelligence-and-music-a-new-GAN-standard-and-fighting-depression.jpg 337 1140 Konrad Budek https://deepsense.ai/wp-content/uploads/2019/04/DS_logo_color.svg Konrad Budek2019-01-07 16:07:132021-01-05 16:46:40AI Monthly digest #4 – artificial intelligence and music, a new GAN standard and fighting depression
Powering up demand forecasting with machine learning

Powering up demand forecasting with machine learning

December 6, 2018/in Machine learning /by Konrad Budek

Demand forecasting is a field of predictive analytics, that aims to predict the demand of customers. It is done by analyzing statistical data and looking for patterns and correlations. Machine learning takes the practice to a higher level.
According to McKinsey study, AI-enhanced supply chain management may lead to improved accuracy by reducing forecasting errors by 20-50%.

Why demand forecasting is essential

There is always a context surrounding customer behavior. It may be an upcoming holiday, the weather or a recent event. As real product demand varies, businesses may face two challenges:

  • Income and profit loss when a product is out of stock or a service is unavailable
  • Cash tied up in stock or
  • The reduced margins that come with getting it out of warehouses

According to the Harvard Business Review, history is littered with missed demand forecasts that came with great losses for the business. To name a few of the most important:

  • In 1974, the U.S. energy sector made plans to double power generation due to a forecast calling for 7% annual growth in demand. But during the 1975-1985 period, growth faltered, coming in at a mere 2%. While many projects were canceled, the industry was heavily hit and consumer prices rose.
  • In 1983 and 1984, the PC market was forecast to explode and see 27-28 million units installed by 1988. As the story goes, by 1986 only 15 million units had been shipped and many PC manufacturers had gone out of business.

Considering the essential role of accurate demand forecasts, it is no surprise that an ARC global market study cited by Forbes predicts that demand applications will account for just under a third of the $2 billion supply planning suites market.

Human-powered demand forecasting methods

Demand is predicted using both qualitative and quantitative methods. In the most basic form, it is done by leveraging the experience of the seller or collective brainstorming in the company. Another popular way is to ask the customers. Conducting market surveys is one of the simplest methods of demand forecasting and may be applied in all industries, no matter the size or market segment. Other common methods include delphi forecasting method that relies on a panel of experts answering questions or employing game theory an leveraging.

Related:  Driverless car or autonomous driving? Tackling the challenges of autonomous vehicles

As companies are relying more and more on the data they collect, it is no surprise that traditional, human-based methods are being replaced by data-powered, automated systems. Using big data demand prediction is enabling a wide range of companies to leverage machine learning models in data exploration and extrapolation. If there is enough data to train a model, it is almost certain to outperform human data analysts and researchers. For proof of that, have a look at Google’s Sunroof project, which makes more accurate predictions of a roof’s sun exposure than any human analyst could do.

Demand prediction done right – case studies

Modern demand forecasting techniques go far beyond simple historical data analysis. When Netflix’s launched the design process for its series House of Cards, the company analyzed the most replayed parts of shows and films featuring Kevin Spacey, the show’s main star.

Powering up demand forecasting with machine learning - Capitol

When Netflix’s launched the design process for its series House of Cards, the company analyzed the most replayed parts of shows and films featuring Kevin Spacey, the show’s main star.

This helped it analyze what the target group was looking for more of. The data mining proved invaluable as the show’s second episode peaked at more than 4 million viewers. And according to the “Esquire”, the absence of Kevin Spacey in the last season “haunts the production” and the show is not the same anymore without the key actor.

Good demand forecasting can be a literal life-saver. In the UK, patients rely on the National Health Service’s (NHS) Blood & Transplant department, which consists of five manufacturing sites and 15 stock holding units. The department is responsible for transporting blood between hospitals, and managing both the donations and deliveries. The NHS’ previous system was relied heavily on manual data management, but it didn’t have the technology to analyze the data. Introducing a data mining-based system enabled the NHS to forecast demand by blood type and quantities it needed. With its new system, the NHS is able to prevent shortages by refreshing stock level readings every 30 minutes and connecting the information from various sources. Participating hospitals reported a 30% reduction in overstock waste and a 20% cut to delivery costs.

Summary

As demand forecasting is statistics-heavy and data-rich, it is ripe for the application of machine learning algorithms. The automation of data flows helps to manage logistics and find new ways to optimize an organization’s performance.

https://deepsense.ai/wp-content/uploads/2018/12/Powering-up-demand-forecasting-with-machine-learning.png 337 1140 Konrad Budek https://deepsense.ai/wp-content/uploads/2019/04/DS_logo_color.svg Konrad Budek2018-12-06 14:41:212021-01-05 16:46:49Powering up demand forecasting with machine learning
AI Monthly digest #3 - artificial intelligence in science getting big

AI Monthly digest #3 – artificial intelligence in science getting big

December 6, 2018/in Machine learning, AI Monthly Digest /by Konrad Budek and Arkadiusz Nowaczynski

November brought a lot of significant AI-related breaking news. Machine learning and deep learning models were folding proteins, deriving the laws of physics from fictional universes and mimicking the human brain in a way never before seen.

This edition of AI Monthly digest looks at scientific improvements made by AI with significant support from tech giants. Stories from November show that AI is clearly a great tool to improve countless lives – and the protein-folding model is the best example of solving problems thought to be unsolvable. Or at least since our universe came into existence.

1. Deepmind’s AlphaFold wins Protein-Folding Contest

It’s a bit ironic that the protein-based human brain designed silicon-based tools to better predict protein folding. That’s precisely the case with AlphaFold, the neural network designed by Deepmind to predict the shape of a protein from the DNA code.
The basic building blocks of all known life forms, proteins come in many forms, shapes and levels of complexity. Their shape and function are encoded within DNA. Yet the DNA itself contains only a sequence of amino acids that form the protein, and the way they will form a structure is not encoded in any way. Thus it is challenging to establish what the protein will look like. As Levintha’s Paradox states, it would take longer than the age of the universe to enumerate the possible configurations of a typical protein before reaching the right structure. Yet, the shape of protein is crucial in the treatment and diagnosis of numerous diseases, including Alzheimer’s, Parkinson’s and Hutchington’s.
Thanks to the relatively high amount of gene-related data, Deepmind was able to build a neural network that could predict the shape of proteins only from a DNA sequence. The algorithm analyzes the distance between particular amino acids and compares that with existing models, preparing estimations about possible shapes and folds.
The effects are seen as one of the most significant breakthroughs and an unprecedented progress on protein folding.
To read more about the model and how it works, see recent Deepmind’s blogpost.

2. Towards building an AI Physicist

A Physicist’s job is to create models of the universe that give us a better understanding of the reality surrounding us. Newton, Galileo, Archimedes and many others had to convert measurements and observations into the fundamental laws of the universe. The ability to discover important answers from data is the best proof of their genius.
Last month, MIT’s Tailin Wu and Max Tegmark presented work reporting on an “AI Physicist” deriving the laws of physics in artificial worlds. To crack the mysterious environments, the AI agent uses four strategies embedded in its architecture: divide-and-conquer, Occam’s razor, Unification and Lifelong Learning, which together allow it to discover and manipulate theories without supervision.
The system was able to produce correct theories for various simulated worlds created only to test its accuracy. The success is significant from a scientific point of view, as it proves that neural networks may be the tools to speed up physical science in more ways than we ever expected.
The Arxiv paper can be read here.

3. SpiNNaker – Human brain supercomputer runs for the first time

Building neural networks is about mimicking the human brain’s activity and processing abilities. Most common neural networks work by communicating in a constant way and exchanging information in a stream. The spiking neural network is more “biological” insofar as it works by forcing artificial neurons to communicate in spikes and exchange information in a “flash”. Instead of sending the information from point A to point B, it supports sending multiple bits of parallel information.

AI Monthly digest #3 - Artificial Intelligence in science getting big - Brain

The spiking neural network is more “biological” insofar as it works by forcing artificial neurons to communicate in spikes and exchange information in a “flash”

This system is mimicked in a SpiNNaker (Spiking Neural Network Machine) built at the University of Manchester’s School of Computer Science, backed by EPSRC and supported by the European Human Brain Project. The most significant outcome of the SpiNNaker is that it builds a working, small-scale model of the human brain, with the manifold scientific possibilities it brings with it.
More information may be read here.

4. New ImageNet state-of-the-art with GPipe

Since its foundations, the greatest challenge in computer science has been insufficient computing power. Multiplying the number of cores is one way to solve the problem, while optimizing the software is another. The challenge comes to bear in neural networks, as training a new one requires gargantuan computing power and no less time.
Thus, Google Brain’s GPipe is a significant improvement that makes neural networks more cost-effective. By using GPipe, neural networks can process significantly more parameters. And that leads to better results in training.
GPipe combines data parallelism and model parallelism, with a high level of automation and memory optimization. In the paper, researcher expanded the AmoebaNet from 155.3 million parameters to 557 million parameters and inserted as input 480×480 ImageNet Images. The result was an improvement in ImageNet Top-1 Accuracy (84.3% vs 83.5%) and Top-5 Accuracy (97.0% vs 96.5%), making the solution the new state-of-the-art.

5. Enterprise machine learning done right – Uber shows its best practices

With machine learning being a recent development, there are no proven and tested methodologies to build new AI-based developments nor best practices with which to dive in and use in development.
Uber’s recent blogpost shared an interesting vision of incorporating AI-driven culture into the company. Instead of building one large ML project to perform one task on an enterprise scale, Uber powers its teams with data scientists and looks for ways to automate their daily work. The company has now done a dozen projects, carried out by a variety of teams – from Uber Eats menu items ranking or marketplace forecasting to customer support.
On the back of this strategy, Uber has gone from a company that did not use machine learning at all to one now heavily infused with AI-based techniques. More details about ML applications in Uber can be seen in their blogpost.

Summary

AI, machine learning and deep learning tend to be seen as buzzwords, applied only by tech giants to solve their hermetic problems. Folding proteins, deriving the laws of physics and simulating the brain as it really works all show that AI is the rule-breaker and disruptor it any field it is applied in. Even if it was dominated by bold minds physicists of all time.

https://deepsense.ai/wp-content/uploads/2018/12/AI-Monthly-digest-3-artificial-intelligence-in-science-getting-big.jpg 337 1140 Konrad Budek https://deepsense.ai/wp-content/uploads/2019/04/DS_logo_color.svg Konrad Budek2018-12-06 14:41:162021-01-05 16:46:53AI Monthly digest #3 – artificial intelligence in science getting big
Driverless car or autonomous driving? Tackling the challenges of autonomous vehicles

Driverless car or autonomous driving? Tackling the challenges of autonomous vehicles

November 26, 2018/in Machine learning, Reinforcement learning /by Konrad Budek

Among both traditional carmakers and cutting-edge tech behemoths, there is massive competition to bring autonomous vehicles to market.

It was a beautiful, sunny day of June 18, 1914 when the brilliant engineer Lawrence Sperry stunned the jury of Concours de la Securité en Aéroplane (Airline Safety Competition) by flying in front of their lodge with his hands held high. It was the first time the public had ever seen a gyroscopic stabilizer, one of the first autopiloting devices. Over a hundred years later, automatic flight control devices and maritime autopilots are common, while cars still require human operation. Thanks to machine learning and autonomous cars, that’s about to change.

What is the future of autonomous vehicles?

According to recent reports, autonomous cars are going to disrupt the private, public and freight transportation industries. A recent Deloitte publication reports that society is putting more and more trust in autonomous vehicles. In 2017, 74% of US, 72% of German and 69% of Canadian respondents declared that fully autonomous cars would not be safe. But those rates have now dropped significantly, to 47%, 45% and 44%, respectively.
Plans for building self-driving cars have been revealed by BMW, Nissan and Ford, while Uber and the Google-affiliated Waymo are also in the thick of the race. Companies aim both to build urban driving vehicles and autonomous trucks, while a startup scene supporting autonomous technology is emerging.
Thanks to the increasing popularity of autonomous cars, up to 40% of mileage could be driven in self-driving vehicles in 2030. But, as always, the devil is in the details.

What is an autonomous car?

To answer that question, the National Highway Traffic Safety Administration uses the autonomous vehicle taxonomy designed by the Society of Automotive Engineers, which lists five levels of automation.

  1. No automation – the driver performs all driving tasks
  2. Driver assistance – the car has built-in functions to assist the driver, who nonetheless must remain engaged in the driving process. Cruise control is one of the best examples.
  3. Partial automation – the vehicle has combined automated functions like acceleration and steering, but the driver must remain engaged. The gyroscopic stabilizer is an example of partial automation.
  4. Conditional automation – a human driver is necessary in totally unpredictable situations, but not required to monitor the environment all the time. BMW currently has a fleet of about 40 level 4 cars unleashed on testing grounds near Munich and in California.
  5. High automation – the car on this level may not even have a steering wheel and can deal with any situation encountered. Fully autonomous vehicles, which do not yet exist, occupy Level 5.

Building level 4 and 5 driverless vehicles is a great challenge because the driving process has a number of complicating factors. Unlike with a plane or ship, drivers usually have little to no time to respond to the changing environment. They must monitor the state of the machine, their surroundings, and the other drivers on the road. What’s more, any mistake can cause an accident – 37.133 people were killed in traffic accidents on American roads in 2017.
While we may not give ourselves credit as drivers, humans’ ability to process signals from various senses to control a car is a super power. It is not only about simply looking at the road – many drivers estimate the distance between cars by looking at reflections in the body of the car in front of it. Many drivers can hear changes in their engine’s performance or sense changing grip strength on various types of road.
To effectively replace human perception, sophisticated assistance systems rely on numerous sensors. GM’s report on autonomous cars  and driving technology safety lists:

  • Cameras – detect and track pedestrians and cyclists, monitor free space and traffic lights
  • Articulating radars –  detect moving vehicles at long range over a wide field of view
  • Short-range radars – monitor objects around the vehicle
  • Long-range radars – detect vehicles and measure velocity
  • Lidars – detect fixed and moving with objects high-precision laser sensors

Handling data from various sources that need to be processed in real time is a perfect task for deep neural networks, especially when it involves simultaneous work on non homogenous data taken from radars, images from cameras and lidar readings.
But building a system that automates driving is an enormous challenge, especially given the sheer number of serious decisions to be made when driving and the fact that a single bad decision can result in disaster.

Two ways how autonomous cars work

There are currently two approaches to building the models that control autonomous vehicles.
A component-based system – the controller is built with several independent models and software components each designed to handle one task, be it road sign recognition, managing the state of the vehicle or interpreting the sensors’ signals.

  • Pros –  dividing the system into subsystems makes building the software easier. Each component can be optimized and developed individually thus improving the system as a whole.
  • Cons – developing the model requires a massive amount of data to be gathered and processed. The image recognition module needs to be fed different data than the engine control device. This makes preparing the dataset to train more than a little challenging.  What’s more, the process of integrating the subsystems may be a challenge in and of itself.

End-to-end system – with this approach, a single model capable of conducting the entire driving process is built – from gathering information from the sensors to steering and reacting accordingly. deepsense.ai is moving ahead with just such a model.

  • Pros – it is easier to perform all the training within the simulation environment. Modern simulators provide the model with a high-quality, diverse urban environment. Using the simulated environment greatly reduces the cost of gathering data.

Although it is possible to label and prepare data gathered with the simulator, the technique requires a bit more effort. What’s more, it is possible to use a pre-trained neural network to mimic the simulated environment (a matrix of sorts) to further reduce the data to gather or generate. We expect the model to perform better than a component-based system would.

  • Cons – this type of model may be harder to interpret or reverse-engineer. When it comes to further tuning the model or reducing the challenge posed by the reality gap (see below) it may be a significant obstacle.

Facing the reality gap

Using a simulator-trained model in a real car is always challenging due to what is known as the reality gap. The reality gap represents all the differences and unexpected situations the model may encounter than the designer was able to predict and therefore prepare it for.
There are countless examples. The position of cameras in a real car may be different than in a simulated one, the simulation physics are necessarily incomplete, and there may be a hidden bug the model could exploit. Furthermore, the sensors’ readings may differ from the real ones concerning calibration or precision. There may be a construction feature that causes the car to behave differently in reality than in a simulation. Even the brightest data scientist is unable to predict all the possible scenarios. What would happen if a bird started to peck at the camera? Or if a car encountered a boy dressed as Superman pretending to fly? Or, more plausibly, after a collision, if there were an oil stain that looked exactly like a puddle, but would obviously have an entirely different effect on the tires’ grip of the road?
To address these challenges, data scientists randomize the data and the training environment to let the model gather more varied experiences. The model will learn how to control the car in changing weather and lighting conditions. By changing the camera and sensor settings, a neural network will gain enough experience to handle the differences or any changes that may occur when the model is being used.

Fighting the gap every lap

Using a simulated environment is one effective way of evaluating a model, but a pronounced reality gap still remains. To acquire better information (and also to have some fun on the job), data scientists evaluate their neural networks by launching them in small-scale models. There is currently an interesting Formula 1/10 autonomous car racing competition being held. Designing the software to control cars and compete against other teams is a challenging (yet fun) way to evaluate models. Small-scale models are tested on tracks with angles and long straights that are perfect for acceleration. Although the cars aren’t driven by humans, the team provides full-time technical assistance and puts the car back on track when it falters.
It’s also a great way to impress the jury in a similar manner to what Lawrence Sperry did more than a hundred years ago!
If this sounds interesting to you, deepsense.ai has recently launched a deep learning workshop where participants will learn to create models to control these small cars. The training will be conducted in cooperation with one of world champion Formula 1/10 racers.
The text was prepared in cooperation with Krzysztof Galias, deepsense.ai data scientist.

https://deepsense.ai/wp-content/uploads/2019/02/Driverless-car-or-autonomous-driving-Tackling-the-challenges-of-autonomous-vehicles-.jpg 337 1140 Konrad Budek https://deepsense.ai/wp-content/uploads/2019/04/DS_logo_color.svg Konrad Budek2018-11-26 11:25:542021-01-05 16:46:58Driverless car or autonomous driving? Tackling the challenges of autonomous vehicles
AI Monthly digest #1 – AI stock trading & Kaggle record

AI Monthly digest #1 – AI stock trading & Kaggle record

October 5, 2018/in Data science, Deep learning, Machine learning, AI Monthly Digest /by Konrad Budek and Arkadiusz Nowaczynski

AI-based stock trading, a record-breaking competition on Kaggle and more stories cherry-picked from all the interesting ML- and AI-related news from September. Right here in the AI Monthly Digest.

The Digest gathers machine learning and AI news to spot the most important and interesting events and developments of the past month. The five events below were curated from last month’s events and chosen by Arkadiusz Nowaczyński and Konrad Budek from deepsense.ai team.

Deep learning takes a deep dive into the stock market

Deep reinforcement learning can be applied as a complete AI solution for algorithmic trading.
The authors of “Deep Reinforcement Learning in Portfolio Management” set out to determine whether methods derived primarily for playing Atari games and continuous control would work on the stock market. The algorithm they used, called deep deterministic policy gradient (DDPG), returned promising results in an offline backtest.
The second paper, “Deep Reinforcement Learning in High Frequency Trading,” provides convincing arguments about why AI stock trading is suitable for trading in a timescale below 1 second (High Frequency Trading). The authors did a solid evaluation of their approach with a few noteworthy tips:

  1. Online learning at test time makes it possible to maintain high accuracy over time;
  2. A small neural network is enough for this problem, meaning AI for trading can be developed on laptops;
  3. Predicting the next 100 ticks from the last 500 ticks works best for them.

Progress remains to be made and questions to be answered. Does this algorithm work when deployed on the real market? How much money can you actually make with it? The lack of answers is certainly intriguing, as is the fact that algorithmic trading may soon be powered mostly by Deep RL, if it’s not already. We think that the potential financial reward will push people to develop further breakthroughs in AI. After all, setting high scores in Atari games isn’t as satisfying as having supersmart AI earning you gobs of money.
For those of you who are new to the topic we recommend a good nice-to-read introduction.

A record-breaking Kaggle competition

Over 8500 data scientists on no fewer than 7000 teams took part in the Kaggle Home Credit Default Risk evaluation record-breaking competition. The goal of the competition was to predict the risk of giving a loan to a particular customer. The teams were provided with rich datasets containing historical and transactional data on the customer’s behavior.

Related:  Keras vs. PyTorch: Alien vs. Predator recognition with transfer learning

Perfectly designed, the competition attracted teams from far and wide, mostly thanks to the outstanding dataset. It allowed the teams to harvest insights and play with data in often surprising ways. Looking to tune up their models and further polish their skills, participants engaged in discussions and peer-reviews long after the competition had ended.
deepsense.ai took part in the competition, with Paweł Godula leading a team that took 5th place overall and finished first on the public leaderboard.

Volvo trucks introduce Vera, the cabless truck

According to PwC data, by 2030 the transport sector will require 138 million fewer cars in Europe and the US, mostly thanks to the rise of autonomous vehicles and the development of new business models. What’s more, it is predicted that by 2030 autonomous vehicles will be driving 40% of all miles driven.
As a proof of concept, Volvo has brought out Vera, the cabless truck to be used in short-haul transportation at logistics centres or ports. With the fleet of vehicles able to communicate and be supervised by a cloud-based management system, the truck is an interesting glimpse of the driverless future.

DARPA announced $2 billion investment in AI

At it’s 60th anniversary conference, the DARPA (Defense Advanced Research Projects Agency) announced that it is going to invest $2 billion in artificial intelligence. The agency is known for developing cutting-edge technology, be it ARPANET, which later evolved into the Internet, or the Aspen Movie Map, which was among the predecessors of Google Street View.
According to John Everrett (via CNNMoney), the deputy director of DARPA’s Information Innovation Office, the agency’s investment is intended to accelerate the development of AI from 20 years down to five years.
DARPA’s investment is not the first a government has made in AI. The most notable example comes from the United Arab Emirates, which has appointed an AI minister.

NIPS conference sold out in less than 13 minutes

NIPS, hosted in Montreal, Canada, is currently the most important machine learning and AI research conference in the world. Initially held as an interdisciplinary meeting of experts interested in sharing their knowledge on neural networks, it has evolved into the machine learning meeting with thousands of papers sent for review. It is also a place to run competitions with the “Learning to run” in 2017 as an example.

Related:  Expert augmented reinforcement learning - agents of Montezuma’s Revenge

In 2017, the tickets sold out in two weeks, a relative eternity compared to the rock concert-like 12 minutes and 38 seconds they flew out in this year. Tickets for last year’s Comic-Con, one of the world’s most beloved pop culture events, sold out in a bit more than an hour.
So, when it comes to selling tickets, Marvel superheroes would appear to have nothing on machine learning. This year’s NIPS conference will feature Henryk Michalewski, visiting professor at Oxford University and a researcher at deepsense.ai, as a co-author of “Reinforcement Learning of Theorem Proving” paper.

Summary

September has clearly shown that AI is one of the most dominant trends in modern tech. Selling out venues faster than pop culture events goes a long way to proving that a scientific conference, or at least this one, can be as exciting as a concert or show – so long as it’s about Artificial Intelligence.

https://deepsense.ai/wp-content/uploads/2019/02/AI-Monthly-digest-1-–-AI-stock-trading-Kaggle-record.jpg 337 1140 Konrad Budek https://deepsense.ai/wp-content/uploads/2019/04/DS_logo_color.svg Konrad Budek2018-10-05 14:11:302021-01-05 16:47:07AI Monthly digest #1 – AI stock trading & Kaggle record
How machine learning improves business efficiency - five practical examples

How machine learning improves business efficiency – five practical examples

August 30, 2018/in Machine learning /by Konrad Budek

Deloitte estimates that in 2021 enterprise spending on artificial intelligence and machine learning projects will reach 57 billion dollars, four times more than in 2017. These technologies are now in every day use, and not only among innovation leaders.

Thanks to digitalization of business processes, organizations command ever greater amounts of data, which, with the help of machine learning, can be used to automate work. At the same time, spending on the following five areas can be limited:

  1. maintenance, thanks to reduced energy consumption
  2. payroll costs, thanks to task automation
  3. raw material and quality assurance costs, thanks to the automation and tightening of quality control
  4. equipment and machinery costs, thanks to the automation of control systems for operations and maintenance
  5. operating costs including marketing and sales

More and more of the business community is catching on to the savings they can harness with artificial intelligence. The evidence for this is clear from the steps individual enterprises are taking, as well as the development of numerous machine learning business examples and the entire ecosystem of companies offering products based on these technologies and support in implementing them.

1. Reducing the costs associated with maintaining and using the energy

As a system’s complexity grows, so too does the challenge of supervising it. Click To Tweet
Consider, for example, the cooling of large server rooms. In terms of energy consumption and CO2 generation, the ICT sector (communication technologies, including telecommunications and IT services) produces two percent of global emissions, which is on a par with the airlines. To reduce its electricity expenses, Google decided to entrust energy management in one of its server rooms to AI, which “learned” the structure of the center and reduced cooling costs by 40 percent.
No new equipment was needed – it was enough to develop new software that leveraged AI. Ultimately, the system is going to be used in all Google server rooms. The British national energy supplier National Grid has also expressed interested in the solution.

Related:  Five trends for business to surf the big data wave

2. Reduction of human costs through automation

Machine learning enables the automation of repetitive, often time-consuming activities, freeing up the teams that had been doing them to take up more profitable tasks. We produced a program for the international research company Nielsen that could find, read and save in a database the composition of the a product’s contents using only a photo of its packaging. This shortened the working time from several minutes dedicated to manually rewriting the composition from the label, to the few seconds required to take a picture of the packaging.
If you need more convincing, consider these figures: If a company employs 46,000 people, helping even half of them save five minutes a day translates into 314 full-time positions each day.
Machine learning business examples

3. Predictive maintenance 4.0 – optimizing machine maintenance costs

Because any hardware failure involves both repair costs and production downtime, what company wouldn’t look for tools that can predict failures and prevent them? Another solution deepsense.ai has prepared, this time for a manufacturer, used data from sensors mounted on machines.
By reviewing and analyzing the signals, the solution can predict upcoming failures up to two weeks before they occur. Another example of predictive maintenance comes thanks to the OneWatt company, which tests the sounds issued by industrial machines. AI steps in where the human ear would be helpless: It detect changes in the sounds the machines produce to predict potential failures.

Related:  Three reasons why data analysts make the perfect data scientists

4. Quality control – fewer mistakes with machine learning

In many industries, quality control comes with huge costs. For semiconductor manufacturers, huge means up to 30 percent of costs. Automating quality control with image recognition tools increases the percentage of defects detected by up to 90 percent.
Unlike automated systems, machine learning-based vision systems can continuously evolve and adapt to new product specifications. Fujitsu implemented a system that both catches defective products and prepares each one for automated assembly at the next production stage. Applying machine learning, the system not only automatically recognizes the parts of the machine but also assesses their compliance with standards in more than 97 percent of cases.
Machine learning business examples

5. The power of data in sales, marketing and customer service

Machine learning is able to process data sets faster and more efficiently than even the most expert analysts. This makes it possible to constantly analyze what is happening, for example, in the company’s sales or transaction system, and also to regularly monitor customer activity. To understand just how beneficial machine learning can be here, consider the customer loyalty survey.
Only one in every 26 clients expresses their dissatisfaction before looking to the competition for what they need. Data science can help you capture the behavior patterns of a dissatisfied customer and react in advance.
Machine learning business examples
Using data more effectively benefits not only business, but the whole of society. A solution developed by deepsense.ai for the city of Portland, Oregon enabled the police to predict in which parts of the city crime would take place.

Machine learning in practice

Machine learning is giving enterprise more opportunities to look for savings and generate additional revenue. AI helps people accomplish complex tasks that under normal conditions would overwhelm them, so great is their complexity. Machine learning also makes it possible to automate activities that, though repetitive and schematic, require maximum focus, so employee productivity tends to fall quickly. AI helps people do their work more effectively and devote more energy to those activities that bring the most value.

https://deepsense.ai/wp-content/uploads/2019/02/How-machine-learning-improves-business-efficiency-–-five-practical-examples.jpg 337 1140 Konrad Budek https://deepsense.ai/wp-content/uploads/2019/04/DS_logo_color.svg Konrad Budek2018-08-30 13:42:072021-01-05 16:47:25How machine learning improves business efficiency – five practical examples
When predictive analytics in football fall short (an example)

When predictive analytics in football fall short (an example)

July 10, 2018/in Machine learning /by Konrad Budek

According to predictions done by Soccerbot 3000, the AI-powered prediction machine, Germany should face Brazil in the finals of the World Cup in Russia – or should have, that is. Then the unthinkable happened.

The short explanation for those not interested in the football matches being played in Russia: the German team – the same one Goldman Sachs picked as the probable world champ – failed to get out of its group for the first time in 80 years. And when the German team was vanquished by South Korea and its brilliant Son Heung-Min, predictions were proved wrong. Not much later, the always ballyhooed Brazilian team was knocked just as far out of the tournament, which is to say, all the way. Indeed, Mr. Neymar and peers, following legends Cristiano Ronaldo and Leo Messi, were sent packing before the semi-finals.
The Financial Times pointed out that Soccerbot 3000 used “200,000 models” that generated “1,000,000 possible evolutions of the tournament”. According to its prediction, this year’s Cup should have gone to Brazil by a nose, or a toe as it were. The conclusion that Machine Learning is still ineffective in predictions seems obvious, but it would be severely biased.

It wasn’t only the model that was surprised

In the detailed report on the current World Cup and among generated vast majority have shown Brazil, France, and Germany were forecast to lead, with 18.5%, 11.3%, and 10.7% chances of bringing home the cup, respectively. None of the other teams garnered more than 10%.
The models used historical data about team characteristics, individual players and recent team performance. The model later learned the correlation between these metrics and the teams’ performance based on World Cup data since 2005. Is that a massive amount of data? Indeed it is. But that’s hardly the entire issue.
The model was unable to predict the weather, player health or the atmosphere prevailing in each team. Football, like any other game, consists of many more variables than researchers are able to predict and insert into a model. Of the one million scenarios it produced, the model predicted almost 200,000 scenarios when Germany didn’t reach the round of 16.
That it didn’t happen shocked the world, not only people who trusted the AI to predict the outcome. Even the famous Gary Lineker, who said that “Football is a simple game – twenty-two men chase a ball for 90 minutes and at the end, the Germans always win.” after Germany have beaten England in Italy in 1990 updated his famous quote.

Football is a simple game. Twenty-two men chase a ball for 90 minutes and at the end, the Germans no longer always win. The previous version is confined to history.

— Gary Lineker (@GaryLineker) June 27, 2018

Racing with probability

The key role machine learning models play is in reducing the randomness of choices based on data processing. The level of accuracy applied to be used in production is highly dependant on the purpose it was designed for.
A fraud detection model that was 80% accurate would never be used in a bank or any other institution. The 20% of the fraud it didn’t catch would be nothing short of a disaster for such an institution.
On the other hand, a model that could return that same 80% processing investment opportunities would earn millions of dollars. Warren Buffet may have missed the investment opportunity in Google and Amazon, but that doesn’t make him an unreliable investor.
When predictive analytics in football fall short (an example)
Considering its 81,25% accuracy in the group phase, the model would be quite reliable as an advisor, even if it was unable to read opinions, use social media, leaked information or just read the news just before each match to make corrections.
When a company has access to more reliable data or even provides all the data possible, the accuracy rises. This can be seen in visual quality control or recognizing diabetic retinopathy from photos. Predicting the outcome of sporting events is a much different business.
Even the Goldman Sachs analysts behind the model cautioned against seeing it as an oracle. In any case, however many analyses or however much data science gets done, the World Cup will be exciting to watch.

https://deepsense.ai/wp-content/uploads/2019/02/when-predictive-analytics-in-football-fall-short-an-example-header.jpg 337 1140 Konrad Budek https://deepsense.ai/wp-content/uploads/2019/04/DS_logo_color.svg Konrad Budek2018-07-10 14:44:512021-01-05 16:47:50When predictive analytics in football fall short (an example)
What is reinforcement learning? The complete guide

What is reinforcement learning? The complete guide

July 5, 2018/in Deep learning, Machine learning, Reinforcement learning, Popular posts /by Błażej Osiński and Konrad Budek

With an estimated market size of 7.35 billion US dollars, artificial intelligence is growing by leaps and bounds. McKinsey predicts that AI techniques (including deep learning and reinforcement learning) have the potential to create between $3.5T and $5.8T in value annually across nine business functions in 19 industries.

Although machine learning is seen as a monolith, this cutting-edge technology is diversified, with various sub-types including machine learning, deep learning, and the state-of-art technology of deep reinforcement learning.

What is reinforcement learning?

Reinforcement learning is the training of machine learning models to make a sequence of decisions. The agent learns to achieve a goal in an uncertain, potentially complex environment. In reinforcement learning, an artificial intelligence faces a game-like situation. The computer employs trial and error to come up with a solution to the problem. To get the machine to do what the programmer wants, the artificial intelligence gets either rewards or penalties for the actions it performs. Its goal is to maximize the total reward.
Although the designer sets the reward policy–that is, the rules of the game–he gives the model no hints or suggestions for how to solve the game. It’s up to the model to figure out how to perform the task to maximize the reward, starting from totally random trials and finishing with sophisticated tactics and superhuman skills. By leveraging the power of search and many trials, reinforcement learning is currently the most effective way to hint machine’s creativity. In contrast to human beings, artificial intelligence can gather experience from thousands of parallel gameplays if a reinforcement learning algorithm is run on a sufficiently powerful computer infrastructure.

Examples of reinforcement learning

Applications of reinforcement learning were in the past limited by weak computer infrastructure. However, as Gerard Tesauro’s backgamon AI superplayer developed in 1990’s shows, progress did happen. That early progress is now rapidly changing with powerful new computational technologies opening the way to completely new inspiring applications.
Training the models that control autonomous cars is an excellent example of a potential application of reinforcement learning. In an ideal situation, the computer should get no instructions on driving the car. The programmer would avoid hard-wiring anything connected with the task and allow the machine to learn from its own errors. In a perfect situation, the only hard-wired element would be the reward function.

  • For example, in usual circumstances we would require an autonomous vehicle to put safety first, minimize ride time, reduce pollution, offer passengers comfort and obey the rules of law. With an autonomous race car, on the other hand, we would emphasize speed much more than the driver’s comfort. The programmer cannot predict everything that could happen on the road. Instead of building lengthy “if-then” instructions, the programmer prepares the reinforcement learning agent to be capable of learning from the system of rewards and penalties. The agent (another name for reinforcement learning algorithms performing the task) gets rewards for reaching specific goals.
  • Another example: deepsense.ai took part in the “Learning to run” project, which aimed to train a virtual runner from scratch. The runner is an advanced and precise musculoskeletal model designed by the Stanford Neuromuscular Biomechanics Laboratory. Learning the agent how to run is a first step in building a new generation of prosthetic legs, ones that automatically recognize people’s walking patterns and tweak themselves to make moving easier and more effective. While it is possible and has been done in Stanford’s labs, hard-wiring all the commands and predicting all possible patterns of walking requires a lot of work from highly skilled programmers.

For more real-life applications of reinforcement learning check this article.

Related:  Learning to run - an example of reinforcement learning

Challenges with reinforcement learning

The main challenge in reinforcement learning lays in preparing the simulation environment, which is highly dependant on the task to be performed. When the model has to go superhuman in Chess, Go or Atari games, preparing the simulation environment is relatively simple. When it comes to building a model capable of driving an autonomous car, building a realistic simulator is crucial before letting the car ride on the street. The model has to figure out how to brake or avoid a collision in a safe environment, where sacrificing even a thousand cars comes at a minimal cost. Transferring the model out of the training environment and into to the real world is where things get tricky.
Scaling and tweaking the neural network controlling the agent is another challenge. There is no way to communicate with the network other than through the system of rewards and penalties.This in particular may lead to catastrophic forgetting, where acquiring new knowledge causes some of the old to be erased from the network (to read up on this issue, see this paper, published during the International Conference on Machine Learning).
Yet another challenge is reaching a local optimum – that is the agent performs the task as it is, but not in the optimal or required way. A “jumper” jumping like a kangaroo instead of doing the thing that was expected of it-walking-is a great example, and is also one that can be found in our recent blog post.
Finally, there are agents that will optimize the prize without performing the task it was designed for. An interesting example can be found in the OpenAI video below, where the agent learned to gain rewards, but not to complete the race.

What distinguishes reinforcement learning from deep learning and machine learning?

In fact, there should be no clear divide between machine learning, deep learning and reinforcement learning. It is like a parallelogram – rectangle – square relation, where machine learning is the broadest category and the deep reinforcement learning the most narrow one.
In the same way, reinforcement learning is a specialized application of machine and deep learning techniques, designed to solve problems in a particular way.

Although the ideas seem to differ, there is no sharp divide between these subtypes. Moreover, they merge within projects, as the models are designed not to stick to a “pure type” but to perform the task in the most effective way possible. So “what precisely distinguishes machine learning, deep learning and reinforcement learning” is actually a tricky question to answer.

  • Machine learning – is a form of AI in which computers are given the ability to progressively improve the performance of a specific task with data, without being directly programmed ( this is Arthur Lee Samuel’s definition. He coined the term “machine learning”, of which there are two types, supervised and unsupervised machine learning

Supervised machine learning happens when a programmer can provide a label for every training input into the machine learning system.

  • Example – by analyzing the historical data taken from coal mines, deepsense.ai prepared an automated system for predicting dangerous seismic events up to 8 hours before they occur. The records of seismic events were taken from 24 coal mines that had collected data for several months. The model was able to recognize the likelihood of an explosion by analyzing the readings from the previous 24 hours.
AAIA16 Data Mining Challenge Seismic Events Height Randomization

Some of the mines can be exactly identified by their main working height values. To obstruct the identification, we added some Gaussian noise

From the AI point of view, a single model was performing a single task on a clarified and normalized dataset. To get more details on the story, read our blog post.
Unsupervised learning takes place when the model is provided only with the input data, but no explicit labels. It has to dig through the data and find the hidden structure or relationships within. The designer might not know what the structure is or what the machine learning model is going to find.

  • An example we employed was for churn prediction. We analyzed customer data and designed an algorithm to group similar customers. However, we didn’t choose the groups ourselves. Later on, we could identify high-risk groups (those with a high churn rate) and our client knew which customers they should approach first.
  • Another example of unsupervised learning is anomaly detection, where the algorithm has to spot the element that doesn’t fit in with the group. It may be a flawed product, potentially fraudulent transaction or any other event associated with breaking the norm.

Deep learning consists of several layers of neural networks, designed to perform more sophisticated tasks. The construction of deep learning models was inspired by the design of the human brain, but simplified. Deep learning models consist of a few neural network layers which are in principle responsible for gradually learning more abstract features about particular data.
Although deep learning solutions are able to provide marvelous results, in terms of scale they are no match for the human brain. Each layer uses the outcome of a previous one as an input and the whole network is trained as a single whole. The core concept of creating an artificial neural network is not new, but only recently has modern hardware provided enough computational power to effectively train such networks by exposing a sufficient number of examples. Extended adoption has brought about frameworks like TensorFlow, Keras and PyTorch, all of which have made building machine learning models much more convenient.

  • Example: deepsense.ai designed a deep learning-based model for the National Oceanic and Atmospheric Administration (NOAA). It was designed to recognize Right whales from aerial photos taken by researchers. For further information about this endangered species and deepsense.ai’s work with the NOAA, read our blog post. From a technical point of view, recognizing a particular specimen of whales from aerial photos is pure deep learning. The solution consists of a few machine learning models performing separate tasks. The first one was in charge of finding the head of the whale in the photograph while the second normalized the photo by cutting and turning it, which ultimately provided a unified view (a passport photo) of a single whale.


The third model was responsible for recognizing particular whales from photos that had been prepared and processed earlier. A network composed of 5 million neurons located the blowhead bonnet-tip. Over 941,000 neurons looked for the head and more than 3 million neurons were used to classify the particular whale. That’s over 9 million neurons performing the task, which may seem like a lot, but pales in comparison to the more than 100 billion neurons at work in the human brain. We later used a similar deep learning-based solution to diagnose diabetic retinopathy using images of patients’ retinas.
Reinforcement learning, as stated above employs a system of rewards and penalties to compel the computer to solve a problem by itself. Human involvement is limited to changing the environment and tweaking the system of rewards and penalties. As the computer maximizes the reward, it is prone to seeking unexpected ways of doing it. Human involvement is focused on preventing it from exploiting the system and motivating the machine to perform the task in the way expected. Reinforcement learning is useful when there is no “proper way” to perform a task, yet there are rules the model has to follow to perform its duties correctly. Take the road code, for example.

  • Example: By tweaking and seeking the optimal policy for deep reinforcement learning, we built an agent that in just 20 minutes reached a superhuman level in playing Atari games. Similar algorithms in principal can be used to build AI for an autonomous car or a prosthetic leg. In fact, one of the best ways to evaluate the reinforcement learning approach is to give the model an Atari video game to play, such as Arkanoid or Space Invaders. According to Google Brain’s Marc G. Bellemare, who introduced Atari video games as a reinforcement learning benchmark, “although challenging, these environments remain simple enough that we can hope to achieve measurable progress as we attempt to solve them”.
Breakout
Initial performance After 15 minutes of training After 30 minutes of training
Playing atari with deep reinforcement learning - 0 Playing atari with deep reinforcement learning - 1 Playing atari with deep reinforcement learning - 2
Assault
Initial performance After 15 minutes of training After 30 minutes of training
Playing atari with deep reinforcement learning - 3 Playing atari with deep reinforcement learning - 4 Playing atari with deep reinforcement learning - 5

In particular, if artificial intelligence is going to drive a car, learning to play some Atari classics can be considered a meaningful intermediate milestone. A potential application of reinforcement learning in autonomous vehicles is the following interesting case. A developer is unable to predict all future road situations, so letting the model train itself with a system of penalties and rewards in a varied environment is possibly the most effective way for the AI to broaden the experience it both has and collects.

Related:  Playing Atari with deep reinforcement learning - deepsense.ai’s approach

Conclusion

The key distinguishing factor of reinforcement learning is how the agent is trained. Instead of inspecting the data provided, the model interacts with the environment, seeking ways to maximize the reward. In the case of deep reinforcement learning, a neural network is in charge of storing the experiences and thus improves the way the task is performed.

Is reinforcement learning the future of machine learning?

Although reinforcement learning, deep learning, and machine learning are interconnected no one of them in particular is going to replace the others. Yann LeCun, the renowned French scientist and head of research at Facebook, jokes that reinforcement learning is the cherry on a great AI cake with machine learning the cake itself and deep learning the icing. Without the previous iterations, the cherry would top nothing.
In many use cases, using classical machine learning methods will suffice. Purely algorithmic methods not involving machine learning tend to be useful in business data processing or managing databases.
Sometimes machine learning is only supporting a process being performed in another way, for example by seeking a way to optimize speed or efficiency.
When a machine has to deal with unstructured and unsorted data, or with various types of data, neural networks can be very useful. How machine learning improved the quality of machine translation has been described by The New York Times.

Summary

Reinforcement learning is no doubt a cutting-edge technology that has the potential to transform our world. However, it need not be used in every case. Nevertheless, reinforcement learning seems to be the most likely way to make a machine creative – as seeking new, innovative ways to perform its tasks is in fact creativity. This is already happening: DeepMind’s now famous AlphaGo played moves that were first considered glitches by human experts, but in fact secured victory against one of the strongest human players, Lee Sedol.
Thus, reinforcement learning has the potential to be a groundbreaking technology and the next step in AI development.

https://deepsense.ai/wp-content/uploads/2019/02/what-is-reinforcement-learning-the-complete-guide.jpg 337 1140 Błażej Osiński https://deepsense.ai/wp-content/uploads/2019/04/DS_logo_color.svg Błażej Osiński2018-07-05 13:57:132021-01-05 16:47:55What is reinforcement learning? The complete guide
Keras or PyTorch as your first deep learning framework

Keras or PyTorch as your first deep learning framework

June 26, 2018/in Data science, Deep learning, Machine learning /by Piotr Migdal and Rafał Jakubanis

So, you want to learn deep learning? Whether you want to start applying it to your business, base your next side project on it, or simply gain marketable skills – picking the right deep learning framework to learn is the essential first step towards reaching your goal.

We strongly recommend that you pick either Keras or PyTorch. These are powerful tools that are enjoyable to learn and experiment with. We know them both from the teacher’s and the student’s perspective. Piotr has delivered corporate workshops on both, while Rafał is currently learning them.
(See the discussion on Hacker News and Reddit).

Introduction

Keras and PyTorch are open-source frameworks for deep learning gaining much popularity among data scientists.

  • Keras is a high-level API capable of running on top of TensorFlow, CNTK, Theano, or MXNet (or as tf.contrib within TensorFlow). Since its initial release in March 2015, it has gained favor for its ease of use and syntactic simplicity, facilitating fast development. It’s supported by Google.
  • PyTorch, released in October 2016, is a lower-level API focused on direct work with array expressions. It has gained immense interest in the last year, becoming a preferred solution for academic research, and applications of deep learning requiring optimizing custom expressions. It’s supported by Facebook.

Before we discuss the nitty-gritty details of both frameworks (well described in this Reddit thread), we want to preemptively disappoint you – there’s no straight answer to the ‘which one is better?’. The choice ultimately comes down to your technical background, needs, and expectations. This article aims to give you a better idea of where each of the two frameworks you should be pick as the first.

TL;DR:

Keras may be easier to get into and experiment with standard layers, in a plug & play spirit.
PyTorch offers a lower-level approach and more flexibility for the more mathematically-inclined users.

Ok, but why not any other framework?

TensorFlow is a popular deep learning framework. Raw TensorFlow, however, abstracts computational graph-building in a way that may seem both verbose and not-explicit. Once you know the basics of deep learning, that is not a problem. But for anyone new to it, sticking with Keras as its officially-supported interface should be easier and more productive.
[Edit: Recently, TensorFlow introduced Eager Execution, enabling the execution of any Python code and making the model training more intuitive for beginners (especially when used with tf.keras API).]
While you may find some Theano tutorials, it is no longer in active development. Caffe lacks flexibility, while Torch uses Lua (though its rewrite is awesome :)). MXNet, Chainer, and CNTK are currently not widely popular.

Keras vs. PyTorch: Ease of use and flexibility

Keras and PyTorch differ in terms of the level of abstraction they operate on.
Keras is a higher-level framework wrapping commonly used deep learning layers and operations into neat, lego-sized building blocks, abstracting the deep learning complexities away from the precious eyes of a data scientist.
PyTorch offers a comparatively lower-level environment for experimentation, giving the user more freedom to write custom layers and look under the hood of numerical optimization tasks. Development of more complex architectures is more straightforward when you can use the full power of Python and access the guts of all functions used. This, naturally, comes at the price of verbosity.
Consider this head-to-head comparison of how a simple convolutional network is defined in Keras and PyTorch:

Keras

model = Sequential()
model.add(Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3)))
model.add(MaxPool2D())
model.add(Conv2D(16, (3, 3), activation='relu'))
model.add(MaxPool2D())
model.add(Flatten())
model.add(Dense(10, activation='softmax'))

PyTorch

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 32, 3)
        self.conv2 = nn.Conv2d(32, 16, 3)
        self.fc1 = nn.Linear(16 * 6 * 6, 10)
        self.pool = nn.MaxPool2d(2, 2)
    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 6 * 6)
        x = F.log_softmax(self.fc1(x), dim=-1)
        return x
model = Net()

The code snippets above give a little taste of the differences between the two frameworks. As for the model training itself – it requires around 20 lines of code in PyTorch, compared to a single line in Keras. Enabling GPU acceleration is handled implicitly in Keras, while PyTorch requires us to specify when to transfer data between the CPU and GPU.
If you’re a beginner, the high-levelness of Keras may seem like a clear advantage. Keras is indeed more readable and concise, allowing you to build your first end-to-end deep learning models faster, while skipping the implementational details. Glossing over these details, however, limits the opportunities for exploration of the inner workings of each computational block in your deep learning pipeline. Working with PyTorch may offer you more food for thought regarding the core deep learning concepts, like backpropagation, and the rest of the training process.
That said, Keras, being much simpler than PyTorch, is by no means a toy – it’s a serious deep learning tool used by beginners, and seasoned data scientists alike.
For instance, in the Dstl Satellite Imagery Feature Detection Kaggle competition, the 3 best teams used Keras in their solutions, while our deepsense.ai team (4th place) used a combination of PyTorch and (to a lesser extend) Keras.
Whether your applications of deep learning will require flexibility beyond what pure Keras has to offer is worth considering. Depending on your needs, Keras might just be that sweet spot following the rule of least power.

SUMMARY

  • Keras – more concise, simpler API
  • PyTorch – more flexible, encouraging deeper understanding of deep learning concepts

Keras vs. PyTorch: Popularity and access to learning resources

A framework’s popularity is not only a proxy of its usability. It is also important for community support – tutorials, repositories with working code, and discussions groups. As of June 2018, Keras and PyTorch are both enjoying growing popularity, both on GitHub and arXiv papers (note that most papers mentioning Keras mention also its TensorFlow backend). According to a KDnuggets survey, Keras and PyTorch are the fastest growing data science tools.
Keras or PyTorch - arvix_graph_cut

Unique mentions of deep learning frameworks in arxiv papers (full text) over time, based on 43K ML papers over last 6 years. So far TF mentioned in 14.3% of all papers, PyTorch 4.7%, Keras 4.0%, Caffe 3.8%, Theano 2.3%, Torch 1.5%, mxnet/chainer/cntk <1%. (cc @fchollet) pic.twitter.com/YOYAvc33iN

— Andrej Karpathy (@karpathy) 10 marca 2018

While both frameworks have satisfactory documentation, PyTorch enjoys stronger community support – their discussion board is a great place to visit to if you get stuck (you will get stuck) and the documentation or StackOverflow don’t provide you with the answers you need.
Anecdotally, we found well-annotated beginner level deep learning courses on a given network architecture easier to come across for Keras than for PyTorch, making the former somewhat more accessible for beginners. The readability of code and the unparalleled ease of experimentation Keras offers may make it the more widely covered by deep learning enthusiasts, tutors and hardcore Kaggle winners.
For examples of great Keras resources and deep learning courses, see “Starting deep learning hands-on: image classification on CIFAR-10“ by Piotr Migdał and “Deep Learning with Python” – a book written by François Chollet, the creator of Keras himself. For PyTorch resources, we recommend the official tutorials, which offer a slightly more challenging, comprehensive approach to learning the inner-workings of neural networks. For a concise overview of PyTorch API, see this article.

SUMMARY

  • Keras – Great access to tutorials and reusable code
  • PyTorch – Excellent community support and active development

Keras vs. PyTorch: Debugging and introspection

Keras, which wraps a lot of computational chunks in abstractions, makes it harder to pin down the exact line that causes you trouble.
PyTorch, being the more verbose framework, allows us to follow the execution of our script, line by line. It’s like debugging NumPy – we have easy access to all objects in our code and are able to use print statements (or any standard Pythonic debugging) to see where our recipe failed.
A Keras user creating a standard network has an order of magnitude fewer opportunities to go wrong than does a PyTorch user. But once something goes wrong, it hurts a lot and often it’s difficult to locate the actual line of code that breaks. PyTorch offers a more direct, unconvoluted debugging experience regardless of model complexity. Moreover, when in doubt, you can readily lookup PyTorch repo to see its readable code.

SUMMARY

  • PyTorch – way better debugging capabilities
  • Keras – (potentially) less frequent need to debug simple networks

Keras vs. PyTorch: Exporting models and cross-platform portability

What are the options for exporting and deploying your trained models in production?
PyTorch saves models in Pickles, which are Python-based and not portable, whereas Keras takes advantages of a safer approach with JSON + H5 files (though saving with custom layers in Keras is generally more difficult). There is also Keras in R, in case you need to collaborate with a data analyst team using R.
Running on Tensorflow, Keras enjoys a wider selection of solid options for deployment to mobile platforms through TensorFlow for Mobile and TensorFlow Lite. Your cool web apps can be deployed with TensorFlow.js or keras.js. As an example, see this deep learning-powered browser plugin detecting trypophobia triggers, developed by Piotr and his students.
Exporting PyTorch models is more taxing due to its Python code, and currently the widely recommended approach is to start by translating your PyTorch model to Caffe2 using ONNX.

SUMMARY

  • Keras – more deployment options (directly and through the TensorFlow backend), easier model export.

Keras vs. PyTorch: Performance

Donald Knuth famously said:

Premature optimization is the root of all evil (or at least most of it) in programming.

In most instances, differences in speed benchmarks should not be the main criterion for choosing a framework, especially when it is being learned. GPU time is much cheaper than a data scientist’s time. Moreover, while learning, performance bottlenecks will be caused by failed experiments, unoptimized networks, and data loading; not by the raw framework speed. Yet, for completeness, we feel compelled to touch on this subject. We recommend these two comparisons:

  • TensorFlow, Keras and PyTorch comparison by Wojtek Rosiński
  • Comparing Deep Learning Frameworks: A Rosetta Stone Approach by Microsoft (make sure to check notebooks to get the taste of different frameworks). For a detailed explanation of the multi-GPU framework comparisons, see this article.

PyTorch is as fast as TensorFlow, and potentially faster for Recurrent Neural Networks. Keras is consistently slower. As the author of the first comparison points out, gains in computational efficiency of higher-performing frameworks (ie. PyTorch & TensorFlow) will in most cases be outweighed by the fast development environment, and the ease of experimentation Keras offers.


SUMMARY:

  • As far as training speed is concerned, PyTorch outperforms Keras

Keras vs. PyTorch: Conclusion

Keras and PyTorch are both excellent choices for your first deep learning framework to learn.

If you’re a mathematician, researcher, or otherwise inclined to understand what your model is really doing, consider choosing PyTorch. It really shines, where more advanced customization (and debugging thereof) is required (e.g. object detection with YOLOv3 or LSTMs with attention) or when we need to optimize array expressions other than neural networks (e.g. matrix decompositions or word2vec algorithms).

Keras is without a doubt the easier option if you want a plug & play framework: to quickly build, train, and evaluate a model, without spending much time on mathematical implementation details.
EDIT: For side-by-side code comparison on a real-life example, see our new article: Keras vs. PyTorch: Alien vs. Predator recognition with transfer learning.

Knowledge of the core concepts of deep learning is transferable. Once you master the basics in one environment, you can apply them elsewhere and hit the ground running as you transition to new deep learning libraries.

We encourage you to try out simple deep learning recipes in both Keras and PyTorch. What are your favourite and least favourite aspects of each? Which framework experience appeals to you more? Let us know in the comment section below!

Would you and your team like to learn more about deep learning in Keras, TensorFlow and PyTorch? See our tailored training offers.

https://deepsense.ai/wp-content/uploads/2019/02/Keras-or-PyTorch.png 337 1140 Piotr Migdal https://deepsense.ai/wp-content/uploads/2019/04/DS_logo_color.svg Piotr Migdal2018-06-26 13:43:322021-01-05 16:48:00Keras or PyTorch as your first deep learning framework
Learning to run - an example of reinforcement learning

Learning to run – an example of reinforcement learning

June 22, 2018/in Deep learning, Machine learning /by Konrad Budek

Turns out a walk in the park is not so simple after all. In fact, it is a complex process done by controlling multiple muscles and coordinating who knows how many motions. If carbon-based lifeforms have been developing these aspects of walking for millions of years, can AI recreate it?

This blog will describe:

  • How reinforcement learning works in practical usage
  • The process used to learn the model
  • Challenges in reinforcement learning
  • How knowledge is transferred between neural networks and why it is important for the development of artificial intelligence

Moving by controlling the muscles attached to bones, as humans do it, is way more complicated and harder to recreate than building a robot that can move with engines and hydraulic cylinders.
Building a model that can run by controlling human muscles recreated in a simulated environment was the goal of a competition organized at the NIPS 2017 conference. Designing the model with reinforcement learning was a part of a scientific project that could potentially be used to build software for sophisticated prostheses, which allow people to live normally after serious injuries.
Software that understands muscle-controlled limb movement would be able to translate the neural signals into instructions for an automated arm or leg. On the other hand, it may also be possible to artificially stimulate the muscles to move in a particular way, allowing paralyzed people to move again.

Why reinforcement learning

Our RL Agent had to move the humanoid by controlling 18 muscles attached to bones. The simulation was done in an OpenSim environment. Such environments are used mainly in medicine to determine how changes in physiology are going to affect a human’s ability to move. For example, if a patient with a shorter tendon or bone will still be able to walk or grab something with his hand. The surprising challenge was the environment itself – OpenSims require a lot of computational power.

Related:  Playing Atari with deep reinforcement learning - deepsense.ai’s approach

Building hard-coded software to control a realistic biomechanical model of a human body would be quite a challenge, even if researchers from Stanford University have done just that. But training a neural network to perform this task proved to be much more efficient and less time-consuming, and didn’t require biomechanical domain specific knowledge.

Run Stephen! Run!

Our reinforcement learning algorithm leverages a system of rewards and punishments to acquire useful behaviour. During the first experiments, our agent (whom we called Stephen)randomly performed his actions, with no hints from the designer. His goal was to maximize the rewards involved by learning which actions, done randomly, yielded the best effect. Basically, the model had to figure out how to walk over the course of a few days, a much shorter time than the few billion years it took carbon-based lifeforms.

In this case, Stephen got a reward for every meter he travelled. During the first trials, he frequently fell over, sometimes forward, sometimes backward. With enough trials, it managed to fall only forward, then to jump or take its first step.

The curriculum, or step-by-step learning

After enough trials, Stephen learned that jumping forward is a good way to maximize the future reward. As a jumper, he was not that bad – he got from point A to point B by effectively controlling his muscles. He didn’t fall and was able to move quickly.
Learning to run - an example of reinforcement learning
But our goal for Stephen was not “learning to hop”- it was “learning to run”. Jumping was a sub-optimal form of locomotion.
This prompted the need for a curriculum, or, in other words, a tutoring program. Instead of training Stephen to avoid obstacles and run at the same time, we would teach him progressively harder skills – first to walk on a straight road, then to run and, finally, to avoid obstacles. Learn to walk before you run, right?

To reduce his tendency to jump and instead find a way to walk, we had to get Stephen to explore different options such as moving his legs separately.
We opted to use a relatively small neural network that would be able to learn to walk on a path without any obstacles. He succeeded at this, but during the process, he had a Jon Snowesque problem with his knee.
Learning to run - an example of reinforcement learning 1
Anyone who has ever aspired to sports stardom will remember a coach admonishing them to bend their knees. Apparently, the failure to do so is common among all walkers, including simulated ones controlled by an artificial neural network. Reshaping the reward function was the only way to communicate with the agent. As the human creators, we of course know just what walking should look like, but the neural network had no clue. So adding an award for Stephen for bending his knees was a good way to improve his performance and find a better policy.
StefanSillyWalk - reinforcement learning example
If any human had his walk from that moment, it would be wise to apply for a government grant to develop it.

When Stephen finally worked out how to walk and run effectively, we added another, bigger neural network to figure out how to avoid obstacles. At that point, one neural network was controlling the running process while the second one figured out how to tweak Stephen’s movement to avoid obstacles and not fall.
This is a novel technique which we called policy blending. The usual way to make a neural network bigger and teach it new skills is behavioral cloning, which is a machine learning interpretation of the master-apprentice relation. The new, bigger deep neural network watches how the smaller one performs its tasks.
For this task, our method of policy blending has been outperforming behavioural cloning. For further information, please read a scientific paper we contributed to. It presents interesting ideas employed during the challenge. After Stephen learned how to move and avoid rocks in his way, we blended another neural network encouraging him to run even faster.
Learning to run - an example of reinforcement learning 3
With policy blending and enough computational power, Stephen managed to run in a human way without falling. With 10 random obstacles to navigate, Stephen fell in less than 8% of trials. When he was moving more carefully (about 20% slower), the falls ratio fell (pardon the pun) to below 0.5%.

After the run – the effects of reinforcement learning

The experiment brought a few significant outcomes.
First, it is possible for a computer to perform the tremendously complicated task of walking with separate and coordinated control of the muscles. The agent was able to figure out how to do that using reinforcement learning alone – it did not need to observe human movement.
Moreover, the policy blending method proved effective and outperformed the standard behaviour cloning approach. Although it is not certain that it will be more efficient in every possible case, it is another, sometimes better way to transfer knowledge from one trained network to another.
Finally, we handled the resource-demanding environment by effectively splitting the computations between nodes of a large cluster. So even within the complex and heavy simulator, reinforcement learning may be not only possible, but effective.

https://deepsense.ai/wp-content/uploads/2018/06/Learning-to-run-an-example-of-reinforcement-learning.png 337 1140 Konrad Budek https://deepsense.ai/wp-content/uploads/2019/04/DS_logo_color.svg Konrad Budek2018-06-22 14:24:272021-01-05 16:48:05Learning to run – an example of reinforcement learning
Page 3 of 6‹12345›»

Start your search here

NEWSLETTER SUBSCRIPTION

    You can modify your privacy settings and unsubscribe from our lists at any time (see our privacy policy).

    This site is protected by reCAPTCHA and the Google privacy policy and terms of service apply.

    THE NEWEST AI MONTHLY DIGEST

    • AI Monthly Digest 20 - TL;DRAI Monthly Digest 20 – TL;DRMay 12, 2020

    CATEGORIES

    • Artificial Intelligence
    • AIOps
    • Big data & Spark
    • Data science
    • Deep learning
    • Machine learning
    • Neptune
    • Reinforcement learning
    • Seahorse
    • Job offer
    • Popular posts
    • AI Monthly Digest
    • Press release

    POPULAR POSTS

    • AI trends for 2021AI trends for 2021January 7, 2021
    • A comprehensive guide to demand forecastingA comprehensive guide to demand forecastingMay 28, 2019
    • What is reinforcement learning? The complete guideWhat is reinforcement learning? The complete guideJuly 5, 2018

    Would you like
    to learn more?

    Contact us!
    • deepsense.ai logo white
    • Industries
    • Retail
    • Manufacturing
    • Financial Services
    • IT Operations
    • TMT and Other
    • Knowledge base
    • Blog
    • Press center
    • deepsense.ai
    • Our story
    • Management
    • Scientific Advisory Board
    • Careers
    • Support
    • Terms of service
    • Privacy policy
    • Contact us
    • Join our community
    • facebook logo linkedin logo twitter logo
    • © deepsense.ai 2014-
    Scroll to top

    This site uses cookies. By continuing to browse the site, you are agreeing to our use of cookies.

    OKLearn more

    Cookie and Privacy Settings



    How we use cookies

    We may request cookies to be set on your device. We use cookies to let us know when you visit our websites, how you interact with us, to enrich your user experience, and to customize your relationship with our website.

    Click on the different category headings to find out more. You can also change some of your preferences. Note that blocking some types of cookies may impact your experience on our websites and the services we are able to offer.

    Essential Website Cookies

    These cookies are strictly necessary to provide you with services available through our website and to use some of its features.

    Because these cookies are strictly necessary to deliver the website, refuseing them will have impact how our site functions. You always can block or delete cookies by changing your browser settings and force blocking all cookies on this website. But this will always prompt you to accept/refuse cookies when revisiting our site.

    We fully respect if you want to refuse cookies but to avoid asking you again and again kindly allow us to store a cookie for that. You are free to opt out any time or opt in for other cookies to get a better experience. If you refuse cookies we will remove all set cookies in our domain.

    We provide you with a list of stored cookies on your computer in our domain so you can check what we stored. Due to security reasons we are not able to show or modify cookies from other domains. You can check these in your browser security settings.

    Other external services

    We also use different external services like Google Webfonts, Google Maps, and external Video providers. Since these providers may collect personal data like your IP address we allow you to block them here. Please be aware that this might heavily reduce the functionality and appearance of our site. Changes will take effect once you reload the page.

    Google Webfont Settings:

    Google Map Settings:

    Google reCaptcha Settings:

    Vimeo and Youtube video embeds:

    Privacy Policy

    You can read about our cookies and privacy settings in detail on our Privacy Policy Page.

    Accept settingsHide notification only