Cost of risk is one of the biggest components in banks’ cost structure. Thus, even a slight improvement in credit risk modelling can translate in huge savings. That’s why machine learning is often implemented in this area.
We would like to share with you some insights from one of our projects, where we applied machine learning to increase credit scoring performance.To illustrate our insights we selected a random pool of 10 000 applications.
How the regular process works
Loan applications are usually assessed through a credit score model, which is most often based on a logistic regression (LR). It is trained on historical data, such as credit history. The model assesses the importance of every attribute provided and translates them into a prediction.
The main limitation of such a model is that it can take into account only linear dependencies between input variables and the predicted variable. On the other hand, it is this very property that makes logistic regression so interpretable. LR is in widespread used in credit risk modelling.
Credit scoring from a logistic regression model
What machine learning brings to the table
Machine learning enables the utilization of more advanced modeling techniques, such as decision trees and neural networks. This introduces non-linearities to the model and allows to detect more complex dependencies between the attributes. We decided to use an XGBoost model fed with features selected with the use of a method called permutation importance.
Credit scoring from tree-based model
However, ML models are usually so sophisticated that they are hard to interpret. Since a lack of interpretability would be a serious issue in such a highly regulated field as credit risk assessment, we opted to combine XGBoost and logistic regression.
Combining the models
We used both scoring engines – logistic regression and the ML based one – to assess all of the loan applications
With a clear correlation between the two assessment approaches, a high score in one model would likely mean a high score in the other.
Loan applications assessed by 2 models
In the original approach, logistic regression was used to assess applications. The acceptance level was set around 60% and the risk resulted at 1%
Initial credit application split (acceptance to portfolio risk)
If we decrease the threshold by a couple of points, the acceptance level hits 70% while the risk jumps to 1,5%
Credit applications’ split after lowering the threshold
We next applied a threshold for an ML model, allowing us to get an acceptance percentage to the original level (60%) while bringing the risk down to 0,75% that is by 25% lower than the risk level resulting from only traditional approach.
Credit applications’ split after applying Machine Learning
Summary
Machine learning is often seen as difficult to apply in banking due to the sheer amount of regulation the industry faces. The facts don’t necessarily back this up. ML is successfully used in numerous, heavily regulated industries. The example above is just one more example of how. Thanks to this innovative approach it is possible to increase the sustainability of the loans sector and make loans even more affordable to bank customers. There’s nothing artificial about that kind of intelligence.
https://deepsense.ai/wp-content/uploads/2020/07/Using-Machine-Learning-in-credit-risk-modelling-to-reduce-risk-costs.jpg3371140deepsense.aihttps://deepsense.ai/wp-content/uploads/2023/10/Logo_black_blue_CLEAN_rgb.pngdeepsense.ai2021-05-05 09:31:322021-05-13 13:36:22Using machine learning in credit risk modelling
AI has become a powerful force in computer vision and it has unleashed tangible business opportunities for 2D visual data such as images and videos. Applying AI can bring tremendous results in a number of fields. To learn more about this exciting area, read our overview of 2D computer vision algorithms and applications.
Despite its popularity, there is nothing inherent to 2D imagery that makes it uniquely suitable for AI application. In fact, artificial intelligence systems can analyze various forms of information, including volumetric data. In spite of the increasing number of companies already using 3D data gathered by lidar or 3D cameras, AI applications aren’t the mainstream in their industries.
In this post, we describe how to leverage 3D data across multiple industries with the use of AI. Later in the article we’ll have a closer look at the nuts and bolts of the technology and we’ll aslo show what it takes to apply AI to 3D data. At the end of the post, you’ll also find an interactive demo to play with.
In the 3D world, there is no Swiss Army Knife
3D data is what we call volumetric information. The most common types include:
2.5D data, including information on depth or the distance to visible objects, but no volumetric information of what’s hidden behind them. Lidar data is an example.
3D data, with full volumetric information. Examples include MRI scans or objects rendered with computer graphics.
4D data, where volumetric information is captured as a sequence, and the outcome is a recording where one can go back and forth in time to see the changes occurring in the volume. We refer to this as 3D + time, which we can treat as the 4th dimension. Such representation enables us to visualize and model dynamic 3D processes, which is especially useful in medical applications such as respiratory or cardiac monitoring.
There are also multiple data representations. These include a compound of 2D images along the normal axis, sparse Point Cloud representation and voxelized representation. Such data could have additional channels, like reflectance in every point of a lidar’s view.
Depending on the business need, there can be different objectives for using AI: object detection and classification, semantic segmentation, instance segmentation and movement parameterization, to name a few. Moreover, every setup has its own characteristics and limitations that should be addressed with a dedicated approach (or, in the case of artificial neural networks, with a sophisticated and thoroughly designed architecture). These are the main reasons our clients come to us, and to take advantage of our experience in the field. We are responsible for delivering the AI part of specific projects, even though the majority of their competencies are built in-house.
Let us have a closer look at a few examples
1. Autonomous driving
Task: 3D object detection and classification,
Data: 2.5 Point clouds captured with a lidar: sparse data, big distances between points
Autonomous driving data are very sparse because:
the distances between objects in outdoor environments are significant
In the majority of cases lidar rays from the front and rear of the car don’t return to lidar, since there are no objects to reflect them.
The resolution of objects gets worse the further they are from the laser scanner. Due to the angular expansion of the beam it’s impossible to determine the precise shape of objects that are far away.
For autonomous driving, we needed a system that can take advantage of data sparsity to infer 3D bounding boxes around objects. One such network is the part-aware and aggregation neural network i.e. Part-A2 net (https://arxiv.org/abs/1907.03670). This is a two-stage network that uses the high separability of objects, which functions as segmentation information.
In the first stage, the network estimates the position of foreground points of objects inside bounding boxes generated by an anchor-based or anchor-free scheme. Then, in the second stage, the network aggregates local information for box refinement and class estimation. The network output is shown below, with the colors of points in bounding boxes showing their relative location as perceived by the Part-A2 net.
Data: Point clouds, sparse data, relatively small distances between points
A different setup is called for in mapping indoor environments, such as we do with instance segmentation of objects in office space or shops (see this dataset for better intuition: S3DIS dataset). Here we employ a relatively high-density representation of a point cloud and BoNet architecture.
In this case the space is divided into a 1- x 1- x 1-meter cubic grid. In each cube, a few thousand points are sampled for further processing. In an autonomous driving scenario, such a grid division would make little sense given the sheer number of cubes produced, many of which are empty and only a few of which contain any relevant information.
The network produces semantic segmentation masks as well as bounding boxes. The inference is a two-stage process. The first produces a global feature vector to predict a fixed number of bounding boxes. It also tallies scores to indicate whether some of the predicted classes are inside those boxes. The point-level and global features derived in the first stage are then used to predict a point-level binary mask with the class assignment. The pictures below show a typical scene with the segmentation masks.
An example from the S3DIS dataset. From left: input image, semantic segmentation labels, instance segmentation labels
3. Medical diagnosis
Task: 3D Semantic segmentation
Data: Stacked 2D images, dense data, small distance between images
This is a highly controlled setup, where all 2D images are carefully and densely stacked together. Such a representation can be treated as a natural extension of a 2D setup. In such cases, modifying existing 2D approaches will deliver satisfactory results.
An example of a modified 2D approach is the 3D U-Net (https://arxiv.org/abs/1606.06650), where all 2D operations for a classical U-Net are replaced by their 3D counterparts. If you want to know more about AI in medicine, check out how it can be used to help with COVID-19 diagnosis and other challenges.
There is also another case, where luckily, it can be relatively straightforward to apply expertise and technology developed for 2D cases in 3D applications. One such scenario is where there are 2D labels available, but the data and the inference products are in 3D. Another is when 3D information can play a supportive role.
In such a case, a depth map produced by 3D cameras can be treated as an additional image channel beyond regular RGB colors. Such additional information increases the sensitivity of neural networks to edge detection and thus yield better object boundaries.
Examples of the projects we have delivered in such a setup include:
Defect detection based on 2D and 3D images.
We developed an AI system for a tire manufacturer to detect diverse types of defects. 3D data played a crucial role as it allowed for ultra-precise detection of submillimeter-size bubbles and scratches.
Object detection in a factory
We designed a system to detect and segment industrial assets in a chemical facility that had been thoroughly scanned with high resolution laser scanners. Combining 2D and 3D information allowed us to digitize the topology of the installation and its pipe system.
3D data needs a mix of competencies
At deepsense.ai, we have a team of data scientists and software engineers handling the algorithmic, visualization, and integration capabilities. Our teams are set up to flexibly adapt to specific business cases and provide tailor-made AI solutions. The solutions they produce are an alternative to pre-made, off-the-shelf products, which often prove too rigid and constrained; they fail once user expectations deviate from the assumptions of their designers.
Processing and visualizing data in near real time with appropriate user experience is no piece of cake. Doing so requires a tough balancing act, including
combining specific business needs, technical limitations resulting from huge data loads and the need to support multiple platforms.
It is always easier to discuss based on an example. Next section shows what it takes to develop an object detection system for autonomous vehicles with outputs accessible from a web browser. The goal is to predict bounding boxes of 3 different classes: car, pedestrian and cyclist, 360 degrees around the car. Such a project can be divided into 4 interconnected components: data processing, algorithms, visualizations and deployment.
Data preprocessing
In our example, we use the KITTI and A2D2 datasets, two common datasets for autonomous driving, and ones our R&D hub rely on heavily. In both datasets, we use data from spinning lidars for inference and cameras for visualization purposes.
Lidars and cameras work independently, capturing data at different rates. To obtain a full picture, all data have to be mapped to a common coordinate system and adjusted for time. This is no easy task. As lidars are constantly spinning, each point is captured at a different time, while simultaneously the position and rotation of the car in relation to world coordinates is changing. Meanwhile, the precise location and angle of the car is not known perfectly due to limitations of geolocation systems such as GPS. These difficulties make it extremely difficult to precisely and stably determine the absolute positions of objects around you (SLAM can be used to tackle some of the problems).
Fortunately, absolute positioning of objects around the vehicle is not always required.
Algorithms
There are a vast number of approaches when it comes to 3D data. However, factors such as the length to and between objects and high sparsity will play an essential role in which algorithm we ultimately settle on. As in the first example above, we used Part-A2 net.
Deployment
We have relied on a complete, in-house solution for visualization, data handling, and UI. We have used expertise in the Unity engine to develop a cross-platform, graphically rich and fully flexible solution. In terms of a platform, we opted for maximum availability, which can be satisfied by a popular web browser like Chrome or Mozilla and WebGL as Unity’s compilation platform.
Visualization/UI
WebGL, while very comfortable for the user, disables drive access and advanced GPU features, limits available RAM to 2GB and processing to a single thread. Additionally, while standalone solutions in Unity may rely on existing libraries for point cloud visualization, making it possible to visualize hundreds of millions of points (thanks to advanced GPU features), this is not the case in WebGL.
Therefore, we have developed an in-house visualization solution enabling real-time, in-browser visualization of up to 70 mln points. Give it a try!
Such visualization could be tailored to the company’s specific needs. In a recent project, we took a different approach: we used AR glasses in visualizing a factory in all its complexity. This enabled our client to reach next level user experience and see the factory in a whole new light.
Summary
We hope that this post has shed some light on how AI can be used with 3D data. If you have a particular 3D use case in mind or you are just curious about the potential for AI solutions in your field, please reach out to us. We’ll be happy to share our experience and discuss potential ways we can help you apply the power of artificial intelligence in your business. Please drop us an email at contact@deepsense.ai.
https://deepsense.ai/wp-content/uploads/2020/05/3D-meets-AI-an-unexplored-world-of-new-business-opportunities.jpg3371140Krzysztof Palczewskihttps://deepsense.ai/wp-content/uploads/2023/10/Logo_black_blue_CLEAN_rgb.pngKrzysztof Palczewski2020-05-22 17:40:362021-05-11 10:55:223D meets AI – an unexplored world of new business opportunities
A trigger-based alerting in infrastructure management is currently insufficient to keep operations smooth and uninterrupted. AI-powered IT operations deliver smart alerts that not only improve a team’s ability to respond swiftly but can also deliver significant automation.
According to Gartner data quoted by the TechBeacon, 40% of organizations are predicted to strategically implement AIOps to enhance performance monitoring by 2022. The reason is obvious: AIOps helps build resilient and reliable IT corporate ecosystems. It also boosts overall company performance, an area that can always be improved. For example, for 34.4% of organizations it takes more than 30 minutes on average to resolve IT incidents impacting consumer-facing digital services.
The standard approach to alerting
In the traditional approach to automating IT operations, sets of previously defined conditions triggered alerts.
A huge drawback of this approach was that problems had to be identified, defined and researched. Rules were then set to react to the problem when it occurred. This approach was effective only in dealing with problems that had already come up, but it was useless when conditions changed. That’s common in IT infrastructure.
Also, manually setting the rules opens the process up to mistakes hidden within the code and rules – the more complicated the system or instruction, the better hidden a malfunction can be. And there it lurks, ready to hit right when the rule should have saved the infrastructure.
This is where AI tools come in handy.
How AIOps makes basic alerting smart alerting
The key difference between trigger-based alerts and AI-powered alerts is that the first comes with a huge amount of manual work, while the second can cut workloads significantly by automating the most tedious tasks (read more about that in our What is AIOps blog post).
Both types of alerts send a constant flow of data in from the system, including logs, events or metrics collected by sensors. However, only in the AI alerts is the data preprocessed and initially analyzed before a human gets it–stripped of the noise and chaos usually seen in the raw data.
Pattern recognition-based alerting
Unlike in less sophisticated systems, AI-powered solutions deliver pattern-based, rather than triggered, alerting. The core difference is in recognizing not a particular, single event or a chain of events that occur within the network, but the overall pattern of a malfunction or attack. Thus, if there is a new channel or technology used, the system will remain vigilant. Read more about that in our AIOps for Network Traffic Analysis (NTA) blog post.
Also, the pattern-based approach not only helps spot known types of attacks but also can be useful when approaching the unknown. Harnessing the power of machine learning or reinforcement learning paradigms of supervised learning and reinforcement learning allows the neural network to learn how IT infrastructure fits into the daily work. This essentially makes it the direct opposite of a trigger-based system. Instead of spotting the signs of malicious activity, machine learning models ensure operations can detect anomalies within the system.
Policy-driven escalation and remediation
Smart alerting is a great tool for DevOps or administration specialists, though it remains only one step of their work. Any alert is a problem that needs to be fixed or at least a condition that requires supervision. In keeping IT infrastructure up and running as well, notification remains only a single step. Services lost even briefly can be very costly.
With AIOps implemented, ml-armed supervisors will not only spot the signs of a malfunction but also respond with the best policy available, according to their previous experience and information implemented in the network.
A simple example would be delivering a mirrored piece of infrastructure and redirecting the traffic while providing an alert to the supervisory team. AIOps handles this with alert escalation workflows and immediate response. Also, the AIOps infrastructure correlates the dependencies based on incoming data and responds not to a single event, but to address the group of alerts continuously. The approach saves the time usually spent grouping alerts into same-event related ones.
Key benefits of AIOps alerting
AIOps solutions have been adopted for their numerous benefits, including the following four.
Savings – any anomalous situation in the network can result in costs skyrocketing, especially in the age of growing cloud workloads. When the company is charged for every byte of information stored or transferred, inefficiencies or instructions add up to unbearable costs, especially in the long term.
Security – alerting delivers knowledge about anomalous situations in the system, and any anomaly can be a sign of an ongoing intrusion or malfunction that can end up shutting the system down.
Scalability – smart alerting systems deliver more clear information about the overall state of the infrastructure. Repeating patterns in network performance are a sign to up- or downscale the entire system, be it once, with on-demand cloud solutions, or permanently by adding new components.
Smart cloud management – delivering a real-time alert is crucial to initiating next steps; in an AIOps alert platform, these can be automated and launched contextually, as the current situation demands, not only according to predefined conditions.
Summary
Smart alerting is the foundation of next-level infrastructure management. Companies can harness the power of AI-powered solutions to optimize operations and deliver better results.
If you’d like to hear more about deepsense.ai’s AIOps platform and our approach, contact us using the form below or drop us a line at aiops@deepsense.ai.
https://deepsense.ai/wp-content/uploads/2020/04/Smart-alerting-with-AIOps.jpg3371140Konrad Budekhttps://deepsense.ai/wp-content/uploads/2023/10/Logo_black_blue_CLEAN_rgb.pngKonrad Budek2020-04-21 12:09:502021-01-05 16:43:13Smart alerting with AIOps
Network Traffic Analysis (NTA) is a key component of modern cybersecurity in companies. With machine learning and artificial intelligence solutions, the sheer amounts of data to analyze is an asset to be used rather than, as was once the case, a challenge to overcome.
According to Markets and Markets data, the global network traffic monitoring software and traffic analysis tool market is projected to grow from $1.9 billion in 2019 to $3.2 billion by 2024. The growth is driven mostly by the increasing demand for sophisticated network monitoring tools and advanced network management systems that can handle the growing traffic and increasing flow of information.
The growth in internal traffic is a direct reflection of global trends. According to Cisco data, nearly 66% of the global population will be online by 2023. The increase in traffic is driven not only by users but also by the myriad of connected devices that form the IoT cloud around us.
The share of Machine-to-Machine (M2M) connections is estimated to grow from 33% in 2018 to 50% in 2023, while the consumer segment will rise to 74% of this share and business segment for 26%.
What is network traffic analysis
In its most basic form, Network Traffic Analysis (NTA) is the process of recording and analyzing network traffic patterns in search of suspicious elements and security threats. The term was originally coined by Gartner to describe a growing industry in the computer security ecosystem.
The foundation of NTA is the assumption that there is a “normal” situation in the system that reflects daily operations. Due to seasonal or general trends, operations fluctuate naturally, but overall the system remains stable and thus internal network monitoring can be done with a traffic analyzer. Knowing the “normal” situation is the first step in spotting signs of malicious activities within the system.
In addition to spotting security threats, NTA is also used to optimize the system, spotting inefficiencies as well as the system’s need for additional components when it arises.
Network Traffic Analysis software tools analyze a system’s communication flow, including
TCP/UDP packets
“Virtual network traffic” done in virtual private networks
Traffic to and from cloud environments (storage, computing power, etc.)
API calls to cloud-based apps or SaaS solutions.
This means that nearly all traffic and information flow can be tracked and analyzed by smart network traffic analysis solutions. Modern solutions often use sophisticated techniques like reinforcement learning.
A key component of network analytics tools is the dashboard used to interface with the team, which receives clear information about the network. The dashboard enables easier network performance monitoring and diagnostics and is a convenient way to convey technical knowledge to those who lack it. Reducing complexity to simplicity, the dashboard will
Play its part in convincing your financial director to spring for a powerful new server or another essential component.
NTA solutions are clearly sophisticated and powerful tools. But what are the direct benefits of network traffic analysis?
The benefits of network traffic analysis
There are at least several benefits:
Avoiding bandwidth and server performance bottlenecks – Armed with knowledge about how information flows in the system, one can analyze network problems, define problems and start looking for solutions.
Discovering apps that gobble up bandwidth – tweaking the system can deliver significant savings when API calls are reduced or information is reused.
Proactively reacting to a changing environment – a key feature when it comes to delivering high-quality services for clients and customers. The company can react to increasing demand or spot signs of an approaching peak to harden the network against it. Advanced network traffic analysis tools are often armed with solutions designed to respond in real-time to network changes much faster than any administrator would.
Managing devices exclusively – with modern network monitoring applications companies can group devices and network components to manage them, effectively making use of network performance analytics done earlier.
Resource usage optimization – With all apps, devices, components, and traffic pinpointed with a dashboard, the company can make more informed decisions about the system’s resources and costs.
The key challenge in computer network management is processing and analyzing the gargantuan amounts of data networks produce. Looking for the proverbial needle in the haystack is an apt metaphor for searching for insights among the data mined from a network.
Using ML tools is the only way to effectively monitor network traffic.
How machine learning can support traffic analysis
The key breakthrough that comes from using machine learning-powered tools in NTA is in automation. The lion’s share of the dull and repetitive yet necessary work is done by machines. Also, in real-time network analysis, time is another component that can be handled only by machines. Machines and neural networks can spot and analyze the hidden patterns in data to deliver a range of advantages for companies. To name just a few:
Intrusion detection
The first and sometimes the only sign of intrusion into a system is suspicious traffic that can be easily overlooked. Intrusions are often detected only after 14 days.
AI-based solutions are tireless, analyzing traffic in real-time. Armed with the knowledge of infrastructure operations, the system can spot any sign of malicious activity.
Reducing false positives
AI-based solutions are less prone to the false-positives that can turn the life of a system administrator into a living hell. AI-based systems significantly enrich ML-supported NTA with false-positive detection and reduction, enabling the team to focus more on real challenges than on verifying every alert.
Workload prediction
With data about ongoing system performance, the solution can deliver information about predicted traffic peaks or downs to optimize spending.
Thus the benefits are twofold. First, the company can manage the costs of infrastructure, be it cloud or on-prem, to handle the real traffic and avoid overpaying. Second, there is much more predictability in the estimated need for resources, so they can be booked in advance or the costs can be optimized in other ways.
Spotting early signs of attack (DDoS)
Distributed Denial of Service attacks attempts to suddenly overload a company’s resources in an effort to take down the website or other online service. The losses are hard to predict – from the company’s reputation being hit as unable to defend itself against cybercrime attacks, to the staggering and quickly accruing losses due to being unavailable for customers.
With the early information about incoming attacks, the company can set up defenses like blocking certain traffic, ports or locations to keep availability on other markets. Also, network traffic reports can be used by various agencies that fight cybercrime and will hunt for those responsible for the attack.
Malicious packet detection
Sometimes it is not about the intrusion and the malicious activity is not aimed directly at the company. A user could have downloaded malware onto a private device connected with an enterprise network via a VPN. With that, the infection can spread or the software itself can leverage the company’s resources, such as computing power, and use it for its own purposes, like mining cryptocurrency without the owner’s consent.
Summary
Network traffic monitoring and analysis is one of the key components of modern enterprise-focuses cybersecurity. The gargantuan amounts of data to process also make it a perfect foundation for ML-based solutions, which thrive on data.
That’s why deepsense.ai delivers a comprehensive AIOps architecture-based platform for network data analytics.
If you have any questions about the AIOps solutions we provide, don’t hesitate to contact Andy Thurai, our Head of US operations via the contact form or aiops@deepsense.ai email address.
https://deepsense.ai/wp-content/uploads/2020/03/aiops-network-traffic-analysis-nta-a-business-guide.jpg3371140Konrad Budekhttps://deepsense.ai/wp-content/uploads/2023/10/Logo_black_blue_CLEAN_rgb.pngKonrad Budek2020-03-27 12:00:122024-04-24 14:20:47AIOps Network Traffic Analysis (NTA) – a business guide
While making predictions may be easy, delivering accurate ones is an altogether different story. That’s why in this column we won’t just be looking at the most important trends of 2020, but we’ll also look at how the ideas we highlighted last year have developed.
In summarizing the trends of 2020, one conclusion we’ve come to is that society is getting increasingly interested in AI technology, in terms of both the threats it poses and common knowledge about other problems that need to be addressed.
AI trends 2019 in review – how accurate were our predictions?
In our AI Trends 2019 blogpost we chronicled last year’s most important trends and directions of development to watch. It was shortly after launching the AI Monthly Digest, a monthly summary of the most significant and exciting machine learning news. Here’s a short summary of what we were right and wrong about in our predictions.
Chatbots and virtual assistants – powered by a focus on the development of Natural Language Processing (NLP), our prediction was accurate–the growth in this market would be robust. The chatbot market was worth $2.6 billion in 2019 and is predicted to reach up to $9.4 billion by 2024.
the time needed for training would fall – the trend gets reflected by larger neural networks being trained in a feasible time, with GPT-2 being the best example.
Autonomous vehicles are on the rise – the best proof is in our own contribution to the matter in a joint-venture with Volkswagen.
Machine learning and artificial intelligence are being democratized and productionized – According to Gartner, 37% of organizations have implemented AI in some form. That’s a 270% increase over the last four years.
AI and ML responsibility and transparency – the trend encompasses the delivering unbiased models and tools. The story of Amazon using an AI-based recruiting tool that turned out to be biased against female applicants made enough waves to highlight the need for further human control and supervision over automated solutions.
Apparently, deepsense.ai’s data science team was up to date and well-informed on these matters.
“It is difficult to make predictions, especially about the future.”
-Niels Bohr
The world is far from slowing down and Artificial Intelligence (AI) appears to be one of the most dominant technologies at work today. The demand for AI talents has doubled in the last two years with technology and the financial sector absorbing 60% of talented employees on the market.
The Artificial Intelligence market itself is predicted to reach $390.9 billion by 2025, mainly by primarily by automating dull and repetitive tasks. It is predicted that AI will resolve around 20% of unmet healthcare demands.
Considering the impact of AI on people’s daily lives, spotting the right trends to follow is even more important. AI is arguably the most important technology trend of 2020, so enjoy our list!
Natural language processing (NLP) – further development
Whether the world was ready for it or not, GPT-2 was released last year, with balance between safety and progress a guiding motif. Initially, OpenAI refused to make the model and dataset public due to the risk of the technology being used for malicious ends.
The organization released versions of the model throughout 2019, with each confirmed to be “hardened against malicious usage”. The model was considered cutting edge, though like most things in tech, another force soon prevailed. At the end of January 2020, Google Brain took the wraps off of Meena, a 2.6-billion parameter end-to-end neural conversational model trained on 341 GB of online text.
The convenience of NLP solutions is enjoyed by users who have embraced virtual assistants like Google Assistant, Alexa or Siri. According to Adroit Market Research, the market of Intelligent Virtual Assistants is predicted to grow at 33% compound annual growth rate between now and 2025. The market was valued at $2.1 billion in 2019. The increasing use of smartphones and other wearable intelligent devices, among other trends, is predicted to be a driver of the growth.
Started with a consumer-centric approach, virtual assistants are predicted to get more involved in business operations, further automating processes as well as tedious and repetitive tasks. According to Computerworld, approximately 40% of business representatives are planning to implement voice technology within 24 months – that is, no later than in 2021. NLP is shaping up to be a major trend not only this year, but well into the future.
Autonomous vehicles
It is 2020 and driverless cars have yet to hit the streets. In hindsight, the Guardian’s prediction that there would be 10 million self-driving cars on the road by 2020 is all too easy to scoff at now.
On the other hand, tremendous progress has been made and with every month the autonomous car gets closer to rolling out.
deepsense.ai has also contributed to the progress, cooperating with Volkswagen on building a reinforcement learning-based model that, when transferred from a simulated to a real environment, managed to safely drive a car.
But deepsense.ai is far from being the only company bringing significant research about autonomous cars and developing the technology in this field. Also, it is a great difference between seeing an autonomous car on busy city streets and in the slightly less demanding highway environment, where we can expect the automation and semi-automation of driving to first get done.
According to the US Department of Transportation, 63.3% of the $1,139 billion of goods shipped in 2017 were moved on roads. Had autonomous vehicles been enlisted to do the hauling, the transport could have been organized more efficiently, and the need for human effort vastly diminished. Machines can drive for hours without losing concentration. Road freight is globally the largest producer of emissions and consumes more than 70% of all energy used for freight. Every optimization made to fuel usage and routes will improve both energy and time management.
AI getting popular – beneath the surface
There is a lot of buzz around how AI-powered solutions impact our daily lives. While the most obvious change may be NLP powering virtual assistants like Google Assistant, Siri or Alexa, the impact on our daily lives runs much deeper, even if it’s not all that visible at first glance. Artificial intelligence-powered solutions have a strong influence on manufacturing, impacting prices and supply chains of goods.
Here are a few applications being used without batting an eye:
Demand forecasting – companies collect tremendous amounts of data on their customer relationships and transactional history. Also, with the e-commerce revolution humming along, retail companies have gained access to gargantuan amounts of data about customer service, products and services. deepsense.ai delivers demand forecasting tools that not only process such data but also combines it with external sources to deliver more accurate predictions than standard heuristics. Helping companies avoid overstocking while continuing to satisfy demand is one essential benefit demand forecasting promises.
Quality control – harnessing the power of image recognition enables companies to deliver more accurate and reliable quality control automation tools. Because machines are domain-agnostic, the tools can be applied in various businesses, from fashion to construction to manufacturing. Any product that can be controlled using human sight can also be placed under the supervision of computer vision-powered tools.
Manufacturing processes optimization – The big data revolution impacts all businesses, but with IoT and the building of intelligent solutions, companies get access to even more data to process. But it is not about gathering and endless processing in search of insights – the data is also the fuel for optimization, sometimes in surprising ways. Thanks solely to optimization, Google reduced its cooling bill by 40% without adding any new components to its system. Beyond cutting costs, companies also use process optimization to boost employee safety and reduce the number of accidents.
Office processes optimization – AI-powered tools can also be used to augment the daily tasks done by various specialists, including lawyers or journalists. Ernst & Young is using an NLP tool to review contracts, enabling their specialists to use their time more efficiently. Reuters, a global media corporation and press agency, uses AI-powered video transcription tools to deliver time-coded speech-to-text tools that are compatible with 11 languages.
Thanks to the versatility and flexibility of such AI-powered solutions, business applications are possible even in the most surprising industries and companies. So even if a person were to completely abandon technology (right…), the services and products delivered to them would still be produced or augmented with AI, be they clothing, food or furniture.
AI getting mainstream in culture and society
The motif of AI is prevalent in the arts, though usually not in a good way. Isaac Asimov was among the first writers to hold that autonomous robots would need to follow a moral code in order not to become dangerous to humans. Of course, literature has offered a number of memorable examples of AI run amok, including Terminator and HAL 9000 from Space Odyssey.
The question of moral principles may once have been elusive and abstract, but autonomous cars have necessitated a legal framework ascribing responsibility for accidents. Amazon learned about the need to control AI models the hard way, albeit in a less mobile environment: a recruiting tool the company was using had to be scrapped due to a bias against women.
The impact of AI applications on people’s daily lives, choices and careers is building pressure to deliver legal regulations on model transparency as well as information not only about outcomes, but also the reasons behind them. Delivering AI in a black-box mode is not the most suitable way to operate, especially as the number of decisions made automatically by AI-powered solutions increases.
Automating the development of AI
Making AI mainstream is not only about making AI systems more common, but widening the availability of AI tools and their accessibility to less-skilled individuals. The number of models delivering solutions to power with the machine and deep learning will only increase.It should therefore come as no surprise that the people responsible for automating others’ jobs are keen to support their own jobs with automation.
Google enters the field with AutoML, a tool that simplifies the process of developing AI and making it available for a wider audience, one that, presumably, is not going to use ML algorithms in some especially non-standard ways. AutoML joins IBM’s autoAI, which supports data preparation.
Also, there are targeted cloud offerings for companies seeking to harness ready-to-use components in their daily jobs with a view to augmenting their standard procedures with machine learning.
Summary
While the 2020 AI Trends themselves are similar to those of 2019, the details have changed immensely, thus refreshing our perspective seemed worth our while. The world is changing, ML is advancing, and AI is ever more ubiquitous in our daily lives.
https://deepsense.ai/wp-content/uploads/2020/03/Five-AI-trends-2020-to-keep-an-eye-on-blogpost-header.jpg3371140Konrad Budekhttps://deepsense.ai/wp-content/uploads/2023/10/Logo_black_blue_CLEAN_rgb.pngKonrad Budek2020-03-09 09:00:252024-04-27 23:22:04Five AI trends 2020 to keep an eye on
Every business now depends on IT. Efficient IT Operations is mandatory for all businesses, especially those operating in a hybrid mode – a mix of existing data centers and multi-cloud locations.
As with any business process, IT operations can be augmented with machine learning-based solutions. IT is particularly fertile ground for AI as it is mostly digital, has seemingly endless processes requiring automation and there are gigantic amounts of data to process.
IT Operations are expensive!
According to Research and Markets data, global IT Ops & service management (ITSM) is predicted to reach $35.98 billion by 2025 with an annual growth rate of 7.5% YoY.
As the importance of IT operations has ramped up, so has the pressure on ITOps teams. A range of issues puts pressure on teams: shrinking budget for IT operations, multi cloud-based applications, dynamic scaling of infrastructure, limited availability of experienced ITOps personnel, the constant threat from outsiders given the nature of cloud applications, the extension of applications to edge locations with IoT and mobile devices.
AIOps is here to support the maintenance teams and provide AIOps tools to solve problems once thought unsolvable.
What is AIOps
AIOps supports infrastructure management and operations with AI-based solutions. It is employed mainly to automate tasks, improve process efficiency, quicken reactions — sometimes even to a real-time response rate — and deliver accurate predictions on upcoming events.
The big data revolution and machine learning technology have driven change, making it possible to process the vast amounts of information IT infrastructure generates. AI can solve the following challenges:
Anomaly detection – despite fluctuations and the dynamic nature of data, the internal infrastructure ecosystem is a stable environment. Thus, any anomaly can signal the existence of a problem. Also, early detection of an anomaly is usually a sign of a problem that has yet to be fully understood.
Event consolidation – An AI model can simplify huge amounts of data, dividing it into multiple layers and finding insights.
Service tickets analytics – when fed data on tickets submitted to a service desk, an ML-based model can predict seasonal spikes and requests. This can help the service desk owner deploy help desk personnel s needed.
Detecting seasonality and trends – when using an AI-powered solution, any trend can be divided into 3 components – seasonality, trend and residual. That increases the predictability of long-term commitments and makes managing them more effective.
Frequent pattern mining – machine-powered analysis delivers insights that are beyond the reach of humans. Machines not only process more data but also , unlike humans, make unbiased decisions. They also find correlations that are impossible for humans to detect.
Time series forecasting – AI-based models can forecast future values such as memory load, network throughput ticket count or other values in the future. This enables AIOps solutions to deliver early alert predictions.
Noise reduction – AIOps solutions eliminate noise and concentrate on the real underlying problems.
AI helps ITOps run smoother
There are currently several major challenges for IT departments.
Fraud Detection/Security
According to IBM data quoted by CSO, the average time to identify a breach in 2019 was 209 days. Such a sizable delay is caused mainly by security teams being overwhelmed with work and the stealth operations of criminals. Cybercrime is a highly profitable venture, with profits reaching up to $1.5 trillion a year. Cybercriminals don’t play favorites, targeting victims of all stripes, from individuals to international corporations. In May 2019 authorities tracked down a group that had availed itself of an estimated $100 million.
Anomaly detecting AI and machine learning-based AIOps solutions can spot even the slightest signs that unexpected events are occurring in a system. AIOps can be trained to learn what “typical operations” look like and spot anything out of place. It can also send real-time notifications to the team.
Eliminate Downtime
According to an ITIC study, for 86% of companies surveyed, a single hour of downtime costs $300,000. For 34%, the cost comes in at a staggering $1 million.
AIOps comes with various tools to help keep the lights on and operations running smoothly. In addition to anomaly detection, time series prediction serves as a benchmark and a tool for designing maintenance flow. It also supports efficient resource management. Pattern mining spots inefficient components and bottlenecks to be optimized. It also enables the mapping of both seasonality and trends, so resources supporting operations can be assigned efficiently.
Capacity planning
Before the cloud, companies were forced to overpay for servers and computing power because they had to stay on top of fluctuations in their seasonal needs for computer power. Today, despite the access they have to endless power and storage in the cloud, IT teams around the world continue to struggle with capacity planning and delivering scalable infrastructure to meet the irregular demands for infrastructure.
Nearly all AIOps functionalities support the goal of delivering a stable and scalable environment. With capacity planning supported by time series forecasting and ticket analysis, IT teams can manage their infrastructure scaling and maintenance not only to avoid downtime but also to minimize costs and utilize their systems as efficiently as possible.
A great example comes from Google, whose AI-based system delivered new operational efficiency recommendations for data center cooling systems, effectively cutting costs by 40%.
Noise reduction and pattern mining deliver clear insights. On the other hand, scraping through the data in real-time enables an AIOps platform to deliver insights faster and using those insights more actionably.
Summary
AIOps machine learning-powered solutions can significantly improve today’s data-heavy IT infrastructure management.
A good way to learn more about the AIOps strategy landscape is to meet our team during the upcoming AIOps conference in Fort Lauderdale, Florida. Our specialists will be more than happy to strategize, assist, answer your questions, and share their expertise from the field. If you’d like to meet us there, just drop us a line or stop by our booth (#1104). Contact us at AIOps@deepsense.AI to see how we can help!
https://deepsense.ai/wp-content/uploads/2020/01/What-is-AIOps-AI-for-IT-operations-explained.jpg3371140Andy Thuraihttps://deepsense.ai/wp-content/uploads/2023/10/Logo_black_blue_CLEAN_rgb.pngAndy Thurai2020-01-31 10:00:322022-10-10 18:40:43What is AIOps – AI for IT operations explained
ICLR (The International Conference on Learning Representations) is one of the most important international machine learning conferences. It’s popularity is growing fast, putting it on a par with such conferences as ICML, NeurIPS or CVPR.
The 2020 conference is slated for next April 26th, but the submissions deadline has already come and gone. 2585 publicly available papers were submitted. That’s about a thousand more than were featured at the 2019 conference.
2nd law of paper-dynamics tells us that the number of submitted papers will reach 100.000 in 24 years. That’s some serious growth!
We analyzed abstracts and keywords of all the ICLR papers submitted within the last three years to see what’s trending and what’s dying out. Brace yourselves! This year, 28% of the papers used or claimed to introduce state-of-the-art algorithms, so be prepared for a great deal of solid machine learning work!
“Deep learning” – have you heard about it?
To say you use deep learning in Computer Vision or Natural Language Processing is like saying fish live in water. Deep learning has revolutionized machine learning and become it’s underpinning. It’s present in almost all fields of ML, including less obvious ones like time series analysis or demand forecasting. This may be why the number of references to deep learning in keywords actually fell – from 19% in ‘18 to just 11% in ‘20–It’s just too obvious to acknowledge.
A revolution in network architecture?
One of the hottest topics this year turned out to be Graph Neural Networks. GNN is a deep learning architecture for graph-structured data. These networks have proved tremendously helpful in some applications in medicine, social network classification and modeling the behavior of dynamic interacting objects. The rise of GNNs is unprecedented, from 12 papers mentioning them in 18’ to 111 in 20’!
All Quiet on the GAN Front
The next topic has been extremely popular in recent years. But what has been called ‘the coolest idea in machine learning in the last twenty years’ has quickly become exploited. Generative Adversarial Networks can learn to mimic any distribution of data – creating impressive never seen artificial images. Yet they are on the decline. Yet they are on the decline, despite being prevalent in the media (deep fakes).
Leave designing your machine learning to… machines
Finding the right architecture for your neural network can be a pain in the neck. Fear not, though: Neural Architecture Search (NAS) will save you. NAS is a method of building network architecture automatically rather than handcrafting it. It has been used in several state-of-the-art algorithms improving image classification, object detection or segmentation models. The number of papers on NAS increased from a mere five in ‘18 to 47 in ‘20!
Reinforcement learning – keeping stable
The percentage of papers on reinforcement learning has remained more or less constant. Interest in the topic remains significant – autonomous vehicles, Alpha Star’s success in playing StarCraft, and advances in robotics were allwidely discussed this year. RL is a stable branch of machine learning, and for good reason: future progress is widely anticipated.
What’s next?
That was just a sample of machine learning trends. What will be on top next year? Even the deepest neural network cannot predict it. But interest in machine learning is still on the rise, and the researchers are nothing if not creative. We shouldn’t be surprised to hear about groundbreaking discoveries next year and a 180-degree change in the trends.
To see a full analysis of the trends papers throughout the last three conferences click the photo below:
https://deepsense.ai/wp-content/uploads/2019/10/key-findings-from-the-international-conference-on-learning-representations-iclr.png3371140Michał Kustoszhttps://deepsense.ai/wp-content/uploads/2023/10/Logo_black_blue_CLEAN_rgb.pngMichał Kustosz2019-10-24 12:49:092024-02-06 19:08:44Trends and fads in machine learning – topics on the rise and in decline in ICLR submissions
September brought us two interesting AI-related stories, both with a surprising social context.
Despite its enormous impact on our daily lives, Artificial Intelligence (AI) is often still regarded as too hermetic and obscure for ordinary people to understand. As a result, an increasing number of people use Natural Language Processing-powered personal assistants, yet only a tiny fraction try to understand how they work and how to use them effectively. This makes them somewhat of a black box.
Making the field more comprehensible and accessible is one aspect of AI researchers’ mission. That’s why research recently done by OpenAI is so interesting.
Hide-and-Seek – the reinforcement learning way
Reinforcement learning has delivered inspiring and breathtaking results. The technique is used in the training models behind autonomous cars and the controlling of sophisticated devices like automated arms and robots.
Unlike in supervised learning, a reinforcement learning model learns by interacting with the environment. The scientist can shape its behavior by applying a policy of rewards and punishments. The mechanism is close to that which humans use to learn.
Reinforcement learning has been used to create super killing agents to go toe-to-toe against human masters in Chess, Go and Starcraft. Now OpenAI, the company behind the GPT-2 model and several other breakthroughs in AI, has created agents that play a version of hide-and-seek, that most basic and ageless of children’s games.
OpenAI researchers divided the agents into two teams, hiders and seekers, and provided them a closed environment with walls and movable objects like boxes, walls and ramps. Any team could “lock” these items to make them unmovable for the opposing team. The teams developed a set of strategies and counter-strategies in a bid to successfully hide from or seek out the other team. The strategies included:
Running – the first and least sophisticated ability, enabling one to avoid the seekers.
Blocking passages – the hider could block passages with a box in order to build a safe shelter.
Using a ramp – to overcome the wall or a box, the seekers team learned to use a ramp to jump over an obstacle or climb a box and see the hider.
Blocking the ramp – to prevent the seekers from using the ramp to climb the box, the hiders could block access to the ramp. The process required a great deal of teamwork, which was not supported by the researchers in any way.
Box surfing – a strategy developed by seekers who were basically exploiting a bug in the system. The seekers not only jumped on a box using a ramp that had been blocked by the hiders, but also devised a way to move it while standing on it.
All-block – the ultimate hider-team teamwork strategy of blocking all the objects on the map and building a shelter.
The research delivered, among other benefits, a mesmerizing visual of little agents running around.
Why does it matter?
The research itself is neither groundbreaking nor breathtaking. From a scientific and developmental point of view, it looks like little more than elaborate fun. Yet it would be unwise to consider the project insignificant.
AI is still considered a hermetic and difficult field. Showing the results of training in the form of friendly, entertaining animations is a way to educate society on the significance of modern AI research.
Also, animation can be inspiring for journalists to write about and may lead youth to take an interest in AI-related career paths. So while the research has brought little if any new knowledge, it could well end up spreading knowledge on what we already know.
AI-generated stock photos available for free
Generative Adversarial Networks have proved to be insanely effective in delivering convincing images of not only hamburgers and dogs, but also human faces. One breakthrough is breathtaking indeed. Not even a year ago the eerie “first AI-generated portrait” was sold on auction for nearly a half-million dollars.
Now, generating faces of non-existent people is as easy as generating any other fake image – a cat, hamburger or landscape. To prove that the technology works, the team behind the 100K faces project delivered a hundred thousand AI-generated faces to use in any stock usage, from business folders, to flyers to presentations. Future use cases could include delivering on-the-go image generators that, powered by a demand forecasting tool, provides an image that best suits demand.
The images added to the free images bank are not perfect. With visible flaws in a model’s hair, teeth or eyes, some are indeed far from it. But that’s nothing a skilled graphic designer can’t handle. Also, there are multiple images that look nearly perfect – especially when there are no teeth visible in the smile.
Many photos are good enough to provide a stock photo as a “virtual assistant” image or to fulfill any need for a random face. This is an early sign that professional models and photographers will see the impact of AI in their daily work sooner than expected.
https://deepsense.ai/wp-content/uploads/2019/10/AI-Monthly-Digest-13-–-an-unexpected-twist-for-the-stock-image-market.jpg3371140Konrad Budekhttps://deepsense.ai/wp-content/uploads/2023/10/Logo_black_blue_CLEAN_rgb.pngKonrad Budek2019-10-07 12:26:182024-02-06 19:08:44AI Monthly Digest #13 – an unexpected twist for the stock image market
Computer vision enables machines to perform once-unimaginable tasks like diagnosing diabetic retinopathy as accurately as a trained physician or supporting engineers by automating their daily work.
Recent advances in computer vision are providing data scientists with tools to automate an ever-wider range of tasks. Yet companies sometimes don’t know how best to employ machine learning in their particular niche. The most common problem is understanding how a machine learning model will perform its task differently than a human would.
What is computer vision?
Computer vision is an interdisciplinary field that enables computers to understand, process and analyze images. The algorithms it uses can process both videos and static images. Practitioners strive to deliver a computer version of human sight while reaping the benefits of automation and digitization. Sub-disciplines of computer vision include object recognition, anomaly detection, and image restoration. While modern computer vision systems rely first and foremost on machine learning, there are also trigger-based solutions for performing simple tasks.
The following case studies show computer vision in action.
5 popular computer vision applications
1. Diagnosing diabetic retinopathy
Diagnosing diabetic retinopathy usually takes a skilled ophthalmologist. With obesity on the rise globally, so too is the threat of diabetes. As the World Bank indicates, obesity is a threat to world development – among Latin America’s countries only Haiti has an average adult Body Mass Index reading below 25 (the upper limit of the healthy weight range). With rising obesity comes a higher risk of diabetes – it is believed that obesity comes with 80-85% risk of developing type 2 diabetes. This results in a skyrocketing need for proper diagnostics.
What is the difference between these two images?
The one on the left has no signs of diabetic retinopathy, while the other one has severe signs of it.
By applying algorithms to analyze digital images of the retina, deepsense.ai delivered a system that diagnosed diabetic retinopathy with the accuracy of a trained human expert. The key was in training the model on a large dataset of healthy and non-healthy retinas.
2. AI movie restoration
The algorithms trained to find the difference between healthy and diseased retinas are equally capable of spotting blemishes on old movies and making the classics shine again.
Recorded on a celluloid film, old movies are endangered by two factors – the fading technology of reading tapes that enable users to watch them and the nature of the tape, which degenerates with age. Moreover, the process of digitizing the movie is no guarantee of flawlessness, as the recorded film comes with multiple new damages.
However, when trained on two versions of a movie – one with digital noise and one that is perfect – the model learns to spot the disturbances and remove them during the AI movie restoration process.
Another example of the push towards digitization comes via industrial installation documentation. Like films, this documentation is riddled with inconsistencies in the symbols used, which can get lost in the myriad of lines and other writing that ends up in the documentation–and must be made sense of by humans. Digitizing industrial documentation that takes a skilled engineer up to ten hours of painstaking work can be reduced to a mere 30 minutes thanks to machine learning.
4. Building digital maps from satellite images
Despite their seeming similarities, satellite images and fully-functional maps that deliver actionable information are two different things. The differences are never as clear as during a natural disaster such as a flood or hurricane, which can quickly if temporarily, render maps irrelevant.
deepsense.ai has also used image recognition technology to develop a solution that instantly turns satellite images into maps, replete with roads, buildings, trees and the countless obstacles that emerge during a crisis situation. The model architecture we used to create the maps is similar to those used to diagnose diabetic retinopathy or restore movies.
Check out the demo:
5. Aerial image recognition
Computer vision techniques can work as well on aerial images as they do on satellite images. deepsense.ai delivered a computer vision system that supports the US NOAA in recognizing individual North Atlantic Right whales from aerial images.
With only about 411 whales alive, the species is highly endangered, so it is crucial that each individual be recognizable so its well-being can be reliably tracked. Before deepsense.ai delivered its AI-based system, identification was handled manually using a catalog of the whales. Tracking whales from aircraft above the ocean is monumentally difficult as the whales dive and rise to the surface, the telltale patterns on their heads obscured by rough seas and other forces of nature.
Bounding box produced by the head localizer
These obstacles made the process both time-consuming and prone to error. deepsense.ai delivered an aerial image recognition solution that improves identification accuracy and takes a mere 2% of the time the NOAA once spent on manual tracking.
The deepsense.ai takeaway
As the above examples show, computer vision is today an essential component of numerous AI-based software development solutions. When combined with natural language processing, it can be used to read the ingredients from product labels and automatically sort them into categories. Alongside reinforcement learning, computer vision powers today’s groundbreaking autonomous vehicles. It can also support demand forecasting and function as a part of an end-to-end machine learning manufacturing support system.
The key difference between human vision and computer vision is the domain of knowledge behind data processing. Machines find no difference in the type of image data they process, be it images of retinas, satellite images or documentation – the key is in providing enough training data to allow the model to spot if a given case fits the pattern. The domain is usually irrelevant.
https://deepsense.ai/wp-content/uploads/2019/07/5-examples-of-the-versatility-of-computer-vision-algorithms-and-applications.jpg3371140Konrad Budekhttps://deepsense.ai/wp-content/uploads/2023/10/Logo_black_blue_CLEAN_rgb.pngKonrad Budek2019-07-25 13:40:162023-11-11 16:37:455 examples of the versatility of computer vision algorithms and applications
With convolutional neural networks and state-of-the-art image recognition techniques it is possible to make old movie classics shine again. Neural networks polish the image, reduce the noise and apply colors to the aged images.
The first movies were created in the late nineteenth century with celluloid photographic film used in conjunction with motion picture cameras.
Skip ahead to 2018, when the global movie industry was worth $41.7 billion globally. Serving entertainment, cultural and social purposes, films are a hugely important heritage to protect. And that’s not always easy. Especially considering the fact that modern movies are produced and screened digitally, with the technology of celluloid tape fading into obsolescence.
Challenges in film preservation
The challenge and importance of preserving the cultural heritage of old movies has been underscored by numerous organizations including the European Commision, which noted that a lack of proper devices to play aging technology on could make it impossible to watch old films.
In deepsense.ai’s experience with restoring film, the first challenge is to remove distortions. Classics are usually recorded in low resolution while the original tapes are obviously aged and filled with noise and cracks. Also, the transition process from celluloid tape to digital format usually damages the material and results in the loss of quality.
By using AI-driven solutions, specifically supervised learning techniques, deepsense.ai’s team removed the cracks and black spots from the digitized version of a film. The model we produced uses deep neural networks trained on a movie with cracks and flaws added manually for training purposes. Having some films in original and broken form, the system learned to remove the flaws. An example of generated noise put on the classic Polish movie “Rejs” and the neural network’s output is displayed below.
The example clearly shows that our neural network can process and restore even a thoroughly damaged source material and make it shine again. The networks start to produce low-quality predictions when the images are so darkened and blurred that the human eye can barely recognize people in the film.
How to convert really old movies into HD
A similar training technique was applied to deliver a neural network used to improve the quality of an old movie. The goal was to deliver missing details and “pump up” the resolution from antiquated to HD quality.
The key challenge lay in reproducing the details, which was nearly impossible. Due to technological development, it is difficult for people to watch lower quality video than what they are used to.
The model was trained by downscaling an HD movie and then conducting a supervised training to deliver the missing details.
Move your mouse cursor over the image to see the difference.
The model performs well thanks to the wide availability of training data. The team could downscale the resolution of any movie, provide the model with the original version and let the neural network learn how to forge and inject the missing detail into the film.
A key misconception about delivering HD versions of old movies is that the neural network will discover the missing details from the original. In fact, there is no way to reclaim lost details because there were none on the originally registered material. The neural network produces them on the go on with the same techniques Thispersondoesnotexist and similar Generative Adversarial Networks use.
So, the source material is enriched with details that only resemble reality, but are in fact not real ones. This can be a challenge (or a problem) if the material is to be used for forensic purposes or detailed research. But when it comes to delivering the movies for entertainment or cultural ends, the technique is more than enough.
Coloring old movies
Another challenge comes with producing color versions of movie classics, technically reviving them for newer audiences. The process was long handled by artists applying color to every frame. The first film colored this way was the British silent movie “The Miracle” (1912).
Because there are countless color movies to draw on, providing a rich training set, a deep neural network can vastly reduce the time required to revive black and white classics. Yet the process is not fully automatic. In fact, putting color on the black and white movie is a titanic undertaking. Consider Disney’s “Tron,” which was shot in black and white and then colored by 200 inkers and painters from Taiwan-based Cuckoo’s Nest Studio.
When choosing colors, a neural network tends to play it safe. An example of how this can be problematic would be when the network misinterprets water as a field of grass. It would do that because it is likely more common for fields than for lakes to appear as a backdrop in a film.
By manually applying colored pixels to single frames, an artist can suggest what colors the AI model should choose.
There is no way to determine the real color of a scarf or a shirt an actor or actress was wearing when a film rendered in black and white was shot. After all these years, does it even matter? In any case, neural networks employ the LAB color standard, leveraging lightness (L) to predict the two remaining channels (A and B respectively).
Transcription and face recognition
Last but not least, transcribing dialogue makes analysis and research much easier – be it for linguistic or cultural studies purposes. With facial recognition software, the solution can attribute all of the lines delivered to the proper characters.
The speech-to-text function processes the sound and transcribes the dialogue while the other network checks which of the people in the video moves his or her lips. When combined with image recognition, the model can both synchronize the subtitles and provide the name of a character or actor speaking.
While the content being produced needs to be supervised, it still vastly reduces the time required for transcription. In the traditional way, the transcription only takes at least the time of a recording and then needs to be validated. The machine transcribes an hour-long movie in a few seconds.
Summary
Using machine learning-based techniques to restore movies takes less time and effort than other methods. It also makes efforts to preserve the cultural heritage more successful and ensures films remain relevant. Machine learning in business gets huge recognition but ML-based techniques remain a novel way to serve the needs of culture and art.. deepsense.ai’s work has proven that AI in the art can serve multiple purposes, including promotion and education. Maybe using it in art and culture will be one of 2020’s AI trends.
Reviving and digitalizing classics improves the access to and availability of cultural goods and ensures that those works remain available, so future generations will, thanks to AI, enjoy Academy-awarded movies of the past as much as, if not more than, we do now.
https://deepsense.ai/wp-content/uploads/2019/07/AI-movie-restoration.jpg3371140Konrad Budekhttps://deepsense.ai/wp-content/uploads/2023/10/Logo_black_blue_CLEAN_rgb.pngKonrad Budek2019-07-18 14:35:042021-10-19 13:17:39AI movie restoration – Scarlett O’Hara HD
