3 Ways That Mathematical Optimization Can Be Used to Improve Machine Learning Applications

Author: Ricardo Baeza-Yates, PhD

Date: 7/2/2020

My career as a practitioner and researcher in the data science space has spanned more than 30 years, and during that time I have seen a lot of new advanced analytics technologies – which were touted as “the latest and greatest,” “cutting-edge,” or “game-changing” or another similar superlative – sizzle and then fizzle. The hype cycles (as Gartner calls them) of these technologies were short – as they failed to deliver real-world business impact and attain long-term commercial viability.

One advanced analytics technology that bucks that trend and has been around ever since I entered the professional arena in the early 1990s (and actually long before that with the introduction of linear programming in the 1940s) is mathematical optimization. For decades, mathematical optimization has been widely used by companies of all sizes and stripes to address their complex business problems. The secret to mathematical optimization’s staying power is that it has consistently demonstrated that it is capable of generating optimal solutions to large-scale, real-world business problems – and has thereby produced significant business value.

Surprisingly, though, many data scientists lack an understanding of what mathematical optimization is and what it can do – and consequently have not added it to their advanced analytics toolboxes. Why is this the case?

The simple answer is: They probably haven’t been exposed to it. Many data scientists study either computer science or statistics in university (while a good number of them are actually self-taught), and their advanced analysis tool of choice is machine learning. Unless they have actually taken some courses in mathematical programming (which most of them, I suspect, have not), they are probably not familiar with mathematical optimization tools and techniques.

Machine Learning and Mathematical Optimization: A Dynamic Duo

In the past, mathematical optimization and machine learning developed in and existed in silos – and these technological tools (as well as their practitioners) did not work together.

But these silos are being broken down and the paradigm is changing: A growing number of companies are starting to use mathematical optimization in combination with machine learning – and seeing how these two technologies can complement and enhance each other.

Fundamentally, this relationship can work in three ways:

1)    Machine learning predictions can serve as input for mathematical optimization solutions.

2)    Mathematical optimization solutions can serve as input for machine learning predictions.

3)    Mathematical optimization can be utilized to generate optimal solutions to classification, prediction, and other problems that are typically solved using machine learning.

For data scientists, this third approach is especially intriguing as it allows them to truly leverage mathematical optimization’s problem-solving power to improve their current machine learning applications.

But – one might wonder – why would data scientists want to use mathematical optimization to address their problems? What can mathematical optimization offer that machine learning can’t?

In this blog, I will discuss the three key benefits that data scientists can realize by using mathematical optimization in their applications.

Benefit #1: With Mathematical Optimization, You Can Add Semantics to Your Model

One of the shortcomings of machine learning solutions is that, although they are capable of taking raw data and producing sophisticated output, they usually don’t consider the semantics of the problem. This means that, in many instances, machine learning applications have to learn things that we already know. Take, for example, natural language processing: A machine learning application has to process and understand so many basic linguistic structures and meanings (which we actually already know) – and it takes some time for the machine learning application to just grasp everything.

With mathematical optimization, you can add semantics to problems in the form of constraints (which are essentially rules and restrictions) that are coded into a formal model. These semantics provide a structure that helps ensure that your subject matter expertise is embedded in the model and that the solutions you generate are not unbound.

It is true that with machine learning you can also add semantics into the application, but this is not a simple process and needs to be done on an ad hoc basis – as these semantics are not a part of a given method.

Mathematical optimization gives you the capability to automatically insert semantics (and thus your subject matter expertise) to your applications – and this is tremendously valuable for data scientists.

Benefit #2: Mathematical Optimization Can Improve the Interpretability of Your Predictions

The second major benefit of using mathematical optimization to address problems that are typically solved using machine learning is that it improves the interpretability of your solutions.

Most data scientists use machine learning as a black box: They have little idea of what’s happening inside their applications and are likely unable to explain why these applications are generating specific predictions. True, data scientists can conduct features analysis and find out which features are the most important factors – but they do not possess a true causal understanding of why the machine learning-based prediction is what it is.

In contrast, the output of mathematical optimization applications is interpretable. Mathematical optimization applications are built on models, which have feature weights that are associated with particular variables and constraints. This makes it possible to analyze the model and pinpoint which variables and constraints are causing it to generate a specific solution.

In this way, mathematical optimization can be used to improve the interpretability of predictions. Having interpretable solutions that can be understood and trusted – by data scientists and subject matter experts alike – is vital, especially when it comes to “life-or-death” classification problems like predicting health problems such as heart disease or diabetes.

By using mathematical optimization to address classification and prediction problems, data scientists can gain a causal understanding of what is happening in the system and gain more confidence in the output it is producing.

Benefit #3: With Mathematical Optimization, You Can Bound the Error of Machine Learning-Based Predictions

There is a degree of uncertainty that is baked into any machine-learning based prediction – it is, after all, a best guess about future conditions based on historical data.

Data scientists try to minimize the errors associated with these predictions – to make sure that they can make the best possible business decisions based on these predictions.

The problem is that data scientists often do not know how far away these predictions are from the minimum total error because traditional predictive models cannot properly handle constraints and typically rely on heuristics. By using these heuristics, data scientists don’t know how close they are to the minimum total error.

By taking the original problem and feeding it into a mathematical optimization solver, data scientists can determine exactly how close they are to an optimal solution. For example, if the total error derived from an heuristics approach might be 60% away from the minimum total error, that means the heuristic is a not an effective predictive approach.

By using mathematical optimization to determine this optimality gap, data scientists can bound and minimize the error of their predictions. This enables them to ensure that their predictions are accurate.

Discovering the Possibilities

Many data scientists today use machine learning like a cannon that they fire at every problem, but as Confucius said, “You shouldn’t use a cannon to kill a fly.” Indeed, machine learning is a powerful technological tool, but it is not necessarily the best solution for every business problem.

To address the many different complex problems in our world today and enhance their machine learning applications, I would encourage data scientists to add new advanced analytics technologies into their toolboxes.

Although mathematical optimization is an established and mature technology with a history stretching back more than 70 years, it is new tool for many data scientists – and I believe that we are only just starting to discover all the possible ways to integrate and utilize mathematical optimization and machine learning together.