Using Mathematical Optimization to Improve Machine Learning Applications

By Ricardo Baeza-Yates, PhD

For decades, mathematical optimization has been widely used by companies of all sizes and stripes to address their complex business problems. The secret to mathematical optimization’s staying power is that it has consistently demonstrated that it is capable of generating optimal solutions to large-scale, real-world business problems – and has thereby produced significant business value.

Surprisingly, though, many data scientists lack an understanding of what mathematical optimization is and what it can do. Why is this the case?

The simple answer is: They probably haven’t been exposed to it. Many data scientists study either computer science or statistics at the university level (while a good number of them are actually self-taught), and their advanced analysis tool of choice is machine learning. Unless they have actually taken some courses in mathematical programming (which most of them, I suspect, have not), they are probably not familiar with mathematical optimization tools and techniques.

Machine Learning and Mathematical Optimization: A Dynamic Duo

In the past, mathematical optimization and machine learning developed in and existed in silos – and these technological tools (as well as their practitioners) did not work together.

But these silos are being broken down and the paradigm is changing: A growing number of companies are starting to use mathematical optimization in combination with machine learning – and seeing how these two technologies can complement and enhance each other.

Fundamentally, this relationship can work in three ways:

1. Machine learning predictions can serve as input for mathematical optimization solutions.
2. Mathematical optimization solutions can serve as input for machine learning predictions.
3. Mathematical optimization can generate optimal solutions to classification, prediction, and other problems typically solved using machine learning.

This third approach is especially intriguing for data scientists as it allows them to truly leverage mathematical optimization’s problem-solving power to improve their current machine-learning applications.

But – one might wonder – why would data scientists want to use mathematical optimization to address their problems? What can mathematical optimization offer that machine learning can’t?

In this article, I will discuss the three key benefits that data scientists can realize by using mathematical optimization in their applications.

Benefit #1: You Can Add Semantics to Your Model

One of the shortcomings of machine learning solutions is that, although they can take raw data and produce sophisticated output, they usually don’t consider the semantics of the problem. This means that, in many instances, machine learning applications have to learn things that we already know. For example, natural language processing: A machine learning application has to process and understand so many basic linguistic structures and meanings (which we already know) – and it takes some time for the machine learning application just to grasp everything.

With mathematical optimization, you can add semantics to problems in the form of constraints (essentially rules and restrictions) coded into a formal model. These semantics provide a structure that helps ensure that your subject matter expertise is embedded in the model and that the solutions you generate are not unbound.

It is true that with machine learning, you can also add semantics into the application, but this is not a simple process and needs to be done on an ad hoc basis – as these semantics are not part of a given method. Mathematical optimization gives you the capability to automatically insert semantics (and thus your subject matter expertise) to your applications – and this is tremendously valuable for data scientists.

Benefit #2: You Can Improve Prediction Interpretability

Most data scientists use machine learning as a black box: They have little idea of what’s happening inside their applications and are likely unable to explain why these applications are generating specific predictions. Data scientists can conduct features analysis and find out which features are the most important factors – but they do not possess a true causal understanding of why the machine learning-based prediction is what it is.

In contrast, the output of mathematical optimization applications is interpretable. Mathematical optimization applications are built on models, which have feature weights that are associated with particular variables and constraints. This makes it possible to analyze the model and pinpoint which variables and constraints are causing it to generate a specific solution.

In this way, mathematical optimization can be used to improve the interpretability of predictions. Having interpretable solutions that can be understood and trusted – by data scientists and subject matter experts alike – is vital, especially when it comes to “life-or-death” classification problems like predicting health problems such as heart disease or diabetes.

By using mathematical optimization to address classification and prediction problems, data scientists can gain a causal understanding of what is happening in the system and gain more confidence in its output.

Benefit #3: You Can Bound the Error of Machine-Learning-Based Predictions

There is a degree of uncertainty that is baked into any machine-learning-based prediction. It is, after all, a best guess about future conditions based on historical data. Data scientists try to minimize the errors associated with these predictions – to make sure that they can make the best possible business decisions based on these predictions.

The problem is that data scientists often do not know how far away these predictions are from the minimum total error because traditional predictive models cannot properly handle constraints and typically rely on heuristics. By using these heuristics, data scientists don’t know how close they are to the minimum total error.

By taking the original problem and feeding it into a mathematical optimization solver, data scientists can determine exactly how close they are to an optimal solution. For example, if the total error derived from a heuristics approach might be 60% away from the minimum total error, that means the heuristic is not an effective predictive approach.

By using mathematical optimization to determine this optimality gap, data scientists can bound and minimize the error of their predictions. This enables them to ensure that their predictions are accurate.

Discovering the Possibilities

Many data scientists today use machine learning like a cannon that they fire at every problem, but as Confucius said, “You shouldn’t use a cannon to kill a fly.” Indeed, machine learning is a powerful technological tool, but it is not necessarily the best solution for every business problem.

To address the many different complex problems in our world today and enhance their machine learning applications, I would encourage data scientists to add new advanced analytics technologies into their toolboxes.

Although mathematical optimization is an established and mature technology with a history stretching back more than 70 years, it is new tool for many data scientists – and I believe that we are only just starting to discover all the possible ways to integrate and utilize mathematical optimization and machine learning together.