Are your machine learning models good enough?

Are your machine learning models good enough?

Are your machine learning models good enough?Imagine you’re the CEO of XYZ Widget company.  Your Chief Marketing Officer (CMO),  Chief Data Officer (CDO) and Chief Operations Officer (COO) just finished their quarterly presentations and were highlighting the success from the various machine learning projects that have been in the works. After the presentations were complete, you begin to wonder – ‘are these machine learning models good enough?’

You’ve invested a significant portion of your annual budget on big data and machine learning projects and based on what your CMO and CDO tell you, things are looking really good. For example, your production and revenue forecasting projects are both delivering some very promising results with recent forecasts being within 2% of actual numbers.

You don’t really understand any of the machine learning stuff though. It seems like magic to you but you trust that the people doing the work understand it and are doing things ‘right’. That said, you have a feeling deep down that something isn’t quite right.  Sure, things look good but just like magic – the output of these machine learning initiatives could just be an illusion.

Are these machine learning models good enough? — Getting past the illusion

While machine learning, deep learning and big data can provide an enormous amount of value to an organization, there is ample opportunity to mess things up dramatically. There are plenty of times where small errors (and even massive errors) can be introduced into the process. For example, during the data munging / exploration phase, a simple error can introduce changes in the data, which could cause massive changes in the results of any modeling.

Additionally, bias can easily be introduced to the process (either on purpose or by accident). This bias can push the results to tell a story that people want the data / models to tell.  It is very easy to fall into the “let’s use statistics to support our view” trap that many fall into.  Rather than look for data and/or  outputs to support your view (and hence build an illusion), your machine learning initiatives (and any other data projects) should be as bias free as possible.

When done right, there’s very little ‘illusion’ in machine learning. The results are the results just like the data is the data.   You either find answers to your questions (and hopefully find more questions) or you don’t.   The results may not be what you wanted to see, but they are what they are…and this is the exact reason you need to be able to trust the process that was used to find those results. You need to understand if (and where) bias was introduced. You need to understand the process in general.

Can your team describe how was the data gathered and cleaned? Where the models used in the process optimized and/or overfit. Can your team explain their rationale for doing what they did?   Your forecasting models are within 2% of actual numbers in recent months, but that doesn’t mean your models are well built and will hold up over time…it could just mean they are overfit and are doing well with very similar numbers to what you’ve given your machine learning algorithm. What do your models really show for things like R-Squared and Mean Absolute Error (MAE)?  Do you understand why R-Squared and MAE are important?  If not, your teams need to make sure they are explained in general terms and describe why those things are important. Also..here’s a few links for you to learn more about R-Squared and MAE.

You don’t have to become an expert

It takes time and a willingness to ‘get your hands dirty’ to get anywhere close to being an expert in machine learning. Most business leaders don’t need to become an expert but you if you spend a little time understanding the basics and the process that your team follows, it might help remove the ‘magic’ aspect associated with machine learning

My suggestion is to spend some time talking to your team(s) about the following topics to get a basic understanding of the three main steps / processes in machine learning.  Below, I’ve outlined the three main areas and included some questions for you to consider.  Note: These aren’t a definitive list of questions / areas but they’ll get you started.

Data Gathering / Preparation / Cleaning

  • How was the data gathered?
  • What data quality measures / methods were undertaken to ensure the data’s accuracy and provenance?
  • What steps were taken to clean / prepare the data?
  • How is new data being gathered / cleaned / prepared for inclusion in existing / new models?
  • Who has access to the data?

Modeling

  • Why was the model (or models) chosen?
  • Were other models considered? If so, why weren’t they used?
  • Did you ‘build your own’ or use existing libraries to build the model?
  • Where the proper data preparation steps taken for the model(s) selected?

Evaluation &Interpretation of Results

  • How do you know the model is ‘good enough’?
  • When and why did you stop iterating on the model / data?
  • What accuracy measures are you using for the model(s)?
  • Are we sure the data isn’t being overfitted? How?
  • Why are the visualizations that are presented used? (Note: the use or non-use of certain visualizations can be a tip-off that something isn’t right about the data / model).

Again – these aren’t meant to be a definitive list of questions / topical areas for you to consider but they should get you started asking good questions of your team.   I particularly love to ask the How do you know the model is good enough question because it sheds a lot of light on the entire process and the mental approach to the problem.

Are these machine learning models good enough?

The answers to the above questions should help you get a better feel for how your team(s) approached the issue at hand and help you (and the rest of your leadership team) understand the approach to data preparation, modeling and evaluation in your machine learning initiatives.

The above questions and answers might not specifically answer the ‘are your machine learning models good enough’ question, but they will get you and your team(s) to a point where they are constantly thinking about whether ‘good enough’ is enough. Sometimes it is…others it isn’t. That’s why you need to understand a bit more about the process to understand whether good enough is good enough.

Of course, if you need help trying to understand all this stuff…you can always hire me to help. Give me a call or drop me an email and let’s discuss your needs.

Deep learning – when should it be used?

Deep Learning vs other approaches

“When should I use deep learning?”

I get asked that question constantly.

The answer to this question is both complicated and simplistic at the same time.

The answer I usually give us something along the lines of ‘if you a lot of data and an interesting / challenging problem, then you should try out deep learning’.

How a much is ‘a lot of data’?  That’s the complicated part.

Let’s use some examples to try to clarify things.

  • If you have 5 years of monthly sales data and want to use deep learning to build a forecaster, you’ll most likely be wasting your time.  Deep learning will work technically, but it generally won’t give you much better results than some simpler machine learning or even more simpler regression techniques.
  • If you have 20 years of real estate sales data with multiple features (e.g., square footage of the house, location, comparables, etc etc) and want to try to predict sales prices within a neighborhood/state/country, then deep learning is definitely an approach to take.  This a wonderful usage for deep learning.
  • If you want to build a forecaster to help develop a budget for your organization, maybe deep learning is a good approach…and maybe it isn’t.
  • If you want to build a “Hotdog Not Hotdog” app, deep learning is the right approach.
  • If you want to forecast how many widgets you’ll need to build next year with the previous 10 years of data, I’d recommend going with regression first then moving into some basic machine learning techniques. Deep learning (e.g., neural networks) could work here but it might not make a lot of sense depending on the size of the data.
  • If you want to predict movements in the stock market using the last 100 years of stock market data combined with hundreds of technical and/or fundamental indicators, deep learning could be a good approach but a good machine learning methodology would work as well. Just be careful of data mining and other bias’ you can introduce when working with time series data like the markets.

Ok. So I didn’t exactly clarify things, but hopefully you get the point.

Andrew Ng uses this graphic to highlight where deep learning make sense (from his Deeplearning.ai Coursera Course):

Deep Learning vs other approaches

 

 

 

 

 

 

 

 

 

 

 

 

 

The various lines on the chart are different approaches (regression, machine learning, deep learning) with the ‘standard’ approaches of regression and machine learning shown in red/orange.  You can see from the left of the chart that these types of approaches are similar performance-wise with small data sets.

Deep learning really begins to diverge in performance when your data-set starts to get sufficiently large. The ‘problem’ here is that ‘sufficiently large’ is hard to define.  That’s why I usually tell people to start with the basics first and try out regression then move to machine learning (Random Forest, SVM’s, etc etc) and then – once you have a feel for your data AND the performance of your approach isn’t delivering the results you expected, then try out deep learning.

That said, there are obviously times that deep learning makes sense initially. When you are looking at things like machine vision, natural language processing, autonomous driving, text translation in real-time, etc you want to investigate deep learning right away. Additionally, you can use deep learning appraoches for any problem you want to, but the performance is best when you have a large data set.

So…when should you consider deep learning? It depends on the challenge you are trying to solve.  Sorry…there’s not an ‘easy’ answer for the question.