But, this can be a challenge, as you run into issues like overfitting, hyperparameter optimization, and model selection.<\/p>\n
One way to overcome these various model evaluation and selection challenges is with cross validation<\/strong><\/em>.<\/p>\n Cross validation is a set of related techniques that we can use to evaluate and optimize our machine learning models.<\/p>\n And, it’s a very important tool in the toolkit of a machine learning developer.<\/p>\n So in this post, I’m going to explain the essentials of cross validation.<\/p>\n I’ll explain what it is, give a crystal clear example to help you understand, and I’ll explain some of the challenges associated with cross validation.<\/p>\n If you need something specific, you can click on any of these following links:<\/p>\n Table of Contents:<\/strong><\/p>\n Let’s jump in.<\/p>\n <\/a><\/p>\n Cross validation is a very important technique for evaluating the performance of machine learning systems. <\/p>\n At its core, cross validation is about assessing how well a machine learning model generalizes to new, previously unseen data.<\/p>\n This is critical, because in many cases, a model may perform very well on the training<\/em> data, but fail to perform well on new data that was outside of the training dataset. This situation, called overfitting, is when a model performs well on the training dataset, but then fails to perform on validation or test sets<\/a> (or worse, performs well on the training data, but fails to perform well in production).<\/p>\n Avoiding overfitting is one of the core problems of machine learning<\/a>. To say that mitigating overfitting is one of the top 5 problems of building machine learning might actually be undestating the problem. It’s that important.<\/p>\n And because it’s so important, you really need to understand how to minimize, correct, or otherwise mitigate overfitting.<\/p>\n Although there are several ways to fix overfitting \u2013\u00a0like using a validation set, regularization, and others \u2013 one of the most important is cross validation.<\/p>\n There are several different types of cross validation (sometimes called CV for short), but let’s start out with a simple explanation of what cross validation is.<\/p>\n The basic concept behind cross validation is taking the training data and splitting it into segments. We use one segment as the validation data (to evaluate the performance of the model), and the rest of the segments to train the model. <\/p>\n So, it’s sort of like a split of the data into training and validation data.<\/p>\n BUT, it’s more complicated than a simple training\/validation split.<\/p>\n To illustrate, let’s look at k-fold cross validation.<\/p>\n K-fold cross validation is a fairly simple process.<\/p>\n In k-fold cross validation, we:<\/p>\n Ok, ok. That might sound a little complicated.<\/p>\n But it’s actually easy to understand with a visual example.<\/p>\n So let’s look at an example of 5-fold cross validation in action.<\/p>\n I think that it’s easiest to understand 5-fold cross validation (and k-fold cross validation generally) with a picture.<\/p>\n At a high level, this is what we’re doing in 5-fold cross validation:<\/p>\n Notice what’s happening.<\/p>\n We take the training data and we split it into folds. In the case of 5-fold cross validation we split the data into 5 folds (as you might expect).<\/p>\n Then we train and validate the model multiple times, during multiple iterations.<\/p>\n During every iteration, we select 1 of the folds as the validation set. Then we train the model on the remaining 4 folds. During every iteration, we select a new fold as the validation set and then train and validate the model again.<\/p>\n For every iteration, we compute the performance of the newly trained model. <\/p>\n After we iterate through all of the folds (such that every fold has been used exactly 1 time as the validation set), then we average the results across all folds.<\/p>\n Although there are some complexities with k-fold CV, as well as some advantages and disadvantages that you need to consider when using it, I think that it’s somewhat easy to understand if you have a good visual image to help explain it.<\/p>\n All that said, let’s discuss why cross validation is so important in machine learning development.<\/p>\n <\/a><\/p>\n Cross validation is critical in machine learning because it’s essential for model evaluation. But because it’s so good at evaluating models, it’s also one of the primary tools that we use to compare different models, and this in turn helps us tune model hyperparameters.<\/p>\n Cross validation is an indispensable tool for model evaluation in machine learning.<\/p>\n And the major reason for this is that cross validation provides a more comprehensive and robust assessment of a model’s performance, compared to a simple train\/validation split.<\/p>\n This advantage comes from how cross validation performs multiple rounds of training and validation. <\/p>\n Recall from my explanation above that in cross validation, we split the data into segments (e.g., folds, in k-fold cross validation) and then iteratively train the model on k-1 of the segments, validate the model on the kth segments, and then repeat that process multiple times, using every segments as the validation set exactly one time.<\/p>\n Effectively, cross validation is like performing a train\/validation split multiple times and then averaging the results.<\/p>\n This evens out the peculiarities that might exist in a single test set.<\/p>\n And in turn, it provides us with a more reliable and robust evaluation of the performance of the model.<\/p>\n Because cross validation provides us with a more robust way to evaluate the performance of individual models, it’s also a more robust way to compare models.<\/p>\n When we use a single train\/validation split, we run the risk that a particular split of the data may favor one model over another. This could be due to certain idiosyncrasies in the training data for that particular split, such that the training data fails to represent the overall data.<\/p>\n In such a scenario, one model or algorithm might perform better on that one, idiosyncratic split. <\/p>\n In turn, this might make it appear that the “better performing” model is better generally<\/em> but in reality, it’s just due to the particular random split that we made.<\/p>\n Cross validation mitigates this issue somewhat by splitting the data into segments (e.g., folds) and rotating those folds into the training and validation sets. <\/p>\n This helps ensure that we evaluate every model across a broader range of possible datasets, which ensures that the evaluation of each model better reflects the model’s ability to generalize. <\/p>\n Further, cross validation plays a critical role in hyperparameter tuning.<\/p>\n Recall that hyperparameters are the parameters that the data scientist or machine learning engineer manually sets prior to the model training process. Hyperparameters contrast with run-of-the-mill “parameters” which are learned during model training. So “parameters” are learned via exposure to data, but “hyperparameters” are manually set by the developer.<\/p>\n The challenge with hyperparameters is selecting the optimal values. This becomes even more complicated when you have multiple hyperparameters. <\/p>\n For example, in a vanilla, feedforward neural network, there are many different hyperparameters, like:<\/p>\n And more.<\/p>\n For most of those hyperparameters, there are multiple different settings or values. For example, there are at least a half dozen activation functions you could use (actually, more than that). For the number of layers, you could have as few as 1 layer and as many as hundreds of layers. The same for the number of neurons.<\/p>\n Because every one of those hyperparameters has multiple values, and there are multiple values for each hyperparameter, there’s potentially tens of thousands … even millions of different combinations of hyperparameter settings.<\/p>\n How do you choose the best combination of hyperparameter settings?<\/p>\n Again, it comes back to being really good at model evaluation. <\/p>\n And in some sense, it’s just a variation of model comparison and selection. But instead of choosing between different algorithms, when you’re evaluating different hyperparameter settings, you’re often trying to choose between models that use the same algorithm, but just different hyperparameter settings. <\/p>\n In order to do this properly, you need to be able to identify the hyperparameter settings that perform well across a range of different training datasets. You need to identify the combination of hyperparameter settings that will generalize to previously unseen data.<\/p>\n Perhaps one of the biggest reasons that cross validation is important for machine learning development is that it helps mitigate overfitting.<\/p>\n Cross-validation helps prevent overfitting by ensuring that the performance of a model generalizes. <\/p>\n Said differently, cross validation makes sure that model performance isn’t overly tailored to a single, particular training dataset. <\/p>\n In traditional model training, a model might perform exceptionally on the training data but perform poorly on new, previously unseen data … a phenomenon known as overfitting. <\/p>\n Cross-validation combats this by dividing the data into multiple sets, and then training and validating the model repeatedly multiple times. <\/p>\n By repeating the training and validation process several times, with different data partitions each time, cross-validation helps ensure that the model generalizes well rather than just memorizing specific training data patterns.<\/p>\n <\/a><\/p>\n Having explained what cross validation is, and why it’s useful, let’s briefly discuss the main types of cross validation.<\/p>\n There are several types of cross validation, but the main ones that you should know are:<\/p>\n All of these forms of cross validation are similar in that they work over multiple iterations. We split the data, then train and validate multiple times.<\/p>\n What’s different about the the different types of cross validation is how they split the data<\/em>.<\/p>\n Let’s quickly discuss each of these.<\/p>\n I explained k-fold cross validation previously in this article, but let me briefly touch on it again.<\/p>\n K-fold cross validation is the most popular form of cross validation.<\/p>\n In k-fold cross validation, we divide the training data into ‘k’ equally-sized subsets, which are known as folds. <\/p>\n Then, during every iteration, one of the folds is held out as the validation set on which we evaluate the model.<\/p>\n And we train the model on the remaining k-1 folds. <\/p>\n We repeat this process k times, with each of the k folds used exactly once as the validation set.<\/p>\n Then we average the results from all k iterations to produce a single estimate of model performance.<\/p>\n Stratified k-fold cross validation is more common in classification tasks (i.e., where the target variable is a categorical label).<\/p>\n In stratified k-fold cross validation, the process ensures that when we split the data into folds, every fold of the dataset has the same proportion of observations with a particular class.<\/p>\n This method of cross validation is particularly useful when we’re dealing with imbalanced datasets.<\/p>\n Leave-one-out cross validation, which is also known as LOOCV, is a special case of k-fold cross validation, where ‘k’ is the number of examples in the dataset. <\/p>\n So if the dataset that you’re using has 100 examples, LOOCV will have 100 folds, and it will operate over 100 iterations.<\/p>\n Therefore, when we use leave-one-out cross validation, during every iteration, we use only one example as the test point on which we validate the model. And all of the remaining examples are used to train the model. <\/p>\n Leave-one-out cross validation can produce robust evaluations of your models, but it’s also very computationally expensive. It’s particularly computationally expensive for large datasets. Therefore, you may want to limit the use of LOOCV to situations where you have a small dataset, or when you have a very large amount of computational resources.<\/p>\n We use grouped cross validation when the data has groups or clusters of related points (i.e., where points within a group are related, and points in separate groups are unrelated or less related).<\/p>\n In such a situation, points within a particular group should be kept together instead of potentially being assigned to different folds (as could happen in k-fold cross validation).<\/p>\n So in grouped cross validation, examples in a particular group are always kept together in either the training set or the test set, but never split into both.<\/p>\n One more type of cross validation is Monte Carlo cross validation (AKA, repeated training\/test splits)<\/p>\n Monte Carlo cross validation is a little different than k-fold.<\/p>\n Instead of splitting the data into k different folds, and then rotating the folds into the validation set (with the remaining data in the training set), Monte Carlo repeatedly splits the data randomly into training and test sets.<\/p>\n So whereas in k-fold cross validation, the algorithm splits the data into k-folds, and those folds are kept the same over every iteration, in Monte Carlo cross validation, there’s a new split of the data at every iteration. The algorithm just randomly splits the data into training and test, multiple times.<\/p>\n This can create some different behavior from k-fold cross validation.<\/p>\n In k-fold cross validation, every example will be in the “test” set exactly one time. But in Monte Carlo cross validation, it’s possible that an example can appear in the test set one time, multiple times, or zero times. It just depends on how the algorithm randomly splits the data in the iterations. <\/p>\n I should note here that different cross validation schemes have different strengths and weaknesses.<\/p>\n Because this is a very big subject, it’s difficult to cover everything here in a single blog post.<\/p>\n That said, I’m going to write more about these different CV schemes individually in separate blog posts, and I’ll also write more comparisons of the different schemes, so you know when to use which type of cross validation.<\/p>\n <\/a><\/p>\n Although cross validation is a powerful tool for model evaluation in machine learning, it also comes with some disadvantages and limitations that wee need to consider to ensure that we use it most effectively.<\/p>\n Let’s talk about them.<\/p>\n One of the biggest issues and limitations of cross validation, particularly methods like leave-one-out and Monte Carlo, is high computational cost.<\/p>\n Every fold or iteration in cross validation requires a separate training and validation step. In many instances, training a model a single<\/em> time can be very time consuming, like when you train a deep neural network or boosted tree model. <\/p>\n Repeating training and validation multiple times then can become very time-consuming and computationally expensive. <\/p>\n This becomes even more of an issue when you’re working with large datasets or complex models.<\/p>\n To mitigate the issues with computational cost, you can:<\/p>\n Of course, all of these have their own costs and\/or tradeoffs.<\/p>\n Cross validation assumes that the data examples are independent and identically distributed (i.i.d.).<\/p>\n However, in many real-world tasks, data has a form that breaks this requirement.<\/p>\n For example, data can have groups, time dependences, or forms of correlation that make it hard or impossible to use certain types of cross validation.<\/p>\n To address this, you can:<\/p>\n <\/a><\/p>\n Before we close, let’s quickly cover some best practices for using cross validation in your machine learning development tasks.<\/p>\n Cross validation is a powerful tool for evaluating the ability of a model to generalize.<\/p>\n That said, cross validation should be used thoughtfully, since it also has increased computational costs and can introduce potential biases in the model.<\/p>\n Ultimately, you need to know the strengths and weaknesses of different cross validation techniques, which will enable you to select the right cross validation technique for your specific machine learning problem.<\/p>\n That said, I’m going to write more in the future about specific types of cross validation, as well as how and when to use it, how to optimize cross validation, how to deal with specific cross validation issues, etc.<\/p>\n <\/a><\/p>\n Do you still have questions about cross validation?<\/p>\n Is there something else that you want me to explain?<\/p>\n I want to hear from you.<\/p>\n Leave your questions, concerns and comments in the comments section at the bottom of the page.<\/p>\n If you want to learn more about data science and machine learning, then sign up for our email list.<\/p>\n Every week, we publish free long-form articles about a variety of topics in machine learning, AI, and data science, including:<\/p>\n When you sign up for our email list, then we’ll deliver those tutorials to you, free, and direct to your inbox.<\/p>\n","protected":false},"excerpt":{"rendered":" In machine learning, making sure that you have a model that performs well is, in some sense, the most important thing. This means that you need to be really good at evaluating different models. But, this can be a challenge, as you run into issues like overfitting, hyperparameter optimization, and model selection. One way to … Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":7886,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":"","om_disable_all_campaigns":false,"footnotes":""},"categories":[19],"tags":[],"publishpress_future_action":{"enabled":false,"date":"2024-05-15 01:03:10","action":"change-status","newStatus":"draft","terms":[],"taxonomy":"category"},"_links":{"self":[{"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/posts\/7447"}],"collection":[{"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/comments?post=7447"}],"version-history":[{"count":1,"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/posts\/7447\/revisions"}],"predecessor-version":[{"id":7881,"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/posts\/7447\/revisions\/7881"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/media\/7886"}],"wp:attachment":[{"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/media?parent=7447"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/categories?post=7447"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/tags?post=7447"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\n
Cross Validation Basics<\/h2>\n
WTF is Cross Validation<\/h3>\n
K-Fold Cross Validation<\/h3>\n
\n
Example of 5-Fold Cross Validation<\/h4>\n
![\"A](\"https:\/\/www.sharpsightlabs.com\/wp-content\/uploads\/2024\/02\/5-fold-cross-validation_SIMPLE-EXAMPLE_v2.png\")
<\/p>\nWhy Cross Validation is Important for Machine Learning<\/h2>\n
Cross Validation is a Critical Model Evaluation Tool<\/h3>\n
Cross Validation Helps us Compare Models<\/h3>\n
Cross Validation is Essential For Hyperparameter Tuning<\/h3>\n
\n
Cross Validation helps Mitigate Overfitting<\/h3>\n
The Main Types of Cross Validation<\/h2>\n
\n
K-Fold Cross-Validation<\/h3>\n
Stratified K-Fold Cross-Validation<\/h3>\n
Leave-One-Out Cross-Validation (LOOCV)<\/h3>\n
Grouped Cross-Validation<\/h3>\n
Monte Carlo Cross Validation (Repeated Training\/Test Splits)<\/h3>\n
Different Cross Validation Schemes have Different Strengths and Weaknesses<\/h3>\n
Special Concerns and Considerations with Cross Validation<\/h2>\n
Cross Validation has High Computational Costs<\/h3>\n
\n
Cross Validation Requires Specific Types of Data<\/h3>\n
\n
Best Practices and Tips for Effective Use of Cross Validation<\/h2>\n
\n
Wrapping Up<\/h2>\n
Leave your Questions Below<\/h3>\n
Sign up for our email list<\/h3>\n
\n