- \n
- Quick Review of Precision-Recall Curves<\/a><\/li>\n
- Setup<\/a><\/li>\n
- Create Data<\/a><\/li>\n
- Plot the Precision-Recall Curve with Seaborn Objects<\/a><\/li>\n
- Why Use Seaborn for Precision-Recall Curves<\/a><\/li>\n
- Frequently Asked Questions<\/a><\/li>\n<\/ul>\n
I’ll show you the actual example further down in the tutorial, but we’ll start with a quick review of what a precision-recall curve is.<\/p>\n
Let’s jump in.<\/p>\n
<\/a><\/p>\n
A Quick Review of Precision-Recall Curves<\/h2>\n
In machine learning, one of the most important tasks is classification<\/em>.<\/p>\n
In classification, we categorize incoming data examples<\/a> into one of several pre-defined groups, which are called “classes” or “labels.”<\/p>\n
There are fairly simple classification systems, like binary classification<\/a>, where there are only 2 classes. And there are more complicated forms of classification, like multi-class classification, where there are more than 2 classes.<\/p>\n
But ultimately, classification is about predicting which class an example belongs to, from a pre-defined list of possible classes.<\/p>\n
Classification Evaluation is Important When You Build a Classifier<\/h3>\n
Building a classifier can be somewhat challenging.<\/p>\n
There are multiple steps, like getting your data, cleaning the data, analyzing the data, preprocessing the data, choosing the right classification algorithm (like logistic regression, decision trees, etc). <\/p>\n
But once you have your classification system, you need to evaluate how it is performing<\/em>.<\/p>\n
Evaluation is critical. <\/p>\n
You need to know how well your classifier is performing, and you need to be able to optimize it given the specific needs of your project.<\/p>\n
That’s where the precision-recall curve comes in.<\/p>\n
A Precision-Recall Curve is a Tool to Evaluate Classifiers<\/h3>\n
Precision-Recall curves are an important tool for classifier evaluation.<\/p>\n
We typically use them to evaluate binary classifiers, although it’s possible to use precision-recall curves with multiclass classification too.<\/p>\n
To explain them simply: precision-recall curves visualize the precision and recall for a classifier across a range of classification thresholds<\/a>.<\/p>\n
![\"An](\"https:\/\/www.sharpsightlabs.com\/wp-content\/uploads\/2023\/12\/precision-recall-curve-python_simple-example.png\")
<\/p>\nIt’s important to emphasize the part about the “range of thresholds.”<\/p>\n
You need to remember that many classification algorithms, and in particular, probabilistic classification algorithms, produce what’s called a “probability score.” The probability score measures the confidence that the classifier has that an incoming example belongs to the positive class.<\/p>\n
In classification systems that produce such a score, the system compares the score to a “threshold” … a cutoff point for the score.<\/p>\n
If the probability score is above the threshold, then the incoming example is predicted as
positive<\/code>.<\/p>\nIf the probability score is below the threshold, then the incoming example is predicted as
negative<\/code>.<\/p>\nSo the threshold acts like a cutoff point for the system to decide when an incoming example should be predicted to be
positive<\/code> ornegative<\/code>.<\/p>\nAnd importantly, the threshold is something we can change. We can increase the threshold (which causes the classifier to become more conservative in making a
positive<\/code> prediction). Or we can lower the threshold (which makes the classifier more liberal in making apositive<\/code> prediction). <\/p>\nMoving the threshold has implications for the number of True Positives<\/a>, True Negatives<\/a>, False Positives<\/a>, and False Negatives<\/a>.<\/p>\n
And in turn, moving the threshold can change both precision and recall.<\/p>\n
A Quick Review of Precision and Recall<\/h3>\n
Precision and recall are themselves classification metrics. We use them to evaluate classifier performance.<\/p>\n
Precision measures the proportion of
positive<\/code> predictions that are actuallypositive<\/code><\/a>.<\/p>\nRecall measures the proportion of actual
positive<\/code> examples that are correctly predicted aspositive<\/code><\/a>.<\/p>\nBoth of these metrics are useful for classification evaluation, but they tend to have strengths and weaknesses.<\/p>\n
Moreover, there’s a tradeoff between them.<\/p>\n
Generally, as precision increases, recall decreases, and vise versa.<\/p>\n
So when we build a classifier, we often need to select the right value for the classification threshold.<\/p>\n
But changing the threshold changes precision and recall.<\/p>\n
So we need to make sure to choose the “best” threshold for a given task that optimizes the tradeoff between precision and recall.<\/p>\n
That’s where the precision-recall curve comes in.<\/p>\n
A Precision-Recall Curve Visualizes the Tradeoff Between Precision and Recall<\/h3>\n
To put it very simply then:<\/p>\n
A precision recall curve visualizes the tradeoff between precision and recall for a classifier, as we change the threshold.<\/p>\n
As we increase the threshold, precision generally increases and recall decreases.<\/p>\n
Conversely, as we decrease the threshold, precision generally decreases and recall increases.<\/p>\n
![\"An](\"https:\/\/www.sharpsightlabs.com\/wp-content\/uploads\/2023\/12\/precision-recall-curve_threshold-explanation.png\")
<\/p>\nAgain: there’s a tradeoff between precision and recall.<\/p>\n
Furthermore, this tradeoff is important because if we optimize for precision, we’re likely to get more False Negatives, and when we optimize for recall, we’re likely to get more False Positives.<\/p>\n
So, we often need to select the right threshold depending on the needs of the task, and also how many False Negatives or False Positives we’re willing to tolerate.<\/p>\n
How do we choose the right threshold?<\/p>\n
One good way to choose is by plotting the precision-recall curve.<\/p>\n
The precision recall curve enables you to see<\/em> the tradeoff between precision and recall, and therefore helps you decide exactly which threshold will be best, given the goals of your project.<\/p>\n
With all that said, let me show you how to create a precision-recall curve in Python.<\/p>\n
<\/a><\/p>\n
Plot a Precision-Recall Curve in Python using Seaborn Objects<\/h2>\n
In the rest of this tutorial, I’ll show you how to plot a precision-recall curve in Python.<\/p>\n
Specifically, I’ll show you how to plot a precision-recall curve using the excellent Seaborn Objects data visualization package (which I love).<\/p>\n
To do that though, we need to perform several steps.<\/p>\n
We will need to:<\/p>\n
- \n
- import the necessary packages<\/li>\n
- create the data for the precision-recall curve<\/li>\n
- plot the curve<\/li>\n<\/ul>\n
We’ll do each of these one at a time.<\/p>\n
(Note: if you already have some precision\/recall data that you want to plot, then feel free to skip to the section where we plot the precision-recall curve with Seaborn<\/a>.<\/p>\n
Import Packages<\/h3>\n
First, we need to import the necessary packages.<\/p>\n
Since we’re going to plot the curve with the Seaborn Objects package, we need to import Seaborn Objects.<\/p>\n
But remember: we’ll also need to generate some data to plot. That means that we’ll need to:<\/p>\n
- \n
- create a classification dataset<\/li>\n
- split the data<\/li>\n
- build a classification model (we’re going to build a logistic regression model)<\/li>\n
- generate the precision\/recall data for the curve itself<\/li>\n<\/ul>\n
All that being the case, we will need to import
make_classification<\/code>,train_test_split<\/code>,LogisticRegression<\/code>, andprecision_recall_curve<\/code> from Scikit Learn in execute all of those steps.<\/p>\n\r\n# seaborn objects\r\nimport seaborn.objects as so\r\n\r\n# scikit learn tools to create model and data\r\nfrom sklearn.datasets import make_classification\r\nfrom sklearn.linear_model import LogisticRegression\r\nfrom sklearn.model_selection import train_test_split\r\nfrom sklearn.metrics import precision_recall_curve\r\n<\/pre>\n
<\/a><\/p>\n
Make Data for the Precision-Recall Curve<\/h3>\n
Now that we have the right packages and tools, we’ll make the data for the precision recall curve.<\/p>\n
Remember that when we plot a precision-recall curve, we’re essentially plotting the precision and recall over a range of classification thresholds for a particular model.<\/p>\n
To get this data, we need to:<\/p>\n
- \n
- create a dataset using sklearn make_classification<\/a> with which we can build a classifier<\/li>\n
- split the data using train_test_split<\/a> from Scikit-learn<\/li>\n
- build the classifier (we’ll create a Logistic Regression model<\/a>)<\/li>\n
- predict the output classes<\/li>\n
- generate the precision\/recall data with the Scikit-Learn precision_recall_curve function<\/a><\/li>\n<\/ul>\n
Since I want to keep this brief, I’m going to move quickly through most of these data creation steps, although you can click on the links above to the related tutorials that will explain how each of these tools\/functions work, if you want more details.<\/p>\n
\r\n# GENERATE CLASSIFICATIONDATASET WITH 2 CLASSES\r\nX, y = make_classification(n_samples=1000, n_classes=2, random_state=1)\r\n\r\n\r\n# SPLIT DATA INTO TRAIN\/TEST\r\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=2)\r\n\r\n\r\n# BUILD & FIT CLASSIFICATION MODEL\r\nmy_logistic_reg = LogisticRegression()\r\nmy_logistic_reg.fit(X_train, y_train)\r\n\r\n\r\n# PREDICT PROBABILITIES FOR TEST SET\r\nprobabilities_logistic_reg = my_logistic_reg.predict_proba(X_test)\r\n\r\n\r\n# KEEP PROBABILITIES FOR ONLY THE POSITIVE OUTCOME\r\nprobabilities_logistic_posclass = probabilities_logistic_reg[:, 1]\r\n\r\n\r\n# CALCULATE PRECISION AND RECALL FOR EACH THRESHOLD\r\nprecisions, recalls, thresholds = precision_recall_curve(y_test, probabilities_logistic_posclass)\r\n\r\n<\/pre>\n
After you run all of the data generation code above to get the precisions and recalls for the logistic regression model, you’ll be ready to plot the precision recall curve.<\/p>\n
<\/a><\/p>\n
Plot the Precision-Recall Curve with Seaborn Objects<\/h3>\n
In this final step, we’re going to plot the precision-recall curve.<\/p>\n
To do this, we’ll use the Seaborn Objects visualization package.<\/p>\n
Let’s run the code, and then I’ll explain.<\/p>\n
\r\n(so.Plot()\r\n .add(so.Path(color = 'red', marker = '.')\r\n ,x = recalls\r\n ,y = precisions\r\n )\r\n .layout(size = (8,5))\r\n)\r\n<\/pre>\n
OUT:<\/p>\n
![\"An](\"https:\/\/www.sharpsightlabs.com\/wp-content\/uploads\/2023\/12\/precision-recall-curve_seaborn-objects.png\")
<\/p>\nExplanation<\/h5>\n
Here, we used Seaborn Objects to create our precision-recall curve.<\/p>\n
But syntactically, what did we do?<\/p>\n
We used the
so.Plot()<\/code> function to initiate the plot.<\/p>\nWe used the
.add()<\/code> method to “add” a layer to the plot. Specifically, we added a “path” layer. This is like a line plot, but it keeps the data in the original order (as opposed to so.Line() which re-sorts the data).<\/p>\nAs you can see, we also needed to specify to put the
recalls<\/code> on the x-axis andprecisions<\/code> on the y-axis. We also changed the color to ‘red’ and added a dot marker over the line withmarker = '.'<\/code>.<\/p>\nIt’s fairly straight-forward.<\/p>\n
<\/a><\/p>\n
Closing Note: Use Seaborn for ML Visualization<\/h3>\n
One final note before I finish the tutorial.<\/p>\n
I recommend that you use Seaborn for your Python<\/a> data visualization<\/a>.<\/p>\n
It’s great for reporting and analytics, but it’s also great for machine learning visualizations.<\/p>\n
But beyond the traditional Seaborn package, I strongly recommend that you start using the new Seaborn Objects visualization system<\/a>.<\/p>\n
Why?<\/p>\n
Seaborn Objects is simple, powerful, and easy to use. If you’re familiar with R, it’s like a ggplot2 or Python.<\/p>\n
I’ve started to use Seaborn Objects for the vast majority of my Python visualizations. <\/p>\n
And for machine learning, as you’ve seen here, you can use it for ML visualizations like precision recall curves, and also for ROC curves (which I show in this tutorial about making ROC curves with Seaborn Objects<\/a>).<\/p>\n
- split the data using train_test_split<\/a> from Scikit-learn<\/li>\n
- create a dataset using sklearn make_classification<\/a> with which we can build a classifier<\/li>\n
- Setup<\/a><\/li>\n