- \n
- Introduction<\/a><\/li>\n
- Syntax<\/a><\/li>\n
- Examples<\/a><\/li>\n
- Frequently Asked Questions<\/a><\/li>\n<\/ul>\n
Ok. Let’s get into the details.<\/p>\n
<\/a><\/p>\n
A quick introduction to the Pandas value_counts method<\/h2>\n
First, let’s just start with an explanation of what the value_counts technique does.<\/p>\n
Essentially, value_counts counts the unique values<\/em> of a Pandas object. We often use this technique to do data wrangling and data exploration in Python<\/a>.<\/p>\n
![\"A](\"https:\/\/www.sharpsightlabs.com\/wp-content\/uploads\/2021\/07\/pandas-value-counts_simple-visual-example.png\")
<\/p>\nThe value_counts method will actually work on several different types of Pandas objects:<\/p>\n
- \n
- Pandas Series<\/li>\n
- Pandas dataframes<\/li>\n
- dataframe columns (which are actually Pandas Series objects)<\/li>\n<\/ul>\n
Having said that, how you use the value_counts method will vary slightly depending on which type of object you’re operating on.<\/p>\n
Additionally, there are some optional parameters that you can use that will change what value_counts does. <\/p>\n
That being the case, let’s look at the syntax.<\/p>\n
<\/a><\/p>\n
The syntax of value_counts<\/h2>\n
Ok. Let’s look at the syntax of the Pandas value_counts technique.<\/p>\n
Here, I’ll divide this up into different sections, so we can look at the syntax for how to use value_counts on Series objects and how to use value counts on dataframes.<\/p>\n
A quick note<\/h4>\n
The following syntax explanations assume that you’ve imported Pandas, and that you’ve already created a Pandas dataframe or a Pandas series.<\/p>\n
You can import Pandas with this code:<\/p>\n
\r\nimport pandas as pd\r\n<\/pre>\n
And for more information about dataframes, you can read our introduction to Pandas dataframes<\/a>.<\/p>\n
syntax to use value_counts on a Pandas dataframe<\/h3>\n
First, let’s look at the syntax for how to use value_counts on a dataframe.<\/p>\n
This is really simple. You just type the name of the dataframe then
.value_counts()<\/code>.<\/p>\n![\"An](\"https:\/\/www.sharpsightlabs.com\/wp-content\/uploads\/2021\/07\/pandas-value-counts_dataframe-syntax.png\")
<\/p>\nWhen you use value_counts on a dataframe, it will count the number of records for every combination of unique values for every column<\/em>.<\/p>\n
This may be more information than you want, and it may be better to subset the dataframe down to only a few columns. I’ll show you some examples of this in the examples section.<\/p>\n
Additionally, there are some optional parameters that you can use, which will modify the behavior of the method. I’ll show you those in the parameters section<\/a>.<\/p>\n
syntax to use value_counts on a Pandas Series<\/h3>\n
Next, let’s look at the syntax to use value_counts on a Series object.<\/p>\n
The syntax for a Series is almost the same as the syntax for a dataframe:<\/p>\n
![\"An](\"https:\/\/www.sharpsightlabs.com\/wp-content\/uploads\/2021\/07\/pandas-value-counts_series-syntax.png\")
<\/p>\nYou simply type the name of the Series object, and then
.value_counts()<\/code>.<\/p>\nAdditionally, there are some optional parameters that you can use, which we’ll discuss in the parameters section<\/a>.<\/p>\n
syntax to use value_counts on a dataframe column<\/h3>\n
Finally, let’ look at how to use value_counts on a column<\/em> inside of a dataframe.<\/p>\n
Remember: individual dataframe columns are<\/em> Series objects.<\/p>\n
So to call value_counts on a column, we first use “dot syntax” to retrieve an individual column. For example, if your dataframe is named
your_dataframe<\/code> and the column you want to retrieve is calledcolumn<\/code>, you would start by typingyour_dataframe.column<\/code>.<\/p>\n![\"An](\"https:\/\/www.sharpsightlabs.com\/wp-content\/uploads\/2021\/07\/pandas-value-counts_dataframe-column-syntax.png\")
<\/p>\nAfter that, you simply type
.value_counts()<\/code> and the method will retrieve the count of the unique values for that individual column.<\/p>\nAnd once again, there are some additional parameters that you can use to change how value_counts works.<\/p>\n
Let’s look at those parameters.<\/p>\n
<\/a><\/p>\n
The parameters of value counts<\/h3>\n
The Pandas value_counts technique has several parameters that you can use which will change how the technique works and what exactly it does.<\/p>\n
- \n
ascending<\/code><\/li>\nsort<\/code><\/li>\nnormalize<\/code><\/li>\nsubset<\/code><\/li>\ndropna<\/code><\/li>\n<\/ul>\nIn addition, there is the
bins<\/code> parameter, which I rarely use and won’t discuss here.<\/p>\nIt’s important to note that all of these parameters are optional<\/em>.<\/p>\n
It’s also important to note that most of these parameters \u2013
ascending<\/code>,sort<\/code>, andnormalize<\/code> \u2013 are used for both the Series syntax and the dataframe syntax.<\/p>\nOn the other hand,
subset<\/code> is only available when you use value_counts on dataframes, anddropna<\/code> is only available when you use value_counts on Series.<\/p>\nHaving said all of that, let’s look at each of these parameters individually.<\/p>\n
ascending<\/code> <\/h5>\nBy default, value_counts will sort the data by numeric count in descending order<\/em>.<\/p>\n
The ascending parameter enables you to change this.<\/p>\n
When you set
ascending = True<\/code>, value counts will sort the data by count from low to high (i.e., ascending order). <\/p>\nI’ll show you an example of this in example 4<\/a>.<\/p>\n
sort<\/code><\/h5>\nThe sort parameter controls how the output is sorted.<\/p>\n
By default, value_counts sorts the data by the numeric count<\/em>.<\/p>\n
You can change this and sort the data by categories instead by setting
sort = False<\/code>.<\/p>\nI’ll show you an example of this in example 5<\/a>.<\/p>\n
normalize<\/code><\/h5>\nThe
normalize<\/code> parameter changes the form of the output.<\/p>\nBy default, value_counts shows the count of the unique values.<\/p>\n
But if you set
normalize = True<\/code>, value_counts will display the proportion of total records<\/em> instead of the raw count. <\/p>\nI’ll show you an example of this in example 6<\/a>.<\/p>\n
subset<\/code><\/h5>\nThe
subset<\/code> parameter enables you to specify a subset of columns on which to apply value_counts, when you use value_counts on a dataframe.<\/p>\nThe argument to this parameter should be a list (or list-like object) of column names.<\/p>\n
So for example, if you want to use value counts on
var_1<\/code> andvar_2<\/code> in a dataframe, you would use the codeyour_dataframe.value_counts(subset = ['val_1','var_2'])<\/code>.<\/p>\nNOTE: again, this parameter is works when you use value_counts on a whole dataframe.<\/p>\n
I’ll show you an example of this in example 7<\/a>.<\/p>\n
dropna<\/code><\/h5>\nThe
dropna<\/code> parameter enables you to show ‘NA’ values (i.e.,NaN<\/code> values).<\/p>\nYou can do this by setting
dropna = False<\/code>. <\/p>\nNOTE: this parameter is only available for Pandas Series objects and individual dataframe columns. This parameter will not work if you use value_counts on a whole dataframe.<\/p>\n
I’ll show you an example of this in example 2<\/a>.<\/p>\n
<\/a><\/p>\n
Examples: Get Value Counts for Pandas Dataframes and Series Objects<\/h2>\n
Now that we’ve looked at the syntax, let’s look at some examples of how to use the value_counts technique.<\/p>\n
Examples:<\/strong><\/p>\n
- \n
- Use value_counts on a dataframe column<\/a><\/li>\n
- Include ‘NA’ values in the counts<\/a><\/li>\n
- Use value_counts on an entire Pandas dataframe<\/a><\/li>\n
- Sort the output in ascending order<\/a><\/li>\n
- Sort by category (instead of count)<\/a><\/li>\n
- Compute proportions (i.e., normalize the value counts)<\/a><\/li>\n
- Operate on a subset of dataframe columns<\/a><\/li>\n<\/ul>\n
Run this code first<\/h4>\n
Before you run the examples, you’ll need to run some preliminary code in order to:<\/p>\n
- \n
- import necessary packages<\/li>\n
- get a dataframe<\/li>\n
- create a dataframe subset that we can work with<\/li>\n<\/ul>\n
Let’s do those one at a time.<\/p>\n
Import Packages<\/h5>\n
First, let’s import two packages that we’ll need.<\/p>\n
Specifically, we’ll need to import Pandas and Seaborn.<\/p>\n
You can do that with the following code:<\/p>\n
\r\nimport pandas as pd\r\nimport seaborn as sns\r\n<\/pre>\n
Obviously, we’ll need Pandas to use the
value_counts()<\/code> technique. But we’ll also need Seaborn, because we’ll be using thetitanic<\/code> dataframe which we can load from Seaborn’s pre-installed datasets.<\/p>\nGet dataframe<\/h5>\n
Next, let’s get the dataframe we’ll be working with.<\/p>\n
In the following examples, we’ll be using the
titanic<\/code> dataset, or some subset of it.<\/p>\nSo here, let’s load the dataset from Seaborn:<\/p>\n
\r\n# GET DATASET\r\ntitanic = sns.load_dataset('titanic')\r\n<\/pre>\nAdditionally, let’s print it out, so we can see the contents:<\/p>\n
\r\nprint(titanic)\r\n<\/pre>\n
OUT:<\/p>\n
\r\n survived pclass sex age sibsp parch fare embarked class who adult_male deck embark_town alive alone \r\n0 0 3 male 22.0 1 0 7.2500 S Third man True NaN Southampton no False \r\n1 1 1 female 38.0 1 0 71.2833 C First woman False C Cherbourg yes False \r\n2 1 3 female 26.0 0 0 7.9250 S Third woman False NaN Southampton yes True \r\n3 1 1 female 35.0 1 0 53.1000 S First woman False C Southampton yes False \r\n4 0 3 male 35.0 0 0 8.0500 S Third man True NaN Southampton no True \r\n.. ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... \r\n886 0 2 male 27.0 0 0 13.0000 S Second man True NaN Southampton no True \r\n887 1 1 female 19.0 0 0 30.0000 S First woman False B Southampton yes True \r\n888 0 3 female NaN 1 2 23.4500 S Third woman False NaN Southampton no False \r\n889 1 1 male 26.0 0 0 30.0000 C First man True C Cherbourg yes True \r\n890 0 3 male 32.0 0 0 7.7500 Q Third man True NaN Queenstown no True \r\n\r\n[891 rows x 15 columns]\r\n<\/pre>\n
There are 15 columns in this dataframe, which will be a little difficult to work with if we’re using the value_counts() technique.<\/p>\n
That said, let’s quickly create a subset that we can use with some of our examples.<\/p>\n
Create Dataframe Subset<\/h5>\n
Now, let’s create a subset of the
titanic<\/code> dataframe.<\/p>\nHere, we’ll create a subset that contains two variables:
sex<\/code> andembarked<\/code>.<\/p>\nTo subset down to these two variables, we’ll use the Pandas filter method<\/a>:<\/p>\n
\r\n#CREATE SUBSET\r\ntitanic_subset = titanic.filter(['sex','embarked'])\r\n<\/pre>\n
For some of our examples, this subset will simply be easier to work with, since it has only 2 variables.<\/p>\n
<\/a><\/p>\n
EXAMPLE 1: Use value_counts on a dataframe column<\/h3>\n
First, let’s use the value_counts technique on a single column.<\/p>\n
Here, we’ll use value_counts on the
embarked<\/code> variable in thetitanic<\/code> dataframe.<\/p>\nLet’s run the code, and then I’ll explain:<\/p>\n
\r\ntitanic.embarked.value_counts()\r\n<\/pre>\n
OUT:<\/p>\n
\r\nS 644\r\nC 168\r\nQ 77\r\nName: embarked, dtype: int64\r\n<\/pre>\n
Explanation<\/h5>\n
The code to perform this operation is a single line of code, but in some sense, it’s a two step process.<\/p>\n
In this code, we’re: <\/p>\n
- \n
- retrieving the
embarked<\/code> variable with “dot syntax”<\/li>\n- calling the
value_counts()<\/code> method<\/li>\n<\/ul>\nSo, we’re retrieving the
embarked<\/code> variable with the codetitanic.embarked<\/code>.<\/p>\nBut after that, we’re calling the value counts method with
.value_counts()<\/code>.<\/p>\nIn the output, you see the unique values of the
embarked<\/code> variable \u2013S<\/code>,C<\/code>, andQ<\/code> \u2013 and the counts associated with each of those values.<\/p>\n<\/a><\/p>\n
EXAMPLE 2: Include ‘NA’ values in the counts (Series only)<\/h3>\n
Next, let’s include the ‘NA’ values (i.e.,
NaN<\/code>) in the output. This will enable us to see the number of ‘missing’ values for the variable, if there are any.<\/p>\nKeep in mind that here, we’re still going to operate on a single dataframe variable.<\/p>\n
\r\ntitanic.embarked.value_counts(dropna = False)\r\n<\/pre>\n
OUT:<\/p>\n
\r\nS 644\r\nC 168\r\nQ 77\r\nNaN 2\r\nName: embarked, dtype: int64\r\n<\/pre>\n
Explanation<\/h5>\n
Here, we’ve called
value_counts()<\/code> just like we did in example 1<\/a>.<\/p>\nThe only difference is that we included the code
dropna = False<\/code> inside the parenthesis.<\/p>\nAs you can see in the output, there is now a count of the number of
NaN<\/code> values (i.e., “missing” values). <\/p>\nThis can be useful if you need to identify missing values to clean them up, etc.<\/p>\n
NOTE<\/strong>: This will only work if you use value_counts() on a Pandas Series or a dataframe column. It will not<\/em> work if you try to use value_counts on an entire Pandas dataframe (like in example 3).<\/p>\n
<\/a><\/p>\n
EXAMPLE 3: Use value_counts on an entire Pandas dataframe<\/h3>\n
In the last two examples, we used value_counts on a single column<\/em> of a dataframe (i.e., a Pandas series object).<\/p>\n
Now, let’s use value_counts on a whole dataframe.<\/p>\n
Here, we’re going to use value counts on the
titanic_subset<\/code> dataframe. (Remember, we created this subset earlier. It has only two variables to make it easier to work with.)<\/p>\nOk. Let’s run the code:<\/p>\n
\r\ntitanic_subset.value_counts()\r\n<\/pre>\n
OUT:<\/p>\n
\r\nsex embarked\r\nmale S 441\r\nfemale S 203\r\nmale C 95\r\nfemale C 73\r\nmale Q 41\r\nfemale Q 36\r\ndtype: int64\r\n<\/pre>\n
Explanation<\/h5>\n
This is really straight forward.<\/p>\n
To do this, we simply typed the name of the dataframe, and then
.value_counts()<\/code>.<\/p>\nYou can see that the output is a count of the unique combinations of the variables in the dataframe.<\/p>\n
Notice as well that the output is sorted in descending order. That’s the default, but we can change it as well, which we’ll do in the next example.<\/p>\n
<\/a><\/p>\n
EXAMPLE 4: Sort the output in ascending order<\/h3>\n
In this example, we’ll sort the output in ascending order.<\/p>\n
Remember that by default, value_counts sorts the output in descending<\/em> order. <\/p>\n
We can change that behavior though with the
ascending<\/code> parameter.<\/p>\nLet’s take a look:<\/p>\n
\r\ntitanic_subset.value_counts(ascending = True)\r\n<\/pre>\n
OUT:<\/p>\n
\r\nsex embarked\r\nfemale Q 36\r\nmale Q 41\r\nfemale C 73\r\nmale C 95\r\nfemale S 203\r\nmale S 441\r\ndtype: int64\r\n<\/pre>\n
Explanation<\/h5>\n
Here, we see the counts of the unique combinations of values in the dataframe. <\/p>\n
But now, because we set
ascending = True<\/code>, the output is sorted in ascending order … it’s sorted from low to high.<\/p>\n<\/a><\/p>\n
EXAMPLE 5: Sort by category (instead of count)<\/h3>\n
Now, let’s remove the sorting altogether.<\/p>\n
To do this, we’ll call the method with
sort = False<\/code>.<\/p>\n\r\ntitanic_subset.value_counts(sort = False)\r\n<\/pre>\n
OUT:<\/p>\n
\r\nsex embarked\r\nfemale C 73\r\n Q 36\r\n S 203\r\nmale C 95\r\n Q 41\r\n S 441\r\ndtype: int64\r\n<\/pre>\n
Explanation<\/h5>\n
Notice in the output, the data are not sorted by the value counts (i.e., the numbers).<\/p>\n
Instead, the data are sorted by the categories. The unique categorical values in both variables are sorted in alphabetical order. <\/p>\n
Personally, I think this is easier to read, but it does depend on what you’re doing.<\/p>\n
There may be some applications where this is better, and there may be some instances where it’s better to sort the data by the numeric counts (like the default behavior).<\/p>\n
In any case, you have a choice. <\/p>\n
<\/a><\/p>\n
EXAMPLE 6: Compute proportions (i.e., normalize the value counts)<\/h3>\n
In this example, let’s compute the proportions of each unique combination of values.<\/p>\n
In the previous examples,
value_counts<\/code> provided a count of the number of values.<\/p>\nHere, we’ll tell
value_counts<\/code> to compute the percent of total records, using thenormalize<\/code> parameter:<\/p>\n\r\ntitanic_subset.value_counts(normalize = True)\r\n<\/pre>\n
OUT:<\/p>\n
\r\nsex embarked\r\nmale S 0.496063\r\nfemale S 0.228346\r\nmale C 0.106862\r\nfemale C 0.082115\r\nmale Q 0.046119\r\nfemale Q 0.040495\r\ndtype: float64\r\n<\/pre>\n
Explanation<\/h5>\n
The output here is somewhat similar to the output for example 3<\/a>, in the sense that it’s sorted in descending order of frequency.<\/p>\n
But instead of showing the raw counts of each unique combination of categories, it’s showing the proportion. Notice that if you add all the numbers up, they add up to 1.<\/p>\n
So again, the numbers represent the proportion of total records accounted for by each unique combination.<\/p>\n
<\/a><\/p>\n
EXAMPLE 7: Operate on a subset of dataframe columns (dataframes only)<\/h3>\n
In the previous examples, I’ve shown you how to use value_counts on a pandas Series, a small Pandas dataframe (with only 2 columns), or a single dataframe column.<\/p>\n
Here, I’ll show you how to operate a large dataframe with many columns. <\/p>\n
But we’ll use the
subset<\/code> parameter to reduce the size and complexity of the output.<\/p>\nSo here, we’ll be working with the full
titanic<\/code> dataframe, which has 15 columns. We’ll use thesubset<\/code> parameter to operate only on two of those variables:sex<\/code> andembarked<\/code>. <\/p>\nLet’s take a look.<\/p>\n
\r\ntitanic.value_counts(subset = ['sex','embarked'])\r\n<\/pre>\n
OUT:<\/p>\n
\r\nsex embarked\r\nmale S 441\r\nfemale S 203\r\nmale C 95\r\nfemale C 73\r\nmale Q 41\r\nfemale Q 36\r\ndtype: int64\r\n<\/pre>\n
Explanation<\/h5>\n
Here, we’re working with the full
titanic<\/code> dataset. Remember: this is the full dataset with 15 variables (instead of the smallertitanic_subset<\/code> dataframe, which only has 2 variables).<\/p>\nSo here, we’re taking the full
titanic<\/code> dataframe with 15 variables and using value_counts on only 2 variables. To do this, we’re settingsubset = ['sex','embarked']<\/code>. <\/p>\nNotice that syntactically, each variable we want to include is presented as a string (inside of quotation marks). And the collection of variable names is organized into a Python list.<\/p>\n
- calling the
- retrieving the
- Include ‘NA’ values in the counts<\/a><\/li>\n
- Syntax<\/a><\/li>\n