- \n
- Syntax Explanation<\/a><\/li>\n
- Examples<\/a><\/li>\n
- Frequently Asked Questions<\/a><\/li>\n<\/ul>\n
Having said that, it will probably be helpful if you read the whole tutorial from start to finish.<\/p>\n
<\/a><\/p>\n
<\/a><\/p>\n
Syntax: How to Recode a Categorical Variable with Pandas<\/h2>\n
Recoding a categorical variable with in Python<\/a> using Pandas can be performed with a single line of code, but really requires two steps:<\/p>\n
- \n
- retrieve the variable and remap the old values to new values,
using Pandas map<\/a><\/li>\n- assign the output of the map step to a new variable,
using Pandas assign<\/a><\/li>\n<\/ol>\nSyntactically, it looks like this:<\/p>\n
![\"An](\"https:\/\/www.sharpsightlabs.com\/wp-content\/uploads\/2023\/07\/pandas-recod-variable_syntax-explanation.png\")
<\/p>\nIt’s somewhat easy to do, but as shown above, you really need to know how to use two different Pandas tools: Pandas assign and Pandas map.<\/p>\n
That said, it’s best to learn how to do this by working through an example.<\/p>\n
<\/a><\/p>\n
Example: How to Recode a Categorical Variable with Pandas<\/h2>\n
In this example, we’re going to recode a categorical variable with a single letter to the full word.<\/p>\n
Specifically, we’re going to recode a “region” variable that contains abbreviated regions (e.g., “N”) and recode them to the full region name (e.g., “North”).<\/p>\n
As noted above, we’ll use a couple of Pandas tools to do this.<\/p>\n
Import Pandas<\/h4>\n
First of all, you need to import Pandas<\/a>.<\/p>\n
You can do that with this code.<\/p>\n
\r\nimport pandas as pd\r\n<\/pre>\n
Create Dataframe<\/h4>\n
Next, we’ll create a dataframe.<\/p>\n
Specifically, we’ll create some dummy sales data that contains 4 variables:<\/p>\n
- \n
name<\/code>: the name of the sales person<\/li>\nregion<\/code>: the region that the sales person operates in<\/li>\nsales<\/code>: their amount of total sales<\/li>\nexpenses<\/code>: their expenses<\/li>\n<\/ul>\n\r\nsales_data = pd.DataFrame({\r\n\"name\":[\"William\",\"Emma\",\"Sofia\",\"Markus\",\"Edward\",\"Thomas\",\"Ethan\",\"Olivia\",\"Arun\",\"Anika\",\"Paulo\"]\r\n,\"sales\":[50000,52000,90000,34000,42000,72000,49000,55000,67000,65000,67000]\r\n,\"expenses\":[42000,43000,50000,44000,38000,39000,42000,60000,39000,44000,45000]\r\n,\"region\":[\"E\",\"N\",\"E\",\"S\",\"W\",\"W\",\"S\",\"W\",\"W\",\"E\",\"S\"]})\r\n<\/pre>\nAnd let’s print out the data:<\/p>\n
\r\nprint(sales_data)\r\n<\/pre>\n
OUT:<\/p>\n
\r\n name sales expenses region\r\n0 William 50000 42000 E\r\n1 Emma 52000 43000 N\r\n2 Sofia 90000 50000 E\r\n3 Markus 34000 44000 S\r\n4 Edward 42000 38000 W\r\n5 Thomas 72000 39000 W\r\n6 Ethan 49000 42000 S\r\n7 Olivia 55000 60000 W\r\n8 Arun 67000 39000 W\r\n9 Anika 65000 44000 E\r\n10 Paulo 67000 45000 S\r\n<\/pre>\n
Notice that the region variable has the following values:<\/p>\n
- \n
N<\/code><\/li>\nS<\/code><\/li>\nE<\/code><\/li>\nW<\/code><\/li>\n<\/ul>\nIt’s probably obvious that these stand for North, South, East, and West.<\/p>\n
In this example, we’re going to recode the abbreviated region values to the full word for the region.<\/p>\n
<\/a><\/p>\n
Create Mapping with Old Vales and New Values<\/h4>\n
Here, we’ll use a dictionary to create a mapping that connects the old values (that are already in the dataframe) to the new values that we want to output.<\/p>\n
\r\nregion_mapping = {'N':'North'\r\n ,'S':'South'\r\n ,'E':'East'\r\n ,'W':'West'\r\n }\r\n<\/pre>\nNotice that the left-hand side of each item is the old value and the right-hand side is the new value.<\/p>\n
<\/a><\/p>\n
Test the Recode<\/h4>\n
Next, we’re going to test<\/em> our variable recode.<\/p>\n
Why?<\/p>\n
Because sometimes, code doesn’t do exactly what you think it will. When you’re changing data, it’s almost always a good idea to test<\/em> your process first, so you make sure that it’s working properly before you overwrite your data.<\/p>\n
(Trust me, if you overwrite your data with something that’s wrong, it can be a pain in the @$$ … you sometimes need to start all over with your data processing.)<\/p>\n
As mentioned elsewhere, we’re going to perform the recode by using the Pandas Map method<\/a>, in concert with the assign Pandas method<\/a>.<\/p>\n
Remember that Pandas map modifies a Pandas series object, and Pandas assign modifies a dataframe.<\/p>\n
So we’re going to:<\/p>\n
- \n
- retrieve the column that we want to operate on using dot syntax<\/li>\n
- use the map method to recode the values in that column<\/li>\n
- use the assign method to assign the output of Pandas map to a variable in our dataframe<\/li>\n<\/ul>\n
Again: to recode a variable in a dataframe, we need to use a couple different tools.<\/p>\n
Ok, let’s do it.<\/p>\n
\r\nsales_data.assign(TEST = sales_data.region.map(region_mapping))\r\n<\/pre>\n
OUT:<\/p>\n
\r\n name sales expenses region TEST\r\n0 William 50000 42000 E East\r\n1 Emma 52000 43000 N North\r\n2 Sofia 90000 50000 E East\r\n3 Markus 34000 44000 S South\r\n4 Edward 42000 38000 W West\r\n5 Thomas 72000 39000 W West\r\n6 Ethan 49000 42000 S South\r\n7 Olivia 55000 60000 W West\r\n8 Arun 67000 39000 W West\r\n9 Anika 65000 44000 E East\r\n10 Paulo 67000 45000 S South\r\n<\/pre>\n
Notice here that I’m assigning the output to a new variable called
TEST<\/code>.<\/p>\nNotice also that here, the original
region<\/code> variable is still in the dataframe. This is useful because we can compare them side by side, and make sure that our recoded values are correct and appropriate.<\/p>\nIf you see anything out of the ordinary, you may need to go back and modify your variable mapping<\/a>.<\/p>\n
Recode the Variable<\/h4>\n
Assuming that everything looks good when you test out the variable recode, then you can finalize the variable recode.<\/p>\n
Again: remember our strategy here.<\/p>\n
We’re using Pandas map to operate to recode the individual column values, and we’re using Pandas assign to assign that new column to our dataframe.<\/p>\n
So specifically, we’re going to assign the newly recoded
region<\/code> data back to the variable name,region<\/code>. Note that this will overwrite the originalregion<\/code> variable. That’s why we needed to test the recode first<\/a>.<\/p>\n\r\nsales_data = sales_data.assign(region = sales_data.region.map(region_mapping))\r\n<\/pre>\n
Notice as well that we’re storing the output of this whole process to the
sales_data<\/code> dataframe. Remember, this overwrites your original data. That’s why you need to test the operation first<\/a>.<\/p>\nCheck the Recode<\/h4>\n
Now, let’s just print out the dataframe so we can check the output.<\/p>\n
\r\nprint(sales_data)\r\n<\/pre>\n
OUT:<\/p>\n
\r\n name sales expenses region\r\n0 William 50000 42000 East\r\n1 Emma 52000 43000 North\r\n2 Sofia 90000 50000 East\r\n3 Markus 34000 44000 South\r\n4 Edward 42000 38000 West\r\n5 Thomas 72000 39000 West\r\n6 Ethan 49000 42000 South\r\n7 Olivia 55000 60000 West\r\n8 Arun 67000 39000 West\r\n9 Anika 65000 44000 East\r\n10 Paulo 67000 45000 South\r\n<\/pre>\n
This looks good.<\/p>\n
All of the one-letter, abbreviated regions have been replaced with the full region name.<\/p>\n
Final Notes<\/h4>\n
This is a somewhat simple example with only 4 levels in the categorical variable.<\/p>\n
In the past, on Twitter<\/a>, I’ve shown some other examples of variable recoding with a larger number of categories.<\/p>\n
If you have a lot of categories, you typically need to get a list of the unique category values, add all of those category mappings to your dictionary mapping, and then carefully review\/test the output.<\/p>\n
- assign the output of the map step to a new variable,
- Examples<\/a><\/li>\n