Although there is a little nuance to that statement, roughly speaking, it’s true. A lot<\/em> of your work will involve getting, cleaning, reshaping, and aggregating your data.<\/p>\n But data manipulation is not only one of the most common tasks you’ll need to perform.<\/p>\n It’s also one of the most challenging tasks for most people trying to learn data science.<\/p>\n Have you ever gotten stuck when trying to wrangle or clean a dataset?<\/p>\n You’re not alone.<\/p>\n Data manipulation is hard.<\/p>\n Having said that, there’s a secret that you should know about data manipulation.<\/p>\n Even though many people get stuck, it’s much easier than you think, once you simplify it down to the core operations that you need to perform.<\/p>\n To do most data manipulation, you only need to be able to perform a few operations with a handful of techniques.<\/p>\n In particular, in Python, there are 19 “core” techniques<\/a> that make up the vast majority of the work. <\/p>\n These 19 techniques are like building blocks …<\/p>\n They are like little components that you can just snap together to do data manipulation tasks.<\/p>\n When you look at it this way, data manipulation becomes much more simple.<\/p>\n For the most part, to master data manipulation, you only need to do two things:<\/p>\n When you think about it that way, it’s pretty easy!<\/p>\n Once you know the simple building blocks, you can put them together to perform fairly complex data manipulation in Python<\/a>.<\/p>\n So what are these techniques?<\/p>\n To master data manipulation in Python using Pandas<\/a>, you need to learn these techniques:<\/p>\n All things considered, it’s a pretty brief list. <\/p>\n You don’t need to learn 1000 things. Only 19. <\/p>\n If you studied hard, you could do it in only a couple of weeks.<\/p>\n Let’s quickly review what each of these do.<\/p>\n When you start wrangling and cleaning your data, one of the first things that you typically need to do is just get<\/em> your data. <\/p>\n Often, your data will be in an external data source, and one of the most common sources is That being said, you should know the Pandas The Keep in mind that once you do this, the data in the dataframe will still be messy. You’ll still need to do a lot of data cleaning to make the data ready for analysis or visualization.<\/p>\n (That’s what most of the other techniques are for.)<\/p>\n After you get your data, you’ll often need to set an “index”<\/a> for the rows of the dataframe.<\/p>\n An index (particularly, a row index) is a number or string that identifies the row.<\/p>\n By default, the index values of a dataframe are integers. These allow you to get the data for row “1” or row “10” (as we’ll see later. <\/p>\n But you can use Pandas Again: indexes make it easier to retrieve and subset your data. They are also used for “join” operations, when you need to join together multiple datasets.<\/p>\n That being the case, you need to know what indexes are, and you need to be able to use the Pandas Sometimes, when you’re wrangling your data, you need to change the index of the dataframe<\/a>, or remove it altogether.<\/p>\n This is particularly true once you start doing more advanced computations … sometimes, you need to change the index or temporarily remove it to create a particular data visualization.<\/p>\n The Pandas Once you have your indexes set properly, you can use the index to create subsets. <\/p>\n This is one of the most common tasks you’ll need to perform.<\/p>\n It’s common to subset your data in the beginning stages of data cleaning (to inspect the data), but it’s also extremely common to create subsets at later stages as well … both later in the data manipulation phase, and also in more advanced data science phases like data visualization and analysis.<\/p>\n The The Again, subsetting is really important for data inspection, but also for data visualization and analysis. You need to know this.<\/p>\n In data manipulation and cleaning, there are times when you need to delete rows of data.<\/p>\n Sometimes, you want to delete rows permanently, because they don’t belong in your data for one reason or another.<\/p>\n Other times, you just want to remove rows temporarily<\/em> so they don’t appear in a data visualization or data aggregation.<\/p>\n In Pandas, you can delete rows of data using the Alternatively, you might only want to drop missing<\/em> data.<\/p>\n So for example, if all of the data are missing for a row, it’s fairly likely that you’ll want to remove that row of data.<\/p>\n In other cases, you might not care if the data are missing for certain variables, but if the data are missing for an “important” variable, you will need to delete the row. <\/p>\n For example, if you have data for customers at a bank, it might be okay if the data for the customer’s age is missing, but it might be really bad if their name is missing. So you want to be able to selectively delete a row if the data are missing for a person’s name.<\/p>\n In Pandas, you can do this with Dealing with missing values is an important part of data cleaning.<\/p>\n As mentioned above, you can delete<\/em> rows with missing data.<\/p>\n But frequently, you will want to fill in the missing data.<\/p>\n With Pandas, you can fill in those missing values with The Once you fix problems in the rows of data, you commonly want to fix issues with the columns<\/em> in your data.<\/p>\n One of the most common problems is that the names of the variables are incorrect or the names simply need to be changed.<\/p>\n You can use the Pandas It’s also common to create new variables.<\/p>\n For example, let’s say that you have data about houses that were sold. The data has one variable called What if you want to do an analysis that involves price per square foot?<\/p>\n You’ll probably need to create new variable that contains price per square foot.<\/p>\n In Python, you can do this with the Pandas The Sometimes, after you clean up the rows and columns, you need to “reshape” your data.<\/p>\n We call this process of “reshaping” your data transposition. <\/p>\n One of the tools for doing this in Pandas is the The pivot method transposes your data from long form to wide form.<\/p>\n The other major tool for reshaping your data is the Whereas pivot transposes from long to wide …<\/p>\n The melt method transposes from wide to long.<\/p>\n If those concepts are a little confusing, that’s okay. Reshaping data with A lot<\/em> of people get completely stuck at this stage.<\/p>\n The truth though, is that transposing data isn’t that hard. You just need someone to show you clear examples that build your intuition, and you need to master these two functions, Once you know how they work, and once you’ve memorized the syntax, transposing your data is fairly straightforward.<\/p>\n Once you’ve cleaned rows and columns and transposed your data into the right shape, you sometimes need to combine data. <\/p>\n This is the case if you have multiple datasets.<\/p>\n One of the ways to combine data is with The So if you have multiple datasets with different rows that you need to combine together, you can use If you have multiple dataframes, you might also need to “merge” your data.<\/p>\n A merge operation is when you need to join two datasets together on an index or on the values for some row. <\/p>\n So let’s say that you have two different datasets. Each dataset has rows of data for particular people, and you want to “join” or combine the data based on the peoples’ names.<\/p>\n You can do this with the Pandas So once you have your data cleaned up, reshaped, and joined together, what do you do?<\/p>\n Often, you need to analyze your data.<\/p>\n This frequently requires you to create new subsets of data, which you can then then sort, group, and aggregate.<\/p>\n You need to be able to filter down to specific columns.<\/p>\n The (To be clear, you also use the During analysis, it’s also extremely common to subset the rows. <\/p>\n Earlier, I mentioned But you can also subset rows based on logical conditions. <\/p>\n For example, if you were analyzing data with bank customers, you might want to retrieve data only for customers who are 18 or older. The logical condition, You can perform operations like this in Python with the Keep in mind that there are other ways to do this in Python besides the To be honest, the other ways to do this are a lot more complicated and harder to read.<\/p>\n If you aren’t already using it, you should be using the Sometimes, when cleaning, inspecting, or analyzing your data, you need to sort the rows based on some variable.<\/p>\n So for example, if you have a dataset with different people, maybe you want to sort the rows of data by age.<\/p>\n In Pandas, you can do this with the This is fairly common in data inspection.<\/p>\n But it’s also very common in data visualization and data analysis. <\/p>\n If you’ve ever needed to display your data in a specific, sorted order, this is the tool that you need.<\/p>\n In data analysis, it is extremely common to aggregate your data by a group.<\/p>\n That is, you commonly need to compute some metric by category. For example, lets say that you have bank data for individual bank customers. If you want to compute the average checking account balance by city, you’ll need to group your data by a To do this in Pandas, you need to use the I need to emphasize how common this is. This is a critical data manipulation tool that you will use over and over again, both in data cleaning, but also particularly in data analysis.<\/p>\n If you feel like you can’t analyze your data well, you probably need to learn how to use this method better.<\/p>\n Finally, there’s In Pandas, we use So if you have a large dataset, and you need to compute the average, maximum, minimum, or sum of some variable, This is a critical tool for data analysis. To analyze your data, you should be using this frequently.<\/p>\n Moreover, it becomes even more powerful once you combine it with That’s about it.<\/p>\n 19 tools.<\/p>\n The 19 tools that I just listed account for the vast<\/em> majority of all data cleaning and data manipulation.<\/p>\n That also includes most of the data manipulation that you need for data visualization and analysis.<\/p>\n Think about it. <\/p>\n You only need to master 19 techniques to do the vast<\/em> majority of data manipulation.<\/p>\n If you learned one technique per day, you could learn them all in less than a month.<\/p>\n Once you master these 19 Pandas techniques, you need to learn how to put them together in Pandas Method Chains<\/a>.<\/p>\n You can string together multiple Pandas methods into a chain. And if you put the whole expression inside of parenthesis, you can perform very complex data manipulations.<\/p>\n This is critical.<\/p>\n It’s the secret<\/em> to data cleaning and data wrangling.<\/p>\n It is also one of the secrets for data visualization and analysis.<\/p>\n So once again: master these 19 Pandas methods, and then master how to put them together to do complex data wrangling work.<\/p>\n If you want a highly effective practice system for memorizing these Pandas data manipulation techniques, you should join our course, Pandas Mastery<\/a>.<\/p>\n Pandas Mastery is our online course that will teach you these critical data manipulation tools, and show you how to memorize them.<\/p>\n Inside the course, you’ll discover our unique practice system that will enable you to memorize all 19 of these data manipulation tools. <\/p>\n And, it will only take a few weeks.<\/p>\n You can find out more here:<\/p>\n <\/p>\n \nLearn More About Pandas Mastery<\/a>\n<\/p>\n <\/strong><\/p>\n","protected":false},"excerpt":{"rendered":" Readers at the Sharp Sight blog will know that I put a lot of emphasis on data manipulation. You’ve heard me repeat the stat many times: 80% of your work as a data scientist will be data manipulation. Although there is a little nuance to that statement, roughly speaking, it’s true. A lot of your … Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":5394,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":"","om_disable_all_campaigns":false,"footnotes":""},"categories":[31,68,58],"tags":[35,69,59],"publishpress_future_action":{"enabled":false,"date":"2024-05-04 19:00:51","action":"change-status","newStatus":"draft","terms":[],"taxonomy":"category"},"_links":{"self":[{"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/posts\/5392"}],"collection":[{"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/comments?post=5392"}],"version-history":[{"count":0,"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/posts\/5392\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/media\/5394"}],"wp:attachment":[{"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/media?parent=5392"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/categories?post=5392"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/tags?post=5392"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}A secret about data manipulation<\/h2>\n
There are 19 “building blocks” for data manipulation in Python<\/h3>\n
\n
The 19 Pandas Methods You Need to Master<\/h3>\n
\n
read csv<\/h4>\n
.csv<\/code> files (“comma separated” text files).<\/p>\nread_csv()<\/code> function and how to use it.<\/p>\nread_csv()<\/code> function takes a .csv<\/code> file and reads the data into a Python dataframe.<\/p>\nset index<\/h4>\n
set_index<\/code> to replace the default index values, and use things that are more meaningful. For example, if you were working data about world countries, you might want to use the 3-letter country codes<\/a> as the row index, assuming you had those codes in your data. <\/p>\nset_index()<\/code> method.<\/p>\nreset index<\/h4>\n
reset_index()<\/code> enables you to do just that. It allows you to remove or change the dataframe index.<\/p>\nloc<\/h4>\n
.loc[]<\/code> method enables you to subset data by character index<\/a>. <\/p>\niloc<\/h4>\n
.iloc[]<\/code> method is similar to .loc[]<\/code> in that it enables you to subset by an index, but when you use .iloc[]<\/code>, you subset by the integer index of the dataframe<\/a>. (Dataframes have integers index values for every row and column.)<\/p>\ndrop<\/h4>\n
drop()<\/code> method.<\/p>\ndropna<\/h4>\n
dropna()<\/code>. The dropna()<\/code> method enables you to delete rows with missing data<\/a> … either if all of the data are missing, or if the data are missing for particular variables.<\/p>\nfillna<\/h4>\n
fillna()<\/code><\/a>.<\/p>\nfillna()<\/code> method lets you “fill in” the missing values with a value of your choice.<\/p>\nrename<\/h4>\n
rename()<\/code> method to rename variables in a Pandas dataframe<\/a>.<\/p>\nassign<\/h4>\n
price<\/code>, which is the sale price of the house. And the data has another variable called square_feet<\/code>, which specifies the square feet of the home.<\/p>\nassign()<\/code> method<\/a>. <\/p>\nassign()<\/code> method is simple. It enables you to create new variables in a Pandas dataframe.<\/p>\npivot<\/h4>\n
pivot()<\/code> function.<\/p>\nmelt<\/h4>\n
melt()<\/code> function.<\/p>\nmelt()<\/code> and pivot()<\/code> often confuse people.<\/p>\nmelt()<\/code> and pivot()<\/code>.<\/p>\nconcat<\/h4>\n
concat()<\/code>.<\/p>\nconcat()<\/code> function stacks multiple dataframes.<\/p>\nconcat()<\/code> to “stack” them into one dataframe.<\/p>\nmerge<\/h4>\n
merge()<\/code> method, although it’s also possible to do it with the join()<\/code> method as well.<\/p>\nfilter<\/h4>\n
filter()<\/code> method enable you to do this. It enables you to retrieve specific columns of a dataframe.<\/p>\nfilter()<\/code> method during data cleaning.)<\/p>\nquery<\/h4>\n
loc[]<\/code> and iloc[]<\/code>, which enable you to subset rows based on index values.<\/p>\nage >= 18<\/code>, acts as a condition for the subset … keep a row if True<\/code> and exclude the row if False<\/code>.<\/p>\nquery()<\/code> method. <\/p>\nquery()<\/code> method. Many people use a complicated “bracket” notation to retrieve data from Pandas dataframes based on logical conditions.<\/p>\nquery()<\/code> method to subset your data based on logical conditions. The code is much easier to read and write than the alternatives.<\/p>\nsort values<\/h4>\n
sort_values()<\/code> method.<\/p>\ngroupby<\/h4>\n
city<\/code> variable.<\/p>\ngroupby()<\/code> method.<\/p>\nagg<\/h4>\n
agg()<\/code>.<\/p>\nagg()<\/code> to compute summary statistics.<\/p>\nagg()<\/code> is the tool you need.<\/p>\ngroupby()<\/code>.<\/p>\nData wrangling is much easier than you think<\/h2>\n
Pandas Method Chains “Puts the Pieces Together”<\/h2>\n
![\"An](\"https:\/\/www.sharpsightlabs.com\/wp-content\/uploads\/2021\/03\/pandas-chain_syntax-explanation.png\")
<\/p>\nDiscover how to memorize Pandas syntax<\/h2>\n