![\"An](\"https:\/\/www.sharpsightlabs.com\/wp-content\/uploads\/2023\/05\/titanic_EDA-for-machine-learning_FEATURED-IMAGE.png\")
<\/p>\nIn this tutorial, I’ll show you how to do some simple exploratory data analysis (EDA) for a machine learning project.<\/p>\n
In this tutorial, we’ll look at the Titanic dataset, which is commonly used in machine learning tutorials, and has previously been used as a Kaggle dataset.<\/p>\n
<\/p>\n
This tutorial will really only scratch the surface. There’s a lot of analysis that we could do, but in the interest of brevity, I’ll show you a few things.<\/p>\n
Table of Contents:<\/strong><\/p>\n <\/a><\/p>\n First, we need to set a few things up before we do our EDA.<\/p>\n Specifically, we need to import some packages and get our data.<\/p>\n First, we need to import some packages.<\/p>\n We’re importing Seaborn, which enables us to create many of the visualizations we’ll use to visualize and analyze our data. We’ll also use the relatively new Seaborn Objects sub-package<\/a> (which you might need to install).<\/p>\n And finally, we’ve imported Pandas, which will give us some tools to wrangle or subset our data.<\/p>\n We also need to load the We’ll eventually need to do a little data manipulation on this dataset, but before we do that, we’ll actually need to get a sense of what’s in here. In turn, that will help us decide on how to modify the data going forward.<\/p>\n <\/a><\/p>\n Now that we have a dataframe, we’ll do some basic data inspection.<\/p>\n Specifically, we will:<\/p>\n Here, we’ll print out a few of the records using the Pandas head method<\/a>.<\/p>\n OUT:<\/p>\n Next, we’ll list the data types in the dataframe. <\/p>\n To do this, we’ll call the dtypes property of the dataframe.<\/p>\n OUT:<\/p>\n Here, you can see that we have a mix of integers, floats, categories, and “objects” (which are commonly strings).<\/p>\n We will need to modify or change a few of these, but it will become more obvious how<\/em> we need to change this as we move forward.<\/p>\n Now, before we move on to some visualizations, we’ll get a count of missing values.<\/p>\n OUT:<\/p>\n Most of these columns have zero or very few missing values. But two of the columns ( We may want to avoid these variables in a model, particularly But we may still take a look at them and see what they contain.<\/p>\n <\/a><\/p>\n And now, we’ll do some high-level visualizations.<\/p>\n Specifically, we’ll create:<\/p>\n We’ll also do a little data manipulation along the way.<\/p>\n Next, we’re going to plot histograms of the numeric variables.<\/p>\n To do this, we’re going to use the Seaborn FacetGrid technique, which creates a small multiple chart (AKA, trellis chart).<\/p>\n But, we also need to use select_dtypes to retrieve the numeric variables with the Pandas melt function. The melt function will restructure the data from “tidy” format to long format. Said differently, Pandas melt will put the dataset into a format that allows us to create the small multiple chart.<\/p>\n Notice that we’re calling sns.histplot<\/a> to actually create the histograms.<\/p>\n OUT:<\/p>\n I’m not going to analyze these plots in detail, but a few things stand out.<\/p>\n First, the But several of the other “numeric” variables are definitely not normal.<\/p>\n In particular, both That said, we’ll quickly recode those variables to behave more like categoricals.<\/p>\n Here, we’re going to recode To do this, we’ll need to use multiple Pandas tools in combination. <\/p>\n Most importantly, we need to use the Pandas pd.Categorical function to create categorical variables. Notice that we’re specifying the categorical values, and using the We’re using the Pandas astype<\/a> to specify that we want to treat these variables as something other than a numeric.<\/p>\n And we’re using the Pandas assign method<\/a> to add the newly constructed variables to our dataframe. <\/p>\n Notice that the whole expression is enclosed in parenthesis. We’re calling multiple functions and methods, and using multiple “assign” operations in series. This is ultimately an example of Pandas method chaining<\/a>, which you really need to know if you want to do complex data manipulations.<\/p>\n I’ll leave you to inspect the new data with a few data inspection methods.<\/p>\n Now that we’ve done a little data cleaning, we’ll visualize the Here, we’re going to create a countplot with Seaborn of the OUT:<\/p>\n You’ll notice that more people died ( We’re going to analyze the Here, we’re going to create a new bar chart of the survived variable, but this time we’re going to break out the data by To do this, we’re going to use the relatively new Seaborn Objects interface (a new Seaborn visualization package) and create a countplot that plots In other words, we’re creating a dodged bar chart. <\/p>\n OUT:<\/p>\n Sorry guys … if you were a male on the Titanic, you were pretty likely to die.<\/p>\n Ladies. A bit more “lucky.” You would have been more likely to live than die.<\/p>\n Importantly, it looks like the Next, let’s visualize survived by This variable encodes the “passenger class,” and has values 1, 2, and 3. You can think of this like first class vs coach for modern plane flights.<\/p>\n Here, we’re going to visualize this as a bar chart, but I’m actually going to calculate the percent of people who survived.<\/p>\n I’m also going to facet this chart out by sex, in order to create a small multiple chart.<\/p>\n To do this, I’m using the Seaborn Objects interface with Note that I’m also faceting this plot with the Seaborn Objects Ok, here’s the code:<\/p>\n And here’s the output:<\/p>\n Fascinating.<\/p>\n If you were female, you were more likely to survive than a male in any passenger class. But females in 1st and 2nd class were more likely to survive than females in 3rd class.<\/p>\n If you were male, you were still pretty likely to die no matter what, but much more<\/em> likely to die if you were in 2nd and 3rd class. The men in 1st class actually had almost a 40% chance of survival, vs under 20% in 2nd and 3rd.<\/p>\n Clearly, passenger class, like sex, had a strong relationship with survival. <\/p>\n Again, this is useful to us when building a machine learning model.<\/p>\n I’m going to stop there for now.<\/p>\n We’ve identified a few variables that appear to be related to the target (both We also identified a few variables that we needed to recode.<\/p>\n Having said that, there’s probably more that we could do.<\/p>\n What else do you want to see for machine learning EDA?<\/p>\n Do you have any suggestions?<\/p>\n What did I miss?<\/p>\n I want to hear from you …<\/p>\n Give me your feedback by leaving a comment in the comments section below.<\/p>\n","protected":false},"excerpt":{"rendered":" In this tutorial, I’ll show you how to do some simple exploratory data analysis (EDA) for a machine learning project. In this tutorial, we’ll look at the Titanic dataset, which is commonly used in machine learning tutorials, and has previously been used as a Kaggle dataset. This tutorial will really only scratch the surface. There’s … Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":7223,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":"","om_disable_all_campaigns":false,"footnotes":""},"categories":[32,24,19,58],"tags":[],"publishpress_future_action":{"enabled":false,"date":"2024-05-04 07:58:04","action":"change-status","newStatus":"draft","terms":[],"taxonomy":"category"},"_links":{"self":[{"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/posts\/7217"}],"collection":[{"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/comments?post=7217"}],"version-history":[{"count":0,"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/posts\/7217\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/media\/7223"}],"wp:attachment":[{"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/media?parent=7217"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/categories?post=7217"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/tags?post=7217"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\n
Project Setup (prior to EDA)<\/h2>\n
IMPORT PACKAGES<\/h3>\n
\r\nimport pandas as pd\r\nimport seaborn as sns\r\nimport seaborn.objects as so\r\n<\/pre>\n
Load Dataset<\/h3>\n
titanic<\/code> dataset, which we’ll be analyzing.<\/p>\n\r\ntitanic = sns.load_dataset('titanic')\r\n<\/pre>\nBasic Data Inspection<\/h2>\n
\n
Print records<\/h4>\n
\r\n# PRINT RECORDS\r\ntitanic.head()\r\n<\/pre>\n
\r\n survived pclass sex age ... deck embark_town alive alone\r\n0 0 3 male 22.0 ... NaN Southampton no False\r\n1 1 1 female 38.0 ... C Cherbourg yes False\r\n2 1 3 female 26.0 ... NaN Southampton yes True\r\n3 1 1 female 35.0 ... C Southampton yes False\r\n4 0 3 male 35.0 ... NaN Southampton no True\r\n\r\n[5 rows x 15 columns]\r\n<\/pre>\n
List Data Types<\/h4>\n
\r\ntitanic.dtypes\r\n<\/pre>\n
\r\nsurvived int64\r\npclass int64\r\nsex object\r\nage float64\r\nsibsp int64\r\nparch int64\r\nfare float64\r\nembarked object\r\nclass category\r\nwho object\r\nadult_male bool\r\ndeck category\r\nembark_town object\r\nalive object\r\nalone bool\r\ndtype: object\r\n<\/pre>\n
Get Count of Missing Values by Column<\/h4>\n
\r\n# GET COUNT OF MISSING\r\n(titanic\r\n .isnull()\r\n .sum()\r\n )\r\n<\/pre>\n
\r\nsurvived 0\r\npclass 0\r\nsex 0\r\nage 177\r\nsibsp 0\r\nparch 0\r\nfare 0\r\nembarked 2\r\nclass 0\r\nwho 0\r\nadult_male 0\r\ndeck 688\r\nembark_town 2\r\nalive 0\r\nalone 0\r\ndtype: int64\r\n<\/pre>\n
age<\/code> and deck<\/code>) have a substantial number.<\/p>\ndeck<\/code>. <\/p>\nHigh-Level Visualizations<\/h2>\n
\n
Create Histograms of Numeric Variables<\/h3>\n
\r\n# CREATE HISTOGRAMS OF NUMERIC VARIABLES\r\nhist_grid = sns.FacetGrid(data = pd.melt(titanic.select_dtypes(include = np.number))\r\n ,col='variable'\r\n ,col_wrap = 3\r\n ,sharex=False\r\n )\r\nhist_grid.map(sns.histplot, 'value', bins=10)\r\n<\/pre>\n
![\"An](\"https:\/\/www.sharpsightlabs.com\/wp-content\/uploads\/2023\/05\/titanic_small-multiple-histogram.png\")
<\/p>\nage<\/code> variable seems not-quite-exactly normal, but it does roughly have a bell shape. This is somewhat more typical of what we might want for a numeric variable.<\/p>\nsurvived<\/code> and pclass<\/code> seem to be categorical variables. They aren’t distributed across many values, but instead have distinct peaks. <\/p>\nRecode Variables to Categoricals<\/h3>\n
survived<\/code> and pclass<\/code> to categorical variables.<\/p>\nordered<\/code> parameter to specify that these categories have a specific order.<\/p>\n\r\ntitanic_new = (titanic\r\n .assign(pclass = pd.Categorical(titanic.pclass.astype(str)\r\n ,categories = ['1', '2', '3']\r\n ,ordered = True))\r\n .assign(survived = pd.Categorical(titanic.survived.astype(str)\r\n ,categories = ['0','1']\r\n ,ordered = True))\r\n )\r\n<\/pre>\n
Visualize the Target Variable (Survived)<\/h3>\n
survived<\/code> variable.<\/p>\nsurvived<\/code> variable.<\/p>\n\r\nsns.countplot(data = titanic_new\r\n ,x = 'survived'\r\n )\r\n<\/pre>\n
![\"An](\"https:\/\/www.sharpsightlabs.com\/wp-content\/uploads\/2023\/05\/countplot_titanic-survived.png\")
<\/p>\n0<\/code>) than survived (1<\/code>). In fact, only about 40% survived.<\/p>\nsurvived<\/code> variable a little further with some additional visualizations.<\/p>\nVisualize Survived by Sex<\/h3>\n
sex<\/code>.<\/p>\nsex<\/code> on the x-axis, and creates a breakout of that data by creating additional bars for survived<\/code>, and “dodging” them to the side.<\/p>\n\r\n(so.Plot(data = titanic_new\r\n ,x = 'sex'\r\n ,color = 'survived'\r\n )\r\n .add(so.Bar(), so.Count(), so.Dodge())\r\n )\r\n<\/pre>\n
![\"An](\"https:\/\/www.sharpsightlabs.com\/wp-content\/uploads\/2023\/05\/dodged-countplot_count-by-sex-by-survived_titanic-data.png\")
<\/p>\nsex<\/code> variable is fairly predictive of survival. This would be important in a classification model<\/a>.<\/p>\nVisualize Survived by Pclass<\/h3>\n
pclass<\/code>.<\/p>\nso.Bar<\/code> to create a bar chart and so.Agg<\/code> to compute the mean. Because we’re also using the astype<\/code> method to treat the survived variable as a binary, 0\/1 integer, computing the mean of this variable will give us a percent.<\/p>\nfacet<\/code> method to create a small multiple chart.<\/p>\n\r\n(so.Plot(data = titanic_new.assign(survived = titanic_new.survived.astype(int))\r\n ,x = 'pclass'\r\n ,y = 'survived'\r\n )\r\n .add(so.Bar(), so.Agg('mean'))\r\n .facet(col = 'sex')\r\n )\r\n<\/pre>\n![\"An](\"https:\/\/www.sharpsightlabs.com\/wp-content\/uploads\/2023\/05\/titanic_small-multiple-bar-chart_pclass-by-survived-percentile.png\")
<\/p>\nThere’s probably more that we could do<\/h2>\n
sex<\/code> and plcass<\/code>).<\/p>\nTell me what you want to know<\/h3>\n