![\"An](\"https:\/\/www.sharpsightlabs.com\/wp-content\/uploads\/2020\/06\/emmanuel-question_26-MAY-2020.png\")
<\/p>\nOne of the common questions that I get from my readers is about “big data.”<\/p>\n
This is particularly true of beginners, who frequently read or hear the term “big data,” and think that it’s extremely important, and that being able to work with big data is necessary for a job in the data industry.<\/p>\n
For example, a couple of weeks ago, reader Emmanuel asked me this question:<\/p>\n
<\/p>\n
Now before I move on, I want to make it clear that by highlighting this, I’m not trying to make fun of Emmanuel. It’s a legitimate question from a sincere beginner, looking for guidance.<\/p>\n
Having said that, I’m going to treat this question with same fair but tough-minded manner that I treat most things here at the Sharp Sight blog.<\/p>\n
If you’re a relative beginner, then 95% of the time, you should forget about big data.<\/p>\n
There are a few reasons for this, but the major reasons are:<\/p>\n
Let’s talk about each of these.<\/p>\n
One of the problems with “big data” is that there’s not really any such thing as big data.<\/p>\n
At least, there’s not any agreed upon definition of big data. <\/p>\n
The best definition that I’ve heard is that big data is “data that won’t fit in memory.” And considering that you can always buy more memory (and that it keeps getting cheaper every year), what’s considered “big” today might be “average” tomorrow.<\/p>\n
What that means is that “big data” is sort of a moving target. What’s considered “big” actually changes over time (and pretty dramatically, I might add).<\/p>\n
Here’s an example.<\/p>\n
I started in the data industry over 15 years ago. For a couple of years, around 2008, I worked at a small, boutique consulting firm in Chicago.<\/p>\n
Technically, we built databases. We would go into a business (like an insurance company, bank, etc), gather all of the data from their old systems, build new databases, and put the old data into the new databases. This process is commonly called extract, transform, and load (ETL for short).<\/p>\n
Technically, that’s what we did. We built databases.<\/p>\n
But the company marketed itself in a very particular way.<\/p>\n
As a company, we were unique amongst our competitors, because we focussed exclusively on “big data” problems. <\/p>\n
Our sales teams would win business by talking to C-suite executives at Fortune 500 countries, explaining that we were experts in “big data.” We’d frequently win business when competing against larger and better-known competitors, like Deloitte. <\/p>\n
We were big data experts, long before big data was really even a thing.<\/p>\n
And we were a successful company largely because we knew how to use this term “big data.”<\/p>\n
Now, here’s the funny part.<\/p>\n
Guess how large “big data” was back then.<\/p>\n
Guess.<\/p>\n
Go ahead … think about it.<\/p>\n
I’ll wait.<\/p>\n
How big was “big data” back in 2008?<\/p>\n
How big was this “big data” that enabled us to win multi-million dollar contracts against major competitors?<\/p>\n
1 Terabyte.<\/p>\n
You read that right.<\/p>\n
The way that our company pitched our services was that we were “experts in ‘big data’ problems,” and in order for us to consider working with you, you needed to have 1 terabyte or more of data.<\/p>\n
For context, right now in 2020, you can buy a 1 terabyte external hard drive on Amazon for $49.99.<\/p>\n
![\"An](\"https:\/\/www.sharpsightlabs.com\/wp-content\/uploads\/2020\/06\/1-TB-on-Amazon-2020.png\")
<\/p>\n
To be fair, building a full 1 TB database is different than just buying raw storage, but it gives you a sense of how much things have changed.<\/p>\n
Ultimately, “big data” is a moving target.<\/p>\n
What’s big today will likely be average in only a few years.<\/p>\n
Another problem is that big data will simply be more confusing.<\/p>\n
When you have very large datasets, it actually becomes harder to understand and work with the data.<\/p>\n
Let’s say that a dataset has 1,000,000 rows and 200 columns. You’ll have a very difficult time just understanding the different columns, the different categories, the average properties of the different variables, etc.<\/p>\n
(And to be clear, 1 million rows and 200 columns is not even that large.)<\/p>\n
When the data gets too large, exploring and understanding your data becomes harder.<\/p>\n
So if you’re a beginner, this adds a whole new level of complexity on to the problem of learning and mastering data science.<\/p>\n
It will be much easier to learn and master data science skills when you have relatively small, clean, easy-to-understand data.<\/p>\n
Finally, a lot of work that’s done by data science and data analytics professionals is done with “average” data. <\/p>\n
This, of course, varies from job to job and company to company. <\/p>\n
But the point stands: a lot of data science and data analytics work deals with a few thousand to a few million rows of data. It’s somewhat more rare to find environments working with massive volumes of data that require special tools.<\/p>\n
This is particularly true for many jobs at junior levels. As a data scientist or data analytics professional, it’s more likely that you’ll be put on projects with smaller datasets anyway. It’s unlikely that in your first job, you’ll be put on projects that involve massive amounts of data (although there are exceptions).<\/p>\n
My point here is that there are a lot of data-related jobs that work with non-big amounts of data … data that fits in a csv file or data that you can pull with a query to a database and pull back to your computer.<\/p>\n
In the end, my advice is that you should learn small data<\/em> before you try to learn big data.<\/p>\n Small data is simply easier to work with, especially as a beginner.<\/p>\n For example, right now, I’m playing around with some machine learning techniques. <\/p>\n One thing that is abundantly clear is how much harder it is to work with a large data set.<\/p>\n Now, the data that I’m working with isn’t that<\/em> large, but it’s large enough that my laptop takes a few minutes to execute some operations that I’m trying to run.<\/p>\n Need to load the dataset? It takes a minute or two.<\/p>\n Need to fit the model? It takes a minute or two.<\/p>\n Cross validation<\/a>? Again…. another minute or two.<\/p>\n It doesn’t sound that bad, but it all adds up. <\/p>\n If you’re like most people, you only have 20 or 30 minutes a day to learn a new skill … only 20 or 30 minutes a day to practice.<\/p>\n Don’t waste that time on processing time.<\/p>\n Use a small dataset that only takes a few seconds to load, fit, and validate.<\/p>\n By working with smaller datasets, you simply get more practice in. You get more repetitions per practice session. <\/p>\n Over time, that increase in repetitions will add up and you’ll make much more rapid progress.<\/p>\n If you really want to master data science fast, start simple and increase the complexity.<\/p>\n Start with simple, small, clean datasets. <\/p>\n Start with data that is easy to understand and work with, and then practice modifying, reshaping, and visualizing that data.<\/p>\n Also, focus on simple operations:<\/p>\n Before you move on to anything advanced, you should be able to do all of these things with your eyes closed. By working with small, clean, simple datasets, you’ll be able to focus on what matters: learning the syntax, and learning the principles for applying the syntax.<\/p>\n Once you know the essentials backwards and forwards, then you can increase complexity.<\/p>\n The best way to increase the complexity is to start working with messy, real-world data. This will be harder to do, because you’ll need to apply the basic techniques (filtering, subsetting, transposing) in new and more complex ways. You’ll need to clean your data and fix things that need fixing. <\/p>\n But if you have the syntax mastered, you’ve at least passed the first bottleneck. You can focus on learning how to apply the syntax to more complicated problems, rather than looking up syntax every 3 minutes.<\/p>\n Finally, once you’re able to work with and clean up messy, real-world data, then<\/em> you can move on to big data.<\/p>\n In many cases, that will mean learning new toolkits and new syntax.<\/p>\n But many of the principles<\/em> that you learned while working with smaller, simpler datasets will still be valid. <\/p>\n Ultimately, even if your goal is to master “big data” (whatever that will mean in 2020 and beyond), your first goal is to master small data.<\/p>\n I get it. It’s not cool. It’s not sexy. You don’t score any points with your friends by saying that you’re studying “small data.”<\/p>\n But honestly: who cares what those people think.<\/p>\n Your goal is to be come a top performer in the shortest possible time. <\/p>\n You can’t think like everyone else.<\/p>\n You need to be disciplined, methodical, and deliberate.<\/p>\n So sit down and focus relentlessly on mastering small data first. Use “small data” to master the fundamental techniques. <\/p>\n As your skill grows, the data can grow too …<\/p>\n … and you’ll be working with big data in no time.<\/p>\n","protected":false},"excerpt":{"rendered":" One of the common questions that I get from my readers is about “big data.” This is particularly true of beginners, who frequently read or hear the term “big data,” and think that it’s extremely important, and that being able to work with big data is necessary for a job in the data industry. For … Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":"","om_disable_all_campaigns":false,"footnotes":""},"categories":[31,55,81],"tags":[78],"publishpress_future_action":{"enabled":false,"date":"2024-05-05 03:42:26","action":"change-status","newStatus":"draft","terms":[],"taxonomy":"category"},"_links":{"self":[{"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/posts\/5337"}],"collection":[{"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/comments?post=5337"}],"version-history":[{"count":0,"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/posts\/5337\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/media?parent=5337"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/categories?post=5337"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.sharpsightlabs.com\/wp-json\/wp\/v2\/tags?post=5337"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Start simple and then increase complexity<\/h3>\n
\n
\nI mean that almost literally. You should know the syntax backwards and forwards. (Once you do, you’ll actually have the skills to be a productive member of a data team).<\/p>\nOnce you know the basics, increase complexity<\/h4>\n
Forget about “big data” … master the fundamentals<\/h2>\n