- \n
- A very quick review of Numpy<\/a><\/li>\n
- Introduction to Numpy standard deviation<\/a><\/li>\n
- The syntax of np.std<\/a><\/li>\n
- Numpy standard deviation examples<\/a><\/li>\n
- Numpy standard deviation FAQ<\/a><\/li>\n<\/ul>\n
Having said that, if you’re relatively new to Numpy, you might want to read the whole tutorial. <\/p>\n
<\/a><\/p>\n
A quick review of Numpy<\/h2>\n
Let’s just start off with a veeeery quick review of Numpy.<\/p>\n
What is Numpy?<\/p>\n
Numpy is a toolkit for working with numeric data<\/h3>\n
To put it simply, Numpy is a toolkit for working with numeric data.<\/p>\n
First, Numpy has a set of tools for creating a data structure called a Numpy array<\/a>.<\/p>\n
You can think of a Numpy array as a row-and-column grid of numbers. Numpy arrays can be 1-dimensional, 2-dimensional, or even n-dimensional.<\/p>\n
A 2D array looks something like this:<\/p>\n
![\"An](\"https:\/\/www.sharpsightlabs.com\/wp-content\/uploads\/2019\/07\/example-of-2D-numpy-array_0-to-7.png\")
<\/p>\nFor simplicity sake, in this tutorial, we’ll stick to 1 or 2-dimentional arrays.<\/p>\n
There are a variety of ways to create different types of arrays with different kinds of numbers. A few other tools for creating Numpy arrays include numpy arrange<\/a>, numpy zeros<\/a>, numpy ones<\/a>, numpy tile<\/a>, and other methods.<\/p>\n
Regardless of how you create your Numpy array, at a high level, they are simply arrays of numbers.<\/p>\n
Numpy provides tools for manipulating Numpy arrays<\/h3>\n
Numpy not only provides tools for creating<\/em> Numpy arrays, Numpy also provides tools for working with<\/em> Numpy arrays.<\/p>\n
Some of the most important of these Numpy tools are Numpy functions for performing calculations.<\/p>\n
There’s a whole set of Numpy functions for doing things like:<\/p>\n
- \n
- computing the sum of a Numpy array<\/a><\/li>\n
- calculating the maximum<\/a><\/li>\n
- calculating the exponential<\/a> of the numbers in an array<\/li>\n
- computing the value x to some power<\/a>, for every value in a Numpy array<\/li>\n<\/ul>\n
… and a variety of other computations.<\/p>\n
The Numpy standard deviation is essentially a lot like these other Numpy tools. It is just used to perform a computation (the standard deviation) of a group of numbers in a Numpy array.<\/p>\n
<\/a><\/p>\n
A quick introduction to Numpy standard deviation<\/h2>\n
At a very high level, standard deviation<\/a> is a measure of the spread of a dataset. In particular, it is a measure of how far the datapoints are from the mean of the data.<\/p>\n
Let’s briefly review the basic calculation.<\/p>\n
Standard deviation is calculated as the square root of the variance.<\/p>\n
So if we have a dataset with
![\"N\"](\"https:\/\/www.sharpsightlabs.com\/wp-content\/ql-cache\/quicklatex.com-a13b200485b8e78d6d1986a266856286_l3.png\" "\\"Rendered")
numbers, the variance will be:<\/p>\n(1) <\/span> <\/span>
![\"\begin{equation*}](\"https:\/\/www.sharpsightlabs.com\/wp-content\/ql-cache\/quicklatex.com-270e482c6119c896b37bcabba9c66763_l3.png\" "\\"Rendered")
<\/p>\n<\/a>
\nAnd the standard deviation will just be the square root of the variance:<\/p>\n(2) <\/span> <\/span>
![\"\begin{equation*}](\"https:\/\/www.sharpsightlabs.com\/wp-content\/ql-cache\/quicklatex.com-d4973b0b5d078460dc71f8c18bcdd5d7_l3.png\" "\\"Rendered")
<\/p>\nWhere:<\/p>\n
\n
![\"x_i\"](\"https:\/\/www.sharpsightlabs.com\/wp-content\/ql-cache\/quicklatex.com-3a7857b57d3ca30f32198442ddfde770_l3.png\" "\\"Rendered")
= the individual values in the dataset
\n = the number of values in the dataset
\n![\"\overline{x}\"](\"https:\/\/www.sharpsightlabs.com\/wp-content\/ql-cache\/quicklatex.com-2df7a312529b9c3fa73b952eeb00533e_l3.png\" "\\"Rendered")
= the mean of the values \n<\/p>\nMost of the time, calculating standard deviation by hand is a little challenging, because you need to compute the mean, the deviations of each datapoint from the mean, then the square of the deviations, etc. Frankly, it’s a little tedious.<\/p>\n
However, if you’re working in Python, you can use the Numpy standard deviation function to perform the calculation for you.<\/p>\n
A quick note if you’re new to statistics<\/h5>\n
Because this blog post is about using the numpy.std() function, I don’t want to get too deep into the weeds about how the calculation is performed by hand. This tutorial is really about how we use the function. So, if you need a quick review of what standard deviation is, you can watch this video<\/a>.<\/p>\n
Ok. Having quickly reviewed what standard deviation is, let’s look at the syntax for np.std.<\/p>\n
<\/a><\/p>\n
The syntax of np.std<\/h2>\n
The syntax of the Numpy standard deviation function is fairly simple.<\/p>\n
I’ll explain it in just a second, but first, I want to tell you one quick note about Numpy syntax.<\/p>\n
A quick note: the exact syntax depends on how you import Numpy<\/h4>\n
Typically, when we write Numpy syntax, we use the alias “np”. That’s the common convention among most data scientists.<\/p>\n
To set that alias, you need to import Numpy like this:<\/p>\n
\r\nimport numpy as np\r\n<\/pre>\n
If we import Numpy with this alias, we’ll can call the Numpy standard deviation function as
np.std()<\/code>.<\/p>\nOk, that being said, let’s take a closer look at the syntax.<\/p>\n
np.std syntax<\/h3>\n
At a high level, the syntax for np.std looks something like this:<\/p>\n
![\"An](\"https:\/\/www.sharpsightlabs.com\/wp-content\/uploads\/2020\/02\/numpy-standard-deviation.png\")
<\/p>\nAs I mentioned earlier, assuming that we’ve imported Numpy with the alias “
np<\/code>” we call the function with the syntaxnp.std()<\/code>.<\/p>\nThen inside of the parenthesis, there are several parameters that allow you to control exactly how the function works.<\/p>\n
Let’s take a look at those parameters.<\/p>\n
<\/a><\/p>\n
The parameters of numpy.std<\/h3>\n
There are a few important parameters you should know:<\/p>\n
- \n
a<\/code><\/li>\naxis<\/code><\/li>\ndtype<\/code><\/li>\nddof<\/code><\/li>\nkeepdims<\/code><\/li>\nout<\/code><\/li>\n<\/ul>\nLet’s take a look at each of them.<\/p>\n
a<\/code>(required)<\/h6>\nThe
a<\/code> parameter specifies the array<\/i> of values over which you want to calculate the standard deviation.<\/p>\nSaid differently, this enables you to specify the input array<\/i> to the function.<\/p>\n
Appropriate inputs include Numpy arrays, but also “array like” objects such as Python lists.<\/p>\n
Importantly, you must provide an input to this parameter<\/b>. An input is required.<\/p>\n
Having said that, the parameter itself can be implicit or explicit. What I mean by that, is that you can directly type the parameter
a=<\/code>, OR you can leave the parameter out of your syntax, and just type the name of your input array.<\/p>\nI’ll show you examples of this in example 1<\/a>.<\/p>\n
axis<\/code><\/h6>\nThe axis parameter enables you to specify an axis along which the standard deviation will be computed.<\/p>\n
To understand this, you really need to understand axes.<\/p>\n
Numpy arrays have axes.<\/p>\n
You can think of an “axis” like a direction along the array.<\/p>\n
In a 2-dimensional array, there will be 2 axes: axis-0 and axis-1.<\/p>\n
In a 2D array, axis-0 points downward along the rows, and axis-1 points horizontally along the columns.<\/p>\n
Visually, you can visualize the axes of a 2D array like this:<\/p>\n
![\"A](\"https:\/\/www.sharpsightlabs.com\/wp-content\/uploads\/2018\/12\/numpy-axis-example_v2.png\")
<\/p>\nUsing the
axis<\/code> parameter, you can compute the standard deviation in a particular direction along the array.<\/p>\nThis is best illustrated with examples, so I’ll show you an example in example 2<\/a>.<\/p>\n
(For a full explanation of Numpy array axes, see our tutorial called Numpy axes explained<\/a>.)<\/p>\n
dtype<\/code><\/h6>\n(optional)<\/i>
\nThedtype<\/code> parameter enables you to specify the data type that you want to use when np.std computes the standard deviation.<\/p>\nIf the data in the input array are integers, then this will default to
float64<\/code>.<\/p>\nOtherwise, if the data in the input array are floats, then this will default to the same float type as the input array.<\/p>\n
ddof<\/code><\/h6>\n(optional)<\/i>
\nThis enables you to specify the “degrees of freedom” for the calculation.<\/p>\nTo understand this, you need to look at equation 2<\/a> again.<\/p>\n
<\/span> <\/span>
![\"\[](\"https:\/\/www.sharpsightlabs.com\/wp-content\/ql-cache\/quicklatex.com-5f35a1284c37c06f8e14069828731442_l3.png\" "\\"Rendered")
<\/p>\nIn this equation, the first term is
![\"\frac{1}{N}\"](\"https:\/\/www.sharpsightlabs.com\/wp-content\/ql-cache\/quicklatex.com-a74a9d81b4b1195de9d45c21a8e2f6fa_l3.png\" "\\"Rendered")
.<\/p>\nRemember:
is the number of values in the array or dataset.<\/p>\nBut if we’re thinking in statistical terms, there’s actually a difference between computing a population standard deviation vs a sample standard deviation<\/a>.<\/p>\n
If we compute a population standard deviation, we use the term
in our equation.<\/p>\nHowever, when we compute the standard deviation on a sample<\/i> of data (a sample of
![\"n\"](\"https:\/\/www.sharpsightlabs.com\/wp-content\/ql-cache\/quicklatex.com-fbdaefcfe46d44fc6862c7ef0b98f40f_l3.png\" "\\"Rendered")
datapoints), then we need to modify the equation so that the leading term is ![\"\frac{1}{n - 1}\"](\"https:\/\/www.sharpsightlabs.com\/wp-content\/ql-cache\/quicklatex.com-957d32b3bb1f7fb59006376cb00abbf2_l3.png\" "\\"Rendered")
. In that case, the equation for a sample<\/i> standard deviation becomes:<\/p>\n(3) <\/span> <\/span>
![\"\begin{equation*}](\"https:\/\/www.sharpsightlabs.com\/wp-content\/ql-cache\/quicklatex.com-cffd3af051cf59b4e286e7cad61e724d_l3.png\" "\\"Rendered")
<\/p>\nHow do we implement this with np.std?<\/p>\n
We can do this with the
ddof<\/code> parameter, by settingddof = 1<\/code>.<\/p>\nAnd in fact, we can set the
ddof<\/code> term more generally. When we useddof<\/code>, it will modify the standard deviation calculation to become:<\/p>\n(4) <\/span> <\/span>
![\"\begin{equation*}](\"https:\/\/www.sharpsightlabs.com\/wp-content\/ql-cache\/quicklatex.com-d47e1d4b0864f703b65aa07feaa57279_l3.png\" "\\"Rendered")
<\/p>\nTo be honest, this is a little technical. If you need to learn more about this, you should watch this video at Khan academy about degrees of freedom, and population vs sample standard deviation<\/a>.<\/p>\n
out<\/code><\/h6>\n(optional)<\/i>
\nTheout<\/code> parameter enables you to specify an alternative array in which to put the output.<\/p>\nIt should have the same shape as the expected output.<\/p>\n
keepdims<\/code><\/h6>\n(optional)<\/i>
\nThekeepdims<\/code> parameter can be used to “keep” the original number of dimensions. When you setkeepdims = True<\/code>, the output will have the same number of dimensions as the input.<\/p>\nRemember: when we compute the standard deviation, the computation will “collapse” the number of dimensions.<\/p>\n
For example, if we input a 2-dimensional array as an input, then by default, np.std will output a number. A scalar value. <\/p>\n
But if we want the output to be a number within a 2D array<\/i> (i.e., an output array with the same dimensions as the input), then we can set
keepdims = True<\/code>.<\/p>\nTo be honest, some of these parameters are a little abstract, and I think they will make a lot more sense with examples.<\/p>\n
Let’s take a look at some examples.<\/p>\n
<\/a><\/p>\n
Examples of how to use Numpy standard deviation<\/h2>\n
Here, we’ll work through a few examples. We’ll start simple and then increase the complexity.<\/p>\n
Examples:<\/strong><\/p>\n
- \n
- Calculate standard deviation of a 1-dimensional array<\/a><\/li>\n
- Calculate the standard deviation of a 2-dimensional array<\/a><\/li>\n
- Use np.std to compute the standard deviations of the columns<\/a><\/li>\n
- Use np.std to compute the standard deviations of the rows<\/a><\/li>\n
- Change the degrees of freedom<\/a><\/li>\n
- Use the keepdims parameter in np.std<\/a><\/li>\n<\/ul>\n
Run this code first<\/h4>\n
Before you run any of the example code, you need to import Numpy.<\/p>\n
To do this, you can run the following code:<\/p>\n
\r\nimport numpy as np\r\n<\/pre>\n
This will import Numpy with the alias “
np<\/code>“.<\/p>\n<\/a><\/p>\n
EXAMPLE 1: Calculate standard deviation of a 1 dimensional array<\/h3>\n
Here, we’ll start simple. <\/p>\n
We’re going to calculate the standard deviation of 1-dimensional Numpy array.<\/p>\n
Create 1D array<\/h6>\n
First, we’ll just create our 1D array:<\/p>\n
\r\narray_1d = np.array([12, 14, 99, 72, 42, 55])\r\n<\/pre>\n
Calculate standard dev<\/h6>\n
Now, we’ll calculate the standard deviation of those numbers.<\/p>\n
\r\nnp.std(array_1d)\r\n<\/pre>\n
OUT:<\/p>\n
\r\n30.84369195367723\r\n<\/pre>\n
So what happened here?<\/p>\n
The np.std function just computed the standard deviation of the numbers
[12, 14, 99, 72, 42, 55]<\/code> using equation 2<\/a> that we saw earlier. Each number is one of the in that equation.<\/p>\nOne quick note<\/h6>\n
In the above example, we did not explicitly use the
a=<\/code> parameter. That is because np.std understands that when we provide an argument to the function like in the codenp.std(array_1d)<\/code>, the input should be passed to thea<\/code> parameter. <\/p>\nAlternatively, you can also explicitly use the
a=<\/code> parameter:<\/p>\n\r\nnp.std(a = array_1d)\r\n<\/pre>\n
OUT:<\/p>\n
\r\n30.84369195367723\r\n<\/pre>\n
<\/a><\/p>\n
EXAMPLE 2: Calculate the standard deviation of a 2-dimensional array<\/h3>\n
Ok. Now, let’s look at an example with a 2-dimensional array.<\/p>\n
Create 2-dimensional array<\/h6>\n
Here, we’re going to create a 2D array, using the np.random.randint function.<\/p>\n
\r\nnp.random.seed(22)\r\narray_2d = np.random.randint(20, size =(3, 4))\r\n<\/pre>\n
This array has 3 rows and 4 columns.<\/p>\n
Let’s print it out, so we can see it.<\/p>\n
\r\nprint(array_2d)\r\n<\/pre>\n
OUT:<\/p>\n
\r\n[[ 4 12 0 4]\r\n [ 6 11 8 4]\r\n [18 14 13 7]]\r\n<\/pre>\n
This is just a 2D array that contains 12 random integers between 0 and 20.<\/p>\n
Compute standard deviation with np.std<\/h6>\n
Okay, let’s compute the standard deviation.<\/p>\n
\r\nnp.std(array_2d)\r\n<\/pre>\n
OUT:<\/p>\n
\r\n5.007633062524539\r\n<\/pre>\n
Here, numpy.std() is just computing the standard deviation of all 12 integers. <\/p>\n
The standard deviation is
5.007633062524539<\/code>.<\/p>\n<\/a><\/p>\n
EXAMPLE 3: Compute the standard deviation of the columns<\/h3>\n
Now, we’re going to compute the standard deviation of the columns. <\/p>\n
To do this, we need to use the
axis<\/code> parameter. (You learned about theaxis<\/code> parameter in the section about the parameters of numpy.std<\/a>)<\/p>\nSpecifically, we need to set
axis = 0<\/code>.<\/p>\nWhy?<\/p>\n
As I mentioned in the explanation of the
axis<\/code> parameter earlier, Numpy arrays have axes.<\/p>\nIn a two dimensional array, axis-0 is the axis that points downwards.<\/p>\n
![\"A](\"https:\/\/www.sharpsightlabs.com\/wp-content\/uploads\/2018\/12\/numpy-axis0.png\")
<\/p>\nWhen we use numpy.std with
axis = 0<\/code>, that will compute the standard deviations downward in the axis-0 direction.<\/p>\nLet’s take a look at an example so you can see what I mean.<\/p>\n
Create 2-dimensional array<\/h5>\n
First, we’ll create a 2D array, using the np.random.randint function. <\/p>\n
(This is the same array that we created in example 2, so if you already created it, you shouldn’t need to create it again.)<\/p>\n
\r\nnp.random.seed(22)\r\narray_2d = np.random.randint(20, size =(3, 4))\r\n<\/pre>\n
Let’s print it out, so we can see it.<\/p>\n
\r\nprint(array_2d)\r\n<\/pre>\n
OUT:<\/p>\n
\r\n[[ 4 12 0 4]\r\n [ 6 11 8 4]\r\n [18 14 13 7]]\r\n<\/pre>\n
This is just a 2D array that contains integers between 0 and 20.<\/p>\n
Use np.std to compute standard deviation of the columns<\/h6>\n
Now, we’ll set
axis = 0<\/code> inside of np.std to compute the standard deviations of the columns.<\/p>\n\r\nnp.std(array_2d, axis = 0)\r\n<\/pre>\n
OUT:<\/p>\n
\r\narray([6.18241233, 1.24721913, 5.35412613, 1.41421356])\r\n<\/pre>\n
Explanation<\/h6>\n
What’s going on here?<\/p>\n
When we use np.std with
axis = 0<\/code>, Numpy will compute the standard deviation downward in the axis-0 direction. Remember, as I mentioned above, axis-0 points downward.<\/p>\nThis has the effect of computing the standard deviation of each column of the Numpy array.<\/p>\n
![\"An](\"https:\/\/www.sharpsightlabs.com\/wp-content\/uploads\/2020\/02\/np-std-column-standard-deviation.png\")
<\/p>\nNow, let’s do a similar example with the row standard deviations.<\/p>\n
<\/a><\/p>\n
EXAMPLE 4: Use np.std to compute the standard deviations of the rows<\/h3>\n
Now, we’re going to use np.std to compute the standard deviations horizontally along a 2D numpy array.<\/p>\n
Remember what I said earlier: numpy arrays have axes. The axes are like directions along the Numpy array. In a 2D array, axis-1 points horizontally, like this:<\/p>\n
![\"An](\"https:\/\/www.sharpsightlabs.com\/wp-content\/uploads\/2018\/12\/numpy-axis-1.png\")
<\/p>\nSo, if we want to compute the standard deviations horizontally, we can set
axis = 1<\/code>. This has the effect of computing the row standard deviations.<\/p>\nLet’s take a look.<\/p>\n
Create 2-dimensional array<\/h5>\n
To run this example, we’ll again need a 2D Numpy array, so we’ll create a 2D array using the np.random.randint function. <\/p>\n
(This is the same array that we created in example 2, so if you already created it, you shouldn’t need to create it again.)<\/p>\n
\r\nnp.random.seed(22)\r\narray_2d = np.random.randint(20, size =(3, 4))\r\n<\/pre>\n
Let’s print it out, so we can see it.<\/p>\n
\r\nprint(array_2d)\r\n<\/pre>\n
OUT:<\/p>\n
\r\n[[ 4 12 0 4]\r\n [ 6 11 8 4]\r\n [18 14 13 7]]\r\n<\/pre>\n
This is just a 2D array that contains integers between 0 and 20.<\/p>\n
Use np.std to compute standard deviation of the rows<\/h6>\n
Now, we’ll use np.std with
axis = 1<\/code> to compute the standard deviations of the rows.<\/p>\n\r\nnp.std(array_2d, axis = 1)\r\n<\/pre>\n
OUT:<\/p>\n
\r\narray([4.35889894, 2.58602011, 3.93700394])\r\n<\/pre>\n
Explanation<\/h6>\n
If you understood example 3<\/a>, this new example should make sense.<\/p>\n
When we use np.std and set
axis = 1<\/code>, Numpy will compute the standard deviations horizontally along axis-1.<\/p>\n![\"An](\"https:\/\/www.sharpsightlabs.com\/wp-content\/uploads\/2020\/02\/numpy-std-row-standard-deviation.png\")
<\/p>\nEffectively, when we use Numpy standard deviation with
axis = 1<\/code>, the function computes the standard deviation of the rows.<\/p>\n<\/a><\/p>\n
EXAMPLE 5: Change the degrees of freedom<\/h3>\n
Now, let’s change the degrees of freedom.<\/p>\n
Here in this example, we’re going to create a large array of numbers, take a sample from that array, and compute the standard deviation on that sample.<\/p>\n
First, let’s create our arrays.<\/p>\n
Create Numpy array<\/h5>\n
First, we’ll just create a normally distributed Numpy array with a mean of 0 and a standard deviation of 10.<\/p>\n
To do this, we’ll use the Numpy random normal function<\/a>. Note that we’re using the Numpy random seed function to set the seed for the random number generator. For more information on this, read our tutorial about np.random.seed<\/a>.<\/p>\n
\r\nnp.random.seed(22)\r\npopulation_array = np.random.normal(size = 100, loc = 0, scale = 10)\r\n<\/pre>\n
Ok. Now we have a Numpy array,
population_array<\/code>, that has 100 elements that have a mean of 0 and a standard deviation of 10.<\/p>\nCreate sample<\/h5>\n
Now, we’ll use Numpy random choice to take a random sample from the Numpy array<\/a>,
population_array<\/code>.<\/p>\n\r\nnp.random.seed(22)\r\nsample_array = np.random.choice(population_array, size = 10)\r\n<\/pre>\n
This new array,
sample_array<\/code>, is a random sample of 10 elements frompopulation_array<\/code>.<\/p>\nWe’ll use
sample_array<\/code> when we calculate our standard deviation using theddof<\/code> parameter.<\/p>\nCalculate the standard deviation of the sample<\/h5>\n
Now, we’ll calculate the standard deviation of the sample.<\/p>\n
Specifically, we’re going to use the Numpy standard deviation function with the
ddof<\/code> parameter set toddof = 1<\/code>.<\/p>\n\r\nnp.std(sample_array, ddof = 1)\r\n<\/pre>\n
OUT:<\/p>\n
\r\n10.703405562234051\r\n<\/pre>\n
Explanation<\/h5>\n
Here, we’ve calculated:<\/p>\n
<\/span> <\/span>
![\"\[](\"https:\/\/www.sharpsightlabs.com\/wp-content\/ql-cache\/quicklatex.com-fd9250acf5d428a8028e53af25668266_l3.png\" "\\"Rendered")
<\/p>\nAnd when we set
ddof = 1<\/code>, the equation evaluates to:<\/p>\n<\/span> <\/span>
![\"\[](\"https:\/\/www.sharpsightlabs.com\/wp-content\/ql-cache\/quicklatex.com-fdeb8e1484c4a9be76ac536990a18b26_l3.png\" "\\"Rendered")
<\/p>\nTo be clear, when you calculate the standard deviation of a sample<\/i>, you will set
ddof = 1<\/code>.<\/p>\nTo be honest, the details about why<\/i> are a little technical (and beyond the scope of this post), so for more information about calculating a sample standard deviation, I recommend that you watch this video<\/a>.<\/p>\n
Keep in mind, that for some other instances, you can set
ddof<\/code> to other values besides 1 or 0. If you don’t use theddof<\/code> parameter at all, it will default to 0.<\/p>\nNo matter what value you select, the Numpy standard deviation function will compute the standard deviation with the equation:<\/p>\n
<\/span> <\/span>
<\/p>\n - Calculate the standard deviation of a 2-dimensional array<\/a><\/li>\n
- calculating the maximum<\/a><\/li>\n
- Introduction to Numpy standard deviation<\/a><\/li>\n