مطالعهٔ pandas Docs
ساخت جدول، خواندن و نوشتن فایل، محاسبهٔ آمار خلاصه
باز کردن مرجع کامل دانلود آرشیو مستندات ناشر
گزیدهای از مستندات pandas Docs · نسخهٔ stable · BSD-3-Clause
ساخت جدول
What kind of data does pandas handle?#
I want to start using pandas
In [1]: import pandas as pd
To load the pandas package and start working with it, import the package. The community agreed alias for pandas is
pd, so loading pandas aspdis assumed standard practice for all of the pandas documentation.
pandas data table representation#
I want to store passenger data of the Titanic. For a number of passengers, I know the name (characters), age (integers) and sex (male/female) data.
In [2]: df = pd.DataFrame( ...: { ...: "Name": [ ...: "Braund, Mr. Owen Harris", ...: "Allen, Mr. William Henry", ...: "Bonnell, Miss Elizabeth", ...: ], ...: "Age": [22, 35, 58], ...: "Sex": ["male", "male", "female"], ...: } ...: ) ...: In [3]: df Out[3]: Name Age Sex 0 Braund, Mr. Owen Harris 22 male 1 Allen, Mr. William Henry 35 male 2 Bonnell, Miss Elizabeth 58 femaleTo manually store data in a table, create a
DataFrame. When using a Python dictionary of lists, the dictionary keys will be used as column headers and the values in each list as columns of theDataFrame.
A DataFrame is a 2-dimensional data structure that can store data of
different types (including characters, integers, floating point values,
categorical data and more) in columns. It is similar to a spreadsheet, a
SQL table or the data.frame in R.
The table has 3 columns, each of them with a column label. The column labels are respectively
Name,AgeandSex.The column
Nameconsists of textual data with each value a string, the columnAgeare numbers and the columnSexis textual data.The index labels each row. By default, this is a sequence of integers starting at 0.
In spreadsheet software, the table representation of our data would look very similar:
Each column in a DataFrame is a Series#
I’m just interested in working with the data in the column
AgeIn [4]: df["Age"] Out[4]: 0 22 1 35 2 58 Name: Age, dtype: int64
When selecting a single column of a pandas
DataFrame, the result is a pandasSeries. To select the column, use the column label in between square brackets[].
Note
If you are familiar with Python dictionaries, the selection of a single column is very similar to the selection of dictionary values based on the key.
You can create a Series from scratch as well:
In [5]: ages = pd.Series([22, 35, 58], name="Age") In [6]: ages Out[6]: 0 22 1 35 2 58 Name: Age, dtype: int64
A pandas Series has no column labels, as it is just a single column
of a DataFrame. A Series does have row labels.
Do something with a DataFrame or Series#
I want to know the maximum Age of the passengers
We can do this on the
DataFrameby selecting theAgecolumn and applyingmax():In [7]: df["Age"].max() Out[7]: 58
Or to the
Series:In [8]: ages.max() Out[8]: 58
As illustrated by the max() method, you can do things with a
DataFrame or Series. pandas provides a lot of functionalities,
each of them a method you can apply to a DataFrame or Series.
As methods are functions, do not forget to use parentheses ().
I’m interested in some basic statistics of the numerical data of my data table
In [9]: df.describe() Out[9]: Age count 3.000000 mean 38.333333 std 18.230012 min 22.000000 25% 28.500000 50% 35.000000 75% 46.500000 max 58.000000The
describe()method provides a quick overview of the numerical data in aDataFrame. As theNameandSexcolumns are textual data, these are by default not taken into account by thedescribe()method.
Many pandas operations return a DataFrame or a Series. The
describe() method is an example of a pandas operation returning a
pandas Series or a pandas DataFrame.
Check more options on describe in the user guide section about aggregations with describe
Note
This is just a starting point. Similar to spreadsheet software, pandas represents data as a table with columns and rows. Apart from the representation, the data manipulations and calculations you would do in spreadsheet software are also supported by pandas. Continue reading the next tutorials to get started!
REMEMBER
Import the package, aka
import pandas as pdA table of data is stored as a pandas
DataFrameEach column in a
DataFrameis aSeriesYou can do things by applying a method on a
DataFrameorSeries
A more extended explanation of DataFrame and Series is provided in the introduction to data structures page.
خواندن و نوشتن فایل
In [1]: import pandas as pdData used for this tutorial:
-
Titanic data
This tutorial uses the Titanic data set, stored as CSV. The data consists of the following data columns:
PassengerId: Id of every passenger.
Survived: Indication whether passenger survived.
0for no and1for yes.Pclass: One out of the 3 ticket classes: Class
1, Class2and Class3.Name: Name of passenger.
Sex: Gender of passenger.
Age: Age of passenger in years.
SibSp: Number of siblings or spouses aboard.
Parch: Number of parents or children aboard.
Ticket: Ticket number of passenger.
Fare: Indicating the fare.
Cabin: Cabin number of passenger.
Embarked: Port of embarkation.
How do I read and write tabular data?#
I want to analyze the Titanic passenger data, available as a CSV file.
In [2]: titanic = pd.read_csv("data/titanic.csv")pandas provides the
read_csv()function to read data stored as a csv file into a pandasDataFrame. pandas supports many different file formats or data sources out of the box (csv, excel, sql, json, parquet, …), each of them with the prefixread_*.
Make sure to always have a check on the data after reading in the
data. When displaying a DataFrame, the first and last 5 rows will be
shown by default:
In [3]: titanic
Out[3]:
PassengerId Survived Pclass ... Fare Cabin Embarked
0 1 0 3 ... 7.2500 NaN S
1 2 1 1 ... 71.2833 C85 C
2 3 1 3 ... 7.9250 NaN S
3 4 1 1 ... 53.1000 C123 S
4 5 0 3 ... 8.0500 NaN S
.. ... ... ... ... ... ... ...
886 887 0 2 ... 13.0000 NaN S
887 888 1 1 ... 30.0000 B42 S
888 889 0 3 ... 23.4500 NaN S
889 890 1 1 ... 30.0000 C148 C
890 891 0 3 ... 7.7500 NaN Q
[891 rows x 12 columns]
I want to see the first 8 rows of a pandas DataFrame.
In [4]: titanic.head(8) Out[4]: PassengerId Survived Pclass ... Fare Cabin Embarked 0 1 0 3 ... 7.2500 NaN S 1 2 1 1 ... 71.2833 C85 C 2 3 1 3 ... 7.9250 NaN S 3 4 1 1 ... 53.1000 C123 S 4 5 0 3 ... 8.0500 NaN S 5 6 0 3 ... 8.4583 NaN Q 6 7 0 1 ... 51.8625 E46 S 7 8 0 3 ... 21.0750 NaN S [8 rows x 12 columns]
To see the first N rows of a
DataFrame, use thehead()method with the required number of rows (in this case 8) as argument.
Note
Interested in the last N rows instead? pandas also provides a
tail() method. For example, titanic.tail(10) will return the last
10 rows of the DataFrame.
A check on how pandas interpreted each of the column data types can be
done by requesting the pandas dtypes attribute:
In [5]: titanic.dtypes Out[5]: PassengerId int64 Survived int64 Pclass int64 Name str Sex str Age float64 SibSp int64 Parch int64 Ticket str Fare float64 Cabin str Embarked str dtype: object
For each of the columns, the used data type is enlisted. The data types
in this DataFrame are integers (int64), floats (float64) and
strings (object).
Note
When asking for the dtypes, no parentheses () are used!
dtypes is an attribute of a DataFrame and Series. Attributes
of a DataFrame or Series do not need (). Attributes
represent a characteristic of a DataFrame/Series, whereas
methods (which require parentheses ()) do something with the
DataFrame/Series as introduced in the first tutorial.
My colleague requested the Titanic data as a spreadsheet.
Note
If you want to use
to_excel()andread_excel(), you need to install an Excel reader as outlined in the Excel files section of the installation documentation.In [6]: titanic.to_excel("titanic.xlsx", sheet_name="passengers", index=False)Whereas
read_*functions are used to read data to pandas, theto_*methods are used to store data. Theto_excel()method stores the data as an excel file. In the example here, thesheet_nameis named passengers instead of the default Sheet1. By settingindex=Falsethe row index labels are not saved in the spreadsheet.
The equivalent read function read_excel() will reload the data to a
DataFrame:
In [7]: titanic = pd.read_excel("titanic.xlsx", sheet_name="passengers", engine="openpyxl")
In [8]: titanic.head() Out[8]: PassengerId Survived Pclass ... Fare Cabin Embarked 0 1 0 3 ... 7.2500 NaN S 1 2 1 1 ... 71.2833 C85 C 2 3 1 3 ... 7.9250 NaN S 3 4 1 1 ... 53.1000 C123 S 4 5 0 3 ... 8.0500 NaN S [5 rows x 12 columns]
I’m interested in a technical summary of a
DataFrameIn [9]: titanic.info() <class 'pandas.DataFrame'> RangeIndex: 891 entries, 0 to 890 Data columns (total 12 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 PassengerId 891 non-null int64 1 Survived 891 non-null int64 2 Pclass 891 non-null int64 3 Name 891 non-null str 4 Sex 891 non-null str 5 Age 714 non-null float64 6 SibSp 891 non-null int64 7 Parch 891 non-null int64 8 Ticket 891 non-null str 9 Fare 891 non-null float64 10 Cabin 204 non-null str 11 Embarked 889 non-null str dtypes: float64(2), int64(5), str(5) memory usage: 118.7 KB
The method
info()provides technical information about aDataFrame, so let’s explain the output in more detail:It is indeed a
DataFrame.There are 891 entries, i.e. 891 rows.
Each row has a row label (aka the
index) with values ranging from 0 to 890.The table has 12 columns. Most columns have a value for each of the rows (all 891 values are
non-null). Some columns do have missing values and less than 891non-nullvalues.The columns
Name,Sex,Ticket,CabinandEmbarkedconsist of textual data (strings, akaobject). The other columns are numerical data, some of them are whole numbers (integer) and others are real numbers (float).The kind of data (characters, integers, …) in the different columns are summarized by listing the
dtypes.The approximate amount of RAM used to hold the DataFrame is provided as well.
REMEMBER
Getting data in to pandas from many different file formats or data sources is supported by
read_*functions.Exporting data out of pandas is provided by different
to_*methods.The
head/tail/infomethods and thedtypesattribute are convenient for a first check.
For a complete overview of the input and output possibilities from and to pandas, see the user guide section about reader and writer functions.
محاسبهٔ آمار خلاصه
In [1]: import pandas as pdData used for this tutorial:
-
Titanic data
This tutorial uses the Titanic data set, stored as CSV. The data consists of the following data columns:
PassengerId: Id of every passenger.
Survived: Indication whether passenger survived.
0for no and1for yes.Pclass: One out of the 3 ticket classes: Class
1, Class2and Class3.Name: Name of passenger.
Sex: Gender of passenger.
Age: Age of passenger in years.
SibSp: Number of siblings or spouses aboard.
Parch: Number of parents or children aboard.
Ticket: Ticket number of passenger.
Fare: Indicating the fare.
Cabin: Cabin number of passenger.
Embarked: Port of embarkation.
In [2]: titanic = pd.read_csv("data/titanic.csv") In [3]: titanic.head() Out[3]: PassengerId Survived Pclass ... Fare Cabin Embarked 0 1 0 3 ... 7.2500 NaN S 1 2 1 1 ... 71.2833 C85 C 2 3 1 3 ... 7.9250 NaN S 3 4 1 1 ... 53.1000 C123 S 4 5 0 3 ... 8.0500 NaN S [5 rows x 12 columns]
How to calculate summary statistics#
Aggregating statistics#
What is the average age of the Titanic passengers?
In [4]: titanic["Age"].mean() Out[4]: np.float64(29.69911764705882)
Different statistics are available and can be applied to columns with numerical data. Operations in general exclude missing data and operate across rows by default.
What is the median age and ticket fare price of the Titanic passengers?
In [5]: titanic[["Age", "Fare"]].median() Out[5]: Age 28.0000 Fare 14.4542 dtype: float64
The statistic applied to multiple columns of a
DataFrame(the selection of two columns returns aDataFrame, see the subset data tutorial) is calculated for each numeric column.
The aggregating statistic can be calculated for multiple columns at the
same time. Remember the describe function from the first tutorial?
In [6]: titanic[["Age", "Fare"]].describe()
Out[6]:
Age Fare
count 714.000000 891.000000
mean 29.699118 32.204208
std 14.526497 49.693429
min 0.420000 0.000000
25% 20.125000 7.910400
50% 28.000000 14.454200
75% 38.000000 31.000000
max 80.000000 512.329200
Instead of the predefined statistics, specific combinations of
aggregating statistics for given columns can be defined using the
DataFrame.agg() method:
In [7]: titanic.agg(
...: {
...: "Age": ["min", "max", "median", "skew"],
...: "Fare": ["min", "max", "median", "mean"],
...: }
...: )
...:
Out[7]:
Age Fare
min 0.420000 0.000000
max 80.000000 512.329200
median 28.000000 14.454200
skew 0.389108 NaN
mean NaN 32.204208
To user guideDetails about descriptive statistics are provided in the user guide section on descriptive statistics.
Aggregating statistics grouped by category#
What is the average age for male versus female Titanic passengers?
In [8]: titanic[["Sex", "Age"]].groupby("Sex").mean() Out[8]: Age Sex female 27.915709 male 30.726645As our interest is the average age for each gender, a subselection on these two columns is made first:
titanic[["Sex", "Age"]]. Next, thegroupby()method is applied on theSexcolumn to make a group per category. The average age for each gender is calculated and returned.
Calculating a given statistic (e.g. mean age) for each category in
a column (e.g. male/female in the Sex column) is a common pattern.
The groupby method is used to support this type of operations. This
fits in the more general split-apply-combine pattern:
Split the data into groups
Apply a function to each group independently
Combine the results into a data structure
The apply and combine steps are typically done together in pandas.
In the previous example, we explicitly selected the 2 columns first. If
not, the mean method is applied to each column containing numerical
columns by passing numeric_only=True:
In [9]: titanic.groupby("Sex").mean(numeric_only=True)
Out[9]:
PassengerId Survived Pclass ... SibSp Parch Fare
Sex ...
female 431.028662 0.742038 2.159236 ... 0.694268 0.649682 44.479818
male 454.147314 0.188908 2.389948 ... 0.429809 0.235702 25.523893
[2 rows x 7 columns]
It does not make much sense to get the average value of the Pclass.
If we are only interested in the average age for each gender, the
selection of columns (square brackets [] as usual) is supported
on the grouped data as well:
In [10]: titanic.groupby("Sex")["Age"].mean()
Out[10]:
Sex
female 27.915709
male 30.726645
Name: Age, dtype: float64
Note
The Pclass column contains numerical data but actually
represents 3 categories (or factors) with respectively the labels ‘1’,
‘2’ and ‘3’. Calculating statistics on these does not make much sense.
Therefore, pandas provides a Categorical data type to handle this
type of data. More information is provided in the user guide
Categorical data section.
What is the mean ticket fare price for each of the sex and cabin class combinations?
In [11]: titanic.groupby(["Sex", "Pclass"])["Fare"].mean() Out[11]: Sex Pclass female 1 106.125798 2 21.970121 3 16.118810 male 1 67.226127 2 19.741782 3 12.661633 Name: Fare, dtype: float64Grouping can be done by multiple columns at the same time. Provide the column names as a list to the
groupby()method.
A full description on the split-apply-combine approach is provided in the user guide section on groupby operations.
Count number of records by category#
What is the number of passengers in each of the cabin classes?
In [12]: titanic["Pclass"].value_counts() Out[12]: Pclass 3 491 1 216 2 184 Name: count, dtype: int64
The
value_counts()method counts the number of records for each category in a column.
The function is a shortcut, it is actually a groupby operation in combination with counting the number of records within each group:
In [13]: titanic.groupby("Pclass")["Pclass"].count()
Out[13]:
Pclass
1 216
2 184
3 491
Name: Pclass, dtype: int64
Note
Both size and count can be used in combination with
groupby. Whereas size includes NaN values and just provides
the number of rows (size of the table), count excludes the missing
values. In the value_counts method, use the dropna argument to
include or exclude the NaN values.
The user guide has a dedicated section on value_counts , see the page on discretization.
REMEMBER
Aggregation statistics can be calculated on entire columns or rows.
groupbyprovides the power of the split-apply-combine pattern.value_countsis a convenient shortcut to count the number of entries in each category of a variable.
A full description on the split-apply-combine approach is provided in the user guide pages about groupby operations.
On this page