مطالعهٔ pandas Docs

ساخت جدول، خواندن و نوشتن فایل، محاسبهٔ آمار خلاصه

باز کردن مرجع کامل دانلود آرشیو مستندات ناشر

گزیده‌ای از مستندات pandas Docs · نسخهٔ stable · BSD-3-Clause

ساخت جدول

What kind of data does pandas handle?#

  • I want to start using pandas

    In [1]: import pandas as pd
    

    To load the pandas package and start working with it, import the package. The community agreed alias for pandas is pd, so loading pandas as pd is assumed standard practice for all of the pandas documentation.

pandas data table representation#

  • I want to store passenger data of the Titanic. For a number of passengers, I know the name (characters), age (integers) and sex (male/female) data.

    In [2]: df = pd.DataFrame(
       ...:     {
       ...:         "Name": [
       ...:             "Braund, Mr. Owen Harris",
       ...:             "Allen, Mr. William Henry",
       ...:             "Bonnell, Miss Elizabeth",
       ...:         ],
       ...:         "Age": [22, 35, 58],
       ...:         "Sex": ["male", "male", "female"],
       ...:     }
       ...: )
       ...: 
    
    In [3]: df
    Out[3]: 
                           Name  Age     Sex
    0   Braund, Mr. Owen Harris   22    male
    1  Allen, Mr. William Henry   35    male
    2   Bonnell, Miss Elizabeth   58  female
    

    To manually store data in a table, create a DataFrame. When using a Python dictionary of lists, the dictionary keys will be used as column headers and the values in each list as columns of the DataFrame.

A DataFrame is a 2-dimensional data structure that can store data of different types (including characters, integers, floating point values, categorical data and more) in columns. It is similar to a spreadsheet, a SQL table or the data.frame in R.

  • The table has 3 columns, each of them with a column label. The column labels are respectively Name, Age and Sex.

  • The column Name consists of textual data with each value a string, the column Age are numbers and the column Sex is textual data.

  • The index labels each row. By default, this is a sequence of integers starting at 0.

In spreadsheet software, the table representation of our data would look very similar:

Each column in a DataFrame is a Series#

  • I’m just interested in working with the data in the column Age

    In [4]: df["Age"]
    Out[4]: 
    0    22
    1    35
    2    58
    Name: Age, dtype: int64
    

    When selecting a single column of a pandas DataFrame, the result is a pandas Series. To select the column, use the column label in between square brackets [].

Note

If you are familiar with Python dictionaries, the selection of a single column is very similar to the selection of dictionary values based on the key.

You can create a Series from scratch as well:

In [5]: ages = pd.Series([22, 35, 58], name="Age")

In [6]: ages
Out[6]: 
0    22
1    35
2    58
Name: Age, dtype: int64

A pandas Series has no column labels, as it is just a single column of a DataFrame. A Series does have row labels.

Do something with a DataFrame or Series#

  • I want to know the maximum Age of the passengers

    We can do this on the DataFrame by selecting the Age column and applying max():

    In [7]: df["Age"].max()
    Out[7]: 58
    

    Or to the Series:

    In [8]: ages.max()
    Out[8]: 58
    

As illustrated by the max() method, you can do things with a DataFrame or Series. pandas provides a lot of functionalities, each of them a method you can apply to a DataFrame or Series. As methods are functions, do not forget to use parentheses ().

  • I’m interested in some basic statistics of the numerical data of my data table

    In [9]: df.describe()
    Out[9]: 
                 Age
    count   3.000000
    mean   38.333333
    std    18.230012
    min    22.000000
    25%    28.500000
    50%    35.000000
    75%    46.500000
    max    58.000000
    

    The describe() method provides a quick overview of the numerical data in a DataFrame. As the Name and Sex columns are textual data, these are by default not taken into account by the describe() method.

Many pandas operations return a DataFrame or a Series. The describe() method is an example of a pandas operation returning a pandas Series or a pandas DataFrame.

To user guide

Check more options on describe in the user guide section about aggregations with describe

Note

This is just a starting point. Similar to spreadsheet software, pandas represents data as a table with columns and rows. Apart from the representation, the data manipulations and calculations you would do in spreadsheet software are also supported by pandas. Continue reading the next tutorials to get started!

REMEMBER

  • Import the package, aka import pandas as pd

  • A table of data is stored as a pandas DataFrame

  • Each column in a DataFrame is a Series

  • You can do things by applying a method on a DataFrame or Series

To user guide

A more extended explanation of DataFrame and Series is provided in the introduction to data structures page.

On this page

خواندن و نوشتن فایل

In [1]: import pandas as pd
Data used for this tutorial:
  • Titanic data

    This tutorial uses the Titanic data set, stored as CSV. The data consists of the following data columns:

    • PassengerId: Id of every passenger.

    • Survived: Indication whether passenger survived. 0 for no and 1 for yes.

    • Pclass: One out of the 3 ticket classes: Class 1, Class 2 and Class 3.

    • Name: Name of passenger.

    • Sex: Gender of passenger.

    • Age: Age of passenger in years.

    • SibSp: Number of siblings or spouses aboard.

    • Parch: Number of parents or children aboard.

    • Ticket: Ticket number of passenger.

    • Fare: Indicating the fare.

    • Cabin: Cabin number of passenger.

    • Embarked: Port of embarkation.

    To raw data

How do I read and write tabular data?#

  • I want to analyze the Titanic passenger data, available as a CSV file.

    In [2]: titanic = pd.read_csv("data/titanic.csv")
    

    pandas provides the read_csv() function to read data stored as a csv file into a pandas DataFrame. pandas supports many different file formats or data sources out of the box (csv, excel, sql, json, parquet, …), each of them with the prefix read_*.

Make sure to always have a check on the data after reading in the data. When displaying a DataFrame, the first and last 5 rows will be shown by default:

In [3]: titanic
Out[3]: 
     PassengerId  Survived  Pclass  ...     Fare Cabin Embarked
0              1         0       3  ...   7.2500   NaN        S
1              2         1       1  ...  71.2833   C85        C
2              3         1       3  ...   7.9250   NaN        S
3              4         1       1  ...  53.1000  C123        S
4              5         0       3  ...   8.0500   NaN        S
..           ...       ...     ...  ...      ...   ...      ...
886          887         0       2  ...  13.0000   NaN        S
887          888         1       1  ...  30.0000   B42        S
888          889         0       3  ...  23.4500   NaN        S
889          890         1       1  ...  30.0000  C148        C
890          891         0       3  ...   7.7500   NaN        Q

[891 rows x 12 columns]
  • I want to see the first 8 rows of a pandas DataFrame.

    In [4]: titanic.head(8)
    Out[4]: 
       PassengerId  Survived  Pclass  ...     Fare Cabin Embarked
    0            1         0       3  ...   7.2500   NaN        S
    1            2         1       1  ...  71.2833   C85        C
    2            3         1       3  ...   7.9250   NaN        S
    3            4         1       1  ...  53.1000  C123        S
    4            5         0       3  ...   8.0500   NaN        S
    5            6         0       3  ...   8.4583   NaN        Q
    6            7         0       1  ...  51.8625   E46        S
    7            8         0       3  ...  21.0750   NaN        S
    
    [8 rows x 12 columns]
    

    To see the first N rows of a DataFrame, use the head() method with the required number of rows (in this case 8) as argument.

Note

Interested in the last N rows instead? pandas also provides a tail() method. For example, titanic.tail(10) will return the last 10 rows of the DataFrame.

A check on how pandas interpreted each of the column data types can be done by requesting the pandas dtypes attribute:

In [5]: titanic.dtypes
Out[5]: 
PassengerId      int64
Survived         int64
Pclass           int64
Name               str
Sex                str
Age            float64
SibSp            int64
Parch            int64
Ticket             str
Fare           float64
Cabin              str
Embarked           str
dtype: object

For each of the columns, the used data type is enlisted. The data types in this DataFrame are integers (int64), floats (float64) and strings (object).

Note

When asking for the dtypes, no parentheses () are used! dtypes is an attribute of a DataFrame and Series. Attributes of a DataFrame or Series do not need (). Attributes represent a characteristic of a DataFrame/Series, whereas methods (which require parentheses ()) do something with the DataFrame/Series as introduced in the first tutorial.

  • My colleague requested the Titanic data as a spreadsheet.

    Note

    If you want to use to_excel() and read_excel(), you need to install an Excel reader as outlined in the Excel files section of the installation documentation.

    In [6]: titanic.to_excel("titanic.xlsx", sheet_name="passengers", index=False)
    

    Whereas read_* functions are used to read data to pandas, the to_* methods are used to store data. The to_excel() method stores the data as an excel file. In the example here, the sheet_name is named passengers instead of the default Sheet1. By setting index=False the row index labels are not saved in the spreadsheet.

The equivalent read function read_excel() will reload the data to a DataFrame:

In [7]: titanic = pd.read_excel("titanic.xlsx", sheet_name="passengers", engine="openpyxl")
In [8]: titanic.head()
Out[8]: 
   PassengerId  Survived  Pclass  ...     Fare Cabin Embarked
0            1         0       3  ...   7.2500   NaN        S
1            2         1       1  ...  71.2833   C85        C
2            3         1       3  ...   7.9250   NaN        S
3            4         1       1  ...  53.1000  C123        S
4            5         0       3  ...   8.0500   NaN        S

[5 rows x 12 columns]
  • I’m interested in a technical summary of a DataFrame

    In [9]: titanic.info()
    <class 'pandas.DataFrame'>
    RangeIndex: 891 entries, 0 to 890
    Data columns (total 12 columns):
     #   Column       Non-Null Count  Dtype  
    ---  ------       --------------  -----  
     0   PassengerId  891 non-null    int64  
     1   Survived     891 non-null    int64  
     2   Pclass       891 non-null    int64  
     3   Name         891 non-null    str    
     4   Sex          891 non-null    str    
     5   Age          714 non-null    float64
     6   SibSp        891 non-null    int64  
     7   Parch        891 non-null    int64  
     8   Ticket       891 non-null    str    
     9   Fare         891 non-null    float64
     10  Cabin        204 non-null    str    
     11  Embarked     889 non-null    str    
    dtypes: float64(2), int64(5), str(5)
    memory usage: 118.7 KB
    

    The method info() provides technical information about a DataFrame, so let’s explain the output in more detail:

    • It is indeed a DataFrame.

    • There are 891 entries, i.e. 891 rows.

    • Each row has a row label (aka the index) with values ranging from 0 to 890.

    • The table has 12 columns. Most columns have a value for each of the rows (all 891 values are non-null). Some columns do have missing values and less than 891 non-null values.

    • The columns Name, Sex, Ticket, Cabin and Embarked consist of textual data (strings, aka object). The other columns are numerical data, some of them are whole numbers (integer) and others are real numbers (float).

    • The kind of data (characters, integers, …) in the different columns are summarized by listing the dtypes.

    • The approximate amount of RAM used to hold the DataFrame is provided as well.

REMEMBER

  • Getting data in to pandas from many different file formats or data sources is supported by read_* functions.

  • Exporting data out of pandas is provided by different to_* methods.

  • The head/tail/info methods and the dtypes attribute are convenient for a first check.

To user guide

For a complete overview of the input and output possibilities from and to pandas, see the user guide section about reader and writer functions.

محاسبهٔ آمار خلاصه

In [1]: import pandas as pd
Data used for this tutorial:
  • Titanic data

    This tutorial uses the Titanic data set, stored as CSV. The data consists of the following data columns:

    • PassengerId: Id of every passenger.

    • Survived: Indication whether passenger survived. 0 for no and 1 for yes.

    • Pclass: One out of the 3 ticket classes: Class 1, Class 2 and Class 3.

    • Name: Name of passenger.

    • Sex: Gender of passenger.

    • Age: Age of passenger in years.

    • SibSp: Number of siblings or spouses aboard.

    • Parch: Number of parents or children aboard.

    • Ticket: Ticket number of passenger.

    • Fare: Indicating the fare.

    • Cabin: Cabin number of passenger.

    • Embarked: Port of embarkation.

    To raw data
    In [2]: titanic = pd.read_csv("data/titanic.csv")
    
    In [3]: titanic.head()
    Out[3]: 
       PassengerId  Survived  Pclass  ...     Fare Cabin Embarked
    0            1         0       3  ...   7.2500   NaN        S
    1            2         1       1  ...  71.2833   C85        C
    2            3         1       3  ...   7.9250   NaN        S
    3            4         1       1  ...  53.1000  C123        S
    4            5         0       3  ...   8.0500   NaN        S
    
    [5 rows x 12 columns]
    

How to calculate summary statistics#

Aggregating statistics#

  • What is the average age of the Titanic passengers?

    In [4]: titanic["Age"].mean()
    Out[4]: np.float64(29.69911764705882)
    

Different statistics are available and can be applied to columns with numerical data. Operations in general exclude missing data and operate across rows by default.

  • What is the median age and ticket fare price of the Titanic passengers?

    In [5]: titanic[["Age", "Fare"]].median()
    Out[5]: 
    Age     28.0000
    Fare    14.4542
    dtype: float64
    

    The statistic applied to multiple columns of a DataFrame (the selection of two columns returns a DataFrame, see the subset data tutorial) is calculated for each numeric column.

The aggregating statistic can be calculated for multiple columns at the same time. Remember the describe function from the first tutorial?

In [6]: titanic[["Age", "Fare"]].describe()
Out[6]: 
              Age        Fare
count  714.000000  891.000000
mean    29.699118   32.204208
std     14.526497   49.693429
min      0.420000    0.000000
25%     20.125000    7.910400
50%     28.000000   14.454200
75%     38.000000   31.000000
max     80.000000  512.329200

Instead of the predefined statistics, specific combinations of aggregating statistics for given columns can be defined using the DataFrame.agg() method:

In [7]: titanic.agg(
   ...:     {
   ...:         "Age": ["min", "max", "median", "skew"],
   ...:         "Fare": ["min", "max", "median", "mean"],
   ...:     }
   ...: )
   ...: 
Out[7]: 
              Age        Fare
min      0.420000    0.000000
max     80.000000  512.329200
median  28.000000   14.454200
skew     0.389108         NaN
mean          NaN   32.204208
To user guide

Details about descriptive statistics are provided in the user guide section on descriptive statistics.

Aggregating statistics grouped by category#

  • What is the average age for male versus female Titanic passengers?

    In [8]: titanic[["Sex", "Age"]].groupby("Sex").mean()
    Out[8]: 
                  Age
    Sex              
    female  27.915709
    male    30.726645
    

    As our interest is the average age for each gender, a subselection on these two columns is made first: titanic[["Sex", "Age"]]. Next, the groupby() method is applied on the Sex column to make a group per category. The average age for each gender is calculated and returned.

Calculating a given statistic (e.g. mean age) for each category in a column (e.g. male/female in the Sex column) is a common pattern. The groupby method is used to support this type of operations. This fits in the more general split-apply-combine pattern:

  • Split the data into groups

  • Apply a function to each group independently

  • Combine the results into a data structure

The apply and combine steps are typically done together in pandas.

In the previous example, we explicitly selected the 2 columns first. If not, the mean method is applied to each column containing numerical columns by passing numeric_only=True:

In [9]: titanic.groupby("Sex").mean(numeric_only=True)
Out[9]: 
        PassengerId  Survived    Pclass  ...     SibSp     Parch       Fare
Sex                                      ...                               
female   431.028662  0.742038  2.159236  ...  0.694268  0.649682  44.479818
male     454.147314  0.188908  2.389948  ...  0.429809  0.235702  25.523893

[2 rows x 7 columns]

It does not make much sense to get the average value of the Pclass. If we are only interested in the average age for each gender, the selection of columns (square brackets [] as usual) is supported on the grouped data as well:

In [10]: titanic.groupby("Sex")["Age"].mean()
Out[10]: 
Sex
female    27.915709
male      30.726645
Name: Age, dtype: float64

Note

The Pclass column contains numerical data but actually represents 3 categories (or factors) with respectively the labels ‘1’, ‘2’ and ‘3’. Calculating statistics on these does not make much sense. Therefore, pandas provides a Categorical data type to handle this type of data. More information is provided in the user guide Categorical data section.

  • What is the mean ticket fare price for each of the sex and cabin class combinations?

    In [11]: titanic.groupby(["Sex", "Pclass"])["Fare"].mean()
    Out[11]: 
    Sex     Pclass
    female  1         106.125798
            2          21.970121
            3          16.118810
    male    1          67.226127
            2          19.741782
            3          12.661633
    Name: Fare, dtype: float64
    

    Grouping can be done by multiple columns at the same time. Provide the column names as a list to the groupby() method.

To user guide

A full description on the split-apply-combine approach is provided in the user guide section on groupby operations.

Count number of records by category#

  • What is the number of passengers in each of the cabin classes?

    In [12]: titanic["Pclass"].value_counts()
    Out[12]: 
    Pclass
    3    491
    1    216
    2    184
    Name: count, dtype: int64
    

    The value_counts() method counts the number of records for each category in a column.

The function is a shortcut, it is actually a groupby operation in combination with counting the number of records within each group:

In [13]: titanic.groupby("Pclass")["Pclass"].count()
Out[13]: 
Pclass
1    216
2    184
3    491
Name: Pclass, dtype: int64

Note

Both size and count can be used in combination with groupby. Whereas size includes NaN values and just provides the number of rows (size of the table), count excludes the missing values. In the value_counts method, use the dropna argument to include or exclude the NaN values.

To user guide

The user guide has a dedicated section on value_counts , see the page on discretization.

REMEMBER

  • Aggregation statistics can be calculated on entire columns or rows.

  • groupby provides the power of the split-apply-combine pattern.

  • value_counts is a convenient shortcut to count the number of entries in each category of a variable.

To user guide

A full description on the split-apply-combine approach is provided in the user guide pages about groupby operations.

On this page