pandas: DataFrame

Ha Khanh Nguyen (hknguyen)

• 1. Importing Data
• 2. What is a DataFrame?
  • 2.1 Creating a DataFrame
  • 2.2 Summary statistics of DataFrame columns
• 3. Selecting Rows/Columns of a DataFrame
  • 3.1 Using []
  • 3.2 Using iloc
  • 3.3 Using loc
  • Exercise
• 4. Filtering DataFrame
• 5. Modifying DataFrame Values

---

1. Importing Data

• We will learn about importing data in depth next week.
• For the purpose of this lecture, we will only work with csv (comma separated value) files.
• Therefore, we will use read_csv() to import data into Python.

import pandas as pd

ramen = pd.read_csv('https://stat430.hknguyen.org/files/datasets/clean-ramen.csv')
ramen

	Brand	Variety	Style	Country	Stars
0	New Touch	T's Restaurant Tantanmen	Cup	Japan	3.75
1	Just Way	Noodles Spicy Hot Sesame Spicy Hot Sesame Guan...	Pack	Taiwan	1.00
2	Nissin	Cup Noodles Chicken Vegetable	Cup	USA	2.25
3	Wei Lih	GGE Ramen Snack Tomato Flavor	Pack	Taiwan	2.75
4	Ching's Secret	Singapore Curry	Pack	India	3.75
...	...	...	...	...	...
2570	Vifon	Hu Tiu Nam Vang ["Phnom Penh" style] Asian Sty...	Bowl	Vietnam	3.50
2571	Wai Wai	Oriental Style Instant Noodles	Pack	Thailand	1.00
2572	Wai Wai	Tom Yum Shrimp	Pack	Thailand	2.00
2573	Wai Wai	Tom Yum Chili Flavor	Pack	Thailand	2.00
2574	Westbrae	Miso Ramen	Pack	USA	0.50

2575 rows × 5 columns

• The output of the read_csv() function is a pandas DataFrame.

---

2. What is a DataFrame?

• A DataFrame is a data structure of table format where:
  • Each row represents each observation.
  • Each column represents each variable/attribute.
• In our ramen dataset, there are 2575 observations and 5 attributes, which are:
  • Brand
  • Variety
  • Style
  • Country
  • Stars

2.1 Creating a DataFrame

• There are many ways to construct a DataFrame, the most common method is to load the data from a file (using function like read_csv().
• Another common method is to use a dictionary of equal-length lists or NumPy arrays:

data = {'state': ['Ohio', 'Ohio', 'Ohio', 'Nevada', 'Nevada', 'Nevada'],
        'year': [2000, 2001, 2002, 2001, 2002, 2003],
        'pop': [1.5, 1.7, 3.6, 2.4, 2.9, 3.2]}
frame = pd.DataFrame(data)

frame

	state	year	pop
0	Ohio	2000	1.5
1	Ohio	2001	1.7
2	Ohio	2002	3.6
3	Nevada	2001	2.4
4	Nevada	2002	2.9
5	Nevada	2003	3.2

• A DataFrame represents a rectangular table of data and contains an ordered collection of columns of the same length, each of which can be a different value type (numeric, string, boolean, etc.).
  • If you have learned R, remind yourself of the definition of a DataFrame in R!
  • Similarly, in Python, DataFrame can be thought of a dictionary of Series all sharing the same index.

ramen['Brand']

0            New Touch
1             Just Way
2               Nissin
3              Wei Lih
4       Ching's Secret
             ...      
2570             Vifon
2571           Wai Wai
2572           Wai Wai
2573           Wai Wai
2574          Westbrae
Name: Brand, Length: 2575, dtype: object

type(ramen['Brand'])

pandas.core.series.Series

2.2 Summary statistics of DataFrame columns

• Use the function describe() of the DataFrame object to get the summary statistics of the variables in the DataFrame.

ramen.describe()

	Stars
count	2575.000000
mean	3.654893
std	1.015641
min	0.000000
25%	3.250000
50%	3.750000
75%	4.250000
max	5.000000

• We also want to know the data type of each variables!

ramen.dtypes

Brand       object
Variety     object
Style       object
Country     object
Stars      float64
dtype: object

• You might wonder: How many different country that produce ramen? How many style of ramen are there? Or how many different brands?
  • Use the unique() function to find out!

---

3. Selecting Rows/Columns of a DataFrame

3.1 Using []

• [] is a very popular operator in Python as we have seen its usage with various data structures.
• With DataFrame, we can use [] to select a column using the column's name:

ramen['Country']

0          Japan
1         Taiwan
2            USA
3         Taiwan
4          India
          ...   
2570     Vietnam
2571    Thailand
2572    Thailand
2573    Thailand
2574         USA
Name: Country, Length: 2575, dtype: object

• Another way to select a column is:

ramen.Country

0          Japan
1         Taiwan
2            USA
3         Taiwan
4          India
          ...   
2570     Vietnam
2571    Thailand
2572    Thailand
2573    Thailand
2574         USA
Name: Country, Length: 2575, dtype: object

• To select multiple columns at the same time, supply a list of column names inside []:

ramen[['Brand', 'Country']]

	Brand	Country
0	New Touch	Japan
1	Just Way	Taiwan
2	Nissin	USA
3	Wei Lih	Taiwan
4	Ching's Secret	India
...	...	...
2570	Vifon	Vietnam
2571	Wai Wai	Thailand
2572	Wai Wai	Thailand
2573	Wai Wai	Thailand
2574	Westbrae	USA

2575 rows × 2 columns

Notes:

• The output of selecting a column using [] is a pandas Series.
• The output of selecting multiple columns using [] is a pandas DataFrame.
• This is similar to how R sometimes returns a vector (or tuple if you use dplyr) or a DataFrame.

BIG NOTES: As DataFrame is a dict of Series (where each key is the column name and each value is a Series), we cannot access the row using [].

3.2 Using iloc

• Now, so how do we access the row?
• One of the methods is to use iloc which is short for interger-location based indexing.
• iloc syntax: dataframe.iloc[<row index>, <column index>].

• To select the Stars rating of the first observation (index 0):

ramen.iloc[0, 4]

3.75

• To select multiple columns of the first observation:

ramen.iloc[0, 1:5]

Variety    T's Restaurant Tantanmen 
Style                            Cup
Country                        Japan
Stars                           3.75
Name: 0, dtype: object

• To select ALL columns of one row:

ramen.iloc[0, :]

Brand                      New Touch
Variety    T's Restaurant Tantanmen 
Style                            Cup
Country                        Japan
Stars                           3.75
Name: 0, dtype: object

• Now, what is the type of the ouput of iloc?

type(ramen.iloc[0, 4])

numpy.float64

type(ramen.iloc[0, 1:5])

pandas.core.series.Series

• It's important to make a habit of know exactly the returned object of a function! Always ask yourself this question when you program!
• Notice the appearance of NumPy data structures and types in pandas? There is a strong relationship between the 2 famous libraries.

• Similarly, we can select multiple rows of the same column:
  • The following code segment returns the rating of the first 5 observations in this dataset.

ramen.iloc[0:5, 4]

0    3.75
1    1.00
2    2.25
3    2.75
4    3.75
Name: Stars, dtype: float64

• Another Series is returned!
• Let's try selecting multiple rows and multiple columns at the same time:

ramen.iloc[[0, 5, 10, 15], :]

	Brand	Variety	Style	Country	Stars
0	New Touch	T's Restaurant Tantanmen	Cup	Japan	3.75
5	Samyang Foods	Kimchi song Song Ramen	Pack	South Korea	4.75
10	Tao Kae Noi	Creamy tom Yum Kung Flavour	Pack	Thailand	5.00
15	KOKA	Mushroom Flavour Instant Noodles	Cup	Singapore	3.50

• The output is now a DataFrame! It makes sense right?
  • One-dimension output: Series
  • Two-dimension output: DataFrame

3.3 Using loc

• The i in iloc stands for integer. That is why with iloc, we always use numbers for indexing.
• With loc, we use label (names) or a Boolean list/array for indexing instead.
• For our dataset, our rows are labeled by integers, so we still use integers as our row labels.

ramen.loc[0, 'Stars']

3.75

Exercise

Try to use loc to select the following rows, columns:

• All columns except Brand for the 1st observation.
• All columns for the last observation.
• Stars for the first 5 observations.
• All columns for the first 5 observations.

---

4. Filtering DataFrame

• Let's say I only want observations of ramen coming from USA!
• Then we need to filter ramen and only print out the rows where Country is USA.

ramen['Country'] == 'USA'

0       False
1       False
2        True
3       False
4       False
        ...  
2570    False
2571    False
2572    False
2573    False
2574     True
Name: Country, Length: 2575, dtype: bool

• This returns a Series of type Boolean. We can use this to filter the DataFrame!
• There are 2 ways to do this:

# method 1
ramen[ramen['Country'] == 'USA']

	Brand	Variety	Style	Country	Stars
2	Nissin	Cup Noodles Chicken Vegetable	Cup	USA	2.25
11	Yamachan	Yokohama Tonkotsu Shoyu	Pack	USA	5.00
17	Yamachan	Tokyo Shoyu Ramen	Pack	USA	5.00
21	Jackpot Teriyaki	Beef Ramen	Pack	USA	5.00
23	Yamachan	Sapporo Miso Ramen	Pack	USA	4.75
...	...	...	...	...	...
2511	Sapporo Ichiban	Chicken Flavor	Pack	USA	3.50
2541	Maruchan	Ramen Noodle Soup Shrimp	Pack	USA	2.00
2552	Nissin	Top Ramen Creamy Chicken	Pack	USA	4.50
2565	Smack	Vegetable Beef	Pack	USA	1.50
2574	Westbrae	Miso Ramen	Pack	USA	0.50

323 rows × 5 columns

# method 2
ramen.loc[ramen['Country'] == 'USA']

	Brand	Variety	Style	Country	Stars
2	Nissin	Cup Noodles Chicken Vegetable	Cup	USA	2.25
11	Yamachan	Yokohama Tonkotsu Shoyu	Pack	USA	5.00
17	Yamachan	Tokyo Shoyu Ramen	Pack	USA	5.00
21	Jackpot Teriyaki	Beef Ramen	Pack	USA	5.00
23	Yamachan	Sapporo Miso Ramen	Pack	USA	4.75
...	...	...	...	...	...
2511	Sapporo Ichiban	Chicken Flavor	Pack	USA	3.50
2541	Maruchan	Ramen Noodle Soup Shrimp	Pack	USA	2.00
2552	Nissin	Top Ramen Creamy Chicken	Pack	USA	4.50
2565	Smack	Vegetable Beef	Pack	USA	1.50
2574	Westbrae	Miso Ramen	Pack	USA	0.50

323 rows × 5 columns

• The 2nd method is PREFERRED when you will be alternating the output!
• Now, what's if I want only the ones where Brand is Nissin?

ramen.loc[(ramen['Country'] == 'USA') & (ramen['Brand'] == 'Nissin')]

	Brand	Variety	Style	Country	Stars
2	Nissin	Cup Noodles Chicken Vegetable	Cup	USA	2.25
38	Nissin	Cup Noodles Very Veggie Spicy Chicken Flavor R...	Cup	USA	5.00
41	Nissin	Cup Noodles Very Veggie Beef Flavor Ramen Nood...	Cup	USA	5.00
44	Nissin	Cup Noodles Very Veggie Chicken Flavor Ramen N...	Cup	USA	5.00
195	Nissin	Cup Noodles Beef Flavor Ramen Noodle Soup (New...	Cup	USA	3.50
...	...	...	...	...	...
2422	Nissin	Creamy Chicken	Cup	USA	1.75
2434	Nissin	Chow Mein Teriyaki Beef	Tray	USA	4.50
2438	Nissin	Bowl Noodles Rich & Savory Chicken	Bowl	USA	1.75
2510	Nissin	Top Ramen Oriental	Pack	USA	2.50
2552	Nissin	Top Ramen Creamy Chicken	Pack	USA	4.50

95 rows × 5 columns

# or
ramen[(ramen['Country'] == 'USA') & (ramen['Brand'] == 'Nissin')]

	Brand	Variety	Style	Country	Stars
2	Nissin	Cup Noodles Chicken Vegetable	Cup	USA	2.25
38	Nissin	Cup Noodles Very Veggie Spicy Chicken Flavor R...	Cup	USA	5.00
41	Nissin	Cup Noodles Very Veggie Beef Flavor Ramen Nood...	Cup	USA	5.00
44	Nissin	Cup Noodles Very Veggie Chicken Flavor Ramen N...	Cup	USA	5.00
195	Nissin	Cup Noodles Beef Flavor Ramen Noodle Soup (New...	Cup	USA	3.50
...	...	...	...	...	...
2422	Nissin	Creamy Chicken	Cup	USA	1.75
2434	Nissin	Chow Mein Teriyaki Beef	Tray	USA	4.50
2438	Nissin	Bowl Noodles Rich & Savory Chicken	Bowl	USA	1.75
2510	Nissin	Top Ramen Oriental	Pack	USA	2.50
2552	Nissin	Top Ramen Creamy Chicken	Pack	USA	4.50

95 rows × 5 columns

---

5. Modifying DataFrame Values

• For example, there was a data entry error and all observations of Nissin brands in the US that received 1.5 stars should receive 2.5 stars instead.
• To do this, we take the following steps:
  • Find all the observations that are of brand Nissin from USA and have 1.5 stars rating.
  • Assign 2.5 as the new stars rating.

# step 1
ramen.loc[(ramen['Brand'] == 'Nissin') & (ramen['Country'] == 'USA') & (ramen['Stars'] == 1.5)]

	Brand	Variety	Style	Country	Stars
2313	Nissin	Chow Mein Kung Pao Chicken	Tray	USA	1.5

# step 2
ramen.loc[(ramen['Brand'] == 'Nissin') & (ramen['Country'] == 'USA') & (ramen['Stars'] == 1.5), 'Stars'] = 2.5
ramen.loc[(ramen['Brand'] == 'Nissin') & (ramen['Country'] == 'USA') & (ramen['Stars'] <= 2.5)]

	Brand	Variety	Style	Country	Stars
2	Nissin	Cup Noodles Chicken Vegetable	Cup	USA	2.250
199	Nissin	Cup Noodles Hot & Spicy Shrimp Flavor Ramen No...	Cup	USA	2.500
2197	Nissin	Chow Mein Thai Peanut	Tray	USA	2.500
2216	Nissin	Cup Noodles Hearty Chicken	Cup	USA	2.500
2238	Nissin	Big Cup Noodles Chicken	Cup	USA	2.250
2242	Nissin	Big Cup Noodles Beef	Cup	USA	2.125
2243	Nissin	Big Cup Noodles Shrimp	Cup	USA	2.250
2313	Nissin	Chow Mein Kung Pao Chicken	Tray	USA	2.500
2393	Nissin	Bowl Noodles Hot & Spicy Chicken	Bowl	USA	2.000
2422	Nissin	Creamy Chicken	Cup	USA	1.750
2438	Nissin	Bowl Noodles Rich & Savory Chicken	Bowl	USA	1.750
2510	Nissin	Top Ramen Oriental	Pack	USA	2.500

• Let's try to do this using [] instead of loc:

ramen = pd.read_csv('https://stat107.hknguyen.org/files/datasets/clean-ramen.csv')
ramen.loc[(ramen['Brand'] == 'Nissin') & (ramen['Country'] == 'USA') & (ramen['Stars'] == 1.5)]

	Brand	Variety	Style	Country	Stars
2313	Nissin	Chow Mein Kung Pao Chicken	Tray	USA	1.5

ramen[(ramen['Brand'] == 'Nissin') & (ramen['Country'] == 'USA') & (ramen['Stars'] == 1.5)]['Stars'] = 2.5

<ipython-input-27-7c925de0f5d3>:1: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  ramen[(ramen['Brand'] == 'Nissin') & (ramen['Country'] == 'USA') & (ramen['Stars'] == 1.5)]['Stars'] = 2.5

ramen.loc[(ramen['Brand'] == 'Nissin') & (ramen['Country'] == 'USA') & (ramen['Stars'] <= 2.5)]

	Brand	Variety	Style	Country	Stars
2	Nissin	Cup Noodles Chicken Vegetable	Cup	USA	2.250
199	Nissin	Cup Noodles Hot & Spicy Shrimp Flavor Ramen No...	Cup	USA	2.500
2197	Nissin	Chow Mein Thai Peanut	Tray	USA	2.500
2216	Nissin	Cup Noodles Hearty Chicken	Cup	USA	2.500
2238	Nissin	Big Cup Noodles Chicken	Cup	USA	2.250
2242	Nissin	Big Cup Noodles Beef	Cup	USA	2.125
2243	Nissin	Big Cup Noodles Shrimp	Cup	USA	2.250
2313	Nissin	Chow Mein Kung Pao Chicken	Tray	USA	1.500
2393	Nissin	Bowl Noodles Hot & Spicy Chicken	Bowl	USA	2.000
2422	Nissin	Creamy Chicken	Cup	USA	1.750
2438	Nissin	Bowl Noodles Rich & Savory Chicken	Bowl	USA	1.750
2510	Nissin	Top Ramen Oriental	Pack	USA	2.500

• What causes this warning?
  • Chained assignment
• To discuss chaining, first, let's define some new terminologies:
  • Assignment: operations that set the value of a variable.
    • ramen = pd.read_csv(...)
  • Access: operations that return the value of a variable/object.
    • ramen['Stars']
  • Indexing: any assignment or access method that references a subset of the data.
    • ramen['Stars']
    • ramen['Stars'] = 0
  • Chaining: the use of more than one indexing operation back-to-back.
    • ramen[ramen['Brand'] == 'Nissin']['Stars']

• Chained assignment throws a warning because pandas suspects that one of the indexing operation returns a copy of the DataFrame instead of a slice of the DataFrame.
• You can read more about it in the pandas documentation page addressing this design choice.

---

This lecture notes reference materials from Chapter 5 of Wes McKinney's Python for Data Analysis 2nd Ed.