Pandas MultiIndex

A MultiIndex in Pandas is a hierarchical indexing structure that allows us to represent and work with higher-dimensional data efficiently.

While a typical index refers to a single column, a MultiIndex contains multiple levels of indexes. Each column in a MultiIndex is linked to one another through a parent/relationship.

Let's take an example of a DataFrame containing the population of different countries.

import pandas as pd

# create a dictionary
data = {
    "Continent": ["North America", "Europe", "Asia", "North America", "Asia", "Europe", "North America", "Asia", "Europe", "Asia"],
    "Country": ["United States", "Germany", "China", "Canada", "Japan", "France", "Mexico", "India", "United Kingdom", "Nepal"],
    "Population": [331002651, 83783942, 1439323776, 37742154, 126476461, 65273511, 128932753, 1380004385, 67886011, 29136808]
}

# create dataframe from dictionary
df = pd.DataFrame(data)

print(df)

Output

       Continent         Country  Population
0  North America   United States   331002651
1         Europe         Germany    83783942
2           Asia           China  1439323776
3  North America          Canada    37742154
4           Asia           Japan   126476461
5         Europe          France    65273511
6  North America          Mexico   128932753
7           Asia           India  1380004385
8         Europe  United Kingdom    67886011
9           Asia           Nepal    29136808

Notice the redundancy in the Continent column. North America and Europe are repeated three times each while Asia is repeated four times.

Additionally, we have arranged the entries in a random order and used integer values as index for the rows, thus complicating the task of locating data for a particular country. This task becomes tedious as the size of the data set grows.

In situations like this, hierarchical indexing, as shown in figure below, makes much more sense.

Here, Continent is the parent column and Country is the child column.

Create MultiIndex in Pandas

In Pandas, we achieve hierarchical indexing using the concept of MultiIndex.

Let's see an example.

import pandas as pd

# create a dictionary
data = {
    "Continent": ["North America", "Europe", "Asia", "North America", "Asia", "Europe", "North America", "Asia", "Europe", "Asia"],
    "Country": ["United States", "Germany", "China", "Canada", "Japan", "France", "Mexico", "India", "United Kingdom", "Nepal"],
    "Population": [331002651, 83783942, 1439323776, 37742154, 126476461, 65273511, 128932753, 1380004385, 67886011, 29136808]
}

# create dataframe from dictionary
df = pd.DataFrame(data)

# sort the data by continent
df.sort_values('Continent', inplace=True)

# create a multiindex
df.set_index(['Continent','Country'], inplace=True)

print(df)

Output

                             Population    
Continent    Country                   
Asia          China           1439323776
              Japan            126476461
              India           1380004385
              Nepal             29136808
Europe        Germany           83783942
              France            65273511
              United Kingdom    67886011
North America United States    331002651
              Canada            37742154
              Mexico           128932753

In the above example, we first sorted the values in the dataframe df based on the Continent column. This groups the entries of the same continent together.

We then created a MultiIndex by passing a list of columns as an argument to the set_index() function.

Notice the order of the columns in the list. Continent comes first as it is the parent column and Country comes second as it is the child of Continent.

Access Rows With MultiIndex

We can access rows with MultiIndex as shown in the example below.

import pandas as pd

# create a dictionary
data = {
    "Continent": ["North America", "Europe", "Asia", "North America", "Asia", "Europe", "North America", "Asia", "Europe", "Asia"],
    "Country": ["United States", "Germany", "China", "Canada", "Japan", "France", "Mexico", "India", "United Kingdom", "Nepal"],
    "Population": [331002651, 83783942, 1439323776, 37742154, 126476461, 65273511, 128932753, 1380004385, 67886011, 29136808]
}

# create dataframe from dictionary
df = pd.DataFrame(data)

# sort the data by continent
df.sort_values('Continent', inplace=True)

# create a multiindex
df.set_index(['Continent','Country'], inplace=True)

# access all entries under Asia
asia = df.loc['Asia']

# access Canada
canada = df.loc[('North America', 'Canada')]

print('Asia\n', asia)
print('\nCanada\n', canada)

Output

Asia
         Population
Country            
China    1439323776
Japan     126476461
India    1380004385
Nepal      29136808

Canada
Population    37742154
Name: (North America, Canada), dtype: int64

In the above example, we accessed all the entries under Asia by passing a single string Asia to df.loc[].

To access a particular row Canada, we passed a tuple ('North America' , 'Canada') to df.loc[].

# correct
df.loc[('North America' , 'Canada')]

# error
df.loc['Canada']

MultiIndex from Arrays

We can also create a MultiIndex from an array of arrays using the from_arrays() method.

Let's see an example.

import pandas as pd

# create arrays
continent = ['Asia', 'Asia', 'Asia', 'Asia', 'Europe', 'Europe', 'Europe', 'North America', 'North America', 'North America']
country = ['China', 'India', 'Japan', 'Nepal', 'France', 'Germany', 'United Kingdom', 'Canada', 'Mexico', 'United States']
population = [1439323776, 1380004385, 126476461, 29136808, 65273511, 83783942, 67886011, 37742154, 128932753, 331002651]

# create array of arrays
index_array = [continent, country]

# create multiindex from array
multi_index = pd.MultiIndex.from_arrays(index_array, names=['Continent', 'Country'])

# create dataframe using multiindex
df = pd.DataFrame({'Population' :population}, index=multi_index)

print(df)

Output

                              Population
Continent     Country                   
Asia          China           1439323776
              India           1380004385
              Japan            126476461
              Nepal             29136808
Europe        France            65273511
              Germany           83783942
              United Kingdom    67886011
North America Canada            37742154
              Mexico           128932753
              United States    331002651

In this example, we created a MultiIndex object named multi_index from two arrays: continent and country.

We then created a DataFrame using the population array and assigned multi_index as its index.