DataMop Tutorial
Welcome to the tutorial for datamop, the ultimate Python package for cleaning and preparing your datasets with minimal effort. Data cleaning can often feel like the most tedious part of any data analysis or machine learning project. Missing values, inconsistent scales, and different data types can slow you down and distract from the real task: extracting insights from your data.
That is where datamop package comes in! This powerful, easy-to-use package automates many of the common data cleaning tasks, like imputing missing values, encoding categorical features and scaling numerical features, saving you time and effort while ensuring your data is consistent, complete, and ready for analysis.
Here we will show example usages for each function in the package, including sweep_nulls, column_encoder, and column_scaler. Your messy data will be ready to use after using this robust package. With datamop, you can focus more on analysis and less on tedious preprocessing.
Imports
Before we get started, let’s install and import the datamop package. We will demonstrate each functions in the datamop package with examples using the Airbnb Open Data from kaggle.
# import modules
import pandas as pd
import numpy as np
from datamop.sweep_nulls import sweep_nulls
from datamop.column_encoder import column_encoder
from datamop.column_scaler import column_scaler
# import Airbnb Open Data
data = pd.read_csv("../src/data/Airbnb_Open_Data.csv")
Pre-check column types
Before imputing, scaling and encoding ensure that numerical columns are in the correct format.
In the Airbnb dataset, the price and service fee columns are objects because they contain a $ sign.
We need to remove the $ sign and convert these columns to floats.
Additionally, we’ll demonstrate scaling on the reviews per month column, which is already a numeric column.
# Clean "price" and "service fee" columns by removing unwanted characters and converting to float
data["price"] = data["price"].str.strip().str.replace(r"[^0-9.]", "", regex=True).astype(float)
data["service fee"] = data["service fee"].str.strip().str.replace(r"[^0-9.]", "", regex=True).astype(float)
# Verify the changes
data[["price", "service fee"]].head()
| price | service fee | |
|---|---|---|
| 0 | 966.0 | 193.0 |
| 1 | 142.0 | 28.0 |
| 2 | 620.0 | 124.0 |
| 3 | 368.0 | 74.0 |
| 4 | 204.0 | 41.0 |
Handling missing values with sweep_nulls()
One of the most common challenges in data cleaning process is dealing with missing values. datamop provides a convenient method called sweep_nulls() to help you handle these issues effortlessly. The sweep_nulls() function scans your dataset for missing values and allows you to handle them using various strategies, including ‘mean’(numeric only), ‘median’(numeric only), ‘mode’, ‘constant’, and ‘drop’.
Let’s start by checking the missing values in the dataset:
data.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 102599 entries, 0 to 102598
Data columns (total 20 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 id 102599 non-null int64
1 host_identity_verified 102310 non-null object
2 neighbourhood group 102570 non-null object
3 neighbourhood 102583 non-null object
4 lat 102591 non-null float64
5 long 102591 non-null float64
6 country 102067 non-null object
7 instant_bookable 102494 non-null object
8 cancellation_policy 102523 non-null object
9 room type 102599 non-null object
10 Construction year 102385 non-null float64
11 price 102352 non-null float64
12 service fee 102326 non-null float64
13 minimum nights 102190 non-null float64
14 number of reviews 102416 non-null float64
15 last review 86706 non-null object
16 reviews per month 86720 non-null float64
17 review rate number 102273 non-null float64
18 calculated host listings count 102280 non-null float64
19 availability 365 102151 non-null float64
dtypes: float64(11), int64(1), object(8)
memory usage: 15.7+ MB
data.isnull().sum()
id 0
host_identity_verified 289
neighbourhood group 29
neighbourhood 16
lat 8
long 8
country 532
instant_bookable 105
cancellation_policy 76
room type 0
Construction year 214
price 247
service fee 273
minimum nights 409
number of reviews 183
last review 15893
reviews per month 15879
review rate number 326
calculated host listings count 319
availability 365 448
dtype: int64
Imputing all columns
When dealing with datasets containing missing values across multiple columns, sweep_nulls()makes it easy to impute all columns simultaneously. This feature ensures consistent handling of missing data throughout the dataset, whether you’re using the mean, median, mode, or a custom value for imputation.
Since ‘mean’ and ‘median’ are designed for numerical features only, it is better to use ‘mode’, ‘constant’ or ‘drop’ when you have mixed data types in the dataset.
# using mode to impute missing value with the most common values in the column
data_fill_mode = data.copy()
sweep_nulls(data_fill_mode, strategy='mode')
data_fill_mode.isnull().sum()
/home/docs/checkouts/readthedocs.org/user_builds/datamop/checkouts/latest/src/datamop/sweep_nulls.py:68: UserWarning: Columns list is empty. Applying strategy to all columns.
warnings.warn("Columns list is empty. Applying strategy to all columns.", UserWarning)
/home/docs/checkouts/readthedocs.org/user_builds/datamop/checkouts/latest/src/datamop/sweep_nulls.py:116: FutureWarning: Downcasting object dtype arrays on .fillna, .ffill, .bfill is deprecated and will change in a future version. Call result.infer_objects(copy=False) instead. To opt-in to the future behavior, set `pd.set_option('future.no_silent_downcasting', True)`
data[column] = data[column].fillna(data[column].mode()[0])
id 0
host_identity_verified 0
neighbourhood group 0
neighbourhood 0
lat 0
long 0
country 0
instant_bookable 0
cancellation_policy 0
room type 0
Construction year 0
price 0
service fee 0
minimum nights 0
number of reviews 0
last review 0
reviews per month 0
review rate number 0
calculated host listings count 0
availability 365 0
dtype: int64
# using constant to impute missing value with a number
data_fill_number = data.copy()
sweep_nulls(data_fill_number, strategy='constant', fill_value = -999)
data_fill_number['host_identity_verified'].head()
/home/docs/checkouts/readthedocs.org/user_builds/datamop/checkouts/latest/src/datamop/sweep_nulls.py:68: UserWarning: Columns list is empty. Applying strategy to all columns.
warnings.warn("Columns list is empty. Applying strategy to all columns.", UserWarning)
0 unconfirmed
1 verified
2 -999
3 unconfirmed
4 verified
Name: host_identity_verified, dtype: object
Imputing specific numerical columns
If you want to focus on imputing missing values in specific numerical columns of your dataset without affecting other columns, you can achieve this by using sweep_nulls() to select the desired columns and apply an imputation strategy only to them.
# using mean to impute price and service fee columns
data_impute_mean = data.copy()
data_impute_mean = sweep_nulls(data_impute_mean, strategy='mean', columns=['price', 'service fee'])
data_impute_mean['price'].iloc[147]
print("Before impute:", data['price'].iloc[147])
print("Mean Price: ", data['price'].mean())
print("After impute:", data_impute_mean['price'].iloc[147])
Before impute: nan
Mean Price: 625.2935360325152
After impute: 625.2935360325152
# using constant to impute price and service fee columns with a negative number
data_impute_constant = data.copy()
data_impute_constant = sweep_nulls(data_impute_constant, strategy='constant', columns=['price', 'service fee'], fill_value=-999)
print("Before impute:", data['price'].iloc[147])
print("After impute:", data_impute_constant['price'].iloc[147])
Before impute: nan
After impute: -999.0
Imputing specific categorical columns
When working with datasets containing missing values in categorical columns, you can impute missing values in specific categorical columns using common strategies like filling with the mode, or a custom value.
# using constant to impute missing value with a string
data_cate = data.copy()
data_cate = sweep_nulls(data_cate, strategy='constant', columns=['host_identity_verified'], fill_value='missing')
data_cate['host_identity_verified'].unique()
array(['unconfirmed', 'verified', 'missing'], dtype=object)
# using mode to impute missing value with the most common values in the column
data_cate_mode = data.copy()
data_cate_mode = sweep_nulls(data_cate_mode, strategy='mode', columns=['country'])
print("Before impute:", data['country'].iloc[156])
print("Mode: ", data['country'].mode())
print("After impute:", data_cate_mode['country'].iloc[156])
Before impute: nan
Mode: 0 United States
Name: country, dtype: object
After impute: United States
Dropping columns
When working with datasets, some columns may have excessive missing values, which makes them unhelpful for analysis. Imputing them can introduce noise, therefore sweep_nulls() allows you to drop missing values.
# dropping one column
data_drop = data.copy()
data_drop = sweep_nulls(data_drop, strategy='drop')
data_drop.isnull().sum()
/home/docs/checkouts/readthedocs.org/user_builds/datamop/checkouts/latest/src/datamop/sweep_nulls.py:68: UserWarning: Columns list is empty. Applying strategy to all columns.
warnings.warn("Columns list is empty. Applying strategy to all columns.", UserWarning)
id 0
host_identity_verified 0
neighbourhood group 0
neighbourhood 0
lat 0
long 0
country 0
instant_bookable 0
cancellation_policy 0
room type 0
Construction year 0
price 0
service fee 0
minimum nights 0
number of reviews 0
last review 0
reviews per month 0
review rate number 0
calculated host listings count 0
availability 365 0
dtype: int64
Scaling Numerical Features with column_scaler()
When working with numerical data, inconsistent scales can distort analysis or machine learning results.
For example, a column measuring price in thousands might dominate another column measuring rating on a 1-5 scale.
To avoid this issue, scaling the numerical data to a consistent range or distribution can mitigate this problem.
The column_scaler() function in the datamop package allows users to scale any numeric column in a dataset. It supports two methods:
Min-Max Scaling: Scale values to a specific range, such as
[0, 1]or[10, 20].Standard Scaling: Transform values to have a mean of
0and a standard deviation of1.
The column_scaler() function allows flexible usage for both in-place scaling (replacing the original column) and creating a new scaled column.
Let’s walk through how to use column_scaler().
Example 1: Min-Max Scaling
Let’s scale the reviews per month column to a range between 0 and 1 using min-max scaling.
The scaled values will replace the original column (inplace=True).
# Using min-max scaling to scale "reviews per month" to a range between 0 and 1
data_minmax = data.copy().dropna()
column_scaler(data_minmax, column="reviews per month", method="minmax", new_min=0, new_max=1, inplace=True)
# Verify the scaled column
data_minmax[["reviews per month"]].head()
| reviews per month | |
|---|---|
| 0 | 0.002222 |
| 1 | 0.004112 |
| 3 | 0.051450 |
| 4 | 0.001000 |
| 5 | 0.006445 |
Example 2: Custom Min-Max Scaling with a New Column
Now let’s scale the price column to a range between 100 and 500. Instead of modifying the original column, we’ll create a new column called price_scaled by setting inplace=False.
# Using min-max scaling to scale "price" to a range between 100 and 500
column_scaler(data, column="price", method="minmax", new_min=100, new_max=500, inplace=False)
# Verify the new scaled column
data[["price", "price_scaled"]].head()
/home/docs/checkouts/readthedocs.org/user_builds/datamop/checkouts/latest/src/datamop/column_scaler.py:86: UserWarning: NaN value detected in column 'price'. They will be unchanged
warnings.warn(
| price | price_scaled | |
|---|---|---|
| 0 | 966.0 | 418.608696 |
| 1 | 142.0 | 132.000000 |
| 2 | 620.0 | 298.260870 |
| 3 | 368.0 | 210.608696 |
| 4 | 204.0 | 153.565217 |
Example 3: Standard Scaling
Let’s scale the service fee column using the standard scaling method,
which transforms the values to have a mean of 0 and standard deviation of 1.
The scaled values will replace the original column (incplace=True).
# Using standard scaling method on "service fee" column
column_scaler(data, column="service fee", method="standard", inplace=True)
# Verify the scaled column
data[["service fee"]].head()
/home/docs/checkouts/readthedocs.org/user_builds/datamop/checkouts/latest/src/datamop/column_scaler.py:86: UserWarning: NaN value detected in column 'service fee'. They will be unchanged
warnings.warn(
| service fee | |
|---|---|
| 0 | 1.024837 |
| 1 | -1.462885 |
| 2 | -0.015483 |
| 3 | -0.769338 |
| 4 | -1.266883 |
Edge Case 1: Scaling Column with Single Unique Value
If a column contains only a single unique value, column_scaler() automatically assigns the midpoint of the range for min-max scaling and issues a warning message.
# Create DataFrame with a single-value column
single_value_df = pd.DataFrame({"price": [100, 100, 100]})
# Scale the column using min-max scaling
scaled_df = column_scaler(single_value_df, column="price", method="minmax", new_min=0, new_max=1)
# Verify the result
scaled_df
/home/docs/checkouts/readthedocs.org/user_builds/datamop/checkouts/latest/src/datamop/column_scaler.py:93: UserWarning: Single-value column detected. All values will be scaled to the midpoint of the `new_min` and `new_max`.
warnings.warn(
| price | |
|---|---|
| 0 | 0.5 |
| 1 | 0.5 |
| 2 | 0.5 |
Edge Case 2: Handling Missing Values (NaN)
If a column contains missing values (NaN), column_scaler() leaves them unchanged and issue a warning. This ensures no data is lost or imputed incorrectly.
# Create a DataFrame with NaN values
nan_df = pd.DataFrame({"reviews per month": [10, np.nan, 20]})
# Scaled the column using min-max scaling
scaled_nan_df = column_scaler(nan_df, column="reviews per month", method="minmax", new_min=0, new_max=1)
# Verify the result
scaled_nan_df
/home/docs/checkouts/readthedocs.org/user_builds/datamop/checkouts/latest/src/datamop/column_scaler.py:86: UserWarning: NaN value detected in column 'reviews per month'. They will be unchanged
warnings.warn(
| reviews per month | |
|---|---|
| 0 | 0.0 |
| 1 | NaN |
| 2 | 1.0 |
Error Case 1: Using Non-Existent Column
If the specified column does not exist in the DataFrame, column_scaler() raises a KeyError
# Pass non existent column in the 'column' argument
try:
column_scaler(data, column="Non_existent", method="minmax")
except KeyError as e:
print(e)
'Column not found in the DataFrame.'
Error Case 2: Using Non-Numeric Columns
If you attempt to scale a non-numeric column, such column of strings, column_scaler() raises a ValueError.
# Pass column of objects (country column) to column scaler
try:
column_scaler(data, column="country", method="minmax")
except ValueError as e:
print(e)
Column must have numeric values.
Error Case 3: Using Invalid Method
If you specify a method other than minmax or standard, column_scaler() raises ValueError.
# Pass invalid method to column scaler
try:
column_scaler(data, column="price", method="invalid_method")
except ValueError as e:
print(e)
Invalid method. Method should be `minmax` or `standard`.
Error Case 4: Using Invalid new_min and new_max Values
For min-max scaling, if the new_min is greater than new_max, column_scaler() raises a ValueError.
# Pass new_min greater than new_max
try:
column_scaler(data, column="price", method="minmax", new_min=10, new_max=5)
except ValueError as e:
print(e)
`new_min` cannot be greater than `new_max`.
/home/docs/checkouts/readthedocs.org/user_builds/datamop/checkouts/latest/src/datamop/column_scaler.py:86: UserWarning: NaN value detected in column 'price'. They will be unchanged
warnings.warn(
Encoding Columns using column_encoder()
The column_encoder() function encodes columns in a pandas DataFrame using one-hot or ordinal encoding based on user preferences. It accepts the DataFrame, a list of columns to encode, the encoding method (one-hot or ordinal), and an custom order for ordinal encoding. The function returns a new DataFrame with the specified columns encoded and provides robust error handling for invalid input types, missing columns, or mismatched category orders. It also issues warnings for potential issues like single unique values or missing data in the columns.
Let’s check how to use column_encoder():
Example 1: One hot encoding:
Let’s select our data for one hot encoding:
#choose our dataset
data_onehot = data.iloc[:,1:4].copy()
data_onehot.head()
| host_identity_verified | neighbourhood group | neighbourhood | |
|---|---|---|---|
| 0 | unconfirmed | Brooklyn | Kensington |
| 1 | verified | Manhattan | Midtown |
| 2 | NaN | Manhattan | Harlem |
| 3 | unconfirmed | Brooklyn | Clinton Hill |
| 4 | verified | Manhattan | East Harlem |
Let’s encode the column neighbourhood group and neighbourhood using method = 'onehot' :
columns = ['neighbourhood group', 'neighbourhood'] #columns to be encoded
encoded_data = column_encoder(data_onehot, columns, method='one-hot')
encoded_data.iloc[:,0:4] #show part of data after encoded
/home/docs/checkouts/readthedocs.org/user_builds/datamop/checkouts/latest/src/datamop/column_encoder.py:123: UserWarning: Missing values detected. They will be left as null
warnings.warn("Missing values detected. They will be left as null", UserWarning)
| host_identity_verified | neighbourhood group_Bronx | neighbourhood group_Brooklyn | neighbourhood group_Manhattan | |
|---|---|---|---|---|
| 0 | unconfirmed | 0 | 1 | 0 |
| 1 | verified | 0 | 0 | 1 |
| 2 | NaN | 0 | 0 | 1 |
| 3 | unconfirmed | 0 | 1 | 0 |
| 4 | verified | 0 | 0 | 1 |
| ... | ... | ... | ... | ... |
| 102594 | verified | 0 | 1 | 0 |
| 102595 | unconfirmed | 0 | 0 | 1 |
| 102596 | unconfirmed | 0 | 1 | 0 |
| 102597 | unconfirmed | 0 | 0 | 0 |
| 102598 | unconfirmed | 0 | 0 | 1 |
102599 rows × 4 columns
Example 2: Ordinal encoding:
Let’s select our data for ordinal encoding:
columns = ['id', 'cancellation_policy'] #columns to be encoded
data_ordinal = data[columns].copy().dropna() #dataset to be encoded
data_ordinal.head()
| id | cancellation_policy | |
|---|---|---|
| 0 | 1001254 | strict |
| 1 | 1002102 | moderate |
| 2 | 1002403 | flexible |
| 3 | 1002755 | moderate |
| 4 | 1003689 | moderate |
Let’s custom the ordinal order of cancellation_policy and encode the data using method = 'ordinal'
custom_order = {
'cancellation_policy': ['flexible', 'moderate', 'strict']
} #custom order, 'flexible' should be the lowest degree and 'strict' is the highest
encoded_data = column_encoder(data_ordinal, columns = ['cancellation_policy'], method='ordinal', order = custom_order)
encoded_data.head()
| id | cancellation_policy | |
|---|---|---|
| 0 | 1001254 | 2 |
| 1 | 1002102 | 1 |
| 2 | 1002403 | 0 |
| 3 | 1002755 | 1 |
| 4 | 1003689 | 1 |
Edge Case 1: 1 unique value for ordinal encoding
If there is only 1 unique value in the column specified for ordinal encoding, the function will raise a warning
one_value_data = data_ordinal.copy()
one_value_data['cancellation_policy'] = 'strict' #change all value to 'strict'
one_value_data.head()
| id | cancellation_policy | |
|---|---|---|
| 0 | 1001254 | strict |
| 1 | 1002102 | strict |
| 2 | 1002403 | strict |
| 3 | 1002755 | strict |
| 4 | 1003689 | strict |
custom_order = {
'cancellation_policy': ['strict']
} # order only have 1 degree
encoded_data = column_encoder(one_value_data, columns = ['cancellation_policy'], method='ordinal', order = custom_order)
encoded_data.head()
/home/docs/checkouts/readthedocs.org/user_builds/datamop/checkouts/latest/src/datamop/column_encoder.py:114: UserWarning: The column 'cancellation_policy' contains only one unique value
warnings.warn(f"The column '{column}' contains only one unique value", UserWarning)
| id | cancellation_policy | |
|---|---|---|
| 0 | 1001254 | 0 |
| 1 | 1002102 | 0 |
| 2 | 1002403 | 0 |
| 3 | 1002755 | 0 |
| 4 | 1003689 | 0 |
Edge Case 2: Missing value
If missing value is in the dataframe, should leave as null value.
missing_value_data = data_ordinal.copy()
missing_value_data.loc[2,'id'] = None # change a value in 'id' to null value
missing_value_data.head()
| id | cancellation_policy | |
|---|---|---|
| 0 | 1001254.0 | strict |
| 1 | 1002102.0 | moderate |
| 2 | NaN | flexible |
| 3 | 1002755.0 | moderate |
| 4 | 1003689.0 | moderate |
custom_order = {
'cancellation_policy': ['flexible', 'moderate', 'strict']
} #custom order
encoded_data = column_encoder(missing_value_data, columns = ['cancellation_policy'], method='ordinal', order = custom_order)
encoded_data.head()
/home/docs/checkouts/readthedocs.org/user_builds/datamop/checkouts/latest/src/datamop/column_encoder.py:123: UserWarning: Missing values detected. They will be left as null
warnings.warn("Missing values detected. They will be left as null", UserWarning)
| id | cancellation_policy | |
|---|---|---|
| 0 | 1001254.0 | 2 |
| 1 | 1002102.0 | 1 |
| 2 | NaN | 0 |
| 3 | 1002755.0 | 1 |
| 4 | 1003689.0 | 1 |
Edge Case 3: Empty dataframe
User input an empty dataframe, should output an empty dataframe
empty_df = pd.DataFrame() #create an empty dataframe
encoded_data = column_encoder(empty_df, columns = [], method='one-hot')
encoded_data.head()
Error Case 1: Set order while using one-hot encoding
User input dataframe, set method = 'one-hot', input order, the output should raise a value error
try:
column_encoder(data, columns = ['cancellation_policy'], method='one-hot', order = custom_order)
except ValueError as e:
print(e)
Order parameter is not applicable for method 'one-hot'
Error Case 2: Unmatch order and column value
User input all parameters, set method = 'ordinal', input order, but order does not match all unique values for the column, the output should raise a value error
ec2_df = data_ordinal.copy()
ec2_df['cancellation_policy'].unique() #check the unique values in the column
array(['strict', 'moderate', 'flexible'], dtype=object)
try:
custom_order = {
'cancellation_policy': ['flexible', 'moderate']
} # create an order without 'strict' value
encoded_data = column_encoder(data_ordinal, columns = ['cancellation_policy'], method='ordinal', order = custom_order)
encoded_data.head()
except ValueError as e:
print(e)
Order for column 'cancellation_policy' does not match its unique values
Error Case 3: Missing or wrong parameters
If a required parameter is missing or incorrect, the output should raise an error
try:
encoded_data = column_encoder(data_ordinal, columns = ['cancellation_policy'], method='ordinal') #create an 'ordinal' encoder but not specify 'order'
except ValueError as e:
print(e)
Order must be specified for ordinal encoding
try:
encoded_data = column_encoder(data_ordinal, columns = None, method='ordinal') #create an 'ordinal' encoder but not specify 'columns'
except TypeError as e:
print(e)
Columns parameter must be a list of strings
Error Case 4: columns does not match actual dataframe
If the user input order does not match the columns in the dataframe, the output should raise a KeyError
data_ordinal.columns ##the columns are 'id' and 'cancellation_policy'
Index(['id', 'cancellation_policy'], dtype='object')
try:
custom_order = {
'cancellation_policy': ['flexible', 'moderate', 'strict']
}
encoded_data = column_encoder(data_ordinal, columns = ['neighbourhood'], method='ordinal', order = custom_order)
# 'neighbourhood' is specified but it is not in the dataframe
except KeyError as e:
print(e)
"The column 'neighbourhood' is not in the dataframe"