{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# DataMop Tutorial\n",
    "\n",
    "Welcome to the tutorial for `datamop`, the ultimate Python package for cleaning and preparing your datasets with minimal effort. Data cleaning can often feel like the most tedious part of any data analysis or machine learning project. Missing values, inconsistent scales, and different data types can slow you down and distract from the real task: extracting insights from your data.\n",
    "\n",
    "That is where `datamop` package comes in! This powerful, easy-to-use package automates many of the common data cleaning tasks, like imputing missing values, encoding categorical features and scaling numerical features, saving you time and effort while ensuring your data is consistent, complete, and ready for analysis.\n",
    "\n",
    "Here we will show example usages for each function in the package, including `sweep_nulls`, `column_encoder`, and `column_scaler`. Your messy data will be ready to use after using this robust package. With `datamop`, you can focus more on analysis and less on tedious preprocessing. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports\n",
    "\n",
    "\n",
    "Before we get started, let's install and import the `datamop` package. We will demonstrate each functions in the `datamop` package with examples using the Airbnb Open Data from kaggle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import modules\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from datamop.sweep_nulls import sweep_nulls\n",
    "from datamop.column_encoder import column_encoder\n",
    "from datamop.column_scaler import column_scaler\n",
    "\n",
    "# import Airbnb Open Data\n",
    "data = pd.read_csv(\"../src/data/Airbnb_Open_Data.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pre-check column types\n",
    "Before imputing, scaling and encoding ensure that numerical columns are in the correct format. \n",
    "In the Airbnb dataset, the `price` and `service fee` columns are objects because they contain a `$` sign. \n",
    "We need to remove the `$` sign and convert these columns to floats. \n",
    "Additionally, we’ll demonstrate scaling on the `reviews per month` column, which is already a numeric column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th>service fee</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>966.0</td>\n",
       "      <td>193.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>142.0</td>\n",
       "      <td>28.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>620.0</td>\n",
       "      <td>124.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>368.0</td>\n",
       "      <td>74.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>204.0</td>\n",
       "      <td>41.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   price  service fee\n",
       "0  966.0        193.0\n",
       "1  142.0         28.0\n",
       "2  620.0        124.0\n",
       "3  368.0         74.0\n",
       "4  204.0         41.0"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Clean \"price\" and \"service fee\" columns by removing unwanted characters and converting to float\n",
    "data[\"price\"] = data[\"price\"].str.strip().str.replace(r\"[^0-9.]\", \"\", regex=True).astype(float)\n",
    "\n",
    "data[\"service fee\"] = data[\"service fee\"].str.strip().str.replace(r\"[^0-9.]\", \"\", regex=True).astype(float)\n",
    "\n",
    "# Verify the changes\n",
    "data[[\"price\", \"service fee\"]].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Handling missing values with `sweep_nulls()`\n",
    "\n",
    "One of the most common challenges in data cleaning process is dealing with missing values. `datamop` provides a convenient method called `sweep_nulls()` to help you handle these issues effortlessly. The `sweep_nulls()` function scans your dataset for missing values and allows you to handle them using various strategies, including 'mean'(numeric only), 'median'(numeric only), 'mode', 'constant', and 'drop'.\n",
    "\n",
    "Let's start by checking the missing values in the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 102599 entries, 0 to 102598\n",
      "Data columns (total 20 columns):\n",
      " #   Column                          Non-Null Count   Dtype  \n",
      "---  ------                          --------------   -----  \n",
      " 0   id                              102599 non-null  int64  \n",
      " 1   host_identity_verified          102310 non-null  object \n",
      " 2   neighbourhood group             102570 non-null  object \n",
      " 3   neighbourhood                   102583 non-null  object \n",
      " 4   lat                             102591 non-null  float64\n",
      " 5   long                            102591 non-null  float64\n",
      " 6   country                         102067 non-null  object \n",
      " 7   instant_bookable                102494 non-null  object \n",
      " 8   cancellation_policy             102523 non-null  object \n",
      " 9   room type                       102599 non-null  object \n",
      " 10  Construction year               102385 non-null  float64\n",
      " 11  price                           102352 non-null  float64\n",
      " 12  service fee                     102326 non-null  float64\n",
      " 13  minimum nights                  102190 non-null  float64\n",
      " 14  number of reviews               102416 non-null  float64\n",
      " 15  last review                     86706 non-null   object \n",
      " 16  reviews per month               86720 non-null   float64\n",
      " 17  review rate number              102273 non-null  float64\n",
      " 18  calculated host listings count  102280 non-null  float64\n",
      " 19  availability 365                102151 non-null  float64\n",
      "dtypes: float64(11), int64(1), object(8)\n",
      "memory usage: 15.7+ MB\n"
     ]
    }
   ],
   "source": [
    "data.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id                                    0\n",
       "host_identity_verified              289\n",
       "neighbourhood group                  29\n",
       "neighbourhood                        16\n",
       "lat                                   8\n",
       "long                                  8\n",
       "country                             532\n",
       "instant_bookable                    105\n",
       "cancellation_policy                  76\n",
       "room type                             0\n",
       "Construction year                   214\n",
       "price                               247\n",
       "service fee                         273\n",
       "minimum nights                      409\n",
       "number of reviews                   183\n",
       "last review                       15893\n",
       "reviews per month                 15879\n",
       "review rate number                  326\n",
       "calculated host listings count      319\n",
       "availability 365                    448\n",
       "dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Imputing all columns\n",
    "\n",
    "When dealing with datasets containing missing values across multiple columns, `sweep_nulls()`makes it easy to impute all columns simultaneously. This feature ensures consistent handling of missing data throughout the dataset, whether you’re using the mean, median, mode, or a custom value for imputation.\n",
    "\n",
    "Since 'mean' and 'median' are designed for numerical features only, it is better to use 'mode', 'constant' or 'drop' when you have mixed data types in the dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/yichi/Documents/Block4/DSCI_524/DataMop_package_group14/src/datamop/sweep_nulls.py:68: UserWarning: Columns list is empty. Applying strategy to all columns.\n",
      "  warnings.warn(\"Columns list is empty. Applying strategy to all columns.\", UserWarning)\n",
      "/Users/yichi/Documents/Block4/DSCI_524/DataMop_package_group14/src/datamop/sweep_nulls.py:116: FutureWarning: Downcasting object dtype arrays on .fillna, .ffill, .bfill is deprecated and will change in a future version. Call result.infer_objects(copy=False) instead. To opt-in to the future behavior, set `pd.set_option('future.no_silent_downcasting', True)`\n",
      "  data[column] = data[column].fillna(data[column].mode()[0])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "id                                0\n",
       "host_identity_verified            0\n",
       "neighbourhood group               0\n",
       "neighbourhood                     0\n",
       "lat                               0\n",
       "long                              0\n",
       "country                           0\n",
       "instant_bookable                  0\n",
       "cancellation_policy               0\n",
       "room type                         0\n",
       "Construction year                 0\n",
       "price                             0\n",
       "service fee                       0\n",
       "minimum nights                    0\n",
       "number of reviews                 0\n",
       "last review                       0\n",
       "reviews per month                 0\n",
       "review rate number                0\n",
       "calculated host listings count    0\n",
       "availability 365                  0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# using mode to impute missing value with the most common values in the column\n",
    "data_fill_mode = data.copy()\n",
    "sweep_nulls(data_fill_mode, strategy='mode')\n",
    "data_fill_mode.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/yichi/Documents/Block4/DSCI_524/DataMop_package_group14/src/datamop/sweep_nulls.py:68: UserWarning: Columns list is empty. Applying strategy to all columns.\n",
      "  warnings.warn(\"Columns list is empty. Applying strategy to all columns.\", UserWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0    unconfirmed\n",
       "1       verified\n",
       "2           -999\n",
       "3    unconfirmed\n",
       "4       verified\n",
       "Name: host_identity_verified, dtype: object"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# using constant to impute missing value with a number\n",
    "data_fill_number = data.copy()\n",
    "sweep_nulls(data_fill_number, strategy='constant', fill_value = -999)\n",
    "data_fill_number['host_identity_verified'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Imputing specific numerical columns\n",
    "\n",
    "If you want to focus on imputing missing values in specific numerical columns of your dataset without affecting other columns, you can achieve this by using `sweep_nulls()` to select the desired columns and apply an imputation strategy only to them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before impute: nan\n",
      "Mean Price:  625.2935360325152\n",
      "After impute: 625.2935360325152\n"
     ]
    }
   ],
   "source": [
    "# using mean to impute price and service fee columns\n",
    "data_impute_mean = data.copy()\n",
    "data_impute_mean = sweep_nulls(data_impute_mean, strategy='mean', columns=['price', 'service fee'])\n",
    "data_impute_mean['price'].iloc[147]\n",
    "print(\"Before impute:\", data['price'].iloc[147])\n",
    "print(\"Mean Price: \", data['price'].mean())\n",
    "print(\"After impute:\", data_impute_mean['price'].iloc[147])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before impute: nan\n",
      "After impute: -999.0\n"
     ]
    }
   ],
   "source": [
    "# using constant to impute price and service fee columns with a negative number\n",
    "data_impute_constant = data.copy()\n",
    "data_impute_constant = sweep_nulls(data_impute_constant, strategy='constant', columns=['price', 'service fee'], fill_value=-999)\n",
    "print(\"Before impute:\", data['price'].iloc[147])\n",
    "print(\"After impute:\", data_impute_constant['price'].iloc[147])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Imputing specific categorical columns\n",
    "\n",
    "When working with datasets containing missing values in categorical columns, you can impute missing values in specific categorical columns using common strategies like filling with the mode, or a custom value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['unconfirmed', 'verified', 'missing'], dtype=object)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# using constant to impute missing value with a string\n",
    "data_cate = data.copy()\n",
    "data_cate = sweep_nulls(data_cate, strategy='constant', columns=['host_identity_verified'], fill_value='missing')\n",
    "data_cate['host_identity_verified'].unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before impute: nan\n",
      "Mode:  0    United States\n",
      "Name: country, dtype: object\n",
      "After impute: United States\n"
     ]
    }
   ],
   "source": [
    "# using mode to impute missing value with the most common values in the column\n",
    "data_cate_mode = data.copy()\n",
    "data_cate_mode = sweep_nulls(data_cate_mode, strategy='mode', columns=['country'])\n",
    "print(\"Before impute:\", data['country'].iloc[156])\n",
    "print(\"Mode: \", data['country'].mode())\n",
    "print(\"After impute:\", data_cate_mode['country'].iloc[156])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dropping columns\n",
    "\n",
    "When working with datasets, some columns may have excessive missing values, which makes them unhelpful for analysis. Imputing them can introduce noise, therefore `sweep_nulls()` allows you to drop missing values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/yichi/Documents/Block4/DSCI_524/DataMop_package_group14/src/datamop/sweep_nulls.py:68: UserWarning: Columns list is empty. Applying strategy to all columns.\n",
      "  warnings.warn(\"Columns list is empty. Applying strategy to all columns.\", UserWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "id                                0\n",
       "host_identity_verified            0\n",
       "neighbourhood group               0\n",
       "neighbourhood                     0\n",
       "lat                               0\n",
       "long                              0\n",
       "country                           0\n",
       "instant_bookable                  0\n",
       "cancellation_policy               0\n",
       "room type                         0\n",
       "Construction year                 0\n",
       "price                             0\n",
       "service fee                       0\n",
       "minimum nights                    0\n",
       "number of reviews                 0\n",
       "last review                       0\n",
       "reviews per month                 0\n",
       "review rate number                0\n",
       "calculated host listings count    0\n",
       "availability 365                  0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# dropping one column\n",
    "data_drop = data.copy()\n",
    "data_drop = sweep_nulls(data_drop, strategy='drop')\n",
    "data_drop.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scaling Numerical Features with `column_scaler()`\n",
    "\n",
    "When working with numerical data, inconsistent scales can distort analysis or machine learning results. \n",
    "For example, a column measuring `price` in thousands might dominate another column measuring `rating` on a 1-5 scale. \n",
    "To avoid this issue, scaling the numerical data to a consistent range or distribution can mitigate this problem.\n",
    "\n",
    "The `column_scaler()` function in the `datamop` package allows users to scale any numeric column in a dataset. It supports two methods:\n",
    "- **Min-Max Scaling**: Scale values to a specific range, such as `[0, 1]` or `[10, 20]`.\n",
    "- **Standard Scaling**: Transform values to have a mean of `0` and a standard deviation of `1`.\n",
    "\n",
    "The `column_scaler()` function allows flexible usage for both in-place scaling (replacing the original column) and creating a new scaled column.\n",
    "\n",
    "Let’s walk through how to use `column_scaler()`.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 1: Min-Max Scaling\n",
    "\n",
    "Let’s scale the `reviews per month` column to a range between 0 and 1 using min-max scaling.\\\n",
    "The scaled values will replace the original column (`inplace=True`).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>reviews per month</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.002222</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.004112</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.051450</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.001000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.006445</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   reviews per month\n",
       "0           0.002222\n",
       "1           0.004112\n",
       "3           0.051450\n",
       "4           0.001000\n",
       "5           0.006445"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Using min-max scaling to scale \"reviews per month\" to a range between 0 and 1\n",
    "data_minmax = data.copy().dropna()\n",
    "column_scaler(data_minmax, column=\"reviews per month\", method=\"minmax\", new_min=0, new_max=1, inplace=True)\n",
    "\n",
    "# Verify the scaled column\n",
    "data_minmax[[\"reviews per month\"]].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2: Custom Min-Max Scaling with a New Column\n",
    "\n",
    "Now let’s scale the `price` column to a range between 100 and 500. Instead of modifying the original column, we’ll create a new column called `price_scaled` by setting `inplace=False`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/yichi/Documents/Block4/DSCI_524/DataMop_package_group14/src/datamop/column_scaler.py:83: UserWarning: NaN value detected in column '{column}'. They will be unchanged\n",
      "  warnings.warn(\"NaN value detected in column '{column}'. They will be unchanged\", UserWarning)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "      <th>price_scaled</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>966.0</td>\n",
       "      <td>418.608696</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>142.0</td>\n",
       "      <td>132.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>620.0</td>\n",
       "      <td>298.260870</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>368.0</td>\n",
       "      <td>210.608696</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>204.0</td>\n",
       "      <td>153.565217</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   price  price_scaled\n",
       "0  966.0    418.608696\n",
       "1  142.0    132.000000\n",
       "2  620.0    298.260870\n",
       "3  368.0    210.608696\n",
       "4  204.0    153.565217"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Using min-max scaling to scale \"price\" to a range between 100 and 500\n",
    "column_scaler(data, column=\"price\", method=\"minmax\", new_min=100, new_max=500, inplace=False)\n",
    "\n",
    "# Verify the new scaled column\n",
    "data[[\"price\", \"price_scaled\"]].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 3: Standard Scaling\n",
    "\n",
    "Let's scale the `service fee` column using the standard scaling method, \n",
    "which transforms the values to have a mean of 0 and standard deviation of 1.\n",
    "The scaled values will replace the original column (`incplace=True`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/yichi/Documents/Block4/DSCI_524/DataMop_package_group14/src/datamop/column_scaler.py:83: UserWarning: NaN value detected in column '{column}'. They will be unchanged\n",
      "  warnings.warn(\"NaN value detected in column '{column}'. They will be unchanged\", UserWarning)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>service fee</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.024837</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1.462885</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.015483</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.769338</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>-1.266883</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   service fee\n",
       "0     1.024837\n",
       "1    -1.462885\n",
       "2    -0.015483\n",
       "3    -0.769338\n",
       "4    -1.266883"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Using standard scaling method on \"service fee\" column\n",
    "column_scaler(data, column=\"service fee\", method=\"standard\", inplace=True)\n",
    "\n",
    "# Verify the scaled column\n",
    "data[[\"service fee\"]].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Edge Case 1: Scaling Column with Single Unique Value\n",
    "\n",
    "If a column contains only a single unique value, `column_scaler()` automatically assigns the midpoint of the range for min-max scaling and issues a warning message."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/yichi/Documents/Block4/DSCI_524/DataMop_package_group14/src/datamop/column_scaler.py:86: UserWarning: Single-value column detected. All values will be scaled to the midpoint of the `new_min` and `new_max`.\n",
      "  warnings.warn(\"Single-value column detected. All values will be scaled to the midpoint of the `new_min` and `new_max`.\", UserWarning)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>price</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   price\n",
       "0    0.5\n",
       "1    0.5\n",
       "2    0.5"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create DataFrame with a single-value column\n",
    "single_value_df = pd.DataFrame({\"price\": [100, 100, 100]})\n",
    "\n",
    "# Scale the column using min-max scaling\n",
    "scaled_df = column_scaler(single_value_df, column=\"price\", method=\"minmax\", new_min=0, new_max=1)\n",
    "\n",
    "# Verify the result\n",
    "scaled_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Edge Case 2: Handling Missing Values (NaN)\n",
    "\n",
    "If a column contains missing values (`NaN`), `column_scaler()` leaves them unchanged and issue a warning. This ensures no data is lost or imputed incorrectly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/yichi/Documents/Block4/DSCI_524/DataMop_package_group14/src/datamop/column_scaler.py:83: UserWarning: NaN value detected in column '{column}'. They will be unchanged\n",
      "  warnings.warn(\"NaN value detected in column '{column}'. They will be unchanged\", UserWarning)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>reviews per month</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   reviews per month\n",
       "0                0.0\n",
       "1                NaN\n",
       "2                1.0"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create a DataFrame with NaN values\n",
    "nan_df = pd.DataFrame({\"reviews per month\": [10, np.nan, 20]})\n",
    "\n",
    "# Scaled the column using min-max scaling\n",
    "scaled_nan_df = column_scaler(nan_df, column=\"reviews per month\", method=\"minmax\", new_min=0, new_max=1)\n",
    "\n",
    "# Verify the result\n",
    "scaled_nan_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Error Case 1: Using Non-Existent Column\n",
    "\n",
    "If the specified column does not exist in the DataFrame, `column_scaler()` raises a `KeyError`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'Column not found in the DataFrame.'\n"
     ]
    }
   ],
   "source": [
    "# Pass non existent column in the 'column' argument\n",
    "try:\n",
    "    column_scaler(data, column=\"Non_existent\", method=\"minmax\")\n",
    "except KeyError as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Error Case 2: Using Non-Numeric Columns\n",
    "\n",
    "If you attempt to scale a non-numeric column, such column of strings, `column_scaler()` raises a `ValueError`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Column must have numeric values.\n"
     ]
    }
   ],
   "source": [
    "# Pass column of objects (country column) to column scaler\n",
    "try:\n",
    "    column_scaler(data, column=\"country\", method=\"minmax\")\n",
    "except ValueError as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Error Case 3: Using Invalid Method\n",
    "\n",
    "If you specify a method other than `minmax` or `standard`, `column_scaler()` raises `ValueError`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Invalid method. Method should be `minmax` or `standard`.\n"
     ]
    }
   ],
   "source": [
    "# Pass invalid method to column scaler\n",
    "try:\n",
    "    column_scaler(data, column=\"price\", method=\"invalid_method\")\n",
    "except ValueError as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Error Case 4: Using Invalid `new_min` and `new_max` Values\n",
    "\n",
    "For min-max scaling, if the `new_min` is greater than `new_max`, `column_scaler()` raises a `ValueError`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "`new_min` cannot be greater than `new_max`.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/yichi/Documents/Block4/DSCI_524/DataMop_package_group14/src/datamop/column_scaler.py:83: UserWarning: NaN value detected in column '{column}'. They will be unchanged\n",
      "  warnings.warn(\"NaN value detected in column '{column}'. They will be unchanged\", UserWarning)\n"
     ]
    }
   ],
   "source": [
    "# Pass new_min greater than new_max\n",
    "try:\n",
    "    column_scaler(data, column=\"price\", method=\"minmax\", new_min=10, new_max=5)\n",
    "except ValueError as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Encoding Columns using `column_encoder()`\n",
    "\n",
    "The `column_encoder()` function encodes columns in a pandas DataFrame using one-hot or ordinal encoding based on user preferences. It accepts the DataFrame, a list of columns to encode, the encoding method (`one-hot` or `ordinal`), and an custom order for ordinal encoding. The function returns a new DataFrame with the specified columns encoded and provides robust error handling for invalid input types, missing columns, or mismatched category orders. It also issues warnings for potential issues like single unique values or missing data in the columns.\n",
    "\n",
    "Let's check how to use column_encoder():\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 1: One hot encoding:\n",
    "Let's select our data for one hot encoding:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>host_identity_verified</th>\n",
       "      <th>neighbourhood group</th>\n",
       "      <th>neighbourhood</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>unconfirmed</td>\n",
       "      <td>Brooklyn</td>\n",
       "      <td>Kensington</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>verified</td>\n",
       "      <td>Manhattan</td>\n",
       "      <td>Midtown</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>Manhattan</td>\n",
       "      <td>Harlem</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>unconfirmed</td>\n",
       "      <td>Brooklyn</td>\n",
       "      <td>Clinton Hill</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>verified</td>\n",
       "      <td>Manhattan</td>\n",
       "      <td>East Harlem</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  host_identity_verified neighbourhood group neighbourhood\n",
       "0            unconfirmed            Brooklyn    Kensington\n",
       "1               verified           Manhattan       Midtown\n",
       "2                    NaN           Manhattan        Harlem\n",
       "3            unconfirmed            Brooklyn  Clinton Hill\n",
       "4               verified           Manhattan   East Harlem"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#choose our dataset\n",
    "data_onehot = data.iloc[:,1:4].copy()\n",
    "data_onehot.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's encode the column `neighbourhood group` and `neighbourhood` using `method = 'onehot'` :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/yichi/Documents/Block4/DSCI_524/DataMop_package_group14/src/datamop/column_encoder.py:124: UserWarning: Missing values detected. They will be left as null\n",
      "  warnings.warn(\"Missing values detected. They will be left as null\", UserWarning)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>host_identity_verified</th>\n",
       "      <th>neighbourhood group_Bronx</th>\n",
       "      <th>neighbourhood group_Brooklyn</th>\n",
       "      <th>neighbourhood group_Manhattan</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>unconfirmed</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>verified</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>unconfirmed</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>verified</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102594</th>\n",
       "      <td>verified</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102595</th>\n",
       "      <td>unconfirmed</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102596</th>\n",
       "      <td>unconfirmed</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102597</th>\n",
       "      <td>unconfirmed</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>102598</th>\n",
       "      <td>unconfirmed</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>102599 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       host_identity_verified  neighbourhood group_Bronx  \\\n",
       "0                 unconfirmed                          0   \n",
       "1                    verified                          0   \n",
       "2                         NaN                          0   \n",
       "3                 unconfirmed                          0   \n",
       "4                    verified                          0   \n",
       "...                       ...                        ...   \n",
       "102594               verified                          0   \n",
       "102595            unconfirmed                          0   \n",
       "102596            unconfirmed                          0   \n",
       "102597            unconfirmed                          0   \n",
       "102598            unconfirmed                          0   \n",
       "\n",
       "        neighbourhood group_Brooklyn  neighbourhood group_Manhattan  \n",
       "0                                  1                              0  \n",
       "1                                  0                              1  \n",
       "2                                  0                              1  \n",
       "3                                  1                              0  \n",
       "4                                  0                              1  \n",
       "...                              ...                            ...  \n",
       "102594                             1                              0  \n",
       "102595                             0                              1  \n",
       "102596                             1                              0  \n",
       "102597                             0                              0  \n",
       "102598                             0                              1  \n",
       "\n",
       "[102599 rows x 4 columns]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "columns = ['neighbourhood group', 'neighbourhood'] #columns to be encoded\n",
    "encoded_data = column_encoder(data_onehot, columns, method='one-hot')\n",
    "encoded_data.iloc[:,0:4] #show part of data after encoded"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example 2: Ordinal encoding:\n",
    "Let's select our data for `ordinal` encoding:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>cancellation_policy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1001254</td>\n",
       "      <td>strict</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1002102</td>\n",
       "      <td>moderate</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1002403</td>\n",
       "      <td>flexible</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1002755</td>\n",
       "      <td>moderate</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1003689</td>\n",
       "      <td>moderate</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id cancellation_policy\n",
       "0  1001254              strict\n",
       "1  1002102            moderate\n",
       "2  1002403            flexible\n",
       "3  1002755            moderate\n",
       "4  1003689            moderate"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "columns = ['id', 'cancellation_policy'] #columns to be encoded\n",
    "data_ordinal = data[columns].copy().dropna() #dataset to be encoded\n",
    "data_ordinal.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's custom the ordinal order of `cancellation_policy` and encode the data using `method = 'ordinal'`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>cancellation_policy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1001254</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1002102</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1002403</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1002755</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1003689</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id  cancellation_policy\n",
       "0  1001254                    2\n",
       "1  1002102                    1\n",
       "2  1002403                    0\n",
       "3  1002755                    1\n",
       "4  1003689                    1"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "custom_order = {\n",
    "    'cancellation_policy': ['flexible', 'moderate', 'strict']\n",
    "} #custom order, 'flexible' should be the lowest degree and 'strict' is the highest\n",
    "encoded_data = column_encoder(data_ordinal, columns = ['cancellation_policy'], method='ordinal', order = custom_order)\n",
    "encoded_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Edge Case 1: 1 unique value for ordinal encoding\n",
    "If there is only 1 unique value in the column specified for `ordinal` encoding, the function will raise a warning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>cancellation_policy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1001254</td>\n",
       "      <td>strict</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1002102</td>\n",
       "      <td>strict</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1002403</td>\n",
       "      <td>strict</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1002755</td>\n",
       "      <td>strict</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1003689</td>\n",
       "      <td>strict</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id cancellation_policy\n",
       "0  1001254              strict\n",
       "1  1002102              strict\n",
       "2  1002403              strict\n",
       "3  1002755              strict\n",
       "4  1003689              strict"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "one_value_data = data_ordinal.copy()\n",
    "one_value_data['cancellation_policy'] = 'strict' #change all value to 'strict'\n",
    "one_value_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/yichi/Documents/Block4/DSCI_524/DataMop_package_group14/src/datamop/column_encoder.py:115: UserWarning: The column 'cancellation_policy' contains only one unique value\n",
      "  warnings.warn(f\"The column '{column}' contains only one unique value\", UserWarning)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>cancellation_policy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1001254</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1002102</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1002403</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1002755</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1003689</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id  cancellation_policy\n",
       "0  1001254                    0\n",
       "1  1002102                    0\n",
       "2  1002403                    0\n",
       "3  1002755                    0\n",
       "4  1003689                    0"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "custom_order = {\n",
    "    'cancellation_policy': ['strict']\n",
    "} # order only have 1 degree\n",
    "encoded_data = column_encoder(one_value_data, columns = ['cancellation_policy'], method='ordinal', order = custom_order)\n",
    "encoded_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Edge Case 2: Missing value\n",
    "If missing value is in the dataframe, should leave as null value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>cancellation_policy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1001254.0</td>\n",
       "      <td>strict</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1002102.0</td>\n",
       "      <td>moderate</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>flexible</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1002755.0</td>\n",
       "      <td>moderate</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1003689.0</td>\n",
       "      <td>moderate</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          id cancellation_policy\n",
       "0  1001254.0              strict\n",
       "1  1002102.0            moderate\n",
       "2        NaN            flexible\n",
       "3  1002755.0            moderate\n",
       "4  1003689.0            moderate"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "missing_value_data = data_ordinal.copy()\n",
    "missing_value_data.loc[2,'id'] = None # change a value in 'id' to null value\n",
    "missing_value_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/yichi/Documents/Block4/DSCI_524/DataMop_package_group14/src/datamop/column_encoder.py:124: UserWarning: Missing values detected. They will be left as null\n",
      "  warnings.warn(\"Missing values detected. They will be left as null\", UserWarning)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>cancellation_policy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1001254.0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1002102.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1002755.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1003689.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          id  cancellation_policy\n",
       "0  1001254.0                    2\n",
       "1  1002102.0                    1\n",
       "2        NaN                    0\n",
       "3  1002755.0                    1\n",
       "4  1003689.0                    1"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "custom_order = {\n",
    "    'cancellation_policy': ['flexible', 'moderate', 'strict']\n",
    "} #custom order\n",
    "encoded_data = column_encoder(missing_value_data, columns = ['cancellation_policy'], method='ordinal', order = custom_order)\n",
    "encoded_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Edge Case 3: Empty dataframe\n",
    "User input an empty dataframe, should output an empty dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: []\n",
       "Index: []"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "empty_df = pd.DataFrame() #create an empty dataframe\n",
    "encoded_data = column_encoder(empty_df, columns = [], method='one-hot')\n",
    "encoded_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Error Case 1: Set `order` while using `one-hot` encoding\n",
    "User input dataframe, set `method = 'one-hot'`, input order, the output should raise a value error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Order parameter is not applicable for method 'one-hot'\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    column_encoder(data, columns = ['cancellation_policy'], method='one-hot', order = custom_order)\n",
    "except ValueError as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Error Case 2: Unmatch `order` and column value\n",
    "User input all parameters, set `method = 'ordinal'`, input order, but `order` does not match all unique values for the column, the output should raise a value error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['strict', 'moderate', 'flexible'], dtype=object)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ec2_df = data_ordinal.copy()\n",
    "ec2_df['cancellation_policy'].unique() #check the unique values in the column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Order for column 'cancellation_policy' does not match its unique values\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    custom_order = {\n",
    "        'cancellation_policy': ['flexible', 'moderate']\n",
    "    } # create an order without 'strict' value\n",
    "    encoded_data = column_encoder(data_ordinal, columns = ['cancellation_policy'], method='ordinal', order = custom_order)\n",
    "    encoded_data.head()\n",
    "except ValueError as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Error Case 3: Missing or wrong parameters\n",
    "If a required parameter is missing or incorrect, the output should raise an error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Order must be specified for ordinal encoding\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    encoded_data = column_encoder(data_ordinal, columns = ['cancellation_policy'], method='ordinal') #create an 'ordinal' encoder but not specify 'order'\n",
    "except ValueError as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Columns parameter must be a list of strings\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    encoded_data = column_encoder(data_ordinal, columns = None, method='ordinal') #create an 'ordinal' encoder but not specify 'columns'\n",
    "except TypeError as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Error Case 4: `columns` does not match actual dataframe\n",
    "If the user input order does not match the columns in the dataframe, the output should raise a KeyError"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['id', 'cancellation_policy'], dtype='object')"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_ordinal.columns ##the columns are 'id' and 'cancellation_policy'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"The column 'neighbourhood' is not in the dataframe\"\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    custom_order = {\n",
    "        'cancellation_policy': ['flexible', 'moderate', 'strict']\n",
    "    }\n",
    "    encoded_data = column_encoder(data_ordinal, columns = ['neighbourhood'], method='ordinal', order = custom_order) \n",
    "    # 'neighbourhood' is specified but it is not in the dataframe\n",
    "except KeyError as e:\n",
    "    print(e)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}