Notebooks from applied-cs/data-science@e5e58ebe

b88d83ec · Christof Kaufmann · c4bac8b7 · b88d83ec · b88d83ec · b88d83ec
Commit b88d83ec authored Jun 12, 2024 by Christof Kaufmann
--- a/07-imputer/01-aim-sol.ipynb
+++ b/07-imputer/01-aim-sol.ipynb
--- a/07-imputer/01-aim.ipynb
+++ b/07-imputer/01-aim.ipynb
--- a/07-imputer/03-impute-train-sol.ipynb
+++ b/07-imputer/03-impute-train-sol.ipynb
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Fehler mit und ohne Imputer für die Trainingsmenge\n",
+    "\n",
+    "Das Laden der Daten ist bereits vorbereitet:"
+   ],
+   "id": "0001-9a691649a9301a1cb609d038b68e1a84d6a0d2120ff69ca180381b5e12c"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "style": "python"
+   },
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "import pandas as pd\n",
+    "from sklearn.neighbors import KNeighborsRegressor\n",
+    "from sklearn.impute import SimpleImputer, KNNImputer\n",
+    "from sklearn.model_selection import train_test_split\n",
+    "from sklearn.metrics import mean_absolute_error, mean_squared_error\n",
+    "from sklearn.preprocessing import robust_scale\n",
+    "\n",
+    "df = pd.read_csv('nations.csv')\n",
+    "\n",
+    "# convert income via categorical to range\n",
+    "income_vals = ['Low income', 'Lower middle income', 'Upper middle income', 'High income', 'High income: OECD']\n",
+    "income_cat_dtype = pd.api.types.CategoricalDtype(categories=income_vals, ordered=True)\n",
+    "df.income = df.income.astype(income_cat_dtype).cat.codes\n",
+    "\n",
+    "# drop samples with missing values in features ['life_expect', 'population', 'birth_rate']\n",
+    "# also drop samples when values are missing in both gdp_percap AND neonat_mortal_rate\n",
+    "core_feat = ['year', 'population', 'birth_rate', 'income']\n",
+    "incomplete_feat = ['gdp_percap', 'neonat_mortal_rate']\n",
+    "target = 'life_expect'\n",
+    "\n",
+    "drop_idx = (df.gdp_percap.isna() & df.neonat_mortal_rate.isna()) | df[core_feat + [target]].isna().any(axis='columns')\n",
+    "df = df[~drop_idx]\n",
+    "\n",
+    "# split into X and y. X has only missing values in incomplete_feat\n",
+    "X = df[core_feat + incomplete_feat]\n",
+    "y = df[target]\n",
+    "\n",
+    "# scale X, otherwise errors are huge\n",
+    "X = pd.DataFrame(robust_scale(X), index=X.index, columns=X.columns)"
+   ],
+   "id": "0002-e5c489be6d21762cddcba2d304863bbb413036df68019bdcd674ec10ac9"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Nun haben Sie `X` und `y`, aber in `X` gibt es fehlende Werte. Wir\n",
+    "definieren eine Testmenge. Wir versuchen zunächst herauszufinden wie\n",
+    "groß der Fehler ist, wenn wir Samples mit fehlenden Werten wegwerfen.\n",
+    "Ihre Aufgabe ist nun auf den vollständigen Samples (also ohne fehlende\n",
+    "Werte), die nicht in der Testmenge sind, ein `KNeighborsRegressor` zu\n",
+    "trainieren und den Fehler (MAE oder RMSE) auszuwerten."
+   ],
+   "id": "0003-14f89fcb50a1fa0d36c91cbe2ff6f54399a587a48a1e812c35f7ab39c90"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "style": "python"
+   },
+   "outputs": [],
+   "source": [
+    "X_cmpl = X.dropna()\n",
+    "y_cmpl = y[X_cmpl.index]\n",
+    "\n",
+    "# split complete samples, use X_test, y_test as common test set also for model trained on imputed values\n",
+    "X_cmpl_train, X_test, y_cmpl_train, y_test = train_test_split(X_cmpl, y_cmpl, test_size=0.2)\n",
+    "\n",
+    "# TODO Exercise 1: train model using complete samples only and evaluate on test set"
+   ],
+   "id": "0004-2e3600b5dc70548bce9012d280aeced64a2b7b2767f0fd03adcd6ffc0c4"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Lösung"
+   ],
+   "id": "0005-94702981911ec4f24f72794f610c502911e5197963070b964d3f3c40038"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "style": "python"
+   },
+   "outputs": [],
+   "source": [
+    "model = KNeighborsRegressor(weights='distance')\n",
+    "model.fit(X_cmpl_train, y_cmpl_train)\n",
+    "y_cmpl_pred = model.predict(X_test)\n",
+    "error = y_cmpl_pred - y_test\n",
+    "mae_cmpl = error.abs().mean()\n",
+    "rmse_cmpl = np.sqrt((error ** 2).mean())"
+   ],
+   "id": "0006-32fe21e10a5bd61d64245e7dec10ffda2ff142d8df6e1b6485f6dc72039"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Als nächstes füllen Sie `X` mit einem simplen Imputer (Mittelwert oder\n",
+    "Median), trainieren nun auf allen Samples, die nicht in der Testmenge\n",
+    "sind, (also auch gefüllte Samples) und werten auf derselben Testmenge\n",
+    "wie zuvor den Fehler aus."
+   ],
+   "id": "0007-e185b61321b945aa97a3f14f62d97f3a2ca2be544052b7ec2f8ea6ced4b"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "style": "python"
+   },
+   "outputs": [],
+   "source": [
+    "X_fill_simple = X.copy()\n",
+    "\n",
+    "# TODO Exercise 2a: impute gdp_percap and neonat_mortal_rate using median or mean\n",
+    "\n",
+    "# TODO Exercise 2b: train model using all samples except test samples and evaluate on test set"
+   ],
+   "id": "0008-4e70cd81b62eb4618125b6b86b08c85de9b3d03f941f397c85d2f743158"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Lösung"
+   ],
+   "id": "0009-5cdeedac3f86c044a9eecadeacc85d72a11512f10a7bb2ec56605e6297a"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "style": "python"
+   },
+   "outputs": [],
+   "source": [
+    "X_fill_simple = X_fill_simple.fillna(X.mean())\n",
+    "model.fit(X_fill_simple.drop(X_test.index), y.drop(y_test.index))\n",
+    "y_simple_pred = model.predict(X_test)\n",
+    "error = y_simple_pred - y_test\n",
+    "mae_simple = error.abs().mean()\n",
+    "rmse_simple = np.sqrt((error ** 2).mean())"
+   ],
+   "id": "0010-a24aeb2fa22291b46a17ecf1046fcc2de4cedd92d91d3f124906aea28b0"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Zuletzt machen Sie nochmals dasselbe, aber mit einem $k$ Nearest\n",
+    "Neighbor Imputer."
+   ],
+   "id": "0011-fe55307b85eff61e6939d7ea4da2f6afc056088b6ba8d51af46ab7ecfea"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "style": "python"
+   },
+   "outputs": [],
+   "source": [
+    "X_fill_knn = X.copy()\n",
+    "\n",
+    "# TODO Exercise 3a: impute gdp_percap and neonat_mortal_rate using KNNImputer\n",
+    "\n",
+    "# TODO Exercise 3b: train model using all samples except test samples and evaluate on test set"
+   ],
+   "id": "0012-1762321c7a3e5806dc42d23642eff7abea763e94d9a2a305139813d05c7"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Lösung"
+   ],
+   "id": "0013-994ce0eff56ef92a6dc375c0d9b4843f6d074065a8f9ff6890853b48681"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "style": "python"
+   },
+   "outputs": [],
+   "source": [
+    "X_fill_knn = KNNImputer().fit_transform(X)\n",
+    "X_fill_knn = pd.DataFrame(X_fill_knn, index=X.index, columns=X.columns)\n",
+    "model.fit(X_fill_knn.drop(X_test.index), y.drop(y_test.index))\n",
+    "y_knn_pred = model.predict(X_test)\n",
+    "error = y_knn_pred - y_test\n",
+    "mae_knn = error.abs().mean()\n",
+    "rmse_knn = np.sqrt((error ** 2).mean())"
+   ],
+   "id": "0014-2d200e90dba955e7b70e8c641c626e0d6d2602cce67568de6f37f06c109"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Auswertung:"
+   ],
+   "id": "0015-b3287664cd4a9f749cfc434debcfe5beb355850c4ca48ff4663a6813c6f"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [
+    {
+     "output_type": "stream",
+     "name": "stdout",
+     "text": [
+      "Drop NA: MAE=0.77, RMSE=1.24\n",
+      "Simple:  MAE=0.77, RMSE=1.23\n",
+      "k-NN:    MAE=0.78, RMSE=1.24"
+     ]
+    }
+   ],
+   "source": [
+    "print(f'Drop NA: MAE={mae_cmpl:.2f}, RMSE={rmse_cmpl:.2f}'); \\\n",
+    "print(f'Simple:  MAE={mae_simple:.2f}, RMSE={rmse_simple:.2f}'); \\\n",
+    "print(f'k-NN:    MAE={mae_knn:.2f}, RMSE={rmse_knn:.2f}')"
+   ],
+   "id": "0016-e515a7b61fcd2bcacbe931d1837e8dd86de58685b76e66ac6758df31583"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Jetzt sollten Sie *für diesen Datensatz* die Frage beantworten können,\n",
+    "ob gefüllte Samples in der Trainingsmenge das Modell verbessern oder\n",
+    "verschlechtern.\n",
+    "\n",
+    "Antwort: Es gibt zwar geringe Unterschiede, aber die sind eher nicht\n",
+    "signifikant. Es lohnt sich also *für diesen Datensatz* nicht Samples in\n",
+    "der Trainingsmenge zu füllen. Ein `dropna` reicht aus."
+   ],
+   "id": "0018-f677ed4224fd64f752931629a671721ec39938b8a7588b4aa37a430def5"
+  }
+ ],
+ "nbformat": 4,
+ "nbformat_minor": 5,
+ "metadata": {}
+}
+%% Cell type:markdown id:0001-9a691649a9301a1cb609d038b68e1a84d6a0d2120ff69ca180381b5e12c tags:
+
+# Fehler mit und ohne Imputer für die Trainingsmenge
+
+Das Laden der Daten ist bereits vorbereitet:
+
+%% Cell type:code id:0002-e5c489be6d21762cddcba2d304863bbb413036df68019bdcd674ec10ac9 tags:
+
+``` 
+import numpy as np
+import pandas as pd
+from sklearn.neighbors import KNeighborsRegressor
+from sklearn.impute import SimpleImputer, KNNImputer
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import mean_absolute_error, mean_squared_error
+from sklearn.preprocessing import robust_scale
+
+df = pd.read_csv('nations.csv')
+
+# convert income via categorical to range
+income_vals = ['Low income', 'Lower middle income', 'Upper middle income', 'High income', 'High income: OECD']
+income_cat_dtype = pd.api.types.CategoricalDtype(categories=income_vals, ordered=True)
+df.income = df.income.astype(income_cat_dtype).cat.codes
+
+# drop samples with missing values in features ['life_expect', 'population', 'birth_rate']
+# also drop samples when values are missing in both gdp_percap AND neonat_mortal_rate
+core_feat = ['year', 'population', 'birth_rate', 'income']
+incomplete_feat = ['gdp_percap', 'neonat_mortal_rate']
+target = 'life_expect'
+
+drop_idx = (df.gdp_percap.isna() & df.neonat_mortal_rate.isna()) | df[core_feat + [target]].isna().any(axis='columns')
+df = df[~drop_idx]
+
+# split into X and y. X has only missing values in incomplete_feat
+X = df[core_feat + incomplete_feat]
+y = df[target]
+
+# scale X, otherwise errors are huge
+X = pd.DataFrame(robust_scale(X), index=X.index, columns=X.columns)
+```
+
+%% Cell type:markdown id:0003-14f89fcb50a1fa0d36c91cbe2ff6f54399a587a48a1e812c35f7ab39c90 tags:
+
+Nun haben Sie `X` und `y`, aber in `X` gibt es fehlende Werte. Wir
+definieren eine Testmenge. Wir versuchen zunächst herauszufinden wie
+groß der Fehler ist, wenn wir Samples mit fehlenden Werten wegwerfen.
+Ihre Aufgabe ist nun auf den vollständigen Samples (also ohne fehlende
+Werte), die nicht in der Testmenge sind, ein `KNeighborsRegressor` zu
+trainieren und den Fehler (MAE oder RMSE) auszuwerten.
+
+%% Cell type:code id:0004-2e3600b5dc70548bce9012d280aeced64a2b7b2767f0fd03adcd6ffc0c4 tags:
+
+``` 
+X_cmpl = X.dropna()
+y_cmpl = y[X_cmpl.index]
+
+# split complete samples, use X_test, y_test as common test set also for model trained on imputed values
+X_cmpl_train, X_test, y_cmpl_train, y_test = train_test_split(X_cmpl, y_cmpl, test_size=0.2)
+
+# TODO Exercise 1: train model using complete samples only and evaluate on test set
+```
+
+%% Cell type:markdown id:0005-94702981911ec4f24f72794f610c502911e5197963070b964d3f3c40038 tags:
+
+## Lösung
+
+%% Cell type:code id:0006-32fe21e10a5bd61d64245e7dec10ffda2ff142d8df6e1b6485f6dc72039 tags:
+
+``` 
+model = KNeighborsRegressor(weights='distance')
+model.fit(X_cmpl_train, y_cmpl_train)
+y_cmpl_pred = model.predict(X_test)
+error = y_cmpl_pred - y_test
+mae_cmpl = error.abs().mean()
+rmse_cmpl = np.sqrt((error ** 2).mean())
+```
+
+%% Cell type:markdown id:0007-e185b61321b945aa97a3f14f62d97f3a2ca2be544052b7ec2f8ea6ced4b tags:
+
+Als nächstes füllen Sie `X` mit einem simplen Imputer (Mittelwert oder
+Median), trainieren nun auf allen Samples, die nicht in der Testmenge
+sind, (also auch gefüllte Samples) und werten auf derselben Testmenge
+wie zuvor den Fehler aus.
+
+%% Cell type:code id:0008-4e70cd81b62eb4618125b6b86b08c85de9b3d03f941f397c85d2f743158 tags:
+
+``` 
+X_fill_simple = X.copy()
+
+# TODO Exercise 2a: impute gdp_percap and neonat_mortal_rate using median or mean
+
+# TODO Exercise 2b: train model using all samples except test samples and evaluate on test set
+```
+
+%% Cell type:markdown id:0009-5cdeedac3f86c044a9eecadeacc85d72a11512f10a7bb2ec56605e6297a tags:
+
+## Lösung
+
+%% Cell type:code id:0010-a24aeb2fa22291b46a17ecf1046fcc2de4cedd92d91d3f124906aea28b0 tags:
+
+``` 
+X_fill_simple = X_fill_simple.fillna(X.mean())
+model.fit(X_fill_simple.drop(X_test.index), y.drop(y_test.index))
+y_simple_pred = model.predict(X_test)
+error = y_simple_pred - y_test
+mae_simple = error.abs().mean()
+rmse_simple = np.sqrt((error ** 2).mean())
+```
+
+%% Cell type:markdown id:0011-fe55307b85eff61e6939d7ea4da2f6afc056088b6ba8d51af46ab7ecfea tags:
+
+Zuletzt machen Sie nochmals dasselbe, aber mit einem $k$ Nearest
+Neighbor Imputer.
+
+%% Cell type:code id:0012-1762321c7a3e5806dc42d23642eff7abea763e94d9a2a305139813d05c7 tags:
+
+``` 
+X_fill_knn = X.copy()
+
+# TODO Exercise 3a: impute gdp_percap and neonat_mortal_rate using KNNImputer
+
+# TODO Exercise 3b: train model using all samples except test samples and evaluate on test set
+```
+
+%% Cell type:markdown id:0013-994ce0eff56ef92a6dc375c0d9b4843f6d074065a8f9ff6890853b48681 tags:
+
+## Lösung
+
+%% Cell type:code id:0014-2d200e90dba955e7b70e8c641c626e0d6d2602cce67568de6f37f06c109 tags:
+
+``` 
+X_fill_knn = KNNImputer().fit_transform(X)
+X_fill_knn = pd.DataFrame(X_fill_knn, index=X.index, columns=X.columns)
+model.fit(X_fill_knn.drop(X_test.index), y.drop(y_test.index))
+y_knn_pred = model.predict(X_test)
+error = y_knn_pred - y_test
+mae_knn = error.abs().mean()
+rmse_knn = np.sqrt((error ** 2).mean())
+```
+
+%% Cell type:markdown id:0015-b3287664cd4a9f749cfc434debcfe5beb355850c4ca48ff4663a6813c6f tags:
+
+Auswertung:
+
+%% Cell type:code id:0016-e515a7b61fcd2bcacbe931d1837e8dd86de58685b76e66ac6758df31583 tags:
+
+``` 
+print(f'Drop NA: MAE={mae_cmpl:.2f}, RMSE={rmse_cmpl:.2f}'); \
+print(f'Simple:  MAE={mae_simple:.2f}, RMSE={rmse_simple:.2f}'); \
+print(f'k-NN:    MAE={mae_knn:.2f}, RMSE={rmse_knn:.2f}')
+```
+
+%% Output
+
+    Drop NA: MAE=0.77, RMSE=1.24
+    Simple:  MAE=0.77, RMSE=1.23
+    k-NN:    MAE=0.78, RMSE=1.24
+
+%% Cell type:markdown id:0018-f677ed4224fd64f752931629a671721ec39938b8a7588b4aa37a430def5 tags:
+
+Jetzt sollten Sie *für diesen Datensatz* die Frage beantworten können,
+ob gefüllte Samples in der Trainingsmenge das Modell verbessern oder
+verschlechtern.
+
+Antwort: Es gibt zwar geringe Unterschiede, aber die sind eher nicht
+signifikant. Es lohnt sich also *für diesen Datensatz* nicht Samples in
+der Trainingsmenge zu füllen. Ein `dropna` reicht aus.
--- a/07-imputer/03-impute-train.ipynb
+++ b/07-imputer/03-impute-train.ipynb
@@ -18,11 +18,13 @@
   },
   "outputs": [],
   "source": [
+    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.neighbors import KNeighborsRegressor\n",
    "from sklearn.impute import SimpleImputer, KNNImputer\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import mean_absolute_error, mean_squared_error\n",
+    "from sklearn.preprocessing import robust_scale\n",
    "\n",
    "df = pd.read_csv('nations.csv')\n",
    "\n",
@@ -42,9 +44,12 @@
    "\n",
    "# split into X and y. X has only missing values in incomplete_feat\n",
    "X = df[core_feat + incomplete_feat]\n",
-    "y = df[target]"
+    "y = df[target]\n",
+    "\n",
+    "# scale X, otherwise errors are huge\n",
+    "X = pd.DataFrame(robust_scale(X), index=X.index, columns=X.columns)"
   ],
-   "id": "0002-e11f96c9f8b6ea98a99253f9057994e4a19c91b310bcb262d754bcee2ac"
+   "id": "0002-e5c489be6d21762cddcba2d304863bbb413036df68019bdcd674ec10ac9"
  },
  {
   "cell_type": "markdown",

 %% Cell type:markdown id:0001-9a691649a9301a1cb609d038b68e1a84d6a0d2120ff69ca180381b5e12c tags:

 # Fehler mit und ohne Imputer für die Trainingsmenge

 Das Laden der Daten ist bereits vorbereitet:

-%% Cell type:code id:0002-e11f96c9f8b6ea98a99253f9057994e4a19c91b310bcb262d754bcee2ac tags:
+%% Cell type:code id:0002-e5c489be6d21762cddcba2d304863bbb413036df68019bdcd674ec10ac9 tags:

 ``` 
+import numpy as np
 import pandas as pd
 from sklearn.neighbors import KNeighborsRegressor
 from sklearn.impute import SimpleImputer, KNNImputer
 from sklearn.model_selection import train_test_split
 from sklearn.metrics import mean_absolute_error, mean_squared_error
+from sklearn.preprocessing import robust_scale

 df = pd.read_csv('nations.csv')

 # convert income via categorical to range
 income_vals = ['Low income', 'Lower middle income', 'Upper middle income', 'High income', 'High income: OECD']
 income_cat_dtype = pd.api.types.CategoricalDtype(categories=income_vals, ordered=True)
 df.income = df.income.astype(income_cat_dtype).cat.codes

 # drop samples with missing values in features ['life_expect', 'population', 'birth_rate']
 # also drop samples when values are missing in both gdp_percap AND neonat_mortal_rate
 core_feat = ['year', 'population', 'birth_rate', 'income']
 incomplete_feat = ['gdp_percap', 'neonat_mortal_rate']
 target = 'life_expect'

 drop_idx = (df.gdp_percap.isna() & df.neonat_mortal_rate.isna()) | df[core_feat + [target]].isna().any(axis='columns')
 df = df[~drop_idx]

 # split into X and y. X has only missing values in incomplete_feat
 X = df[core_feat + incomplete_feat]
 y = df[target]
+
+# scale X, otherwise errors are huge
+X = pd.DataFrame(robust_scale(X), index=X.index, columns=X.columns)
 ```

 %% Cell type:markdown id:0003-14f89fcb50a1fa0d36c91cbe2ff6f54399a587a48a1e812c35f7ab39c90 tags:

 Nun haben Sie `X` und `y`, aber in `X` gibt es fehlende Werte. Wir
 definieren eine Testmenge. Wir versuchen zunächst herauszufinden wie
 groß der Fehler ist, wenn wir Samples mit fehlenden Werten wegwerfen.
 Ihre Aufgabe ist nun auf den vollständigen Samples (also ohne fehlende
 Werte), die nicht in der Testmenge sind, ein `KNeighborsRegressor` zu
 trainieren und den Fehler (MAE oder RMSE) auszuwerten.

 %% Cell type:code id:0004-2e3600b5dc70548bce9012d280aeced64a2b7b2767f0fd03adcd6ffc0c4 tags:

 ``` 
 X_cmpl = X.dropna()
 y_cmpl = y[X_cmpl.index]

 # split complete samples, use X_test, y_test as common test set also for model trained on imputed values
 X_cmpl_train, X_test, y_cmpl_train, y_test = train_test_split(X_cmpl, y_cmpl, test_size=0.2)

 # TODO Exercise 1: train model using complete samples only and evaluate on test set
 ```

 %% Cell type:markdown id:0005-e185b61321b945aa97a3f14f62d97f3a2ca2be544052b7ec2f8ea6ced4b tags:

 Als nächstes füllen Sie `X` mit einem simplen Imputer (Mittelwert oder
 Median), trainieren nun auf allen Samples, die nicht in der Testmenge
 sind, (also auch gefüllte Samples) und werten auf derselben Testmenge
 wie zuvor den Fehler aus.

 %% Cell type:code id:0006-4e70cd81b62eb4618125b6b86b08c85de9b3d03f941f397c85d2f743158 tags:

 ``` 
 X_fill_simple = X.copy()

 # TODO Exercise 2a: impute gdp_percap and neonat_mortal_rate using median or mean

 # TODO Exercise 2b: train model using all samples except test samples and evaluate on test set
 ```

 %% Cell type:markdown id:0007-fe55307b85eff61e6939d7ea4da2f6afc056088b6ba8d51af46ab7ecfea tags:

 Zuletzt machen Sie nochmals dasselbe, aber mit einem $k$ Nearest
 Neighbor Imputer.

 %% Cell type:code id:0008-1762321c7a3e5806dc42d23642eff7abea763e94d9a2a305139813d05c7 tags:

 ``` 
 X_fill_knn = X.copy()

 # TODO Exercise 3a: impute gdp_percap and neonat_mortal_rate using KNNImputer

 # TODO Exercise 3b: train model using all samples except test samples and evaluate on test set
 ```

 %% Cell type:markdown id:0009-9c9ced6f052ed3afdc869c83fff2567e5ec5065832cd369dc2c65e59e52 tags:

 Jetzt sollten Sie *für diesen Datensatz* die Frage beantworten können,
 ob gefüllte Samples in der Trainingsmenge das Modell verbessern oder
 verschlechtern.

--- a/07-imputer/04-impute-test-sol.ipynb
+++ b/07-imputer/04-impute-test-sol.ipynb
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Fehler mit und ohne Imputer für die Testmenge\n",
+    "\n",
+    "Das Laden der Daten ist bereits vorbereitet:"
+   ],
+   "id": "0001-3c25ae28e6c0657f9cfc7b0066a7add26c991ca53c40803277a38c0e72d"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "style": "python"
+   },
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "import pandas as pd\n",
+    "from sklearn.neighbors import KNeighborsRegressor\n",
+    "from sklearn.impute import SimpleImputer, KNNImputer\n",
+    "from sklearn.metrics import mean_absolute_error, mean_squared_error\n",
+    "from sklearn.preprocessing import robust_scale\n",
+    "\n",
+    "df = pd.read_csv('nations.csv')\n",
+    "\n",
+    "# convert income via categorical to range\n",
+    "income_vals = ['Low income', 'Lower middle income', 'Upper middle income', 'High income', 'High income: OECD']\n",
+    "income_cat_dtype = pd.api.types.CategoricalDtype(categories=income_vals, ordered=True)\n",
+    "df.income = df.income.astype(income_cat_dtype).cat.codes\n",
+    "\n",
+    "# drop samples with missing values in features ['life_expect', 'population', 'birth_rate']\n",
+    "# also drop samples when values are missing in both gdp_percap AND neonat_mortal_rate\n",
+    "core_feat = ['year', 'population', 'birth_rate', 'income']\n",
+    "incomplete_feat = ['gdp_percap', 'neonat_mortal_rate']\n",
+    "target = 'life_expect'\n",
+    "\n",
+    "drop_idx = (df.gdp_percap.isna() & df.neonat_mortal_rate.isna()) | df[core_feat + [target]].isna().any(axis='columns')\n",
+    "df = df[~drop_idx]\n",
+    "\n",
+    "# split into X and y. X has only missing values in incomplete_feat\n",
+    "X = df[core_feat + incomplete_feat]\n",
+    "y = df[target]\n",
+    "\n",
+    "# scale X, otherwise errors are huge\n",
+    "X = pd.DataFrame(robust_scale(X), index=X.index, columns=X.columns)"
+   ],
+   "id": "0002-f5e678b61651241061177705ddea02e21bcb8793fa1bf16e1c09a6a1ad4"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Nun haben Sie `X` und `y`, aber in `X` gibt es fehlende Werte. Wir\n",
+    "definieren die Testmenge, als die Menge der Samples bei denen mind. ein\n",
+    "Wert fehlt. Wir versuchen zunächst herauszufinden wie groß der Fehler\n",
+    "ist, wenn wir Features mit fehlenden Werten wegwerfen. Ihre Aufgabe ist\n",
+    "nun auf den vollständigen Features (also diejenigen ohne fehlende Werte)\n",
+    "ein `KNeighborsRegressor` zu trainieren und den Fehler (MAE oder RMSE)\n",
+    "auszuwerten."
+   ],
+   "id": "0003-adf7aa36716dcd8289f4291070e2e4af1c34878b1c50c570b425b138935"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "style": "python"
+   },
+   "outputs": [],
+   "source": [
+    "X_cmpl = X.dropna()\n",
+    "y_cmpl = y[X_cmpl.index]\n",
+    "\n",
+    "# test set contains only incomplete samples\n",
+    "y_test = y.drop(X_cmpl.index)\n",
+    "\n",
+    "# TODO Exercise 1a: train model on complete samples with dropped incomplete features\n",
+    "\n",
+    "# TODO Exercise 1b: evaluate on test set (drop incomplete features)"
+   ],
+   "id": "0004-2e44b46fb461aa34b89bb051f44d9f40708759418f9777ac3e3067be52a"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Lösung"
+   ],
+   "id": "0005-94702981911ec4f24f72794f610c502911e5197963070b964d3f3c40038"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "style": "python"
+   },
+   "outputs": [],
+   "source": [
+    "model = KNeighborsRegressor(weights='distance')\n",
+    "model.fit(X_cmpl[core_feat], y_cmpl)\n",
+    "y_drop = model.predict(X.loc[y_test.index, core_feat])\n",
+    "error = y_drop - y_test\n",
+    "mae_drop = error.abs().mean()\n",
+    "rmse_drop = np.sqrt((error ** 2).mean())"
+   ],
+   "id": "0006-266e99bb3211b2c2a717e65161e38889c330169497a0dc527f851f543de"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Wir wollen gleich Untersuchen, ob das Füllen der Testmenge den Fehler\n",
+    "verringert oder erhöht. Das machen wir einmal mit einem simplen Imputer\n",
+    "und einmal mit einem $k$-NN. In beiden Fällen ist jedoch das Modell\n",
+    "dasselbe. Daher können Sie nun schon mal einen `KNeighborsRegressor` auf\n",
+    "allen Features trainieren."
+   ],
+   "id": "0007-facaa96e1a021f3a63651e20a81752665271f8f3e55167e6784350141ad"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "style": "python"
+   },
+   "outputs": [],
+   "source": [
+    "# TODO Exercise 2: train model on all complete samples"
+   ],
+   "id": "0008-2b246d6be116cdc87429e01dd209927bc251258dae8b6b6aa00da953939"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Lösung"
+   ],
+   "id": "0009-5cdeedac3f86c044a9eecadeacc85d72a11512f10a7bb2ec56605e6297a"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "style": "python"
+   },
+   "outputs": [],
+   "source": [
+    "model.fit(X_cmpl, y_cmpl)"
+   ],
+   "id": "0010-089d039f187bea04ff56ce1de21e7a79139c415e39b8656432fda248d59"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Als nächstes füllen Sie `X` mit einem simplen Imputer (Mittelwert oder\n",
+    "Median) und werten auf gefüllten Testmenge den Fehler aus."
+   ],
+   "id": "0011-29464d9aa828481b9d1a94768e82a15f13fc68d66929044b49650b6c7fa"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "style": "python"
+   },
+   "outputs": [],
+   "source": [
+    "X_fill_simple = X.copy()\n",
+    "\n",
+    "# TODO Exercise 3a: impute gdp_percap and neonat_mortal_rate using median or mean\n",
+    "# TODO Exercise 3b: evaluate model on filled test set"
+   ],
+   "id": "0012-2c3c93b51c1909a78077b4929022c8b1b5bfa42183c8d005bf1581528ee"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Lösung"
+   ],
+   "id": "0013-994ce0eff56ef92a6dc375c0d9b4843f6d074065a8f9ff6890853b48681"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "style": "python"
+   },
+   "outputs": [],
+   "source": [
+    "X_fill_simple = X.fillna(X.median())\n",
+    "y_simple = model.predict(X_fill_simple.loc[y_test.index])\n",
+    "error = y_simple - y_test\n",
+    "mae_simple = error.abs().mean()\n",
+    "rmse_simple = np.sqrt((error ** 2).mean())"
+   ],
+   "id": "0014-c996550e79026fe2751c7a3429041c49b176d538b0c50f318798fd71491"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Zuletzt machen Sie nochmals dasselbe, aber mit einem $k$ Nearest\n",
+    "Neighbor Imputer."
+   ],
+   "id": "0015-fe55307b85eff61e6939d7ea4da2f6afc056088b6ba8d51af46ab7ecfea"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "style": "python"
+   },
+   "outputs": [],
+   "source": [
+    "X_fill_knn = X.copy()\n",
+    "\n",
+    "# TODO Exercise 4a: impute gdp_percap and neonat_mortal_rate using KNNImputer\n",
+    "# TODO Exercise 4b: evaluate model on filled test set"
+   ],
+   "id": "0016-c8dc15d1646bc841284cf0497b1a0483cd0a4ab70a1ac18f12c39825f40"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Lösung"
+   ],
+   "id": "0017-a3e6eaa86d762cf4283a965e129e1f725971ce14b306289209de26c4ca3"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "style": "python"
+   },
+   "outputs": [],
+   "source": [
+    "X_fill_knn = KNNImputer().fit_transform(X)\n",
+    "X_fill_knn = pd.DataFrame(X_fill_knn, index=X.index, columns=X.columns)\n",
+    "y_knn = model.predict(X_fill_knn.loc[y_test.index])\n",
+    "error = y_knn - y_test\n",
+    "mae_knn = error.abs().mean()\n",
+    "rmse_knn = np.sqrt((error ** 2).mean())"
+   ],
+   "id": "0018-95a2bf3e8370a2e64069b6cb6369da598ce6357513e007f42a676a2f14d"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Auswertung:"
+   ],
+   "id": "0019-b3287664cd4a9f749cfc434debcfe5beb355850c4ca48ff4663a6813c6f"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [
+    {
+     "output_type": "stream",
+     "name": "stdout",
+     "text": [
+      "Drop Features:  MAE=3.96, RMSE=5.12\n",
+      "Simple Imputer: MAE=4.28, RMSE=5.19\n",
+      "k-NN Imputer:   MAE=3.42, RMSE=4.33"
+     ]
+    }
+   ],
+   "source": [
+    "print(f'Drop Features:  MAE={mae_drop:.2f}, RMSE={rmse_drop:.2f}'); \\\n",
+    "print(f'Simple Imputer: MAE={mae_simple:.2f}, RMSE={rmse_simple:.2f}'); \\\n",
+    "print(f'k-NN Imputer:   MAE={mae_knn:.2f}, RMSE={rmse_knn:.2f}')"
+   ],
+   "id": "0020-8bcb78415ee333fdff6dba47e484cf51354ec13ba4f1175155c99e096fe"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Jetzt sollten Sie *für diesen Datensatz* die Frage beantworten können,\n",
+    "ob, wenn Sie für unvollständige Daten Vorhersagen durchführen müssen,\n",
+    "gefüllte Features in der Testmenge das Modell verbessern oder\n",
+    "verschlechtern.\n",
+    "\n",
+    "Antwort: Der Fehler ist im Vergleich zu der anderen Aufgabe recht groß.\n",
+    "Interessant ist, dass der einfache Imputer den Fehler im Vergleich zum\n",
+    "Wegwerfen der Features vergrößert, der $k$ Nearest Neighbor Imputer den\n",
+    "Fehler jedoch im Vergleich zum Wegwerfen der Features verringert. Das\n",
+    "heißt, *für diesen Datensatz* lohnt sich nur ein guter Imputer."
+   ],
+   "id": "0022-c9a78506d1968859eabbe8fc291a75a979953b5c94f0b7997eb3758df66"
+  }
+ ],
+ "nbformat": 4,
+ "nbformat_minor": 5,
+ "metadata": {}
+}
+%% Cell type:markdown id:0001-3c25ae28e6c0657f9cfc7b0066a7add26c991ca53c40803277a38c0e72d tags:
+
+# Fehler mit und ohne Imputer für die Testmenge
+
+Das Laden der Daten ist bereits vorbereitet:
+
+%% Cell type:code id:0002-f5e678b61651241061177705ddea02e21bcb8793fa1bf16e1c09a6a1ad4 tags:
+
+``` 
+import numpy as np
+import pandas as pd
+from sklearn.neighbors import KNeighborsRegressor
+from sklearn.impute import SimpleImputer, KNNImputer
+from sklearn.metrics import mean_absolute_error, mean_squared_error
+from sklearn.preprocessing import robust_scale
+
+df = pd.read_csv('nations.csv')
+
+# convert income via categorical to range
+income_vals = ['Low income', 'Lower middle income', 'Upper middle income', 'High income', 'High income: OECD']
+income_cat_dtype = pd.api.types.CategoricalDtype(categories=income_vals, ordered=True)
+df.income = df.income.astype(income_cat_dtype).cat.codes
+
+# drop samples with missing values in features ['life_expect', 'population', 'birth_rate']
+# also drop samples when values are missing in both gdp_percap AND neonat_mortal_rate
+core_feat = ['year', 'population', 'birth_rate', 'income']
+incomplete_feat = ['gdp_percap', 'neonat_mortal_rate']
+target = 'life_expect'
+
+drop_idx = (df.gdp_percap.isna() & df.neonat_mortal_rate.isna()) | df[core_feat + [target]].isna().any(axis='columns')
+df = df[~drop_idx]
+
+# split into X and y. X has only missing values in incomplete_feat
+X = df[core_feat + incomplete_feat]
+y = df[target]
+
+# scale X, otherwise errors are huge
+X = pd.DataFrame(robust_scale(X), index=X.index, columns=X.columns)
+```
+
+%% Cell type:markdown id:0003-adf7aa36716dcd8289f4291070e2e4af1c34878b1c50c570b425b138935 tags:
+
+Nun haben Sie `X` und `y`, aber in `X` gibt es fehlende Werte. Wir
+definieren die Testmenge, als die Menge der Samples bei denen mind. ein
+Wert fehlt. Wir versuchen zunächst herauszufinden wie groß der Fehler
+ist, wenn wir Features mit fehlenden Werten wegwerfen. Ihre Aufgabe ist
+nun auf den vollständigen Features (also diejenigen ohne fehlende Werte)
+ein `KNeighborsRegressor` zu trainieren und den Fehler (MAE oder RMSE)
+auszuwerten.
+
+%% Cell type:code id:0004-2e44b46fb461aa34b89bb051f44d9f40708759418f9777ac3e3067be52a tags:
+
+``` 
+X_cmpl = X.dropna()
+y_cmpl = y[X_cmpl.index]
+
+# test set contains only incomplete samples
+y_test = y.drop(X_cmpl.index)
+
+# TODO Exercise 1a: train model on complete samples with dropped incomplete features
+
+# TODO Exercise 1b: evaluate on test set (drop incomplete features)
+```
+
+%% Cell type:markdown id:0005-94702981911ec4f24f72794f610c502911e5197963070b964d3f3c40038 tags:
+
+## Lösung
+
+%% Cell type:code id:0006-266e99bb3211b2c2a717e65161e38889c330169497a0dc527f851f543de tags:
+
+``` 
+model = KNeighborsRegressor(weights='distance')
+model.fit(X_cmpl[core_feat], y_cmpl)
+y_drop = model.predict(X.loc[y_test.index, core_feat])
+error = y_drop - y_test
+mae_drop = error.abs().mean()
+rmse_drop = np.sqrt((error ** 2).mean())
+```
+
+%% Cell type:markdown id:0007-facaa96e1a021f3a63651e20a81752665271f8f3e55167e6784350141ad tags:
+
+Wir wollen gleich Untersuchen, ob das Füllen der Testmenge den Fehler
+verringert oder erhöht. Das machen wir einmal mit einem simplen Imputer
+und einmal mit einem $k$-NN. In beiden Fällen ist jedoch das Modell
+dasselbe. Daher können Sie nun schon mal einen `KNeighborsRegressor` auf
+allen Features trainieren.
+
+%% Cell type:code id:0008-2b246d6be116cdc87429e01dd209927bc251258dae8b6b6aa00da953939 tags:
+
+``` 
+# TODO Exercise 2: train model on all complete samples
+```
+
+%% Cell type:markdown id:0009-5cdeedac3f86c044a9eecadeacc85d72a11512f10a7bb2ec56605e6297a tags:
+
+## Lösung
+
+%% Cell type:code id:0010-089d039f187bea04ff56ce1de21e7a79139c415e39b8656432fda248d59 tags:
+
+``` 
+model.fit(X_cmpl, y_cmpl)
+```
+
+%% Cell type:markdown id:0011-29464d9aa828481b9d1a94768e82a15f13fc68d66929044b49650b6c7fa tags:
+
+Als nächstes füllen Sie `X` mit einem simplen Imputer (Mittelwert oder
+Median) und werten auf gefüllten Testmenge den Fehler aus.
+
+%% Cell type:code id:0012-2c3c93b51c1909a78077b4929022c8b1b5bfa42183c8d005bf1581528ee tags:
+
+``` 
+X_fill_simple = X.copy()
+
+# TODO Exercise 3a: impute gdp_percap and neonat_mortal_rate using median or mean
+# TODO Exercise 3b: evaluate model on filled test set
+```
+
+%% Cell type:markdown id:0013-994ce0eff56ef92a6dc375c0d9b4843f6d074065a8f9ff6890853b48681 tags:
+
+## Lösung
+
+%% Cell type:code id:0014-c996550e79026fe2751c7a3429041c49b176d538b0c50f318798fd71491 tags:
+
+``` 
+X_fill_simple = X.fillna(X.median())
+y_simple = model.predict(X_fill_simple.loc[y_test.index])
+error = y_simple - y_test
+mae_simple = error.abs().mean()
+rmse_simple = np.sqrt((error ** 2).mean())
+```
+
+%% Cell type:markdown id:0015-fe55307b85eff61e6939d7ea4da2f6afc056088b6ba8d51af46ab7ecfea tags:
+
+Zuletzt machen Sie nochmals dasselbe, aber mit einem $k$ Nearest
+Neighbor Imputer.
+
+%% Cell type:code id:0016-c8dc15d1646bc841284cf0497b1a0483cd0a4ab70a1ac18f12c39825f40 tags:
+
+``` 
+X_fill_knn = X.copy()
+
+# TODO Exercise 4a: impute gdp_percap and neonat_mortal_rate using KNNImputer
+# TODO Exercise 4b: evaluate model on filled test set
+```
+
+%% Cell type:markdown id:0017-a3e6eaa86d762cf4283a965e129e1f725971ce14b306289209de26c4ca3 tags:
+
+## Lösung
+
+%% Cell type:code id:0018-95a2bf3e8370a2e64069b6cb6369da598ce6357513e007f42a676a2f14d tags:
+
+``` 
+X_fill_knn = KNNImputer().fit_transform(X)
+X_fill_knn = pd.DataFrame(X_fill_knn, index=X.index, columns=X.columns)
+y_knn = model.predict(X_fill_knn.loc[y_test.index])
+error = y_knn - y_test
+mae_knn = error.abs().mean()
+rmse_knn = np.sqrt((error ** 2).mean())
+```
+
+%% Cell type:markdown id:0019-b3287664cd4a9f749cfc434debcfe5beb355850c4ca48ff4663a6813c6f tags:
+
+Auswertung:
+
+%% Cell type:code id:0020-8bcb78415ee333fdff6dba47e484cf51354ec13ba4f1175155c99e096fe tags:
+
+``` 
+print(f'Drop Features:  MAE={mae_drop:.2f}, RMSE={rmse_drop:.2f}'); \
+print(f'Simple Imputer: MAE={mae_simple:.2f}, RMSE={rmse_simple:.2f}'); \
+print(f'k-NN Imputer:   MAE={mae_knn:.2f}, RMSE={rmse_knn:.2f}')
+```
+
+%% Output
+
+    Drop Features:  MAE=3.96, RMSE=5.12
+    Simple Imputer: MAE=4.28, RMSE=5.19
+    k-NN Imputer:   MAE=3.42, RMSE=4.33
+
+%% Cell type:markdown id:0022-c9a78506d1968859eabbe8fc291a75a979953b5c94f0b7997eb3758df66 tags:
+
+Jetzt sollten Sie *für diesen Datensatz* die Frage beantworten können,
+ob, wenn Sie für unvollständige Daten Vorhersagen durchführen müssen,
+gefüllte Features in der Testmenge das Modell verbessern oder
+verschlechtern.
+
+Antwort: Der Fehler ist im Vergleich zu der anderen Aufgabe recht groß.
+Interessant ist, dass der einfache Imputer den Fehler im Vergleich zum
+Wegwerfen der Features vergrößert, der $k$ Nearest Neighbor Imputer den
+Fehler jedoch im Vergleich zum Wegwerfen der Features verringert. Das
+heißt, *für diesen Datensatz* lohnt sich nur ein guter Imputer.
--- a/07-imputer/04-impute-test.ipynb
+++ b/07-imputer/04-impute-test.ipynb
@@ -18,11 +18,12 @@
   },
   "outputs": [],
   "source": [
+    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.neighbors import KNeighborsRegressor\n",
    "from sklearn.impute import SimpleImputer, KNNImputer\n",
-    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import mean_absolute_error, mean_squared_error\n",
+    "from sklearn.preprocessing import robust_scale\n",
    "\n",
    "df = pd.read_csv('nations.csv')\n",
    "\n",
@@ -42,9 +43,12 @@
    "\n",
    "# split into X and y. X has only missing values in incomplete_feat\n",
    "X = df[core_feat + incomplete_feat]\n",
-    "y = df[target]"
+    "y = df[target]\n",
+    "\n",
+    "# scale X, otherwise errors are huge\n",
+    "X = pd.DataFrame(robust_scale(X), index=X.index, columns=X.columns)"
   ],
-   "id": "0002-e11f96c9f8b6ea98a99253f9057994e4a19c91b310bcb262d754bcee2ac"
+   "id": "0002-f5e678b61651241061177705ddea02e21bcb8793fa1bf16e1c09a6a1ad4"
  },
  {
   "cell_type": "markdown",
@@ -54,11 +58,11 @@
    "definieren die Testmenge, als die Menge der Samples bei denen mind. ein\n",
    "Wert fehlt. Wir versuchen zunächst herauszufinden wie groß der Fehler\n",
    "ist, wenn wir Features mit fehlenden Werten wegwerfen. Ihre Aufgabe ist\n",
-    "nun auf den vollständigen Features (also ohne fehlende Werte) ein\n",
-    "`KNeighborsRegressor` zu trainieren und den Fehler (MAE oder RMSE)\n",
+    "nun auf den vollständigen Features (also diejenigen ohne fehlende Werte)\n",
+    "ein `KNeighborsRegressor` zu trainieren und den Fehler (MAE oder RMSE)\n",
    "auszuwerten."
   ],
-   "id": "0003-6824753586f2898b08a189062eaf58ae145f256c8f3eb584a873cb62539"
+   "id": "0003-adf7aa36716dcd8289f4291070e2e4af1c34878b1c50c570b425b138935"
  },
  {
   "cell_type": "code",

 %% Cell type:markdown id:0001-3c25ae28e6c0657f9cfc7b0066a7add26c991ca53c40803277a38c0e72d tags:

 # Fehler mit und ohne Imputer für die Testmenge

 Das Laden der Daten ist bereits vorbereitet:

-%% Cell type:code id:0002-e11f96c9f8b6ea98a99253f9057994e4a19c91b310bcb262d754bcee2ac tags:
+%% Cell type:code id:0002-f5e678b61651241061177705ddea02e21bcb8793fa1bf16e1c09a6a1ad4 tags:

 ``` 
+import numpy as np
 import pandas as pd
 from sklearn.neighbors import KNeighborsRegressor
 from sklearn.impute import SimpleImputer, KNNImputer
-from sklearn.model_selection import train_test_split
 from sklearn.metrics import mean_absolute_error, mean_squared_error
+from sklearn.preprocessing import robust_scale

 df = pd.read_csv('nations.csv')

 # convert income via categorical to range
 income_vals = ['Low income', 'Lower middle income', 'Upper middle income', 'High income', 'High income: OECD']
 income_cat_dtype = pd.api.types.CategoricalDtype(categories=income_vals, ordered=True)
 df.income = df.income.astype(income_cat_dtype).cat.codes

 # drop samples with missing values in features ['life_expect', 'population', 'birth_rate']
 # also drop samples when values are missing in both gdp_percap AND neonat_mortal_rate
 core_feat = ['year', 'population', 'birth_rate', 'income']
 incomplete_feat = ['gdp_percap', 'neonat_mortal_rate']
 target = 'life_expect'

 drop_idx = (df.gdp_percap.isna() & df.neonat_mortal_rate.isna()) | df[core_feat + [target]].isna().any(axis='columns')
 df = df[~drop_idx]

 # split into X and y. X has only missing values in incomplete_feat
 X = df[core_feat + incomplete_feat]
 y = df[target]
+
+# scale X, otherwise errors are huge
+X = pd.DataFrame(robust_scale(X), index=X.index, columns=X.columns)
 ```

-%% Cell type:markdown id:0003-6824753586f2898b08a189062eaf58ae145f256c8f3eb584a873cb62539 tags:
+%% Cell type:markdown id:0003-adf7aa36716dcd8289f4291070e2e4af1c34878b1c50c570b425b138935 tags:

 Nun haben Sie `X` und `y`, aber in `X` gibt es fehlende Werte. Wir
 definieren die Testmenge, als die Menge der Samples bei denen mind. ein
 Wert fehlt. Wir versuchen zunächst herauszufinden wie groß der Fehler
 ist, wenn wir Features mit fehlenden Werten wegwerfen. Ihre Aufgabe ist
-nun auf den vollständigen Features (also ohne fehlende Werte) ein
-`KNeighborsRegressor` zu trainieren und den Fehler (MAE oder RMSE)
+nun auf den vollständigen Features (also diejenigen ohne fehlende Werte)
+ein `KNeighborsRegressor` zu trainieren und den Fehler (MAE oder RMSE)
 auszuwerten.

 %% Cell type:code id:0004-2e44b46fb461aa34b89bb051f44d9f40708759418f9777ac3e3067be52a tags:

 ``` 
 X_cmpl = X.dropna()
 y_cmpl = y[X_cmpl.index]

 # test set contains only incomplete samples
 y_test = y.drop(X_cmpl.index)

 # TODO Exercise 1a: train model on complete samples with dropped incomplete features

 # TODO Exercise 1b: evaluate on test set (drop incomplete features)
 ```

 %% Cell type:markdown id:0005-facaa96e1a021f3a63651e20a81752665271f8f3e55167e6784350141ad tags:

 Wir wollen gleich Untersuchen, ob das Füllen der Testmenge den Fehler
 verringert oder erhöht. Das machen wir einmal mit einem simplen Imputer
 und einmal mit einem $k$-NN. In beiden Fällen ist jedoch das Modell
 dasselbe. Daher können Sie nun schon mal einen `KNeighborsRegressor` auf
 allen Features trainieren.

 %% Cell type:code id:0006-2b246d6be116cdc87429e01dd209927bc251258dae8b6b6aa00da953939 tags:

 ``` 
 # TODO Exercise 2: train model on all complete samples
 ```

 %% Cell type:markdown id:0007-29464d9aa828481b9d1a94768e82a15f13fc68d66929044b49650b6c7fa tags:

 Als nächstes füllen Sie `X` mit einem simplen Imputer (Mittelwert oder
 Median) und werten auf gefüllten Testmenge den Fehler aus.

 %% Cell type:code id:0008-2c3c93b51c1909a78077b4929022c8b1b5bfa42183c8d005bf1581528ee tags:

 ``` 
 X_fill_simple = X.copy()

 # TODO Exercise 3a: impute gdp_percap and neonat_mortal_rate using median or mean
 # TODO Exercise 3b: evaluate model on filled test set
 ```

 %% Cell type:markdown id:0009-fe55307b85eff61e6939d7ea4da2f6afc056088b6ba8d51af46ab7ecfea tags:

 Zuletzt machen Sie nochmals dasselbe, aber mit einem $k$ Nearest
 Neighbor Imputer.

 %% Cell type:code id:0010-c8dc15d1646bc841284cf0497b1a0483cd0a4ab70a1ac18f12c39825f40 tags:

 ``` 
 X_fill_knn = X.copy()

 # TODO Exercise 4a: impute gdp_percap and neonat_mortal_rate using KNNImputer
 # TODO Exercise 4b: evaluate model on filled test set
 ```

 %% Cell type:markdown id:0011-146751deaf5a1f6ed1a9581e8a2de68567f7fb6842dd76893fb7ee5fee6 tags:

 Jetzt sollten Sie *für diesen Datensatz* die Frage beantworten können,
 ob, wenn Sie für unvollständige Daten Vorhersagen durchführen müssen,
 gefüllte Features in der Testmenge das Modell verbessern oder
 verschlechtern.