Notebooks from applied-cs/data-science@7134bb74

c4bac8b7 · Christof Kaufmann · 7b74d2c5 · c4bac8b7 · c4bac8b7 · c4bac8b7
Commit c4bac8b7 authored 1 year ago by Christof Kaufmann
--- a/07-imputer/01-aim-sol.ipynb
+++ b/07-imputer/01-aim-sol.ipynb
--- a/07-imputer/01-aim.ipynb
+++ b/07-imputer/01-aim.ipynb
--- a/07-imputer/02-imputer-beispiel.ipynb
+++ b/07-imputer/02-imputer-beispiel.ipynb
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Imputer-Beispiel\n",
+    "\n",
+    "Hier ist das Beispiel aus den Folien für Sie zum herumspielen."
+   ],
+   "id": "0001-e7a70edb35e0c1581dca0175d38bbd8b2353be7ee09df3f218d3756f76e"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "style": "python"
+   },
+   "outputs": [],
+   "source": [
+    "from sklearn.neighbors import KNeighborsRegressor\n",
+    "import numpy as np\n",
+    "import pandas as pd\n",
+    "from IPython.display import display\n",
+    "\n",
+    "a = np.arange(0, 1.05, 0.1)\n",
+    "b = (a - 0.2) ** 2\n",
+    "\n",
+    "a[3] = np.nan\n",
+    "b[8] = np.nan\n",
+    "\n",
+    "df = pd.DataFrame({'a': a, 'b': b})"
+   ],
+   "id": "0002-cf0883ec19f01bd632d0b34731487f6a33f9dfab30bbcb1be4c93cdf965"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [
+    {
+     "output_type": "stream",
+     "name": "stdout",
+     "text": [
+      "      a     b\n",
+      "0   0.0  0.04\n",
+      "1   0.1  0.01\n",
+      "2   0.2  0.00\n",
+      "3   NaN  0.01\n",
+      "4   0.4  0.04\n",
+      "5   0.5  0.09\n",
+      "6   0.6  0.16\n",
+      "7   0.7  0.25\n",
+      "8   0.8   NaN\n",
+      "9   0.9  0.49\n",
+      "10  1.0  0.64"
+     ]
+    }
+   ],
+   "source": [
+    "display(df)"
+   ],
+   "id": "0003-e6141b805dd1474c212e189a002ed6471840a8225d95bf031bc58e5b64a"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [
+    {
+     "output_type": "stream",
+     "name": "stdout",
+     "text": [
+      "mean:\n",
+      "a    0.520\n",
+      "b    0.173\n",
+      "dtype: float64"
+     ]
+    }
+   ],
+   "source": [
+    "print('mean:'); \\\n",
+    "print(df.mean())"
+   ],
+   "id": "0004-43dd5cd0b7d6c0caf296201efea81c9fbd00175e4a5b2a3fca47b9ef6b4"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [
+    {
+     "output_type": "stream",
+     "name": "stdout",
+     "text": [
+      "median:\n",
+      "a    0.550\n",
+      "b    0.065\n",
+      "dtype: float64"
+     ]
+    }
+   ],
+   "source": [
+    "print(f'median:'); \\\n",
+    "print(df.median())"
+   ],
+   "id": "0005-ad45fb9e0f769caaecc6997d8ee5b807ec0014753e08c71dad5d164a92d"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "style": "python"
+   },
+   "outputs": [],
+   "source": [
+    "knn_results = {'a': {}, 'b': {}}\n",
+    "\n",
+    "X_fill = df.loc[df['a'].isna(), ['b']]\n",
+    "X_train = df.dropna()[['b']]\n",
+    "Y_train = df.dropna()[['a']]\n",
+    "for weighting in ('uniform', 'distance'):\n",
+    "    knn = KNeighborsRegressor(4, weights=weighting)\n",
+    "    knn.fit(X_train, Y_train)\n",
+    "    a_pred = knn.predict(X_fill).squeeze()\n",
+    "    knn_results['a'][weighting] = a_pred\n",
+    "\n",
+    "X_fill = df.loc[df['b'].isna(), ['a']]\n",
+    "X_train = df.dropna()[['a']]\n",
+    "Y_train = df.dropna()[['b']]\n",
+    "for weighting in ('uniform', 'distance'):\n",
+    "    knn = KNeighborsRegressor(4, weights=weighting)\n",
+    "    knn.fit(X_train, Y_train)\n",
+    "    b_pred = knn.predict(X_fill).squeeze()\n",
+    "    knn_results['b'][weighting] = b_pred"
+   ],
+   "id": "0006-c259838f9487b6c11884d3c01fc090996ad33c53ef0ad93c20c559524a2"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [
+    {
+     "output_type": "stream",
+     "name": "stdout",
+     "text": [
+      "{'a': {'uniform': array(0.175), 'distance': array(0.1)},\n",
+      " 'b': {'uniform': array(0.385), 'distance': array(0.38)}}"
+     ]
+    }
+   ],
+   "source": [
+    "display(knn_results)"
+   ],
+   "id": "0007-406d6aef8dac4ca4c882f63b12e35c9a3d63f8c1bada846d870aa0ee57a"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Hier Ihr Code:"
+   ],
+   "id": "0008-4f90926ab6960e65168fdff6f5e151194200dd69b468fb4eeb302fa84f1"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "style": "python"
+   },
+   "outputs": [],
+   "source": [],
+   "id": "0009-44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
+  }
+ ],
+ "nbformat": 4,
+ "nbformat_minor": 5,
+ "metadata": {}
+}
+%% Cell type:markdown id:0001-e7a70edb35e0c1581dca0175d38bbd8b2353be7ee09df3f218d3756f76e tags:
+# Imputer-Beispiel
+Hier ist das Beispiel aus den Folien für Sie zum herumspielen.
+%% Cell type:code id:0002-cf0883ec19f01bd632d0b34731487f6a33f9dfab30bbcb1be4c93cdf965 tags:
+``` 
+from sklearn.neighbors import KNeighborsRegressor
+import numpy as np
+import pandas as pd
+from IPython.display import display
+a = np.arange(0, 1.05, 0.1)
+b = (a - 0.2) ** 2
+a[3] = np.nan
+b[8] = np.nan
+df = pd.DataFrame({'a': a, 'b': b})
+```
+%% Cell type:code id:0003-e6141b805dd1474c212e189a002ed6471840a8225d95bf031bc58e5b64a tags:
+``` 
+display(df)
+```
+%% Output
+          a     b
+    0   0.0  0.04
+    1   0.1  0.01
+    2   0.2  0.00
+    3   NaN  0.01
+    4   0.4  0.04
+    5   0.5  0.09
+    6   0.6  0.16
+    7   0.7  0.25
+    8   0.8   NaN
+    9   0.9  0.49
+    10  1.0  0.64
+%% Cell type:code id:0004-43dd5cd0b7d6c0caf296201efea81c9fbd00175e4a5b2a3fca47b9ef6b4 tags:
+``` 
+print('mean:'); \
+print(df.mean())
+```
+%% Output
+    mean:
+    a    0.520
+    b    0.173
+    dtype: float64
+%% Cell type:code id:0005-ad45fb9e0f769caaecc6997d8ee5b807ec0014753e08c71dad5d164a92d tags:
+``` 
+print(f'median:'); \
+print(df.median())
+```
+%% Output
+    median:
+    a    0.550
+    b    0.065
+    dtype: float64
+%% Cell type:code id:0006-c259838f9487b6c11884d3c01fc090996ad33c53ef0ad93c20c559524a2 tags:
+``` 
+knn_results = {'a': {}, 'b': {}}
+X_fill = df.loc[df['a'].isna(), ['b']]
+X_train = df.dropna()[['b']]
+Y_train = df.dropna()[['a']]
+for weighting in ('uniform', 'distance'):
+    knn = KNeighborsRegressor(4, weights=weighting)
+    knn.fit(X_train, Y_train)
+    a_pred = knn.predict(X_fill).squeeze()
+    knn_results['a'][weighting] = a_pred
+X_fill = df.loc[df['b'].isna(), ['a']]
+X_train = df.dropna()[['a']]
+Y_train = df.dropna()[['b']]
+for weighting in ('uniform', 'distance'):
+    knn = KNeighborsRegressor(4, weights=weighting)
+    knn.fit(X_train, Y_train)
+    b_pred = knn.predict(X_fill).squeeze()
+    knn_results['b'][weighting] = b_pred
+```
+%% Cell type:code id:0007-406d6aef8dac4ca4c882f63b12e35c9a3d63f8c1bada846d870aa0ee57a tags:
+``` 
+display(knn_results)
+```
+%% Output
+    {'a': {'uniform': array(0.175), 'distance': array(0.1)},
+     'b': {'uniform': array(0.385), 'distance': array(0.38)}}
+%% Cell type:markdown id:0008-4f90926ab6960e65168fdff6f5e151194200dd69b468fb4eeb302fa84f1 tags:
+Hier Ihr Code:
+%% Cell type:code id:0009-44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 tags:
+``` 
+```
--- a/07-imputer/03-impute-train.ipynb
+++ b/07-imputer/03-impute-train.ipynb
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Fehler mit und ohne Imputer für die Trainingsmenge\n",
+    "\n",
+    "Das Laden der Daten ist bereits vorbereitet:"
+   ],
+   "id": "0001-9a691649a9301a1cb609d038b68e1a84d6a0d2120ff69ca180381b5e12c"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "style": "python"
+   },
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "from sklearn.neighbors import KNeighborsRegressor\n",
+    "from sklearn.impute import SimpleImputer, KNNImputer\n",
+    "from sklearn.model_selection import train_test_split\n",
+    "from sklearn.metrics import mean_absolute_error, mean_squared_error\n",
+    "\n",
+    "df = pd.read_csv('nations.csv')\n",
+    "\n",
+    "# convert income via categorical to range\n",
+    "income_vals = ['Low income', 'Lower middle income', 'Upper middle income', 'High income', 'High income: OECD']\n",
+    "income_cat_dtype = pd.api.types.CategoricalDtype(categories=income_vals, ordered=True)\n",
+    "df.income = df.income.astype(income_cat_dtype).cat.codes\n",
+    "\n",
+    "# drop samples with missing values in features ['life_expect', 'population', 'birth_rate']\n",
+    "# also drop samples when values are missing in both gdp_percap AND neonat_mortal_rate\n",
+    "core_feat = ['year', 'population', 'birth_rate', 'income']\n",
+    "incomplete_feat = ['gdp_percap', 'neonat_mortal_rate']\n",
+    "target = 'life_expect'\n",
+    "\n",
+    "drop_idx = (df.gdp_percap.isna() & df.neonat_mortal_rate.isna()) | df[core_feat + [target]].isna().any(axis='columns')\n",
+    "df = df[~drop_idx]\n",
+    "\n",
+    "# split into X and y. X has only missing values in incomplete_feat\n",
+    "X = df[core_feat + incomplete_feat]\n",
+    "y = df[target]"
+   ],
+   "id": "0002-e11f96c9f8b6ea98a99253f9057994e4a19c91b310bcb262d754bcee2ac"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Nun haben Sie `X` und `y`, aber in `X` gibt es fehlende Werte. Wir\n",
+    "definieren eine Testmenge. Wir versuchen zunächst herauszufinden wie\n",
+    "groß der Fehler ist, wenn wir Samples mit fehlenden Werten wegwerfen.\n",
+    "Ihre Aufgabe ist nun auf den vollständigen Samples (also ohne fehlende\n",
+    "Werte), die nicht in der Testmenge sind, ein `KNeighborsRegressor` zu\n",
+    "trainieren und den Fehler (MAE oder RMSE) auszuwerten."
+   ],
+   "id": "0003-14f89fcb50a1fa0d36c91cbe2ff6f54399a587a48a1e812c35f7ab39c90"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "style": "python"
+   },
+   "outputs": [],
+   "source": [
+    "X_cmpl = X.dropna()\n",
+    "y_cmpl = y[X_cmpl.index]\n",
+    "\n",
+    "# split complete samples, use X_test, y_test as common test set also for model trained on imputed values\n",
+    "X_cmpl_train, X_test, y_cmpl_train, y_test = train_test_split(X_cmpl, y_cmpl, test_size=0.2)\n",
+    "\n",
+    "# TODO Exercise 1: train model using complete samples only and evaluate on test set"
+   ],
+   "id": "0004-2e3600b5dc70548bce9012d280aeced64a2b7b2767f0fd03adcd6ffc0c4"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Als nächstes füllen Sie `X` mit einem simplen Imputer (Mittelwert oder\n",
+    "Median), trainieren nun auf allen Samples, die nicht in der Testmenge\n",
+    "sind, (also auch gefüllte Samples) und werten auf derselben Testmenge\n",
+    "wie zuvor den Fehler aus."
+   ],
+   "id": "0005-e185b61321b945aa97a3f14f62d97f3a2ca2be544052b7ec2f8ea6ced4b"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "style": "python"
+   },
+   "outputs": [],
+   "source": [
+    "X_fill_simple = X.copy()\n",
+    "\n",
+    "# TODO Exercise 2a: impute gdp_percap and neonat_mortal_rate using median or mean\n",
+    "\n",
+    "# TODO Exercise 2b: train model using all samples except test samples and evaluate on test set"
+   ],
+   "id": "0006-4e70cd81b62eb4618125b6b86b08c85de9b3d03f941f397c85d2f743158"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Zuletzt machen Sie nochmals dasselbe, aber mit einem $k$ Nearest\n",
+    "Neighbor Imputer."
+   ],
+   "id": "0007-fe55307b85eff61e6939d7ea4da2f6afc056088b6ba8d51af46ab7ecfea"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "style": "python"
+   },
+   "outputs": [],
+   "source": [
+    "X_fill_knn = X.copy()\n",
+    "\n",
+    "# TODO Exercise 3a: impute gdp_percap and neonat_mortal_rate using KNNImputer\n",
+    "\n",
+    "# TODO Exercise 3b: train model using all samples except test samples and evaluate on test set"
+   ],
+   "id": "0008-1762321c7a3e5806dc42d23642eff7abea763e94d9a2a305139813d05c7"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Jetzt sollten Sie *für diesen Datensatz* die Frage beantworten können,\n",
+    "ob gefüllte Samples in der Trainingsmenge das Modell verbessern oder\n",
+    "verschlechtern."
+   ],
+   "id": "0009-9c9ced6f052ed3afdc869c83fff2567e5ec5065832cd369dc2c65e59e52"
+  }
+ ],
+ "nbformat": 4,
+ "nbformat_minor": 5,
+ "metadata": {}
+}
+%% Cell type:markdown id:0001-9a691649a9301a1cb609d038b68e1a84d6a0d2120ff69ca180381b5e12c tags:
+# Fehler mit und ohne Imputer für die Trainingsmenge
+Das Laden der Daten ist bereits vorbereitet:
+%% Cell type:code id:0002-e11f96c9f8b6ea98a99253f9057994e4a19c91b310bcb262d754bcee2ac tags:
+``` 
+import pandas as pd
+from sklearn.neighbors import KNeighborsRegressor
+from sklearn.impute import SimpleImputer, KNNImputer
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import mean_absolute_error, mean_squared_error
+df = pd.read_csv('nations.csv')
+# convert income via categorical to range
+income_vals = ['Low income', 'Lower middle income', 'Upper middle income', 'High income', 'High income: OECD']
+income_cat_dtype = pd.api.types.CategoricalDtype(categories=income_vals, ordered=True)
+df.income = df.income.astype(income_cat_dtype).cat.codes
+# drop samples with missing values in features ['life_expect', 'population', 'birth_rate']
+# also drop samples when values are missing in both gdp_percap AND neonat_mortal_rate
+core_feat = ['year', 'population', 'birth_rate', 'income']
+incomplete_feat = ['gdp_percap', 'neonat_mortal_rate']
+target = 'life_expect'
+drop_idx = (df.gdp_percap.isna() & df.neonat_mortal_rate.isna()) | df[core_feat + [target]].isna().any(axis='columns')
+df = df[~drop_idx]
+# split into X and y. X has only missing values in incomplete_feat
+X = df[core_feat + incomplete_feat]
+y = df[target]
+```
+%% Cell type:markdown id:0003-14f89fcb50a1fa0d36c91cbe2ff6f54399a587a48a1e812c35f7ab39c90 tags:
+Nun haben Sie `X` und `y`, aber in `X` gibt es fehlende Werte. Wir
+definieren eine Testmenge. Wir versuchen zunächst herauszufinden wie
+groß der Fehler ist, wenn wir Samples mit fehlenden Werten wegwerfen.
+Ihre Aufgabe ist nun auf den vollständigen Samples (also ohne fehlende
+Werte), die nicht in der Testmenge sind, ein `KNeighborsRegressor` zu
+trainieren und den Fehler (MAE oder RMSE) auszuwerten.
+%% Cell type:code id:0004-2e3600b5dc70548bce9012d280aeced64a2b7b2767f0fd03adcd6ffc0c4 tags:
+``` 
+X_cmpl = X.dropna()
+y_cmpl = y[X_cmpl.index]
+# split complete samples, use X_test, y_test as common test set also for model trained on imputed values
+X_cmpl_train, X_test, y_cmpl_train, y_test = train_test_split(X_cmpl, y_cmpl, test_size=0.2)
+# TODO Exercise 1: train model using complete samples only and evaluate on test set
+```
+%% Cell type:markdown id:0005-e185b61321b945aa97a3f14f62d97f3a2ca2be544052b7ec2f8ea6ced4b tags:
+Als nächstes füllen Sie `X` mit einem simplen Imputer (Mittelwert oder
+Median), trainieren nun auf allen Samples, die nicht in der Testmenge
+sind, (also auch gefüllte Samples) und werten auf derselben Testmenge
+wie zuvor den Fehler aus.
+%% Cell type:code id:0006-4e70cd81b62eb4618125b6b86b08c85de9b3d03f941f397c85d2f743158 tags:
+``` 
+X_fill_simple = X.copy()
+# TODO Exercise 2a: impute gdp_percap and neonat_mortal_rate using median or mean
+# TODO Exercise 2b: train model using all samples except test samples and evaluate on test set
+```
+%% Cell type:markdown id:0007-fe55307b85eff61e6939d7ea4da2f6afc056088b6ba8d51af46ab7ecfea tags:
+Zuletzt machen Sie nochmals dasselbe, aber mit einem $k$ Nearest
+Neighbor Imputer.
+%% Cell type:code id:0008-1762321c7a3e5806dc42d23642eff7abea763e94d9a2a305139813d05c7 tags:
+``` 
+X_fill_knn = X.copy()
+# TODO Exercise 3a: impute gdp_percap and neonat_mortal_rate using KNNImputer
+# TODO Exercise 3b: train model using all samples except test samples and evaluate on test set
+```
+%% Cell type:markdown id:0009-9c9ced6f052ed3afdc869c83fff2567e5ec5065832cd369dc2c65e59e52 tags:
+Jetzt sollten Sie *für diesen Datensatz* die Frage beantworten können,
+ob gefüllte Samples in der Trainingsmenge das Modell verbessern oder
+verschlechtern.
--- a/07-imputer/04-impute-test.ipynb
+++ b/07-imputer/04-impute-test.ipynb
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Fehler mit und ohne Imputer für die Testmenge\n",
+    "\n",
+    "Das Laden der Daten ist bereits vorbereitet:"
+   ],
+   "id": "0001-3c25ae28e6c0657f9cfc7b0066a7add26c991ca53c40803277a38c0e72d"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "style": "python"
+   },
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "from sklearn.neighbors import KNeighborsRegressor\n",
+    "from sklearn.impute import SimpleImputer, KNNImputer\n",
+    "from sklearn.model_selection import train_test_split\n",
+    "from sklearn.metrics import mean_absolute_error, mean_squared_error\n",
+    "\n",
+    "df = pd.read_csv('nations.csv')\n",
+    "\n",
+    "# convert income via categorical to range\n",
+    "income_vals = ['Low income', 'Lower middle income', 'Upper middle income', 'High income', 'High income: OECD']\n",
+    "income_cat_dtype = pd.api.types.CategoricalDtype(categories=income_vals, ordered=True)\n",
+    "df.income = df.income.astype(income_cat_dtype).cat.codes\n",
+    "\n",
+    "# drop samples with missing values in features ['life_expect', 'population', 'birth_rate']\n",
+    "# also drop samples when values are missing in both gdp_percap AND neonat_mortal_rate\n",
+    "core_feat = ['year', 'population', 'birth_rate', 'income']\n",
+    "incomplete_feat = ['gdp_percap', 'neonat_mortal_rate']\n",
+    "target = 'life_expect'\n",
+    "\n",
+    "drop_idx = (df.gdp_percap.isna() & df.neonat_mortal_rate.isna()) | df[core_feat + [target]].isna().any(axis='columns')\n",
+    "df = df[~drop_idx]\n",
+    "\n",
+    "# split into X and y. X has only missing values in incomplete_feat\n",
+    "X = df[core_feat + incomplete_feat]\n",
+    "y = df[target]"
+   ],
+   "id": "0002-e11f96c9f8b6ea98a99253f9057994e4a19c91b310bcb262d754bcee2ac"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Nun haben Sie `X` und `y`, aber in `X` gibt es fehlende Werte. Wir\n",
+    "definieren die Testmenge, als die Menge der Samples bei denen mind. ein\n",
+    "Wert fehlt. Wir versuchen zunächst herauszufinden wie groß der Fehler\n",
+    "ist, wenn wir Features mit fehlenden Werten wegwerfen. Ihre Aufgabe ist\n",
+    "nun auf den vollständigen Features (also ohne fehlende Werte) ein\n",
+    "`KNeighborsRegressor` zu trainieren und den Fehler (MAE oder RMSE)\n",
+    "auszuwerten."
+   ],
+   "id": "0003-6824753586f2898b08a189062eaf58ae145f256c8f3eb584a873cb62539"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "style": "python"
+   },
+   "outputs": [],
+   "source": [
+    "X_cmpl = X.dropna()\n",
+    "y_cmpl = y[X_cmpl.index]\n",
+    "\n",
+    "# test set contains only incomplete samples\n",
+    "y_test = y.drop(X_cmpl.index)\n",
+    "\n",
+    "# TODO Exercise 1a: train model on complete samples with dropped incomplete features\n",
+    "\n",
+    "# TODO Exercise 1b: evaluate on test set (drop incomplete features)"
+   ],
+   "id": "0004-2e44b46fb461aa34b89bb051f44d9f40708759418f9777ac3e3067be52a"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Wir wollen gleich Untersuchen, ob das Füllen der Testmenge den Fehler\n",
+    "verringert oder erhöht. Das machen wir einmal mit einem simplen Imputer\n",
+    "und einmal mit einem $k$-NN. In beiden Fällen ist jedoch das Modell\n",
+    "dasselbe. Daher können Sie nun schon mal einen `KNeighborsRegressor` auf\n",
+    "allen Features trainieren."
+   ],
+   "id": "0005-facaa96e1a021f3a63651e20a81752665271f8f3e55167e6784350141ad"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "style": "python"
+   },
+   "outputs": [],
+   "source": [
+    "# TODO Exercise 2: train model on all complete samples"
+   ],
+   "id": "0006-2b246d6be116cdc87429e01dd209927bc251258dae8b6b6aa00da953939"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Als nächstes füllen Sie `X` mit einem simplen Imputer (Mittelwert oder\n",
+    "Median) und werten auf gefüllten Testmenge den Fehler aus."
+   ],
+   "id": "0007-29464d9aa828481b9d1a94768e82a15f13fc68d66929044b49650b6c7fa"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "style": "python"
+   },
+   "outputs": [],
+   "source": [
+    "X_fill_simple = X.copy()\n",
+    "\n",
+    "# TODO Exercise 3a: impute gdp_percap and neonat_mortal_rate using median or mean\n",
+    "# TODO Exercise 3b: evaluate model on filled test set"
+   ],
+   "id": "0008-2c3c93b51c1909a78077b4929022c8b1b5bfa42183c8d005bf1581528ee"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Zuletzt machen Sie nochmals dasselbe, aber mit einem $k$ Nearest\n",
+    "Neighbor Imputer."
+   ],
+   "id": "0009-fe55307b85eff61e6939d7ea4da2f6afc056088b6ba8d51af46ab7ecfea"
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "style": "python"
+   },
+   "outputs": [],
+   "source": [
+    "X_fill_knn = X.copy()\n",
+    "\n",
+    "# TODO Exercise 4a: impute gdp_percap and neonat_mortal_rate using KNNImputer\n",
+    "# TODO Exercise 4b: evaluate model on filled test set"
+   ],
+   "id": "0010-c8dc15d1646bc841284cf0497b1a0483cd0a4ab70a1ac18f12c39825f40"
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Jetzt sollten Sie *für diesen Datensatz* die Frage beantworten können,\n",
+    "ob, wenn Sie für unvollständige Daten Vorhersagen durchführen müssen,\n",
+    "gefüllte Features in der Testmenge das Modell verbessern oder\n",
+    "verschlechtern."
+   ],
+   "id": "0011-146751deaf5a1f6ed1a9581e8a2de68567f7fb6842dd76893fb7ee5fee6"
+  }
+ ],
+ "nbformat": 4,
+ "nbformat_minor": 5,
+ "metadata": {}
+}
+%% Cell type:markdown id:0001-3c25ae28e6c0657f9cfc7b0066a7add26c991ca53c40803277a38c0e72d tags:
+# Fehler mit und ohne Imputer für die Testmenge
+Das Laden der Daten ist bereits vorbereitet:
+%% Cell type:code id:0002-e11f96c9f8b6ea98a99253f9057994e4a19c91b310bcb262d754bcee2ac tags:
+``` 
+import pandas as pd
+from sklearn.neighbors import KNeighborsRegressor
+from sklearn.impute import SimpleImputer, KNNImputer
+from sklearn.model_selection import train_test_split
+from sklearn.metrics import mean_absolute_error, mean_squared_error
+df = pd.read_csv('nations.csv')
+# convert income via categorical to range
+income_vals = ['Low income', 'Lower middle income', 'Upper middle income', 'High income', 'High income: OECD']
+income_cat_dtype = pd.api.types.CategoricalDtype(categories=income_vals, ordered=True)
+df.income = df.income.astype(income_cat_dtype).cat.codes
+# drop samples with missing values in features ['life_expect', 'population', 'birth_rate']
+# also drop samples when values are missing in both gdp_percap AND neonat_mortal_rate
+core_feat = ['year', 'population', 'birth_rate', 'income']
+incomplete_feat = ['gdp_percap', 'neonat_mortal_rate']
+target = 'life_expect'
+drop_idx = (df.gdp_percap.isna() & df.neonat_mortal_rate.isna()) | df[core_feat + [target]].isna().any(axis='columns')
+df = df[~drop_idx]
+# split into X and y. X has only missing values in incomplete_feat
+X = df[core_feat + incomplete_feat]
+y = df[target]
+```
+%% Cell type:markdown id:0003-6824753586f2898b08a189062eaf58ae145f256c8f3eb584a873cb62539 tags:
+Nun haben Sie `X` und `y`, aber in `X` gibt es fehlende Werte. Wir
+definieren die Testmenge, als die Menge der Samples bei denen mind. ein
+Wert fehlt. Wir versuchen zunächst herauszufinden wie groß der Fehler
+ist, wenn wir Features mit fehlenden Werten wegwerfen. Ihre Aufgabe ist
+nun auf den vollständigen Features (also ohne fehlende Werte) ein
+`KNeighborsRegressor` zu trainieren und den Fehler (MAE oder RMSE)
+auszuwerten.
+%% Cell type:code id:0004-2e44b46fb461aa34b89bb051f44d9f40708759418f9777ac3e3067be52a tags:
+``` 
+X_cmpl = X.dropna()
+y_cmpl = y[X_cmpl.index]
+# test set contains only incomplete samples
+y_test = y.drop(X_cmpl.index)
+# TODO Exercise 1a: train model on complete samples with dropped incomplete features
+# TODO Exercise 1b: evaluate on test set (drop incomplete features)
+```
+%% Cell type:markdown id:0005-facaa96e1a021f3a63651e20a81752665271f8f3e55167e6784350141ad tags:
+Wir wollen gleich Untersuchen, ob das Füllen der Testmenge den Fehler
+verringert oder erhöht. Das machen wir einmal mit einem simplen Imputer
+und einmal mit einem $k$-NN. In beiden Fällen ist jedoch das Modell
+dasselbe. Daher können Sie nun schon mal einen `KNeighborsRegressor` auf
+allen Features trainieren.
+%% Cell type:code id:0006-2b246d6be116cdc87429e01dd209927bc251258dae8b6b6aa00da953939 tags:
+``` 
+# TODO Exercise 2: train model on all complete samples
+```
+%% Cell type:markdown id:0007-29464d9aa828481b9d1a94768e82a15f13fc68d66929044b49650b6c7fa tags:
+Als nächstes füllen Sie `X` mit einem simplen Imputer (Mittelwert oder
+Median) und werten auf gefüllten Testmenge den Fehler aus.
+%% Cell type:code id:0008-2c3c93b51c1909a78077b4929022c8b1b5bfa42183c8d005bf1581528ee tags:
+``` 
+X_fill_simple = X.copy()
+# TODO Exercise 3a: impute gdp_percap and neonat_mortal_rate using median or mean
+# TODO Exercise 3b: evaluate model on filled test set
+```
+%% Cell type:markdown id:0009-fe55307b85eff61e6939d7ea4da2f6afc056088b6ba8d51af46ab7ecfea tags:
+Zuletzt machen Sie nochmals dasselbe, aber mit einem $k$ Nearest
+Neighbor Imputer.
+%% Cell type:code id:0010-c8dc15d1646bc841284cf0497b1a0483cd0a4ab70a1ac18f12c39825f40 tags:
+``` 
+X_fill_knn = X.copy()
+# TODO Exercise 4a: impute gdp_percap and neonat_mortal_rate using KNNImputer
+# TODO Exercise 4b: evaluate model on filled test set
+```
+%% Cell type:markdown id:0011-146751deaf5a1f6ed1a9581e8a2de68567f7fb6842dd76893fb7ee5fee6 tags:
+Jetzt sollten Sie *für diesen Datensatz* die Frage beantworten können,
+ob, wenn Sie für unvollständige Daten Vorhersagen durchführen müssen,
+gefüllte Features in der Testmenge das Modell verbessern oder
+verschlechtern.
--- a/07-imputer/nations.csv
+++ b/07-imputer/nations.csv
--- a/07-imputer/points.parquet
+++ b/07-imputer/points.parquet