{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Exercise: Bayesian neural networks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## A simple classification problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`scikit-learn` includes various random [sample generators](https://scikit-learn.org/stable/datasets/index.html#generated-datasets) that can be used to build artificial datasets of controlled size and complexity.\n",
    "\n",
    "For example, [`make_blobs`](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_blobs.html) generates two (or more) Gaussian distributions of data that correspond to different classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import datasets, linear_model\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Data sets from `scikit-learn`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, t = datasets.make_blobs(n_samples=20, cluster_std = 1.0, \\\n",
    "                           centers=[(-1.0, 1.0), (1.0,-1.0)], n_features=2,random_state=2019)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, t_train, t_test = \\\n",
    "          train_test_split(X, t, test_size=0.5, random_state=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Plot the training data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task 1: Logistic regression using `scikit-learn`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implement a logistic regression binary classifier using `scikit-learn`\n",
    "* Use an L2 regularizer with weight decay $\\alpha = 1.0$.\n",
    "* Print the best-fit parameters.\n",
    "* Create a grid in the $(x_1, x_2)$-plane and plot the decision boundary ($y=0.5$) for the binary classifier together with both the training and the test data.\n",
    "* Add also levels that correspond to the activation $a=\\pm1,\\pm2$ (what class probabilities do these activations correspond to?)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Task 2: Bayesian logistic regression using MCMC sampling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Implement instead a Bayesian binary classifier by considering the probability distributions for the three parameters of the single neuron (bias $w_0$ and weights $w_1,w_2$).\n",
    "1. You will need to define the single neuron as a function that takes data $\\boldsymbol{x}$ and parameters $\\boldsymbol{w}$ as input and returns the output $y$.\n",
    "1. You will also need to define the log prior for the parameters (use a pdf that is consistent with the choice of an L2-regularizer in the logistic regression implementation) and a log-likelihood for the data.\n",
    "1. Use an MCMC sampler to make draws from the posterior distribution of the neuron parameters and make a corner plot.\n",
    "1. Use a subset of the samples ($\\sim 50$) to make predictions on the $x_1,x_2$-grid that was created in task 1. \n",
    "  - Extract the mean and the standard deviation of the predictions for these sampled neurons on the grid (remember that each sample correspond to a neuron with those specific parameters).\n",
    "  - Plot the decision boundaries for ~ten of those samples. \n",
    "1. Finally, compare the predictions (mean and standard deviation) of your Bayesian binary classifier with those from the logistic regression approach."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Task 3: Bayesian logistic regression using Variational Inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To be added."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}