{ "cells": [ { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# Demonstration: Variational Inference and Bayesian Neural Networks" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "(c) 2016-2018 by Thomas Wiecki, updated by Maxim Kochurov\n", "Comments by Christian Forssén, 2019-2020.\n", "\n", "Original blog post: https://twiecki.github.io/blog/2016/06/01/bayesian-deep-learning/\n", "and udated version: https://docs.pymc.io/notebooks/bayesian_neural_network_advi.html." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This demonstration relies on the following additional packages:\n", "* `theano`: Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently, featuring tight integration with NumPy, transparent use of a GPU, efficient symbolic differentiation, speed and stability optimizations and dynamic C code generation.\n", "* `pymc3`: PyMC3 is a library for probabilistic programming in python. It allows you to write down models using an intuitive syntax to describe a data generating process.\n", "* `arviz`: ArviZ (pronounced \"AR-vees\") is a Python package for exploratory analysis of Bayesian models. Includes functions for posterior analysis, model checking, comparison and diagnostics.\n", "\n", "They can be installed with: `conda install `. \n", "\n", "You might have to specify using the conda-forge channel: `conda install -c conda-forge `." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "## Current trends in Machine Learning\n", "\n", "In this demonstration, it is shown how to use **Variational Inference** in PyMC3 to fit a simple Bayesian Neural Network. The authors will also discuss how bridging Probabilistic Programming and Deep Learning can open up very interesting avenues to explore in future research.\n", "\n", "### Probabilistic Programming at scale\n", "**Probabilistic Programming** allows very flexible creation of custom probabilistic models and is mainly concerned with **insight** and learning from your data. The approach is inherently **Bayesian** so we can specify **priors** to inform and constrain our models and get uncertainty estimation in form of a **posterior** distribution. Using [MCMC sampling algorithms](https://twiecki.github.io/blog/2015/11/10/mcmc-sampling/) we can draw samples from this posterior to very flexibly estimate these models. PyMC3 and [Stan](http://mc-stan.org/) are the current state-of-the-art tools to consruct and estimate these models. One major drawback of sampling, however, is that it's often very slow, especially for high-dimensional models. That's why more recently, **variational inference** algorithms have been developed that are almost as flexible as MCMC but much faster. Instead of drawing samples from the posterior, these algorithms instead fit a distribution (e.g. normal) to the posterior turning a sampling problem into and optimization problem. [ADVI](http://arxiv.org/abs/1506.03431) -- Automatic Differentation Variational Inference -- is implemented in PyMC3 and [Stan](http://mc-stan.org/), as well as a new package called [Edward](https://github.com/blei-lab/edward/) which is mainly concerned with Variational Inference. \n", "\n", "Unfortunately, when it comes to traditional ML problems like classification or (non-linear) regression, Probabilistic Programming often plays second fiddle (in terms of accuracy and scalability) to more algorithmic approaches like [ensemble learning](https://en.wikipedia.org/wiki/Ensemble_learning) (e.g. [random forests](https://en.wikipedia.org/wiki/Random_forest) or [gradient boosted regression trees](https://en.wikipedia.org/wiki/Boosting_(machine_learning))." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Deep Learning\n", "\n", "Now in its third renaissance, deep learning has been making headlines repeatadly by dominating almost any object recognition benchmark, [kicking ass at Atari games](https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf), and [beating the world-champion Lee Sedol at Go](http://www.nature.com/nature/journal/v529/n7587/full/nature16961.html). From a statistical point, Neural Networks are extremely good non-linear function approximators and representation learners. While mostly known for classification, they have been extended to unsupervised learning with [AutoEncoders](https://arxiv.org/abs/1312.6114) and in all sorts of other interesting ways (e.g. [Recurrent Networks](https://en.wikipedia.org/wiki/Recurrent_neural_network), or [MDNs](http://cbonnett.github.io/MDN_EDWARD_KERAS_TF.html) to estimate multimodal distributions). Why do they work so well? No one really knows as the statistical properties are still not fully understood.\n", "\n", "A large part of the innoviation in deep learning is the ability to train these extremely complex models. This rests on several pillars:\n", "* Speed: facilitating the GPU allowed for much faster processing.\n", "* Software: frameworks like [Theano](http://deeplearning.net/software/theano/) and [TensorFlow](https://www.tensorflow.org/) allow flexible creation of abstract models that can then be optimized and compiled to CPU or GPU.\n", "* Learning algorithms: training on sub-sets of the data -- stochastic gradient descent -- allows us to train these models on massive amounts of data. Techniques like drop-out avoid overfitting.\n", "* Architectural: A lot of innovation comes from changing the input layers, like for convolutional neural nets, or the output layers, like for [MDNs](http://cbonnett.github.io/MDN_EDWARD_KERAS_TF.html)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Bridging Deep Learning and Probabilistic Programming\n", "On one hand we have Probabilistic Programming which allows us to build rather small and focused models in a very principled and well-understood way to gain insight into our data; on the other hand we have deep learning which uses many heuristics to train huge and highly complex models that are amazing at prediction. Recent innovations in variational inference allow probabilistic programming to scale model complexity as well as data size. We are thus at the cusp of being able to combine these two approaches to hopefully unlock new innovations in Machine Learning. For more motivation, see also [Dustin Tran's](https://twitter.com/dustinvtran) recent [blog post](http://dustintran.com/blog/a-quick-update-edward-and-some-motivations/)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "While this would allow Probabilistic Programming to be applied to a much wider set of interesting problems, I believe this bridging also holds great promise for innovations in Deep Learning. Some ideas are:\n", "* **Uncertainty in predictions**: As we will see below, the Bayesian Neural Network informs us about the uncertainty in its predictions. I think uncertainty is an underappreciated concept in Machine Learning as it's clearly important for real-world applications. But it could also be useful in training. For example, we could train the model specifically on samples it is most uncertain about.\n", "* **Uncertainty in representations**: We also get uncertainty estimates of our weights which could inform us about the stability of the learned representations of the network.\n", "* **Regularization with priors**: Weights are often L2-regularized to avoid overfitting, this very naturally becomes a Gaussian prior for the weight coefficients. We could, however, imagine all kinds of other priors, like spike-and-slab to enforce sparsity (this would be more like using the L1-norm).\n", "* **Transfer learning with informed priors**: If we wanted to train a network on a new object recognition data set, we could bootstrap the learning by placing informed priors centered around weights retrieved from other pre-trained networks, like [GoogLeNet](https://arxiv.org/abs/1409.4842). \n", "* **Hierarchical Neural Networks**: A very powerful approach in Probabilistic Programming is hierarchical modeling that allows pooling of things that were learned on sub-groups to the overall population (see my tutorial on [Hierarchical Linear Regression in PyMC3](https://twiecki.github.io/blog/2014/03/17/bayesian-glms-3/)). Applied to Neural Networks, in hierarchical data sets, we could train individual neural nets to specialize on sub-groups while still being informed about representations of the overall population. For example, imagine a network trained to classify car models from pictures of cars. We could train a hierarchical neural network where a sub-neural network is trained to tell apart models from only a single manufacturer. The intuition being that all cars from a certain manufactures share certain similarities so it would make sense to train individual networks that specialize on brands. However, due to the individual networks being connected at a higher layer, they would still share information with the other specialized sub-networks about features that are useful to all brands. Interestingly, different layers of the network could be informed by various levels of the hierarchy -- e.g. early layers that extract visual lines could be identical in all sub-networks while the higher-order representations would be different. The hierarchical model would learn all that from the data.\n", "* **Other hybrid architectures**: We can more freely build all kinds of neural networks. For example, Bayesian non-parametrics could be used to flexibly adjust the size and shape of the hidden layers to optimally scale the network architecture to the problem at hand during training. Currently, this requires costly hyper-parameter optimization and a lot of tribal knowledge." ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "## Bayesian Neural Networks in PyMC3" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Generating data\n", "\n", "First, lets generate some toy data -- a simple binary classification problem that's not linearly separable." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", "import sklearn\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "#import seaborn as sns\n", "from warnings import filterwarnings\n", "filterwarnings('ignore')\n", "\n", "import seaborn as sns\n", "sns.set()\n", "sns.set_context(\"talk\")\n", "\n", "#sns.set_style('presentation')\n", "from sklearn import datasets\n", "from sklearn.preprocessing import scale\n", "from sklearn.model_selection import train_test_split\n", "from sklearn.datasets import make_moons" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "#### Import theano and pymc3" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "ename": "ModuleNotFoundError", "evalue": "No module named 'theano'", "output_type": "error", "traceback": [ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[0;31mModuleNotFoundError\u001b[0m Traceback (most recent call last)", "Cell \u001b[0;32mIn[2], line 2\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[38;5;66;03m# suppress some theano compilation errors with MacOSX clang compiler\u001b[39;00m\n\u001b[0;32m----> 2\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mtheano\u001b[39;00m\n\u001b[1;32m 3\u001b[0m \u001b[38;5;66;03m# CF added (https://stackoverflow.com/questions/51238578/error-non-constant-expression-cannot-be-narrowed-from-type-npy-intp-to-int)\u001b[39;00m\n\u001b[1;32m 4\u001b[0m theano\u001b[38;5;241m.\u001b[39mconfig\u001b[38;5;241m.\u001b[39mgcc\u001b[38;5;241m.\u001b[39mcxxflags \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m-Wno-c++11-narrowing\u001b[39m\u001b[38;5;124m\"\u001b[39m\n", "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'theano'" ] } ], "source": [ "# suppress some theano compilation errors with MacOSX clang compiler\n", "import theano\n", "# CF added (https://stackoverflow.com/questions/51238578/error-non-constant-expression-cannot-be-narrowed-from-type-npy-intp-to-int)\n", "theano.config.gcc.cxxflags = \"-Wno-c++11-narrowing\"\n", "floatX = theano.config.floatX\n", "import pymc3 as pm\n", "import theano.tensor as T" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "X, Y = make_moons(noise=0.2, random_state=0, n_samples=1000)\n", "X = scale(X)\n", "X = X.astype(floatX)\n", "Y = Y.astype(floatX)\n", "X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=.5)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "slideshow": { "slide_type": "subslide" } }, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize=(8,8))\n", "ax.scatter(X[Y==0, 0], X[Y==0, 1], label='Class 0')\n", "ax.scatter(X[Y==1, 0], X[Y==1, 1], color='r', label='Class 1')\n", "sns.despine(); ax.legend()\n", "ax.set(xlabel='X', ylabel='Y', title='Toy binary classification data set');" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "subslide" } }, "source": [ "### Model specification\n", "\n", "A neural network is quite simple. The basic unit is a [perceptron](https://en.wikipedia.org/wiki/Perceptron) which is nothing more than [logistic regression](http://pymc-devs.github.io/pymc3/notebooks/posterior_predictive.html#Prediction). We use many of these in parallel and then stack them up to get hidden layers. Here we will use 2 hidden layers with 5 neurons each which is sufficient for such a simple problem." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def construct_nn(ann_input, ann_output):\n", " n_hidden = 5\n", "\n", " # Initialize random weights between each layer\n", " init_1 = np.random.randn(X.shape[1], n_hidden).astype(floatX)\n", " init_2 = np.random.randn(n_hidden, n_hidden).astype(floatX)\n", " init_out = np.random.randn(n_hidden).astype(floatX)\n", "\n", " with pm.Model() as neural_network:\n", " # Trick: Turn inputs and outputs into shared variables using the data container pm.Data\n", " # It's still the same thing, but we can later change the values of the shared variable\n", " # (to switch in the test-data later) and pymc3 will just use the new data.\n", " # Kind-of like a pointer we can redirect.\n", " # For more info, see: http://deeplearning.net/software/theano/library/compile/shared.html\n", " ann_input = pm.Data('ann_input', X_train)\n", " ann_output = pm.Data('ann_output', Y_train)\n", "\n", " # Weights from input to hidden layer\n", " weights_in_1 = pm.Normal('w_in_1', 0, sigma=1,\n", " shape=(X.shape[1], n_hidden),\n", " testval=init_1)\n", "\n", " # Weights from 1st to 2nd layer\n", " weights_1_2 = pm.Normal('w_1_2', 0, sigma=1,\n", " shape=(n_hidden, n_hidden),\n", " testval=init_2)\n", "\n", " # Weights from hidden layer to output\n", " weights_2_out = pm.Normal('w_2_out', 0, sigma=1,\n", " shape=(n_hidden,),\n", " testval=init_out)\n", "\n", " # Build neural-network using tanh activation function\n", " act_1 = pm.math.tanh(pm.math.dot(ann_input,\n", " weights_in_1))\n", " act_2 = pm.math.tanh(pm.math.dot(act_1,\n", " weights_1_2))\n", " act_out = pm.math.sigmoid(pm.math.dot(act_2,\n", " weights_2_out))\n", "\n", " # Binary classification -> Bernoulli likelihood\n", " out = pm.Bernoulli('out',\n", " act_out,\n", " observed=ann_output,\n", " total_size=Y_train.shape[0] # IMPORTANT for minibatches\n", " )\n", " return neural_network\n", "\n", "neural_network = construct_nn(X_train, Y_train)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "That's not so bad. The `Normal` priors help regularize the weights. Usually we would add a constant `b` to the inputs but I omitted it here to keep the code cleaner." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Variational Inference: Scaling model complexity\n", "\n", "We could now just run a MCMC sampler like NUTS which works pretty well in this case, but as I already mentioned, this will become very slow as we scale our model up to deeper architectures with more layers.\n", "\n", "Instead, we will use [ADVI](https://arxiv.org/abs/1603.00788) variational inference algorithm which was recently added to `PyMC3`, and updated to use the operator variational inference (OPVI) framework. This is much faster and will scale better." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Note**, however, that this is a mean-field approximation so we **ignore correlations** in the posterior." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from pymc3.theanof import set_tt_rng, MRG_RandomStreams\n", "set_tt_rng(MRG_RandomStreams(42))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%time\n", "\n", "with neural_network:\n", " inference = pm.ADVI()\n", " approx = pm.fit(n=30000, method=inference)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Performance wise that's pretty good considering that NUTS is having a really hard time. Further below we make this even faster. To make it really fly, we probably want to run the Neural Network on the GPU.\n", "\n", "As samples are more convenient to work with, we can very quickly draw samples from the variational approximation using the `sample` method (this is just sampling from Normal distributions, so not at all the same like MCMC):" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Plotting the objective function (ELBO) we can see that the optimization slowly improves the fit over time." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize=(8,6))\n", "\n", "ax.plot(-inference.hist, label='new ADVI', alpha=.3)\n", "ax.legend()\n", "ax.set_ylabel('ELBO')\n", "ax.set_xlabel('iteration')\n", "plt.tight_layout();" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig.savefig('fig/ADVI-classifier_ELBO.png')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "trace = approx.sample(draws=5000)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now that we trained our model, lets predict on the hold-out set using a posterior predictive check (PPC). \n", "1. We can use `sample_posterior_predictive() <../api/inference.rst>`__ to generate new data (in this case class predictions) from the posterior (sampled from the variational estimation).\n", "1. It is better to get the node directly and build theano graph using our approximation (approx.sample_node) , we get a lot of speed up" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# We can get predicted probability from model\n", "neural_network.out.distribution.p" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# create symbolic input\n", "x = T.matrix('X')\n", "# symbolic number of samples is supported, we build vectorized posterior on the fly\n", "n = T.iscalar('n')\n", "# Do not forget test_values or set theano.config.compute_test_value = 'off'\n", "x.tag.test_value = np.empty_like(X_train[:10])\n", "n.tag.test_value = 100\n", "_sample_proba = approx.sample_node(neural_network.out.distribution.p,\n", " size=n,\n", " more_replacements={neural_network['ann_input']: x})\n", "# It is time to compile the function\n", "# No updates are needed for Approximation random generator\n", "# Efficient vectorized form of sampling is used\n", "sample_proba = theano.function([x, n], _sample_proba)\n", "\n", "# Create bechmark functions\n", "def production_step1():\n", " pm.set_data(new_data={'ann_input': X_test, 'ann_output': Y_test}, model=neural_network)\n", " ppc = pm.sample_posterior_predictive(trace, samples=500, progressbar=False, model=neural_network)\n", "\n", " # Use probability of > 0.5 to assume prediction of class 1\n", " pred = ppc['out'].mean(axis=0) > 0.5\n", "\n", "def production_step2():\n", " sample_proba(X_test, 500).mean(0) > 0.5" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "See the difference" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%timeit production_step1()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%timeit production_step2()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let’s go ahead and generate predictions:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "pred = sample_proba(X_test, 500).mean(0) > 0.5" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize=(8,8))\n", "ax.scatter(X_test[pred==0, 0], X_test[pred==0, 1])\n", "ax.scatter(X_test[pred==1, 0], X_test[pred==1, 1], color='r')\n", "sns.despine()\n", "ax.set(title='Predicted labels in testing set', xlabel='X', ylabel='Y');" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print('Accuracy = {}%'.format((Y_test == pred).mean() * 100))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Hey, our neural network did all right!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Lets look at what the classifier has learned\n", "\n", "For this, we evaluate the class probability predictions on a grid over the whole input space." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "grid = pm.floatX(np.mgrid[-3:3:100j,-3:3:100j])\n", "grid_2d = grid.reshape(2, -1).T\n", "dummy_out = np.ones(grid.shape[1], dtype=np.int8)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ppc = sample_proba(grid_2d ,500)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Probability surface" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "cmap = sns.diverging_palette(250, 12, s=85, l=25, as_cmap=True)\n", "fig, ax = plt.subplots(figsize=(12, 9))\n", "contour = ax.contourf(grid[0], grid[1], ppc.mean(axis=0).reshape(100, 100), cmap=cmap)\n", "ax.scatter(X_test[pred==0, 0], X_test[pred==0, 1])\n", "ax.scatter(X_test[pred==1, 0], X_test[pred==1, 1], color='r')\n", "cbar = plt.colorbar(contour, ax=ax)\n", "_ = ax.set(xlim=(-3, 3), ylim=(-3, 3), xlabel='X', ylabel='Y');\n", "cbar.ax.set_ylabel('Posterior predictive mean probability of class label = 0');" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Uncertainty in predicted value\n", "\n", "So far, everything I showed we could have done with a non-Bayesian Neural Network. The mean of the posterior predictive for each class-label should be identical to maximum likelihood predicted values. However, we can also look at the standard deviation of the posterior predictive to get a sense for the uncertainty in our predictions. Here is what that looks like:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "cmap = sns.cubehelix_palette(light=1, as_cmap=True)\n", "fig, ax = plt.subplots(figsize=(12, 9))\n", "contour = ax.contourf(grid[0], grid[1], ppc.std(axis=0).reshape(100, 100), cmap=cmap)\n", "ax.scatter(X_test[pred==0, 0], X_test[pred==0, 1])\n", "ax.scatter(X_test[pred==1, 0], X_test[pred==1, 1], color='r')\n", "cbar = plt.colorbar(contour, ax=ax)\n", "_ = ax.set(xlim=(-3, 3), ylim=(-3, 3), xlabel='X', ylabel='Y');\n", "cbar.ax.set_ylabel('Uncertainty (posterior predictive standard deviation)');" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see that very close to the decision boundary, our uncertainty as to which label to predict is highest. You can imagine that associating predictions with uncertainty is a critical property for many applications like health care. To further maximize accuracy, we might want to train the model primarily on samples from that high-uncertainty region." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "It is also clear that the uncertainty is large in the region where there is no training data. That is what should be expected, and it is good that our network shows this explicitly. The normal neural network would not give any such signals." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Mini-batch ADVI\n", "\n", "So far, we have trained our model on all data at once. Obviously this won't scale to something like ImageNet. Moreover, training on mini-batches of data (stochastic gradient descent) avoids local minima and can lead to faster convergence.\n", "\n", "Fortunately, ADVI can be run on mini-batches as well. It just requires some setting up:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "minibatch_x = pm.Minibatch(X_train, batch_size=50)\n", "minibatch_y = pm.Minibatch(Y_train, batch_size=50)\n", "neural_network_minibatch = construct_nn(minibatch_x, minibatch_y)\n", "with neural_network_minibatch:\n", " approx = pm.fit(40000, method=pm.ADVI())" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize=(8,6))\n", "\n", "ax.plot(-inference.hist)\n", "ax.set_ylabel('ELBO')\n", "ax.set_xlabel('iteration');" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As you can see, mini-batch ADVI's running time is much lower. It also seems to converge faster.\n", "\n", "For fun, we can also look at the trace. The point is that we also get uncertainty of our Neural Network weights." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The plotting of traces in pyMC3 requires the ArviZ module." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "pm.traceplot(trace);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## Next steps\n", "\n", "[`Theano`](http://deeplearning.net/software/theano/), which is used by `PyMC3` as its computational backend, was mainly developed for estimating neural networks and there are great libraries like [`Lasagne`](https://github.com/Lasagne/Lasagne) that build on top of `Theano` to make construction of the most common neural network architectures easy. There is a [follow-up blog post on how to use Lasagne together with PyMC3](https://twiecki.github.io/blog/2016/07/05/bayesian-deep-learning/).\n", "\n", "You should be able to run this example on the GPU by setting `device = gpu` and `floatX = float32` in your `.theanorc`.\n", "\n", "You might argue that the above network isn't really deep, but note that we could easily extend it to have more layers, including convolutional ones to train on more challenging data sets, as demonstrated [here]([follow-up blog post on how to use Lasagne together with PyMC3](https://twiecki.github.io/blog/2016/07/05/bayesian-deep-learning/)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Acknowledgements\n", "\n", "This notebook is very much inspired by a blog post written by Thomas Wiecki (and updated by Maxim Kochurov). The\n", "original posts are available [here](http://twiecki.github.io/blog/2016/06/01/bayesian-deep-learning/) and [here](https://docs.pymc.io/notebooks/bayesian_neural_network_advi.html).\n", "\n", "This example uses the [PyMC3](https://docs.pymc.io/) package for probabilistic programming, and relies in particular on the [ADVI](http://pymc-devs.github.io/pymc3/api.html#advi) implementation by [Taku Yoshioka](https://github.com/taku-y)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "anaconda-cloud": {}, "celltoolbar": "Slideshow", "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.17" }, "latex_envs": { "bibliofile": "biblio.bib", "cite_by": "apalike", "current_citInitial": 1, "eqLabelWithNumbers": true, "eqNumInitial": 0 }, "nav_menu": {}, "toc": { "navigate_menu": true, "number_sections": true, "sideBar": true, "threshold": 6, "toc_cell": false, "toc_section_display": "block", "toc_window_display": false } }, "nbformat": 4, "nbformat_minor": 4 }