Browse Source

align intros of notebooks

Robin Gutzen 5 years ago
parent
commit
aaa16ab074
2 changed files with 52 additions and 63 deletions
  1. 17 32
      NetworkUnit_examples.ipynb
  2. 35 31
      generate_validation_results.ipynb

+ 17 - 32
NetworkUnit_examples.ipynb

@@ -5,7 +5,7 @@
    "metadata": {},
    "source": [
     "# Quantitative comparison of experimental data sets with NetworkUnit\n",
-    "Supplementary notebook for *Gutzen et al. (2018) Reproducible neural network simulations: statistical methods for model validation on the level of network activity data, Frontiers in Neuroinformatics*."
+    "Supplementary notebook for *Gutzen, R., von Papen, M., Trensch, G., Quaglio, P., Grün, S., and Denker, M. (2018) Reproducible neural network simulations: statistical methods for model validation on the level of network activity data*."
    ]
   },
   {
@@ -13,13 +13,13 @@
    "metadata": {},
    "source": [
     "## Table of Contents\n",
-    "1. [Introduction](#intro)  \n",
+    "* [Introduction](#intro)  \n",
     "* [Setup](#set)  \n",
     "* [NetworkUnit classes](#cla)  \n",
     "* [Results](#res)  \n",
-    "    3.1. [Firing rates](#fir)  \n",
-    "    3.2. [Inter spike intervals](#int)  \n",
-    "    3.3. [Local coefficient of variation](#loc)  \n",
+    "    * [Firing rates](#fir)  \n",
+    "    * [Inter spike intervals](#int)  \n",
+    "    * [Local coefficient of variation](#loc)  \n",
     "* [Conclusions](#con)  \n",
     "* [References](#ref)  "
    ]
@@ -29,7 +29,7 @@
    "metadata": {},
    "source": [
     "# Introduction<a id='intro'></a>\n",
-    "This notebook showcases the basic workflow of [NetworkUnit](https://github.com/INM-6/NetworkUnit) applied to experimental data and provides an example of how to use [NetworkUnit](https://github.com/INM-6/NetworkUnit) to quantify the statistical difference between data sets (instead of model validation). **The difference between experimental datasets is the basis to define the acceptable agreement for a model** that aims to explain these datasets. The quantification of the statistical difference between the underlying experimental data is therefore an important first step in the general validation process. However, care must be taken in interpreting these results due to the large number of factors contributing to the observed variability. Such factors include amongst others a slightly different electrode location within the motor cortex for the two monkeys and differences in the signal-to-noise ratio of the measurements which affects spike sorting and can result in different response profiles.\n",
+    "This notebook showcases the basic workflow of [NetworkUnit](https://github.com/INM-6/NetworkUnit) applied to experimental data and provides an example of how to use [NetworkUnit](https://github.com/INM-6/NetworkUnit) to quantify the statistical difference between data sets (instead of model validation). **The quantified difference between experimental datasets can be used to guide the definiton of the acceptable agreement for a model** that aims to explain these datasets. The quantification of the statistical difference between the underlying experimental data is therefore an important first step in the general validation process. However, care must be taken in interpreting these results due to the large number of factors contributing to the observed variability. Such factors include amongst others a slightly different electrode location within the motor cortex for the two monkeys and differences in the signal-to-noise ratio of the measurements which affects spike sorting and can result in different response profiles.\n",
     "\n",
     "The experimental data for this example is taken from [*Brochier et al.* (2018)](#brochier2018). The provided datasets were recorded in the motor cortex of two macaque monkeys using Utah multi-electrode arrays. The underlying task was an instructed delayed reach-to-grasp task, the details of which are described in [*Riehle et al.* (2013)](#riehle2013) and [*Brochier et al.* (2018)](#brochier2018)."
    ]
@@ -48,7 +48,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Clone NetworkUnit from git\n",
+    "## Clone NetworkUnit from GitHub\n",
     "Clone the newest version of the repository NetworkUnit to current directory and adds the corresponding path to `sys`."
    ]
   },
@@ -61,29 +61,14 @@
      "start_time": "2018-10-24T10:22:52.610372Z"
     }
    },
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Klone nach 'NetworkUnit' ...\n",
-      "remote: Enumerating objects: 5, done.\u001b[K\n",
-      "remote: Counting objects: 100% (5/5), done.\u001b[K\n",
-      "remote: Compressing objects: 100% (5/5), done.\u001b[K\n",
-      "remote: Total 522 (delta 0), reused 1 (delta 0), pack-reused 517\u001b[K\n",
-      "Empfange Objekte: 100% (522/522), 4.50 MiB | 2.44 MiB/s, Fertig.\n",
-      "Löse Unterschiede auf: 100% (339/339), Fertig.\n",
-      "Prüfe Konnektivität ... Fertig.\n",
-      "Already up-to-date.\n"
-     ]
-    }
-   ],
+   "outputs": [],
    "source": [
+    "%%capture\n",
     "import sys, os\n",
     "\n",
-    "# Clone repository and pull latest version\n",
+    "# Clone repository and pull commit at time of paper submission\n",
     "!git clone https://github.com/INM-6/NetworkUnit.git\n",
-    "!cd NetworkUnit/; git fetch; git pull\n",
+    "!cd NetworkUnit/; git checkout 'f9d5c6deabe56c2419b9698d285a62af254cfc56'; git fetch; git pull\n",
     "\n",
     "# add to path\n",
     "sys.path.insert(0, './NetworkUnit')"
@@ -977,21 +962,21 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "Python 2",
+   "display_name": "Python 3",
    "language": "python",
-   "name": "python2"
+   "name": "python3"
   },
   "language_info": {
    "codemirror_mode": {
     "name": "ipython",
-    "version": 2
+    "version": 3
    },
    "file_extension": ".py",
    "mimetype": "text/x-python",
    "name": "python",
    "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython2",
-   "version": "2.7.12"
+   "pygments_lexer": "ipython3",
+   "version": "3.6.6"
   },
   "toc": {
    "colors": {
@@ -1026,5 +1011,5 @@
   }
  },
  "nbformat": 4,
- "nbformat_minor": 1
+ "nbformat_minor": 2
 }

+ 35 - 31
generate_validation_results.ipynb

@@ -4,18 +4,11 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Demonstration of validation testing with NetworkUnit by replicating the results in Gutzen et al. (sub.)\n",
-    "This notebook showcases the basic validation workflow with the [SciUnit](https://github.com/scidash/sciunit) package and the test repository [NetworkUnit](https://github.com/INM-6/NetworkUnit).\n",
+    "# Demonstration of validation testing with NetworkUnit by replicating the results in Gutzen et al. (2018)\n",
+    "This notebook should showcase the basic validation workflow with the [SciUnit](https://github.com/scidash/sciunit) package and the test repository [NetworkUnit](https://github.com/INM-6/NetworkUnit). Here, the results are replicated reported in\n",
     "\n",
-    "Requirements (>= Python 2.7.14):\n",
-    "\n",
-    "        elephant == 0.5.0\n",
-    "        sciunit == 0.2.0.2\n",
-    "        jupyter == 1.0.0\n",
-    "        tabulate == 0.8.2\n",
-    "        networkx == 2.1\n",
-    "        fastcluster == 1.1.24\n",
-    "        seaborn == 0.8.1\n",
+    "Gutzen, R., von Papen, M., Trensch, G., Quaglio, P., Grün, S., and Denker, M. (2018). \n",
+    "*Reproducible neural network simulations: statistical methods for model validation on the level of network activity data*\n",
     "        \n",
     "**Note:** because of the large number of neurons, the computations involving the calculation of correlation coefficients for all pairs of neurons are very expensive, i.e.very time consuming."
    ]
@@ -24,7 +17,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Table of Contents\n",
+    "## Table of Contents\n",
     "- [Setup](#setup)\n",
     "1. [Polychrony model class](#poly_model)\n",
     "1. [Iteration I](#it1)\n",
@@ -47,8 +40,19 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Setup <a id='setup'></a>\n",
-    "importing the required packages"
+    "# Setup <a id='setup'></a>\n",
+    "## Requirements\n",
+    "The notebook is executable by \n",
+    "* cloning this [repository](https://web.gin.g-node.org/INM-6/network_validation) \n",
+    "* and installing the necessary packages defined in the [requirements file](https://web.gin.g-node.org/INM-6/network_validation/src/master/requirements.txt).\n",
+    "* It runs with Python 2.7 and Python 3."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Importing the required packages"
    ]
   },
   {
@@ -76,7 +80,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Load NetworkUnit from GitHub (only required once)\n",
+    "### Load NetworkUnit from GitHub (only required once)\n",
     "Since the module is not yet pip-installable it needs to be cloned from GitHub."
    ]
   },
@@ -164,7 +168,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Define the polychronization model class<a id='poly_model'></a>\n",
+    "## Define the polychronization model class<a id='poly_model'></a>\n",
     "Within the SciUnit framework the models are represented as class objects. These model classes inherit from a capability class (here `ProducesSpikeTrains`) and possibly other model classes. NetworkUnit provides an abstract model class `spiketrain_data` which already has the capability of `ProducesSpikeTrains` and several other utility functions for spiketrain data such as aligning the spiketrains to 0 ms.\n",
     "\n",
     "The essential property of the model class is to generate a simulation outcome, either by executing the simulation or by loading the data of a previously run simulation. Here we do the latter and thus equip the class with a load() function and the variable file_path."
@@ -224,14 +228,14 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Iteration I<a id='it1'></a>"
+    "## Iteration I<a id='it1'></a>"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Define the model instances<a id='model1'></a>\n",
+    "### Define the model instances<a id='model1'></a>\n",
     "The abstract parent class polychrony_data provides the loading routine for the simulation data. As this is equivalent for both simulators, this way the same routine is simply inherited by simulation specific child classes.\n",
     "The child classes thus only need to specify the file path to the corresponding simulation outcome."
    ]
@@ -281,7 +285,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Show rasterplots\n",
+    "### Show rasterplots\n",
     "This is equivalent to the spiking activity displayed in Fig. 5 in Gutzen et al."
    ]
   },
@@ -316,7 +320,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Define test classes<a id='test1'></a>\n",
+    "### Define test classes<a id='test1'></a>\n",
     "The abstract base classes for these tests are implemented in NetworkUnit so that here the tests can be easily defined by inherting from these test classes and setting the parameters and pair the test with a score class. To perfom a hypothesis test the `effect_size` score can be replaced for example with the `ks_distance` or the `mwu_statistic`.\n",
     "The inhereted `TestM2M` class adapts the test such that the tests don't need to be initiliazed with experimental data. "
    ]
@@ -353,7 +357,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### How to generate the test prediction\n",
+    "### How to generate the test prediction\n",
     "As the essential attribute of a model is to be able to generate predictions, the main functionality of the test class if the generate_prediction() function. This function calculates the measure(s), corresponding to the specific test (here firing rates), given by the simulation or experimental data set."
    ]
   },
@@ -372,7 +376,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### How to generate the test score\n",
+    "### How to generate the test score\n",
     "In order to not only generate the predictions but to also compare them, i.e. perform the validation test, the judge function is called. \n",
     "The `judge()` function does:\n",
     "    1. checks if the model has all the required capabilities. If a model  \n",
@@ -565,7 +569,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Visualization<a id='viz'></a>\n",
+    "### Visualization<a id='viz'></a>\n",
     "The visual inspection of the samples of comparison measures is a very relevant part of the validation workflow. Thus, all tests have the inherent functionality to visualize their predictions by the function `visualize_samples()`.\n",
     "\n",
     "These plots are equivalent to those displayed in Fig. 5 in Gutzen et al.."
@@ -758,14 +762,14 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Iteration II<a id='poly_model'></a>"
+    "## Iteration II<a id='poly_model'></a>"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Define the model (-instances)<a id='model2'></a>\n",
+    "### Define the model (-instances)<a id='model2'></a>\n",
     "\n",
     "As before the specific simulations are represented by child classes of `polychrony_data` with the corresponding file paths to the simulation results. Here, we even another generation for the simulations of every network state."
    ]
@@ -939,14 +943,14 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Iteration III<a id='it3'></a>"
+    "## Iteration III<a id='it3'></a>"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Define the model (-instances)<a id='model3'></a>"
+    "### Define the model (-instances)<a id='model3'></a>"
    ]
   },
   {
@@ -1017,7 +1021,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Define additional tests<a id='test3'></a>\n",
+    "### Define additional tests<a id='test3'></a>\n",
     "We add additional validation tests in order to get a more comprehensive assesment of the network dynamics, quantify the overall similarity/difference more accurately, and ideally increase the confidence we have in ther (qualitative) equivalence."
    ]
   },
@@ -1053,7 +1057,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### perform tests and average over the 5 network states<a id='test3_avg'></a>"
+    "### perform tests and average over the 5 network states<a id='test3_avg'></a>"
    ]
   },
   {
@@ -1168,7 +1172,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Performing further testing of the correlation structure<a id='test3_struct'></a>\n",
+    "### Performing further testing of the correlation structure<a id='test3_struct'></a>\n",
     "To also take into account the organisation or correlations in the two simulations, we compare the rate correlation matrices by their normed scalar product and assess the significance by comparison with surrogate data. This score is not implemented in NetworkUnit, but we use the NetworkUnit test formalism to generate and visualize the predictions (the correlation matrices). This figure is equivalent to Fig. 8 in Gutzen et al."
    ]
   },
@@ -1308,7 +1312,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Calculate the power spectrum<a id='test3_power'></a>"
+    "### Calculate the power spectrum<a id='test3_power'></a>"
    ]
   },
   {