added more explanatory comments

NetworkUnit_examples.ipynb

@@ -5,7 +5,7 @@
    "metadata": {},
    "source": [
     "# Quantitative comparison of experimental data sets with NetworkUnit\n",
-    "Supplementary notebook for *Gutzen et al. (2018) Reproducible neural network simulations: model validation on the level of network activity data, Frontiers in Neuroinformatics, submitted*."
+    "Supplementary notebook for *Gutzen et al. (2018) Reproducible neural network simulations: statistical methods for model validation on the level of network activity data, Frontiers in Neuroinformatics, 2018*."
    ]
   },
   {
@@ -394,7 +394,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Summary"
+    "## Summarize data"
    ]
   },
   {

generate_validation_results.ipynb

@@ -4,8 +4,8 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## Demonstration of validation testing with NetworkUnit by replicating the results in Gutzen et al. (sub.)\n",
-    "This notebook should showcase the basic validation workflow with the sciunit package and the test repository networkunit.\n",
+    "# Demonstration of validation testing with NetworkUnit by replicating the results in Gutzen et al. (2018)\n",
+    "This notebook showcases the basic validation workflow of [NetworkUnit](https://github.com/INM-6/NetworkUnit) based on the functionality provided by the [SciUnit](https://github.com/scidash/sciunit) package. A detailed explanation of the validation workflow and the corresponding tests can be found in *Gutzen et al. (2018) Reproducible neural network simulations: statistical methods for model validation on the level of network activity data, Frontiers in Neuroinformatics, 2018*.\n",
     "\n",
     "\n",
     "\n",
@@ -24,14 +24,14 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Table of Contents\n",
+    "## Table of Contents\n",
     "- [Setup](#setup)\n",
     "1. [Polychrony model class](#poly_model)\n",
     "1. [Iteration I](#it1)\n",
     "    - [Define model classes](#model1)\n",
     "    1. [Define test classes and how to perform a test](#test1)\n",
     "    1. [Visualization](#viz)\n",
-    "    1. [Artefact detection](#artfcts)\n",
+    "    1. [Analysis of spike train cross-correlation](#xcor)\n",
     "1. [Iteration II](#it2)\n",
     "    - [Define model classes](#model2)\n",
     "    1. [Perform validation tests and average over network states](#test2)\n",
@@ -47,7 +47,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Setup <a id='setup'></a>"
+    "## Setup <a id='setup'></a>"
    ]
   },
   {
@@ -73,15 +73,13 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### load a version of NetworkUnit (only required once)"
+    "### Load a version of NetworkUnit (only required once)"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 2,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "%%capture\n",
@@ -91,9 +89,7 @@
   {
    "cell_type": "code",
    "execution_count": 3,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "%%capture\n",
@@ -103,9 +99,7 @@
   {
    "cell_type": "code",
    "execution_count": 4,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "%%capture\n",
@@ -116,9 +110,7 @@
   {
    "cell_type": "code",
    "execution_count": 5,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "%%capture\n",
@@ -153,7 +145,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Define the polychronization model class<a id='poly_model'></a>\n",
+    "## Define the polychronization model class<a id='poly_model'></a>\n",
     "NetworkUnit does not yet have an interface to HPC resources or SpiNNaker to perform the simulations automatically. Therefore, the outcomes of manually performed simulations are loaded into the model class."
    ]
   },
@@ -211,14 +203,14 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Iteration I<a id='it1'></a>"
+    "## Iteration I<a id='it1'></a>"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Define the model instances<a id='model1'></a>"
+    "### Define the model instances<a id='model1'></a>"
    ]
   },
   {
@@ -253,9 +245,7 @@
   {
    "cell_type": "code",
    "execution_count": 10,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "C = C_sim(name='C')\n",
@@ -266,7 +256,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Show rasterplots"
+    "### Show rasterplots"
    ]
   },
   {
@@ -300,9 +290,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Define test classes<a id='test1'></a>\n",
-    "The abstract base classes for these tests are implemented in NetworkUnit so that here only the parameters have to be set and the test is paired with a score class. To perfom a hypothesis test the effect_size score can be replaced for example with the ks_distance or the mwu_statistic.\n",
-    "The inhereted TestM2M class adapts the test such that the tests don't need to be initiliazed with experimental data. "
+    "### Define test classes<a id='test1'></a>\n",
+    "The abstract base classes for these tests are implemented in NetworkUnit so that here only the parameters have to be set and the test is paired with a score class. To perfom a hypothesis test the `effect_size` score can be replaced for example with the `ks_distance` or the `mwu_statistic`.\n",
+    "The inhereted `TestM2M` class adapts the test such that the tests don't need to be initialized with experimental data. "
    ]
   },
   {
@@ -337,8 +327,8 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### How to generate the test prediction\n",
-    "This calculates and retruns the firing rates for the loaded spike trains."
+    "### How to generate the test prediction\n",
+    "This section calculates and returns the firing rates for the loaded spike trains."
    ]
   },
   {
@@ -356,7 +346,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "##### how to generate the test score\n",
+    "#### How to generate the test score\n",
     "The validation test is performed by calling the judge function, which\n",
     "    1. checks if the model has all the required capabilities. If a model  \n",
     "       does not, and skip_incapable=False, then a CapabilityError is raised.\n",
@@ -371,15 +361,13 @@
     "       e) custom metadata defined in bind_score()\n",
     "    6. returns the score.\n",
     "    \n",
-    "A model-to-model test (TestM2M), requires a list of two or more model classes to be passed to the judge function. The resulting scores are returned in a pandas.Dataframe. Individual entries can be accessed via .iloc[ , ]"
+    "A model-to-model test (`TestM2M`), requires a list of two or more model classes to be passed to the judge function. The resulting scores are returned in a `pandas.Dataframe`. Individual entries can be accessed via `.iloc[ , ]`"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 14,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "name": "stdout",
@@ -411,15 +399,13 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "##### comparing multiple models at once"
+    "#### Comparing multiple models at once"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 15,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "name": "stderr",
@@ -511,15 +497,13 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "##### accessing the confidence intervals of the effect size scores"
+    "#### Accessing the confidence intervals of the effect size scores"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 16,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "name": "stdout",
@@ -540,15 +524,15 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "##### this process is equivalent for the LV_test and the CC_test"
+    "**This process is equivalent for the LV_test and the CC_test.**"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Visualization<a id='viz'></a>\n",
-    "Visualize the prediction with the class function visualize_samples()"
+    "### Visualization<a id='viz'></a>\n",
+    "Visualize the prediction with the class function `visualize_samples()`. Note that the execution of this cell takes approximately 10 mins because all necessary analyses are performed prior to visualization."
    ]
   },
   {
@@ -599,15 +583,17 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "While visualize_samples() displays the predictions of the test, visualize_score() can visualize the test outcome if the linked score has a plot function."
+    "While `visualize_samples()` displays the predictions of the test, `visualize_score()` can visualize the test outcome if the linked score has a plot function."
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Detection of artefacts<a id='artfcts'></a>\n",
-    "Define test using the cross-correlation coefficient as correlation measure"
+    "### Analysis of spike train cross-correlation<a id='xcor'></a>\n",
+    "Define test using the cross-correlation coefficient as correlation measure. This will also lead to the identification of artefacts in the form of *overactive neurons*.\n",
+    "\n",
+    "**NOTE: This analysis takes several hours on a standard laptop machine!** It is therefore commented out by default and the reader is invited to continue with [Iteration II](#it2)."
    ]
   },
   {
@@ -618,15 +604,15 @@
    },
    "outputs": [],
    "source": [
-    "class cross_correlation_struct_test_class(sciunit.TestM2M, tests.generalized_correlation_matrix_test):\n",
-    "    score_type = scores.eigenangle # not used\n",
-    "    params = {'binsize': 2*ms,\n",
-    "              'bin_num': 30000,\n",
-    "              'maxlag': 50,\n",
-    "              'cluster_matrix': False,\n",
-    "              'time_reduction': 'sum'}\n",
+    "# class cross_correlation_struct_test_class(sciunit.TestM2M, tests.generalized_correlation_matrix_test):\n",
+    "#     score_type = scores.eigenangle # not used\n",
+    "#     params = {'binsize': 2*ms,\n",
+    "#               'bin_num': 30000,\n",
+    "#               'maxlag': 50,\n",
+    "#               'cluster_matrix': False,\n",
+    "#               'time_reduction': 'sum'}\n",
     "\n",
-    "ccc_struct_test = cross_correlation_struct_test_class()"
+    "# ccc_struct_test = cross_correlation_struct_test_class()"
    ]
   },
   {
@@ -646,21 +632,19 @@
     }
    ],
    "source": [
-    "triu_idx = np.triu_indices(800, 1)\n",
-    "ax = plots.sample_histogram(ccc_struct_test.generate_prediction(C)[triu_idx[0],triu_idx[1]],\n",
-    "                            ccc_struct_test.generate_prediction(S_sims[-1])[triu_idx[0],triu_idx[1]],\n",
-    "                            sample_names=[C.name, S_sims[-1].name], var_name='cross-correlation coefficient',\n",
-    "                            palette=[C.params['color'], S_sims[-1].params['color']])\n",
-    "ax.set_yscale('log')\n",
-    "plt.show()"
+    "# triu_idx = np.triu_indices(800, 1)\n",
+    "# ax = plots.sample_histogram(ccc_struct_test.generate_prediction(C)[triu_idx[0],triu_idx[1]],\n",
+    "#                             ccc_struct_test.generate_prediction(S_sims[-1])[triu_idx[0],triu_idx[1]],\n",
+    "#                             sample_names=[C.name, S_sims[-1].name], var_name='cross-correlation coefficient',\n",
+    "#                             palette=[C.params['color'], S_sims[-1].params['color']])\n",
+    "# ax.set_yscale('log')\n",
+    "# plt.show()"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 20,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "data": {
@@ -674,28 +658,26 @@
     }
    ],
    "source": [
-    "fig, ax = plt.subplots(ncols=2, figsize=(10,5))\n",
+    "# fig, ax = plt.subplots(ncols=2, figsize=(10,5))\n",
     "\n",
-    "ccc_struct_test.visualize_samples(C, S_sims[-1], ax=ax, cluster=True, cluster_as=0, limit_to_1=False,\n",
-    "                                  square=True, cmap='coolwarm', vmax=2, vmin=-1, labels=['']*800)\n",
-    "ax[0].axis('off')\n",
-    "ax[1].axis('off')\n",
-    "plt.show()"
+    "# ccc_struct_test.visualize_samples(C, S_sims[-1], ax=ax, cluster=True, cluster_as=0, limit_to_1=False,\n",
+    "#                                   square=True, cmap='coolwarm', vmax=2, vmin=-1, labels=['']*800)\n",
+    "# ax[0].axis('off')\n",
+    "# ax[1].axis('off')\n",
+    "# plt.show()"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "##### Identifying the overactive neurons"
+    "#### Identifying the overactive neurons"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 21,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "name": "stdout",
@@ -716,29 +698,29 @@
     }
    ],
    "source": [
-    "m = ccc_struct_test.generate_prediction(S_sims[-1]) \n",
-    "# threshold 1.5 clearly separates the 'normal' \n",
-    "# and the very large correlations (see matrix above)\n",
-    "hist, edges = np.histogram(np.append(np.where(m > 1.5)[0],np.where(m > 1.5)[1]), bins=np.linspace(0,800,801))\n",
-    "plt.plot(np.arange(800), hist)\n",
-    "plt.gca().set_xlabel('neuron id')\n",
-    "plt.gca().set_ylabel('count of high correlation interactions')\n",
-    "overactive_neurons = np.where(hist > 20)[0]\n",
-    "print(overactive_neurons)"
+    "# m = ccc_struct_test.generate_prediction(S_sims[-1]) \n",
+    "# # threshold 1.5 clearly separates the 'normal' \n",
+    "# # and the very large correlations (see matrix above)\n",
+    "# hist, edges = np.histogram(np.append(np.where(m > 1.5)[0],np.where(m > 1.5)[1]), bins=np.linspace(0,800,801))\n",
+    "# plt.plot(np.arange(800), hist)\n",
+    "# plt.gca().set_xlabel('neuron id')\n",
+    "# plt.gca().set_ylabel('count of high correlation interactions')\n",
+    "# overactive_neurons = np.where(hist > 20)[0]\n",
+    "# print(overactive_neurons)"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Iteration II<a id='poly_model'></a>"
+    "## Iteration II<a id='it2'></a>"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Define the model (-instances)<a id='model2'></a>"
+    "### Define the model (-instances)<a id='model2'></a>"
    ]
   },
   {
@@ -809,7 +791,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### perform the validation tests and average over the 5 network states<a id='test2'></a>"
+    "### Perform the validation tests and average over the 5 network states<a id='test2'></a>"
    ]
   },
   {
@@ -834,9 +816,7 @@
   {
    "cell_type": "code",
    "execution_count": 24,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "name": "stdout",
@@ -896,14 +876,14 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "### Iteration III<a id='it3'></a>"
+    "## Iteration III<a id='it3'></a>"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Define the model (-instances)<a id='model3'></a>"
+    "### Define the model (-instances)<a id='model3'></a>"
    ]
   },
   {
@@ -974,7 +954,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Define additional tests<a id='test3'></a>"
+    "### Define additional tests<a id='test3'></a>"
    ]
   },
   {
@@ -1009,15 +989,13 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### perform tests and average over the 5 network states<a id='test3_avg'></a>"
+    "### Perform tests and average over the 5 network states<a id='test3_avg'></a>"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 28,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "%%capture\n",
@@ -1040,9 +1018,7 @@
   {
    "cell_type": "code",
    "execution_count": 29,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "name": "stdout",
@@ -1110,7 +1086,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Perfrom further testing of the correlation structure<a id='test3_struct'></a>"
+    "### Perform further testing of the correlation structure<a id='test3_struct'></a>"
    ]
   },
   {
@@ -1133,9 +1109,7 @@
   {
    "cell_type": "code",
    "execution_count": 33,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "data": {
@@ -1160,9 +1134,7 @@
   {
    "cell_type": "code",
    "execution_count": 34,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "name": "stdout",
@@ -1204,9 +1176,7 @@
   {
    "cell_type": "code",
    "execution_count": 35,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "name": "stdout",
@@ -1228,15 +1198,13 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "#### Calculate the power spectrum<a id='test3_power'></a>"
+    "### Calculate the power spectrum<a id='test3_power'></a>"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": 36,
-   "metadata": {
-    "collapsed": false
-   },
+   "metadata": {},
    "outputs": [
     {
      "data": {
@@ -1283,8 +1251,15 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "For the results of the SPADE analysis refer to the folder 'spade analysis'"
+    "For the results of the SPADE analysis the reader is referred to the folder 'spade_analysis' in this repository."
    ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
   }
  ],
  "metadata": {
@@ -1296,14 +1271,38 @@
   "language_info": {
    "codemirror_mode": {
     "name": "ipython",
-    "version": 3
+    "version": 2
    },
    "file_extension": ".py",
    "mimetype": "text/x-python",
    "name": "python",
    "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.7.0"
+   "pygments_lexer": "ipython2",
+   "version": "2.7.12"
+  },
+  "toc": {
+   "colors": {
+    "hover_highlight": "#DAA520",
+    "navigate_num": "#000000",
+    "navigate_text": "#333333",
+    "running_highlight": "#FF0000",
+    "selected_highlight": "#FFD700",
+    "sidebar_border": "#EEEEEE",
+    "wrapper_background": "#FFFFFF"
+   },
+   "moveMenuLeft": true,
+   "nav_menu": {
+    "height": "399.333px",
+    "width": "252px"
+   },
+   "navigate_menu": true,
+   "number_sections": false,
+   "sideBar": true,
+   "threshold": "3",
+   "toc_cell": false,
+   "toc_section_display": "block",
+   "toc_window_display": false,
+   "widenNotebook": false
   }
  },
  "nbformat": 4,