{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Demonstration of validation testing with NetworkUnit by replicating the results in Gutzen et al. (sub.)\n",
"This notebook should showcase the basic validation workflow with the sciunit package and the test repository networkunit.\n",
"\n",
"\n",
"\n",
"Requirements (Python 2.7.14):\n",
"\n",
" elephant == 0.5.0\n",
" sciunit == 0.2.0.2\n",
" jupyter == 1.0.0\n",
" tabulate == 0.8.2\n",
" networkx == 2.1\n",
" fastcluster == 1.1.24\n",
" seaborn == 0.8.1"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Table of Contents\n",
"- [Setup](#setup)\n",
"1. [Polychrony model class](#poly_model)\n",
"1. [Iteration I](#it1)\n",
" - [Define model classes](#model1)\n",
" 1. [Define test classes and how to perform test](#test1)\n",
" 1. [Visualization and artefact detection](#viz)\n",
"1. [Iteration II](#it2)\n",
" - [Define model classes](#model2)\n",
" 1. [Perform validation tests and average over network states](#test2)\n",
"1. [Iteration III](#it3)\n",
" - [Define model classes](#model3)\n",
" 1. [Define additional test classes](#test3)\n",
" 1. [Perform validation tests and average over network states](#test3_avg)\n",
" 1. [Comparing the correlation structure](#test3_struct)\n",
" 1. [Calculating the power spectrum](#test3_power) "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Setup "
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"%matplotlib inline\n",
"import matplotlib.pyplot as plt\n",
"import sys\n",
"import sciunit\n",
"import elephant\n",
"import numpy as np\n",
"from quantities import ms\n",
"from neo.core import SpikeTrain\n",
"from copy import copy"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### load a version of NetworkUnit (only required once)"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"%%capture\n",
"!git clone -n https://github.com/INM-6/NetworkUnit.git"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"%%capture\n",
"cd NetworkUnit"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"%%capture\n",
"!git checkout '060b30372e86f1337da768537cadaa9a2d58ea43' # latest commit at time of paper submission \n",
"!git fetch; git pull"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"%%capture\n",
"cd .."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"sys.path.insert(0, './NetworkUnit')\n",
"from networkunit import tests, scores, plots, models"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"data_path = './simulation_data/'"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Define the polychronization model class\n",
"NetworkUnit does not yet have an interface to HPC resources or SpiNNaker to perform the simulations automatically. Therefore, the outcomes of manually performed simulations are loaded into the model class."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"class polychrony_data(models.spiketrain_data):\n",
" file_path = '' # to be added in child class\n",
" \n",
" params = {'align_to_0': True,\n",
" 'filter_inh': True,\n",
" }\n",
" \n",
" def load(self, file_path, simulator, t_start=0, t_stop=60000, filter_inh=False, **kwargs):\n",
" f = open(file_path, 'r')\n",
" lines = f.readlines()\n",
" \n",
" N = 1000 # neurons \n",
" \n",
" # Read Spike Times\n",
" spike_times = [[]] * N\n",
" for line in lines:\n",
" sec, msec, n = line.split(' ')[:3]\n",
" t = float(sec)*1000. + float(msec)\n",
" n = int(n)\n",
" if t > t_stop:\n",
" break\n",
" spike_times[n] = spike_times[n] + [t]\n",
"\n",
" # Fill Spike Trains\n",
" nbr_neurons = N\n",
" if filter_inh:\n",
" nbr_neurons = 800\n",
" \n",
" spiketrains = [[]] * nbr_neurons\n",
"\n",
" for n, st in enumerate(spike_times):\n",
" if n < 800:\n",
" n_type = 'exc'\n",
" else:\n",
" n_type = 'inh'\n",
" if not filter_inh or n_type == 'exc':\n",
" spiketrains[n] = SpikeTrain(np.sort(st), units='ms', \n",
" t_start=t_start, t_stop=t_stop,\n",
" n_type=n_type, unitID=n)\n",
" return spiketrains"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Iteration I"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Define the model instances"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"class C_sim(polychrony_data):\n",
" file_path = data_path + 'iteration_I/60s_simulation_runs/C/out_firings_after5h.dat'\n",
" params = copy(polychrony_data.params)\n",
" params.update(color='#01589F', simulator='C')\n",
" \n",
"class S_sim_i(polychrony_data):\n",
" file_path = data_path + 'iteration_I/60s_simulation_runs/SpiNNaker_i/out_firings_after5h.dat'\n",
" params = copy(polychrony_data.params)\n",
" params.update(color='#1B6145', simulator='SpiNNaker')\n",
"\n",
"class S_sim_ii(polychrony_data):\n",
" file_path = data_path + 'iteration_I/60s_simulation_runs/SpiNNaker_ii/out_firings_after5h.dat'\n",
" params = copy(polychrony_data.params)\n",
" params.update(color='#1B6145', simulator='SpiNNaker')\n",
" \n",
"class S_sim_iii(polychrony_data):\n",
" file_path = data_path + 'iteration_I/60s_simulation_runs/SpiNNaker_iii/out_firings_after5h.dat'\n",
" params = copy(polychrony_data.params)\n",
" params.update(color='#1B6145', simulator='SpiNNaker')"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"C = C_sim(name='C')\n",
"S_sims = [S_sim_i(name='SpiNNaker (i)'), S_sim_ii(name='SpiNNaker (ii)'), S_sim_iii(name='SpiNNaker (iii)')]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Define test classes\n",
"The abstract base classes for these tests are implemented in NetworkUnit so that here only the parameters have to be set and the test is paired with a score class. To perfrom a hypothesis test the effect_size score can be replaced for example with the ks_distance or the mwu_statistic.\n",
"The inhereted TestM2M class adapts the test such that the tests don't need to be initiliazed with experimental data. "
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"class FR_test_class(sciunit.TestM2M, tests.firing_rate_test):\n",
" score_type = scores.effect_size # ks_distance, mwu_statistic, students_t\n",
"\n",
"class LV_test_class(sciunit.TestM2M, tests.isi_variation_test):\n",
" score_type = scores.effect_size\n",
" params = {'variation_measure': 'lv'}\n",
"\n",
"class CC_test_class(sciunit.TestM2M, tests.correlation_dist_test):\n",
" score_type = scores.effect_size\n",
" params = {'binsize': 2*ms}\n",
"\n",
"FR_test = FR_test_class()\n",
"LV_test = LV_test_class()\n",
"CC_test = CC_test_class()\n",
"\n",
"rate_score = [0] * 3\n",
"isi_score = [0] * 3\n",
"cc_score = [0] * 3"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### how to generate the test prediction\n",
"This calculates and retruns the firing rates for the loaded spike trains."
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"FR_test.generate_prediction(C);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### how to generate the test score\n",
"The validation test is performed by calling the judge function, which\n",
" 1. checks if the model has all the required capabilities. If a model \n",
" does not, and skip_incapable=False, then a CapabilityError is raised.\n",
" 2. calls generate_prediction() to generate a prediction.\n",
" 3. calls compute_score() to generate a score.\n",
" 4. checks that the score is of score_type, else raising an InvalidScoreError.\n",
" 5. equips the score with metadata:\n",
" a) a reference to the model\n",
" b) a reference to the test.\n",
" c) a reference to the prediction\n",
" d) a reference to the observation\n",
" e) custom metadata defined in bind_score()\n",
" 6. returns the score.\n",
" \n",
"A model-to-model test (TestM2M), requires a list of two or more model classes to be passed to the judge function. The resulting scores are returned in a pandas.Dataframe. Individual entries can be accessed via .iloc[ , ]"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"\n",
"\u001b[4mEffect Size\u001b[0m\n",
"\tdatasize: 800 \t 800\n",
"\tEffect Size = 3.410 \t CI = (3.257, 3.564)\n",
"\n",
"\n"
]
}
],
"source": [
"print FR_test.judge([C, S_sims[0]]).iloc[0,1]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"##### comparing multiple models at once"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"
\n",
" \n",
" \n",
" | \n",
" C | \n",
" SpiNNaker (i) | \n",
" SpiNNaker (ii) | \n",
" SpiNNaker (iii) | \n",
"
\n",
" \n",
" \n",
" \n",
" | C | \n",
" 0.000000 | \n",
" 3.410068 | \n",
" 1.338312 | \n",
" 0.902216 | \n",
"
\n",
" \n",
" | SpiNNaker (i) | \n",
" 3.410068 | \n",
" 0.000000 | \n",
" 2.461155 | \n",
" 2.046497 | \n",
"
\n",
" \n",
" | SpiNNaker (ii) | \n",
" 1.338312 | \n",
" 2.461155 | \n",
" 0.000000 | \n",
" 0.258701 | \n",
"
\n",
" \n",
" | SpiNNaker (iii) | \n",
" 0.902216 | \n",
" 2.046497 | \n",
" 0.258701 | \n",
" 0.000000 | \n",
"
\n",
" \n",
"
\n",
"