change layout, fix and extend requirements.txt, place it under git not annex (as any other code)

See individual commits for more explanation on any of the changes.

With this we get following layout

❯ tree -L 3
.
├── rawdata
│   ├── code
│   │   ├── create_rawdata.py
│   │   ├── requirements.txt
│   │   └── venvs
│   ├── dataset_description.json
│   ├── participants.tsv
│   ├── sourcedata -> ../sourcedata
│   └── sub-Neuropixel001
│       ├── ses-210101
│       └── sub-Neuropixel001_sessions.tsv
└── sourcedata
    ├── neuropixPhase3A_kilosortChanMap.mat -> ...
    └── sub-Neuropixel001
        └── ses-210101

I would even just place sourcedata/ under rawdata/ and not bother with symlink there ATM since it is pertinent only for that original sourcedata/ - users care only to see the rawdata/ -- the BIDS dataset. @TheChymera thought differently. This is our mid-point. See commit for more detail/pointers
Now this folder could become a BIDS dataset itself, e.g. a derivative dataset which would have that "rawdata/" -- the BIDS dataset it is created from, or populate a number of derivatives/ subdatasets.

We also fixed up the code script so it would not care where it is ran from.

Unfortunately, with freshly instantiated venv with the requirements.txt dependencies - conversion fails following way:

(git)smaug:~/proj/bids/BEPS/BEP032-examples[relayout]ephys_neuropixel/rawdata
$> datalad run -m "Redo the conversion on Yarik's laptop" --input ../sourcedata/ --output . code/create_rawdata.py
[INFO   ] Making sure inputs are available (this may take some time) 
[INFO   ] == Command start (output follows) ===== 
Traceback (most recent call last):
  File "/home/yoh/proj/bids/BEPS/BEP032-examples/ephys_neuropixel/rawdata/code/create_rawdata.py", line 120, in <module>
    gen.generate_all_metadata_files()
  File "/home/yoh/proj/bids/BEPS/BEP032-examples/ephys_neuropixel/rawdata/code/venv/lib/python3.9/site-packages/bep032tools/generator/BEP032Generator.py", line 297, in generate_all_metadata_files
    self.generate_metadata_file_contacts(dest_path / (stem + '_contacts'))
  File "/home/yoh/proj/bids/BEPS/BEP032-examples/ephys_neuropixel/rawdata/code/create_rawdata.py", line 62, in generate_metadata_file_contacts
    save_tsv(df, output)
  File "/home/yoh/proj/bids/BEPS/BEP032-examples/ephys_neuropixel/rawdata/code/venv/lib/python3.9/site-packages/bep032tools/generator/utils.py", line 29, in save_tsv
    merged_dfs = merge_dfs_by_index(existing_df, dataframe)
  File "/home/yoh/proj/bids/BEPS/BEP032-examples/ephys_neuropixel/rawdata/code/venv/lib/python3.9/site-packages/bep032tools/generator/utils.py", line 151, in merge_dfs_by_index
    raise ValueError('Dataframes have incompatible values: '
ValueError: Dataframes have incompatible values:            1-indexed-contact_id       connected       shank_id          x           y      
                           self other      self other     self other self other  self other
contact_id                                                                                 
0                             1     1         1     1        1     1   43    43    20    20
1                             2     2         1     1        1     1   11    11    20    20
2                             3     3         1     1        1     1   59    59    40    40
3                             4     4         1     1        1     1   27    27    40    40
4                             5     5         1     1        1     1   43    43    60    60
...                         ...   ...       ...   ...      ...   ...  ...   ...   ...   ...
379                         380   380         0     0        1     1   27    27  3800  3800
380                         381   381         1     1        1     1   43    43  3820  3820
381                         382   382         1     1        1     1   11    11  3820  3820
382                         383   383         1     1        1     1   59    59  3840  3840
383                         384   384         1     1        1     1   27    27  3840  3840

[384 rows x 10 columns]
[INFO   ] == Command exit (modification check follows) ===== 
[INFO   ] The command had a non-zero exit code. If this is expected, you can save the changes with 'datalad save -d . -r -F .git/COMMIT_EDITMSG' 
run(error): /home/yoh/proj/bids/BEPS/BEP032-examples (dataset) [code/create_rawdata.py]

any immediate ideas on how to address?

See individual commits for more explanation on any of the changes. With this we get following layout ``` ❯ tree -L 3 . ├── rawdata │ ├── code │ │ ├── create_rawdata.py │ │ ├── requirements.txt │ │ └── venvs │ ├── dataset_description.json │ ├── participants.tsv │ ├── sourcedata -> ../sourcedata │ └── sub-Neuropixel001 │ ├── ses-210101 │ └── sub-Neuropixel001_sessions.tsv └── sourcedata ├── neuropixPhase3A_kilosortChanMap.mat -> ... └── sub-Neuropixel001 └── ses-210101 ``` - I would even just place `sourcedata/` under `rawdata/` and not bother with symlink there ATM since it is pertinent only for that original `sourcedata/` - users care only to see the `rawdata/` -- the BIDS dataset. @TheChymera thought differently. This is our mid-point. See commit for more detail/pointers - Now this folder could become a BIDS dataset itself, e.g. a derivative dataset which would have that "rawdata/" -- the BIDS dataset it is created from, or populate a number of `derivatives/` subdatasets. We also fixed up the code script so it would not care where it is ran from. Unfortunately, with freshly instantiated venv with the requirements.txt dependencies - conversion fails following way: ```shell (git)smaug:~/proj/bids/BEPS/BEP032-examples[relayout]ephys_neuropixel/rawdata $> datalad run -m "Redo the conversion on Yarik's laptop" --input ../sourcedata/ --output . code/create_rawdata.py [INFO ] Making sure inputs are available (this may take some time) [INFO ] == Command start (output follows) ===== Traceback (most recent call last): File "/home/yoh/proj/bids/BEPS/BEP032-examples/ephys_neuropixel/rawdata/code/create_rawdata.py", line 120, in <module> gen.generate_all_metadata_files() File "/home/yoh/proj/bids/BEPS/BEP032-examples/ephys_neuropixel/rawdata/code/venv/lib/python3.9/site-packages/bep032tools/generator/BEP032Generator.py", line 297, in generate_all_metadata_files self.generate_metadata_file_contacts(dest_path / (stem + '_contacts')) File "/home/yoh/proj/bids/BEPS/BEP032-examples/ephys_neuropixel/rawdata/code/create_rawdata.py", line 62, in generate_metadata_file_contacts save_tsv(df, output) File "/home/yoh/proj/bids/BEPS/BEP032-examples/ephys_neuropixel/rawdata/code/venv/lib/python3.9/site-packages/bep032tools/generator/utils.py", line 29, in save_tsv merged_dfs = merge_dfs_by_index(existing_df, dataframe) File "/home/yoh/proj/bids/BEPS/BEP032-examples/ephys_neuropixel/rawdata/code/venv/lib/python3.9/site-packages/bep032tools/generator/utils.py", line 151, in merge_dfs_by_index raise ValueError('Dataframes have incompatible values: ' ValueError: Dataframes have incompatible values: 1-indexed-contact_id connected shank_id x y self other self other self other self other self other contact_id 0 1 1 1 1 1 1 43 43 20 20 1 2 2 1 1 1 1 11 11 20 20 2 3 3 1 1 1 1 59 59 40 40 3 4 4 1 1 1 1 27 27 40 40 4 5 5 1 1 1 1 43 43 60 60 ... ... ... ... ... ... ... ... ... ... ... 379 380 380 0 0 1 1 27 27 3800 3800 380 381 381 1 1 1 1 43 43 3820 3820 381 382 382 1 1 1 1 11 11 3820 3820 382 383 383 1 1 1 1 59 59 3840 3840 383 384 384 1 1 1 1 27 27 3840 3840 [384 rows x 10 columns] [INFO ] == Command exit (modification check follows) ===== [INFO ] The command had a non-zero exit code. If this is expected, you can save the changes with 'datalad save -d . -r -F .git/COMMIT_EDITMSG' run(error): /home/yoh/proj/bids/BEPS/BEP032-examples (dataset) [code/create_rawdata.py] ``` any immediate ideas on how to address?

Tento požadavek na natažení může být automaticky sloučen.

Přihlaste se pro zapojení do konverzace.

Bez štítku

Bez milníku

Bez zpracovatele

1 účastníků

#13 change layout, fix and extend requirements.txt, place it under git not annex (as any other code)