Primoz
15d792089d
Changes in cleaning script:
...
- target extracted from config to remove rows where target is nan
- prepared sns.heatmap for further missing values analysis
- necessary changes in config and participant p01
- picture of heatmap which shows the values state after cleaning
2022-09-01 10:33:36 +00:00
Primoz
cb351e0ff6
Unnecessary line (rows with no target value will be removed in cleaning script).
2022-09-01 10:06:57 +00:00
Primoz
86299d346b
Impute phone and sms NAs with 0
2022-09-01 09:57:21 +00:00
Primoz
3f7ec80c18
Preparation a) phone_calls 0 imputation b) remove rows with NaN target
2022-08-31 10:18:50 +00:00
Primoz
f78aa3e7b3
Preparation for cleaning & imputation
2022-08-26 10:56:14 +00:00
Primoz
c498ecb742
Include baseline models (+corrections), disable columns drop in cleaning function.
2022-08-23 14:12:14 +00:00
Primoz
f088e9586f
Handle empty ACC.csv
2022-08-22 14:20:47 +00:00
Primoz
0aa0e82673
Handle empty Empatica csv files.
2022-08-22 14:18:12 +00:00
Primoz
4cfe5a3a98
Disable discarding rows if DATA_YIELD_RATIO_THRESHOLD==0.
2022-08-19 13:10:56 +00:00
Primoz
607da820f2
Configuration and cleaning changes
2022-08-18 14:21:05 +00:00
Primoz
fb577bc9ad
Squashed commit of the following:
...
commit 43ecc243cb62bb31eed85cb477ca4131555c7fe7
Author: Primoz <sisko.primoz@gmail.com>
Date: Fri Jul 22 15:26:09 2022 +0000
Adding TODO comments
commit 2df1ebf90c3a93812b112b8ed0ee4e23cd74533f
Author: Primoz <sisko.primoz@gmail.com>
Date: Thu Jul 21 13:59:23 2022 +0000
README update
commit 5182c2b16dff3537aad42984b8ea5214743cdb32
Author: Primoz <sisko.primoz@gmail.com>
Date: Thu Jul 21 11:03:01 2022 +0000
Few corrections for all_cleaning
commit 3d9254c1b3bed6e95e631d4e0402548830a19534
Author: Primoz <sisko.primoz@gmail.com>
Date: Thu Jul 21 10:28:05 2022 +0000
Adding the min overlap for corr threshold and preservation of esm cols.
commit e27c49cc8fa4c51f9fe8e593a8d25e9a032ab393
Author: Primoz <sisko.primoz@gmail.com>
Date: Thu Jul 21 09:02:00 2022 +0000
Commenting and cleaning.
commit 31a47a5ee4569264e39d7c445525a6e64bb7700a
Author: Primoz <sisko.primoz@gmail.com>
Date: Wed Jul 20 13:49:22 2022 +0000
Environment version change.
commit 5b274ed8993f58e783bda6d82fce936764209c28
Author: Primoz <sisko.primoz@gmail.com>
Date: Tue Jul 19 16:10:07 2022 +0000
Enabled cleaning for all participants + standardization files.
commit 203fdb31e0f3c647ef8c8a60cb9531831b7ab924
Author: Primoz <sisko.primoz@gmail.com>
Date: Tue Jul 19 14:14:51 2022 +0000
Features cleaning fixes after testing. Visualization script for phone features values.
commit 176178d73b154c30b9eb9eb4a67514f00d6a924e
Author: Primoz <sisko.primoz@gmail.com>
Date: Tue Jul 19 09:05:14 2022 +0000
Revert "Necessary config changes."
This reverts commit 6ec1ef50430d2e1f5ce4670d505d5e84ac47f0a0.
commit 26ea6512c9d512f95837e7b047fe510c1d196403
Author: Primoz <sisko.primoz@gmail.com>
Date: Mon Jul 18 13:19:47 2022 +0000
Adding cleaning function condition and cleaning functionality.
commit 575c29eef9c21e6f2d7832871e73bc0941643734
Author: Primoz <sisko.primoz@gmail.com>
Date: Mon Jul 18 12:51:56 2022 +0000
Translation of the cleaning individual RAPIDS function from R to py.
commit 6ec1ef50430d2e1f5ce4670d505d5e84ac47f0a0
Author: Primoz <sisko.primoz@gmail.com>
Date: Mon Jul 18 12:02:18 2022 +0000
Necessary config changes.
commit b5669f51612fbd8378848615d639677851ab032f
Author: Primoz <sisko.primoz@gmail.com>
Date: Fri Jul 15 15:26:00 2022 +0000
Modified snakemake rule to dynamically choose script extention.
commit 66636be1e8ae4828228b37c59b9df1faf3fc3d3d
Author: Primoz <sisko.primoz@gmail.com>
Date: Fri Jul 15 14:43:08 2022 +0000
Trying to modify the snakefile rule to execute scripts in two languages depended on the provider.
commit 574778b00f3cbb368ef4bc74de15cf5070c65ea9
Author: Primoz <sisko.primoz@gmail.com>
Date: Fri Jul 15 09:49:41 2022 +0000
gitignore: adding required files so that RAPIDS can be run successfully.
commit 71018ab178256970535e78961602ab8c7f0ebb14
Author: Primoz <sisko.primoz@gmail.com>
Date: Fri Jul 15 08:34:19 2022 +0000
Standardization bug fixes
commit 6253c470a624e6bfbb02e0c453b652452eb2dbbc
Author: Primoz <sisko.primoz@gmail.com>
Date: Thu Jul 14 15:28:02 2022 +0000
Seperate rules for empatica vs. nonempatica standardization.
Parameter in config that controls the creation of standardized merged files for individual and all participants..
commit 90f902778565e0896d3bae22ae8551be8b487e67
Author: Primoz <sisko.primoz@gmail.com>
Date: Tue Jul 12 14:23:03 2022 +0000
Preparing for final csvs' standardization.
commit d25dde3998786a9a582f5cda544ee104386778f9
Author: Primoz <sisko.primoz@gmail.com>
Date: Mon Jul 11 12:08:47 2022 +0000
Revert "Changes in config to be reverted."
This reverts commit bea7608e7095021fb7c53a9afa07074448fe4313.
commit 6b23e70857e63deda98eb98d190af9090626c84b
Author: Primoz <sisko.primoz@gmail.com>
Date: Mon Jul 11 12:08:26 2022 +0000
Enabled standardization for rest (previously active) phone features.
Testing still needed.
commit 8ec58a6f34ba3d42e5cc71d26e6d91837472ca5f
Author: Primoz <sisko.primoz@gmail.com>
Date: Mon Jul 11 09:07:55 2022 +0000
Enabled standardization for phone calls.
All steps completed and tested.
commit bea7608e7095021fb7c53a9afa07074448fe4313
Author: Primoz <sisko.primoz@gmail.com>
Date: Mon Jul 11 07:47:51 2022 +0000
Changes in config to be reverted.
commit 4e84ca0e51bf709bff56fd09437b95310ec6bedd
Author: Primoz <sisko.primoz@gmail.com>
Date: Fri Jul 8 14:11:24 2022 +0000
Standardization for the rest of the features.
commit cc581aa788e3d5c17131af8f3d5dd6b0c3b5aff7
Author: Primoz <sisko.primoz@gmail.com>
Date: Fri Jul 8 14:11:08 2022 +0000
README update again
2022-07-22 15:31:30 +00:00
Primoz
6ba4a66deb
Squashed commit of the following:
...
commit 31a47a5ee4569264e39d7c445525a6e64bb7700a
Author: Primoz <sisko.primoz@gmail.com>
Date: Wed Jul 20 13:49:22 2022 +0000
Environment version change.
commit 5b274ed8993f58e783bda6d82fce936764209c28
Author: Primoz <sisko.primoz@gmail.com>
Date: Tue Jul 19 16:10:07 2022 +0000
Enabled cleaning for all participants + standardization files.
commit 203fdb31e0f3c647ef8c8a60cb9531831b7ab924
Author: Primoz <sisko.primoz@gmail.com>
Date: Tue Jul 19 14:14:51 2022 +0000
Features cleaning fixes after testing. Visualization script for phone features values.
commit 176178d73b154c30b9eb9eb4a67514f00d6a924e
Author: Primoz <sisko.primoz@gmail.com>
Date: Tue Jul 19 09:05:14 2022 +0000
Revert "Necessary config changes."
This reverts commit 6ec1ef50430d2e1f5ce4670d505d5e84ac47f0a0.
commit 26ea6512c9d512f95837e7b047fe510c1d196403
Author: Primoz <sisko.primoz@gmail.com>
Date: Mon Jul 18 13:19:47 2022 +0000
Adding cleaning function condition and cleaning functionality.
commit 575c29eef9c21e6f2d7832871e73bc0941643734
Author: Primoz <sisko.primoz@gmail.com>
Date: Mon Jul 18 12:51:56 2022 +0000
Translation of the cleaning individual RAPIDS function from R to py.
commit 6ec1ef50430d2e1f5ce4670d505d5e84ac47f0a0
Author: Primoz <sisko.primoz@gmail.com>
Date: Mon Jul 18 12:02:18 2022 +0000
Necessary config changes.
commit b5669f51612fbd8378848615d639677851ab032f
Author: Primoz <sisko.primoz@gmail.com>
Date: Fri Jul 15 15:26:00 2022 +0000
Modified snakemake rule to dynamically choose script extention.
commit 66636be1e8ae4828228b37c59b9df1faf3fc3d3d
Author: Primoz <sisko.primoz@gmail.com>
Date: Fri Jul 15 14:43:08 2022 +0000
Trying to modify the snakefile rule to execute scripts in two languages depended on the provider.
commit 574778b00f3cbb368ef4bc74de15cf5070c65ea9
Author: Primoz <sisko.primoz@gmail.com>
Date: Fri Jul 15 09:49:41 2022 +0000
gitignore: adding required files so that RAPIDS can be run successfully.
commit 71018ab178256970535e78961602ab8c7f0ebb14
Author: Primoz <sisko.primoz@gmail.com>
Date: Fri Jul 15 08:34:19 2022 +0000
Standardization bug fixes
commit 6253c470a624e6bfbb02e0c453b652452eb2dbbc
Author: Primoz <sisko.primoz@gmail.com>
Date: Thu Jul 14 15:28:02 2022 +0000
Seperate rules for empatica vs. nonempatica standardization.
Parameter in config that controls the creation of standardized merged files for individual and all participants..
commit 90f902778565e0896d3bae22ae8551be8b487e67
Author: Primoz <sisko.primoz@gmail.com>
Date: Tue Jul 12 14:23:03 2022 +0000
Preparing for final csvs' standardization.
commit d25dde3998786a9a582f5cda544ee104386778f9
Author: Primoz <sisko.primoz@gmail.com>
Date: Mon Jul 11 12:08:47 2022 +0000
Revert "Changes in config to be reverted."
This reverts commit bea7608e7095021fb7c53a9afa07074448fe4313.
commit 6b23e70857e63deda98eb98d190af9090626c84b
Author: Primoz <sisko.primoz@gmail.com>
Date: Mon Jul 11 12:08:26 2022 +0000
Enabled standardization for rest (previously active) phone features.
Testing still needed.
commit 8ec58a6f34ba3d42e5cc71d26e6d91837472ca5f
Author: Primoz <sisko.primoz@gmail.com>
Date: Mon Jul 11 09:07:55 2022 +0000
Enabled standardization for phone calls.
All steps completed and tested.
commit bea7608e7095021fb7c53a9afa07074448fe4313
Author: Primoz <sisko.primoz@gmail.com>
Date: Mon Jul 11 07:47:51 2022 +0000
Changes in config to be reverted.
commit 4e84ca0e51bf709bff56fd09437b95310ec6bedd
Author: Primoz <sisko.primoz@gmail.com>
Date: Fri Jul 8 14:11:24 2022 +0000
Standardization for the rest of the features.
commit cc581aa788e3d5c17131af8f3d5dd6b0c3b5aff7
Author: Primoz <sisko.primoz@gmail.com>
Date: Fri Jul 8 14:11:08 2022 +0000
README update again
2022-07-20 13:51:22 +00:00
Primoz
788ac31190
Bug fix: if df has no rows write an empty zscore file.
2022-07-08 10:40:45 +00:00
Primoz
5a777ac79f
Working version that integrates both phone and empatica feature calculations.
2022-07-07 15:00:47 +00:00
Primoz
0425403951
Merge branch 'master' of https://repo.ijs.si/junoslukan/rapids
2022-07-06 11:53:31 +00:00
Primoz
887fd7dc72
Merge branch 'empatica_calculating_features'
2022-07-06 11:53:21 +00:00
Primoz
d2758eef46
Set not NaN sum insted of 0 sum for HRV features windows.
2022-07-06 07:36:35 +00:00
Primoz
2d5d23b615
Testing files change and remove standardization from hrv sensors main files.
2022-07-06 07:35:39 +00:00
Primoz
505c3a86b9
Testing different EDA findPeaks parameters.
2022-06-30 15:15:37 +00:00
Primoz
c851ab0763
Fill EDA NaN values where numPeak is zero. Other small changes.
2022-06-21 14:09:49 +00:00
Primoz
dda4554d46
Various small changes.
2022-06-15 13:57:46 +00:00
Primoz
212cf300f8
Debugging EDA signal - preliminary step for imputation.
2022-06-14 15:09:14 +00:00
Primoz
9ea39dc557
Standardization as a Snakefile's rule enabled for all E4 sensors.
2022-06-13 18:17:30 +00:00
Primoz
402059871f
Making standardization as a rule. WIP: done only for BVP.
2022-06-13 14:12:03 +00:00
Primoz
094743244d
Added SO feature for sum all rows that are non zero for BVP and IBI sensors.
2022-06-13 10:51:22 +00:00
primoz
e1d7607de4
Extraction of additional SO features. Min/max has been changed to nsmallest/nlargest means.
2022-06-10 12:34:48 +00:00
primoz
f371249b99
First order features standardization WIP
2022-06-09 13:35:15 +00:00
primoz
2c7ac21465
Added standardization on SO features.
2022-06-06 13:51:15 +00:00
primoz
2acf6ff9fb
Exception handling in case of empty ibi. Changes of the method EDA uses in main.py. Other small corrections.
2022-06-03 12:34:36 +00:00
primoz
d300f0f8f0
Fixed RAPIDS bug: error when IBI.csv is empty.
2022-06-02 11:43:49 +00:00
Primoz
5532043b1f
Patching IBI with BVP - completed.
2022-05-25 19:39:47 +00:00
Primoz
bb62497ba6
Patching IBI with BVP - selecting appropriate pipeline entry point. WIP
2022-05-24 11:07:18 +00:00
Primoz
2a8f58f5c8
Patching IBI with BVP. WIP
2022-05-20 13:18:45 +00:00
Primoz
6864cfe775
Changes after thorough testing with available data.
2022-05-13 13:35:34 +00:00
Primoz
c1564f0cae
Changed wrapper method calculate_feature to its newest version (for TEMP and ACC).
2022-05-11 14:21:21 +00:00
Primoz
31e36e7400
Alternating Second order and full segment features corresponding to config settings.
2022-05-11 08:50:15 +00:00
Primoz
9cf9e1fe14
Testing and modifying the code with different E4 data.
2022-05-10 11:36:49 +00:00
Primoz
f62a1302dd
Cr-features corrections for ACC and TEMP sensors
2022-05-09 11:01:52 +00:00
Primoz
5638367999
Implementation of the second order features.
2022-04-25 13:07:03 +00:00
Primoz
66451160e9
Calculating HRV features with IBI.csv.
2022-04-20 10:44:51 +00:00
=
8c8fe1fec7
Modifications, mostly imports, after changes in cr-features module.
2022-04-19 13:24:46 +00:00
=
075c64d1e5
HRV: changed wrapper calcFeat method with specialized one.
2022-04-14 11:51:53 +00:00
junos
c05b047c2d
Correct outstanding baseline feature mistake.
2022-04-13 17:05:16 +02:00
junos
53ec52a954
Disable (SOME) feature cleaning for ESM data.
2022-04-13 16:01:31 +02:00
=
3c058e4463
Add option to calculate features within windows and store it in CSV (all sensors).
2022-04-13 13:18:23 +00:00
junos
144f0d0dcf
Account for missing baseline data.
2022-04-13 14:56:28 +02:00
junos
ed5314aa98
Merge remote-tracking branch 'origin/master'
2022-04-12 17:27:25 +02:00
junos
a6a37c7bd9
Drop NaN targets.
...
This mirrors INNER join in merge_features_and_targets_for_individual_model.py:
data = pd.concat([sensor_features, targets[["target"]]], axis=1, join="inner")
2022-04-12 17:01:49 +02:00
junos
9f5edf1c2b
Revert "Add a rule for model baselines."
...
The example was for a classification rather than regression problem.
This reverts commit 9ab0c8f289
.
# Conflicts:
# rules/models.smk
2022-04-12 16:59:42 +02:00
junos
4ad261fae5
Rename baseline features AGAIN.
...
Correct other mistakes.
2022-04-12 16:55:01 +02:00
=
74cf4ada1c
Cr-feat window length for all empaticas sensors.
2022-04-12 14:00:44 +00:00
junos
9ab0c8f289
Add a rule for model baselines.
...
Add baselines and helper functions to main models dir.
2022-04-12 14:23:58 +02:00
junos
f5688f6154
Add a rule to merge sensor and baseline features.
...
And select target as before.
2022-04-08 15:42:04 +02:00
junos
b1f356c3f7
Extract a function to be used elsewhere.
2022-04-08 15:36:32 +02:00
junos
7ff3dcf5fc
Move and rename target variable.
2022-04-06 18:21:09 +02:00
junos
50c0defca7
Select target columns (no parsing necessary).
2022-04-06 18:16:49 +02:00
junos
ac86221662
[WIP] Add a rule to parse targets.
...
Does nothing for now.
2022-04-06 17:47:03 +02:00
junos
d326a1b09d
Include the constant directly in main.py.
2022-04-05 19:08:43 +02:00
junos
2e545e81f0
Include feature calculations for different scales.
2022-04-05 19:05:34 +02:00
junos
cbc8ae4e03
Add necessary checks for empty data frames.
2022-04-05 18:58:09 +02:00
junos
751b04f3f4
Pass scale names to Snakemake correctly.
2022-04-05 18:14:37 +02:00
junos
99245afca3
Try a different approach for preprocessing ESMs.
...
It is important that this follows generic RAPIDS pattern.
In the subsequent step of calculating features,
there is an expected file and folder structure of data/interim.
See rules/common.smk/find_features_files()
2022-04-05 18:02:31 +02:00
junos
ed298a9479
Implement the basic feature extraction steps.
2022-04-05 15:46:02 +02:00
Primoz
c050174ca3
Various minimal changes.
2022-03-31 09:16:00 +00:00
Primoz
a357138f6e
Added CF for HRV and shortened test data
2022-03-30 15:01:24 +00:00
Primoz
470993eeb0
Modification of getSampleRate method for all CF scripts.
2022-03-30 15:00:11 +00:00
junos
3af8de6235
Create feature provider script.
2022-03-30 10:40:53 +02:00
junos
9478dc94f2
Add an else.
...
This is to make sure that in case the reversing fails, we do not get any output items.
Snakemake will inform us of an error in this event.
2022-03-30 10:40:53 +02:00
=
ab0b9227d7
Added ACC calculated features and shorter version of ACC data.
2022-03-29 09:41:51 +00:00
=
a9244a60fc
Corrections for TEMP cf src script.
2022-03-28 14:26:37 +00:00
=
8b76c96e47
Cleaning existing CF mains' and preparing src script for ACC.
2022-03-28 14:18:29 +00:00
=
ca59a54d8f
Get a sample rate from two sequential timestamps.
2022-03-28 13:50:08 +00:00
=
393dab72f5
Added components for the temperature features extraction.
2022-03-28 12:37:02 +00:00
Primoz
f389ac9d89
Delete CF features folder
2022-03-25 16:24:52 +00:00
Primoz
191e53e543
Added cf provider for EDA feature processing.
2022-03-23 15:13:53 +00:00
Primoz
2da0911d4c
Skeleton file main.py for EDA CalcFt. integration.
2022-03-22 12:48:43 +00:00
junos
c6144f8403
Reverse JCQ items.
2022-03-16 18:55:46 +01:00
junos
23f0aaba3a
Get the name of the questionnaire from Snakefile.
2022-03-16 18:28:57 +01:00
junos
679f00dc19
Enable selecting any questionnaire as target.
2022-03-16 17:55:44 +01:00
junos
1374eda171
Flatten questionnaire ID dict.
2022-03-16 17:38:09 +01:00
junos
19b9da0ba3
Separate function definitions from main.
2022-03-16 16:49:28 +01:00
junos
ef57103bac
Add questionnaire ID key.
2022-03-15 13:41:33 +01:00
junos
5f293211a7
Reformat.
2022-03-15 13:28:51 +01:00
junos
d470eef27e
Add a rule to preprocess and clean ESM.
2022-03-09 18:38:46 +01:00
junos
d4a4bbbff0
Remove unused columns.
2022-03-09 17:58:36 +01:00
junos
085a6d144b
Add files to compute and create an empty script.
2022-03-09 17:32:02 +01:00
junos
42d62f16d0
Add RAPIDS mandatory columns for ESM.
2022-03-09 17:31:37 +01:00
junos
2bef86b1da
Add a format for ESM and add to config.
2022-03-08 15:43:25 +01:00
junos
d8e9a309f7
Rename features and write baseline_interim.
2022-03-08 15:10:36 +01:00
junos
a3a4f04ffe
Setting with : produces NaNs.
2022-03-01 12:02:57 +01:00
junos
aedb8b6785
Write questionnaire data to data/interim.
2022-03-01 12:02:36 +01:00
junos
b5a6317f4b
Calculate JCQ control and demand control ratio.
...
Include norms and corresponding quartile.
2022-02-28 18:51:47 +01:00
junos
2fed962644
Calculate JCQ demand score.
...
Hardcode question IDs to be reversed.
2022-02-28 18:30:41 +01:00
junos
30ac8b1cd5
Start calculating demand control features.
2022-02-23 19:08:10 +01:00
junos
9a74e74d08
Add the baseline features rule to snakefile.
...
Correct age calculation for a single value instead of dataframe.
2022-02-23 18:15:26 +01:00
junos
07da6be398
Add age, gender, and language as features.
...
Move calculation of age from merge_baseline_data.py to baseline_features.py.
2022-02-23 18:05:23 +01:00
junos
176367631b
Prepare baseline feature rule.
2022-02-23 11:09:33 +01:00
junos
bf9c764c97
Split baseline data to participants.
...
And some csv I/O settings.
2022-02-04 18:37:57 +01:00
junos
16e608db74
First merge baseline datasets.
2022-02-04 18:21:42 +01:00
junos
204f6f50b0
Read the relevant files.
2022-02-04 18:06:02 +01:00