Primoz
437459648f
Errors fix: individual script - treat participants missing data.
2022-10-05 13:35:05 +00:00
Primoz
53f6cc60d5
Config and cleaning script necessary changes ...
2022-10-03 13:06:39 +00:00
Primoz
bbeabeee6f
Last changes before processing on the server.
2022-10-03 12:53:31 +00:00
Primoz
44531c6d94
Code cleaning, reworking cleaning individual based on changes in overall script. Changes in thresholds.
2022-09-30 10:04:07 +00:00
Primoz
7ac7cd5a37
Preparation of the overall cleaning script.
2022-09-29 14:33:21 +00:00
Primoz
68fd69dada
Cleaning script for individuals: corrections and comments.
2022-09-29 11:55:25 +00:00
Primoz
a4f0d056a0
Fillna for app foreground and activity recognition
2022-09-29 11:44:27 +00:00
Primoz
6286e7a44c
firstuseafter column removed from contextual imputation
2022-09-28 12:47:08 +00:00
Primoz
9b3447febd
Contextual imputation correction
2022-09-28 12:40:05 +00:00
Primoz
d6adda30cf
Contextual imputation on time(first/last) features.
2022-09-28 12:37:51 +00:00
Primoz
8af4ef11dc
Contextual imputation by feature type.
2022-09-28 10:02:47 +00:00
Primoz
536b9494cd
Cleaning script corrections
2022-09-27 14:12:08 +00:00
Primoz
f0b87c9dd0
Debugging of the empatica data yield integration.
2022-09-27 09:54:15 +00:00
Primoz
bd53dc1684
Empatica data yield usage in the cleaning script.
2022-09-26 15:54:00 +00:00
Primoz
d9a574c550
Changes in the cleaning script and preparation of empatica data yield method.
2022-09-23 13:24:50 +00:00
Primoz
19aa8707c0
Redefined cleaning steps after revision
2022-09-22 13:45:51 +00:00
Primoz
247d758cb7
Merge branch 'imputation_and_cleaning' of https://repo.ijs.si/junoslukan/rapids into imputation_and_cleaning
2022-09-21 07:18:01 +00:00
Primoz
90ee99e4b9
Remove TODO comments
2022-09-21 07:16:00 +00:00
Primoz
7493aaa643
Small changes in cleaning scrtipt and missing vals testing.
2022-09-20 12:57:55 +00:00
Primoz
eaf4340afd
Small imputation and cleaning corrections.
2022-09-20 08:03:48 +00:00
Primoz
a96ea508c6
Fill NaN of Empatica's SD second order feature (must be tested).
2022-09-19 07:34:02 +00:00
Primoz
52e11cdcab
Configurations for new standardization path.
2022-09-19 07:25:54 +00:00
Primoz
92aff93e65
Remove standardization script.
2022-09-19 07:25:16 +00:00
Primoz
62982866cd
Phone wifi visible inspection (WIP)
2022-09-16 13:24:21 +00:00
Primoz
0ce6da5444
kNN imputation relocation and execution only on specific columns.
2022-09-16 11:30:08 +00:00
Primoz
e3b78c8a85
Impute selected phone features with 0.
...
Wifi visible, screen, and light.
2022-09-16 10:58:57 +00:00
Primoz
18002f59e1
Doryab bluetooth and locations features fill in NaN values.
2022-09-15 10:48:59 +00:00
Primoz
3cf7ca41aa
Merge branch 'imputation_and_cleaning' of https://repo.ijs.si/junoslukan/rapids into imputation_and_cleaning
2022-09-14 15:38:32 +00:00
Primoz
d5ab5a0394
Writing testing scripts to determine the point of manual imputation.
2022-09-14 14:13:03 +00:00
Primoz
d27a4a71c8
Reorganisation and reordering of the cleaning script.
2022-09-12 13:44:17 +00:00
Primoz
15d792089d
Changes in cleaning script:
...
- target extracted from config to remove rows where target is nan
- prepared sns.heatmap for further missing values analysis
- necessary changes in config and participant p01
- picture of heatmap which shows the values state after cleaning
2022-09-01 10:33:36 +00:00
Primoz
cb351e0ff6
Unnecessary line (rows with no target value will be removed in cleaning script).
2022-09-01 10:06:57 +00:00
Primoz
86299d346b
Impute phone and sms NAs with 0
2022-09-01 09:57:21 +00:00
Primoz
3f7ec80c18
Preparation a) phone_calls 0 imputation b) remove rows with NaN target
2022-08-31 10:18:50 +00:00
Primoz
f78aa3e7b3
Preparation for cleaning & imputation
2022-08-26 10:56:14 +00:00
Primoz
c498ecb742
Include baseline models (+corrections), disable columns drop in cleaning function.
2022-08-23 14:12:14 +00:00
Primoz
f088e9586f
Handle empty ACC.csv
2022-08-22 14:20:47 +00:00
Primoz
0aa0e82673
Handle empty Empatica csv files.
2022-08-22 14:18:12 +00:00
Primoz
4cfe5a3a98
Disable discarding rows if DATA_YIELD_RATIO_THRESHOLD==0.
2022-08-19 13:10:56 +00:00
Primoz
607da820f2
Configuration and cleaning changes
2022-08-18 14:21:05 +00:00
Primoz
fb577bc9ad
Squashed commit of the following:
...
commit 43ecc243cb62bb31eed85cb477ca4131555c7fe7
Author: Primoz <sisko.primoz@gmail.com>
Date: Fri Jul 22 15:26:09 2022 +0000
Adding TODO comments
commit 2df1ebf90c3a93812b112b8ed0ee4e23cd74533f
Author: Primoz <sisko.primoz@gmail.com>
Date: Thu Jul 21 13:59:23 2022 +0000
README update
commit 5182c2b16dff3537aad42984b8ea5214743cdb32
Author: Primoz <sisko.primoz@gmail.com>
Date: Thu Jul 21 11:03:01 2022 +0000
Few corrections for all_cleaning
commit 3d9254c1b3bed6e95e631d4e0402548830a19534
Author: Primoz <sisko.primoz@gmail.com>
Date: Thu Jul 21 10:28:05 2022 +0000
Adding the min overlap for corr threshold and preservation of esm cols.
commit e27c49cc8fa4c51f9fe8e593a8d25e9a032ab393
Author: Primoz <sisko.primoz@gmail.com>
Date: Thu Jul 21 09:02:00 2022 +0000
Commenting and cleaning.
commit 31a47a5ee4569264e39d7c445525a6e64bb7700a
Author: Primoz <sisko.primoz@gmail.com>
Date: Wed Jul 20 13:49:22 2022 +0000
Environment version change.
commit 5b274ed8993f58e783bda6d82fce936764209c28
Author: Primoz <sisko.primoz@gmail.com>
Date: Tue Jul 19 16:10:07 2022 +0000
Enabled cleaning for all participants + standardization files.
commit 203fdb31e0f3c647ef8c8a60cb9531831b7ab924
Author: Primoz <sisko.primoz@gmail.com>
Date: Tue Jul 19 14:14:51 2022 +0000
Features cleaning fixes after testing. Visualization script for phone features values.
commit 176178d73b154c30b9eb9eb4a67514f00d6a924e
Author: Primoz <sisko.primoz@gmail.com>
Date: Tue Jul 19 09:05:14 2022 +0000
Revert "Necessary config changes."
This reverts commit 6ec1ef50430d2e1f5ce4670d505d5e84ac47f0a0.
commit 26ea6512c9d512f95837e7b047fe510c1d196403
Author: Primoz <sisko.primoz@gmail.com>
Date: Mon Jul 18 13:19:47 2022 +0000
Adding cleaning function condition and cleaning functionality.
commit 575c29eef9c21e6f2d7832871e73bc0941643734
Author: Primoz <sisko.primoz@gmail.com>
Date: Mon Jul 18 12:51:56 2022 +0000
Translation of the cleaning individual RAPIDS function from R to py.
commit 6ec1ef50430d2e1f5ce4670d505d5e84ac47f0a0
Author: Primoz <sisko.primoz@gmail.com>
Date: Mon Jul 18 12:02:18 2022 +0000
Necessary config changes.
commit b5669f51612fbd8378848615d639677851ab032f
Author: Primoz <sisko.primoz@gmail.com>
Date: Fri Jul 15 15:26:00 2022 +0000
Modified snakemake rule to dynamically choose script extention.
commit 66636be1e8ae4828228b37c59b9df1faf3fc3d3d
Author: Primoz <sisko.primoz@gmail.com>
Date: Fri Jul 15 14:43:08 2022 +0000
Trying to modify the snakefile rule to execute scripts in two languages depended on the provider.
commit 574778b00f3cbb368ef4bc74de15cf5070c65ea9
Author: Primoz <sisko.primoz@gmail.com>
Date: Fri Jul 15 09:49:41 2022 +0000
gitignore: adding required files so that RAPIDS can be run successfully.
commit 71018ab178256970535e78961602ab8c7f0ebb14
Author: Primoz <sisko.primoz@gmail.com>
Date: Fri Jul 15 08:34:19 2022 +0000
Standardization bug fixes
commit 6253c470a624e6bfbb02e0c453b652452eb2dbbc
Author: Primoz <sisko.primoz@gmail.com>
Date: Thu Jul 14 15:28:02 2022 +0000
Seperate rules for empatica vs. nonempatica standardization.
Parameter in config that controls the creation of standardized merged files for individual and all participants..
commit 90f902778565e0896d3bae22ae8551be8b487e67
Author: Primoz <sisko.primoz@gmail.com>
Date: Tue Jul 12 14:23:03 2022 +0000
Preparing for final csvs' standardization.
commit d25dde3998786a9a582f5cda544ee104386778f9
Author: Primoz <sisko.primoz@gmail.com>
Date: Mon Jul 11 12:08:47 2022 +0000
Revert "Changes in config to be reverted."
This reverts commit bea7608e7095021fb7c53a9afa07074448fe4313.
commit 6b23e70857e63deda98eb98d190af9090626c84b
Author: Primoz <sisko.primoz@gmail.com>
Date: Mon Jul 11 12:08:26 2022 +0000
Enabled standardization for rest (previously active) phone features.
Testing still needed.
commit 8ec58a6f34ba3d42e5cc71d26e6d91837472ca5f
Author: Primoz <sisko.primoz@gmail.com>
Date: Mon Jul 11 09:07:55 2022 +0000
Enabled standardization for phone calls.
All steps completed and tested.
commit bea7608e7095021fb7c53a9afa07074448fe4313
Author: Primoz <sisko.primoz@gmail.com>
Date: Mon Jul 11 07:47:51 2022 +0000
Changes in config to be reverted.
commit 4e84ca0e51bf709bff56fd09437b95310ec6bedd
Author: Primoz <sisko.primoz@gmail.com>
Date: Fri Jul 8 14:11:24 2022 +0000
Standardization for the rest of the features.
commit cc581aa788e3d5c17131af8f3d5dd6b0c3b5aff7
Author: Primoz <sisko.primoz@gmail.com>
Date: Fri Jul 8 14:11:08 2022 +0000
README update again
2022-07-22 15:31:30 +00:00
Primoz
6ba4a66deb
Squashed commit of the following:
...
commit 31a47a5ee4569264e39d7c445525a6e64bb7700a
Author: Primoz <sisko.primoz@gmail.com>
Date: Wed Jul 20 13:49:22 2022 +0000
Environment version change.
commit 5b274ed8993f58e783bda6d82fce936764209c28
Author: Primoz <sisko.primoz@gmail.com>
Date: Tue Jul 19 16:10:07 2022 +0000
Enabled cleaning for all participants + standardization files.
commit 203fdb31e0f3c647ef8c8a60cb9531831b7ab924
Author: Primoz <sisko.primoz@gmail.com>
Date: Tue Jul 19 14:14:51 2022 +0000
Features cleaning fixes after testing. Visualization script for phone features values.
commit 176178d73b154c30b9eb9eb4a67514f00d6a924e
Author: Primoz <sisko.primoz@gmail.com>
Date: Tue Jul 19 09:05:14 2022 +0000
Revert "Necessary config changes."
This reverts commit 6ec1ef50430d2e1f5ce4670d505d5e84ac47f0a0.
commit 26ea6512c9d512f95837e7b047fe510c1d196403
Author: Primoz <sisko.primoz@gmail.com>
Date: Mon Jul 18 13:19:47 2022 +0000
Adding cleaning function condition and cleaning functionality.
commit 575c29eef9c21e6f2d7832871e73bc0941643734
Author: Primoz <sisko.primoz@gmail.com>
Date: Mon Jul 18 12:51:56 2022 +0000
Translation of the cleaning individual RAPIDS function from R to py.
commit 6ec1ef50430d2e1f5ce4670d505d5e84ac47f0a0
Author: Primoz <sisko.primoz@gmail.com>
Date: Mon Jul 18 12:02:18 2022 +0000
Necessary config changes.
commit b5669f51612fbd8378848615d639677851ab032f
Author: Primoz <sisko.primoz@gmail.com>
Date: Fri Jul 15 15:26:00 2022 +0000
Modified snakemake rule to dynamically choose script extention.
commit 66636be1e8ae4828228b37c59b9df1faf3fc3d3d
Author: Primoz <sisko.primoz@gmail.com>
Date: Fri Jul 15 14:43:08 2022 +0000
Trying to modify the snakefile rule to execute scripts in two languages depended on the provider.
commit 574778b00f3cbb368ef4bc74de15cf5070c65ea9
Author: Primoz <sisko.primoz@gmail.com>
Date: Fri Jul 15 09:49:41 2022 +0000
gitignore: adding required files so that RAPIDS can be run successfully.
commit 71018ab178256970535e78961602ab8c7f0ebb14
Author: Primoz <sisko.primoz@gmail.com>
Date: Fri Jul 15 08:34:19 2022 +0000
Standardization bug fixes
commit 6253c470a624e6bfbb02e0c453b652452eb2dbbc
Author: Primoz <sisko.primoz@gmail.com>
Date: Thu Jul 14 15:28:02 2022 +0000
Seperate rules for empatica vs. nonempatica standardization.
Parameter in config that controls the creation of standardized merged files for individual and all participants..
commit 90f902778565e0896d3bae22ae8551be8b487e67
Author: Primoz <sisko.primoz@gmail.com>
Date: Tue Jul 12 14:23:03 2022 +0000
Preparing for final csvs' standardization.
commit d25dde3998786a9a582f5cda544ee104386778f9
Author: Primoz <sisko.primoz@gmail.com>
Date: Mon Jul 11 12:08:47 2022 +0000
Revert "Changes in config to be reverted."
This reverts commit bea7608e7095021fb7c53a9afa07074448fe4313.
commit 6b23e70857e63deda98eb98d190af9090626c84b
Author: Primoz <sisko.primoz@gmail.com>
Date: Mon Jul 11 12:08:26 2022 +0000
Enabled standardization for rest (previously active) phone features.
Testing still needed.
commit 8ec58a6f34ba3d42e5cc71d26e6d91837472ca5f
Author: Primoz <sisko.primoz@gmail.com>
Date: Mon Jul 11 09:07:55 2022 +0000
Enabled standardization for phone calls.
All steps completed and tested.
commit bea7608e7095021fb7c53a9afa07074448fe4313
Author: Primoz <sisko.primoz@gmail.com>
Date: Mon Jul 11 07:47:51 2022 +0000
Changes in config to be reverted.
commit 4e84ca0e51bf709bff56fd09437b95310ec6bedd
Author: Primoz <sisko.primoz@gmail.com>
Date: Fri Jul 8 14:11:24 2022 +0000
Standardization for the rest of the features.
commit cc581aa788e3d5c17131af8f3d5dd6b0c3b5aff7
Author: Primoz <sisko.primoz@gmail.com>
Date: Fri Jul 8 14:11:08 2022 +0000
README update again
2022-07-20 13:51:22 +00:00
Primoz
788ac31190
Bug fix: if df has no rows write an empty zscore file.
2022-07-08 10:40:45 +00:00
Primoz
5a777ac79f
Working version that integrates both phone and empatica feature calculations.
2022-07-07 15:00:47 +00:00
Primoz
0425403951
Merge branch 'master' of https://repo.ijs.si/junoslukan/rapids
2022-07-06 11:53:31 +00:00
Primoz
887fd7dc72
Merge branch 'empatica_calculating_features'
2022-07-06 11:53:21 +00:00
Primoz
d2758eef46
Set not NaN sum insted of 0 sum for HRV features windows.
2022-07-06 07:36:35 +00:00
Primoz
2d5d23b615
Testing files change and remove standardization from hrv sensors main files.
2022-07-06 07:35:39 +00:00
Primoz
505c3a86b9
Testing different EDA findPeaks parameters.
2022-06-30 15:15:37 +00:00
Primoz
c851ab0763
Fill EDA NaN values where numPeak is zero. Other small changes.
2022-06-21 14:09:49 +00:00
Primoz
dda4554d46
Various small changes.
2022-06-15 13:57:46 +00:00
Primoz
212cf300f8
Debugging EDA signal - preliminary step for imputation.
2022-06-14 15:09:14 +00:00
Primoz
9ea39dc557
Standardization as a Snakefile's rule enabled for all E4 sensors.
2022-06-13 18:17:30 +00:00
Primoz
402059871f
Making standardization as a rule. WIP: done only for BVP.
2022-06-13 14:12:03 +00:00
Primoz
094743244d
Added SO feature for sum all rows that are non zero for BVP and IBI sensors.
2022-06-13 10:51:22 +00:00
primoz
e1d7607de4
Extraction of additional SO features. Min/max has been changed to nsmallest/nlargest means.
2022-06-10 12:34:48 +00:00
primoz
f371249b99
First order features standardization WIP
2022-06-09 13:35:15 +00:00
primoz
2c7ac21465
Added standardization on SO features.
2022-06-06 13:51:15 +00:00
primoz
2acf6ff9fb
Exception handling in case of empty ibi. Changes of the method EDA uses in main.py. Other small corrections.
2022-06-03 12:34:36 +00:00
primoz
d300f0f8f0
Fixed RAPIDS bug: error when IBI.csv is empty.
2022-06-02 11:43:49 +00:00
Primoz
5532043b1f
Patching IBI with BVP - completed.
2022-05-25 19:39:47 +00:00
Primoz
bb62497ba6
Patching IBI with BVP - selecting appropriate pipeline entry point. WIP
2022-05-24 11:07:18 +00:00
Primoz
2a8f58f5c8
Patching IBI with BVP. WIP
2022-05-20 13:18:45 +00:00
Primoz
6864cfe775
Changes after thorough testing with available data.
2022-05-13 13:35:34 +00:00
Primoz
c1564f0cae
Changed wrapper method calculate_feature to its newest version (for TEMP and ACC).
2022-05-11 14:21:21 +00:00
Primoz
31e36e7400
Alternating Second order and full segment features corresponding to config settings.
2022-05-11 08:50:15 +00:00
Primoz
9cf9e1fe14
Testing and modifying the code with different E4 data.
2022-05-10 11:36:49 +00:00
Primoz
f62a1302dd
Cr-features corrections for ACC and TEMP sensors
2022-05-09 11:01:52 +00:00
Primoz
5638367999
Implementation of the second order features.
2022-04-25 13:07:03 +00:00
Primoz
66451160e9
Calculating HRV features with IBI.csv.
2022-04-20 10:44:51 +00:00
=
8c8fe1fec7
Modifications, mostly imports, after changes in cr-features module.
2022-04-19 13:24:46 +00:00
=
075c64d1e5
HRV: changed wrapper calcFeat method with specialized one.
2022-04-14 11:51:53 +00:00
junos
c05b047c2d
Correct outstanding baseline feature mistake.
2022-04-13 17:05:16 +02:00
junos
53ec52a954
Disable (SOME) feature cleaning for ESM data.
2022-04-13 16:01:31 +02:00
=
3c058e4463
Add option to calculate features within windows and store it in CSV (all sensors).
2022-04-13 13:18:23 +00:00
junos
144f0d0dcf
Account for missing baseline data.
2022-04-13 14:56:28 +02:00
junos
ed5314aa98
Merge remote-tracking branch 'origin/master'
2022-04-12 17:27:25 +02:00
junos
a6a37c7bd9
Drop NaN targets.
...
This mirrors INNER join in merge_features_and_targets_for_individual_model.py:
data = pd.concat([sensor_features, targets[["target"]]], axis=1, join="inner")
2022-04-12 17:01:49 +02:00
junos
9f5edf1c2b
Revert "Add a rule for model baselines."
...
The example was for a classification rather than regression problem.
This reverts commit 9ab0c8f289
.
# Conflicts:
# rules/models.smk
2022-04-12 16:59:42 +02:00
junos
4ad261fae5
Rename baseline features AGAIN.
...
Correct other mistakes.
2022-04-12 16:55:01 +02:00
=
74cf4ada1c
Cr-feat window length for all empaticas sensors.
2022-04-12 14:00:44 +00:00
junos
9ab0c8f289
Add a rule for model baselines.
...
Add baselines and helper functions to main models dir.
2022-04-12 14:23:58 +02:00
junos
f5688f6154
Add a rule to merge sensor and baseline features.
...
And select target as before.
2022-04-08 15:42:04 +02:00
junos
b1f356c3f7
Extract a function to be used elsewhere.
2022-04-08 15:36:32 +02:00
junos
7ff3dcf5fc
Move and rename target variable.
2022-04-06 18:21:09 +02:00
junos
50c0defca7
Select target columns (no parsing necessary).
2022-04-06 18:16:49 +02:00
junos
ac86221662
[WIP] Add a rule to parse targets.
...
Does nothing for now.
2022-04-06 17:47:03 +02:00
junos
d326a1b09d
Include the constant directly in main.py.
2022-04-05 19:08:43 +02:00
junos
2e545e81f0
Include feature calculations for different scales.
2022-04-05 19:05:34 +02:00
junos
cbc8ae4e03
Add necessary checks for empty data frames.
2022-04-05 18:58:09 +02:00
junos
751b04f3f4
Pass scale names to Snakemake correctly.
2022-04-05 18:14:37 +02:00
junos
99245afca3
Try a different approach for preprocessing ESMs.
...
It is important that this follows generic RAPIDS pattern.
In the subsequent step of calculating features,
there is an expected file and folder structure of data/interim.
See rules/common.smk/find_features_files()
2022-04-05 18:02:31 +02:00
junos
ed298a9479
Implement the basic feature extraction steps.
2022-04-05 15:46:02 +02:00
Primoz
c050174ca3
Various minimal changes.
2022-03-31 09:16:00 +00:00
Primoz
a357138f6e
Added CF for HRV and shortened test data
2022-03-30 15:01:24 +00:00
Primoz
470993eeb0
Modification of getSampleRate method for all CF scripts.
2022-03-30 15:00:11 +00:00
junos
3af8de6235
Create feature provider script.
2022-03-30 10:40:53 +02:00
junos
9478dc94f2
Add an else.
...
This is to make sure that in case the reversing fails, we do not get any output items.
Snakemake will inform us of an error in this event.
2022-03-30 10:40:53 +02:00
=
ab0b9227d7
Added ACC calculated features and shorter version of ACC data.
2022-03-29 09:41:51 +00:00
=
a9244a60fc
Corrections for TEMP cf src script.
2022-03-28 14:26:37 +00:00