Commit Graph

682 Commits (7832d7d098cc0acc501a79a5a03a3f6985626d5e)

Author SHA1 Message Date
junos 2bef86b1da Add a format for ESM and add to config. 2022-03-08 15:43:25 +01:00
junos d8e9a309f7 Rename features and write baseline_interim. 2022-03-08 15:10:36 +01:00
junos a3a4f04ffe Setting with : produces NaNs. 2022-03-01 12:02:57 +01:00
junos aedb8b6785 Write questionnaire data to data/interim. 2022-03-01 12:02:36 +01:00
junos b5a6317f4b Calculate JCQ control and demand control ratio.
Include norms and corresponding quartile.
2022-02-28 18:51:47 +01:00
junos 2fed962644 Calculate JCQ demand score.
Hardcode question IDs to be reversed.
2022-02-28 18:30:41 +01:00
junos 30ac8b1cd5 Start calculating demand control features. 2022-02-23 19:08:10 +01:00
junos 9a74e74d08 Add the baseline features rule to snakefile.
Correct age calculation for a single value instead of dataframe.
2022-02-23 18:15:26 +01:00
junos 07da6be398 Add age, gender, and language as features.
Move calculation of age from merge_baseline_data.py to baseline_features.py.
2022-02-23 18:05:23 +01:00
junos 176367631b Prepare baseline feature rule. 2022-02-23 11:09:33 +01:00
junos bf9c764c97 Split baseline data to participants.
And some csv I/O settings.
2022-02-04 18:37:57 +01:00
junos 16e608db74 First merge baseline datasets. 2022-02-04 18:21:42 +01:00
junos 204f6f50b0 Read the relevant files. 2022-02-04 18:06:02 +01:00
junos 685ed6a546 Set up demographic data download. 2022-02-04 17:37:00 +01:00
Meng Li 463ac0a2aa
Fix bug#169 (#174) 2022-01-27 11:27:32 -05:00
Sam 10e896ca1d
Add data stream for AWARE Micro server (#173)
* Add data stream for AWARE Micro server

* Fix one documentation typo and one ommission
2022-01-27 10:47:50 -05:00
junos afa3b8546f Mutate data in an R script.
The Python script did not read the timestamp correctly for some reason. All timestamps were 0.
2022-01-26 16:34:19 +01:00
Sam e5dbbfce44
Avoid NA problem in barnett location evaluation (#172)
* Avoid occasional issue where does_not_span evaluates to NA, which breaks the if()

* Restored original warning
2022-01-18 10:16:37 -05:00
Sam 8ae26fb845
Fixes issue where 'duration' in the 'ios_calls' dataframe is seen as a character type. (#171) 2022-01-18 10:15:53 -05:00
junos b17a7eff1a Deal with inexplicable snakemake failure. 2022-01-07 18:11:38 +01:00
junos e1499a5ae2 Account for missing device_ids. 2021-12-15 20:41:28 +01:00
junos 5a9252e46e Merge remote-tracking branch 'origin/master' 2021-12-15 18:32:36 +01:00
junos 352598f3da Use absolute path to avoid RuleException. 2021-12-15 17:27:13 +01:00
junos 70cada8bb8 Consider a subset of columns when dropping. 2021-12-15 16:14:33 +01:00
junos d2ed73dccf Debug ValueError for index.
See exploration/debug_heatmap.py for illustration.
2021-12-15 16:03:04 +01:00
junos 6f451e05ac Bring back application_name.
This column still needs to be in the data, so add it in app_add_name.py.
Later, join categories by package hash.
2021-12-15 12:58:27 +01:00
junos 4485c4c95e Delete columns we don't have.
Rename light table.
Correct timesegments.
2021-12-08 20:02:47 +01:00
junos 37b3460b76 Use Empatica wristband numbers as provided in CSV. 2021-12-01 17:20:57 +01:00
junos 22f9e0722d Start preparing the true usernames CSV file. 2021-12-01 11:29:22 +01:00
junos 0be4cd5a8f Remove unnecessary library. 2021-11-30 17:08:07 +01:00
junos 04ad2d0b81 Source specific container script.
It is probably not worth the effort of making this general.
2021-11-29 18:19:47 +01:00
junos da5ff0f36e Correct small errors in settings. 2021-11-29 18:04:06 +01:00
junos 35d9779026 Prepare the tibble in requested format.
Write it to a CSV file.
2021-11-29 17:54:16 +01:00
junos 32025cbd8c Start with a tibble from CSV. 2021-11-29 17:51:07 +01:00
junos 181e4f0118 Add parameters to yaml file.
And use these in the prepare_participants_file function.
2021-11-29 16:57:50 +01:00
junos ab84109d55 Prepare a function to compile participants data.
It combines functions from container.R
2021-11-24 19:07:56 +01:00
junos f9863ec622 Fix small mistakes. 2021-11-24 19:01:30 +01:00
junos c1f56c61e8 Add a function to pull start and end datetimes. 2021-11-24 18:33:06 +01:00
junos 3acf6ece14 Add a function to pull device IDs. 2021-11-24 18:23:53 +01:00
junos 8b2717122d Add a function to get participants' IDs. 2021-11-24 18:05:17 +01:00
Meng Li 5bad3eb8b5
Data cleaning (#166)
* Refactor data cleaning module: move it from example workflow to main directory

* Replace NAs with 0 in selected event-based features

* Add one step to drop highly correlated features

Co-authored-by: Weiyu <weiyuhuang7@gmail.com>
2021-11-19 10:34:36 -05:00
Meng Li 296960f425 Fix the bug of location doryab features when a participant is moving during the whole time segment 2021-11-18 18:42:19 -05:00
Meng Li 3d34036eae
Add firststeptime and laststeptime features to FITBIT_STEPS_INTRADAY RAPIDS provider (#168)
* Add firststeptime and laststeptime features to FITBIT_STEPS_INTRADAY RAPIDS provider

* Update test config files
2021-11-18 18:35:27 -05:00
junos ed193d2290 Revert "Correct the name of a field."
This reverts commit b335561a55.

It was actually correct.
2021-11-17 19:16:35 +01:00
junos b335561a55 Correct the name of a field. 2021-11-17 18:50:06 +01:00
junos fcec3e2f93 Implement the necessary functions for PSQL. 2021-11-17 18:49:25 +01:00
junos 7a1e4f7139 Add the format file copied from MySQL. 2021-11-17 18:46:16 +01:00
Meng Li f340b89c58 Temporary revert PHONE_LOCATIONS BARNETT provider to use R script 2021-09-23 18:16:13 -04:00
Meng Li a3fb718aea Refactor PHONE_LOCATIONS DORYAB provider to compute features based on location episodes 2021-09-23 17:40:06 -04:00
Meng Li a8a178486b Refactor PHONE_CALLS RAPIDS provider to compute features based on call episodes or events 2021-09-15 10:28:37 -04:00
JulioV 2e553dc9e7 Add tqdm package to environment.yaml 2021-08-16 11:04:03 -04:00
Meng Li 3ac12e7dad Fix the bug of step intraday features when INCLUDE_ZERO_STEP_ROWS is False 2021-08-11 12:40:40 -04:00
Weiyu 35eebe8a51 Bug fixed: set ratiovalidyielded mins/hours value to the range 0 to 1 2021-08-09 17:56:29 -04:00
JulioV 3e69966c91 Update error message 2021-08-04 15:33:02 -04:00
Shirley 4ddb2845a6 Update initialize_params 2021-08-04 15:33:02 -04:00
JulioV 834bd3b93d Refactor in Python of Barnett provider
Co-authored-by: Shirley Hayati <sahayati@ucdavis.edu>
Co-authored-by: JulioV <JulioV@users.noreply.github.com>
2021-08-04 15:33:02 -04:00
Weiyu 7f1c502ea0 Fixed bug: Added local_segment column if no data left after filtered 2021-08-04 11:06:31 -04:00
Hannah Roberts b52059b027 Ensure date/time format is maintained
Within the 'determine which is home' for loop, 'xx' is the midpoint of two datetime objects. When the midpoint is calculated to be midnight, only the date is returned. This can be replicated with:

mydates <- as.POSIXct("2018-01-01 00:00:00", tz = "UTC")
mydates
[1] "2018-01-01 UTC"

This results in 'hourofday' being NA as an hour cannot be found. By adding the suggested format wrapper, the time is maintained and 'hourofday' can be determined. It can then successfully be applied to the embedded if-statement within the loop.

mydates <- format(as.POSIXct("2018-01-01 00:00:00", tz = "UTC"), "%Y-%m-%d %H:%M:%S")
mydates
[1] "01-01-2018 00:00:00"
2021-07-23 10:12:11 -04:00
Weiyu 5a465873c4 Tested fitbit heartrate intraday feature 2021-07-21 10:24:02 -04:00
JulioV 6fa1875bf3 Add app foreground episode count 2021-07-01 16:20:16 -04:00
JulioV bc5c0c9a4f Fix app episode length bug 2021-07-01 16:20:16 -04:00
JulioV 065a926a87 Change own to custom categories name 2021-07-01 16:20:16 -04:00
JulioV e74c745f86 Add own categories to app foreground features 2021-07-01 16:20:16 -04:00
JulioV 5892b6d838 Fix create_participants_files.R to handle numeric PIDs 2021-07-01 16:20:16 -04:00
Meng Li 97ef8a8368 Set color range and avoid SettingWithCopyWarning 2021-06-29 09:50:19 -04:00
Meng Li 1c57320ab3 Update segment labels and fix the bug when we do not have any labels for event segments 2021-06-29 09:49:24 -04:00
Meng Li cefcb0635b Update heatmap of recorded phone sensors 2021-06-29 09:49:24 -04:00
Meng Li bc06477d89 Update heatmap of sensor row count 2021-06-29 09:49:24 -04:00
Meng Li e98a8ff7ca Update histogram of phone data yield 2021-06-29 09:49:24 -04:00
Meng Li f436f1f530 Update heatmap of correlation matrix 2021-06-29 09:49:23 -04:00
Meng Li 4d37696158 Update heatmaps of overall data yield 2021-06-29 09:48:30 -04:00
Weiyu f374c67bd5 Bug fixed: Added unknown activity case 2021-06-23 19:04:55 -04:00
Weiyu 3e4d167adc Bug fixed: sort bt_address alphabetically before picking the most frequent bt_address 2021-06-22 17:40:00 -04:00
Meng Li f248b6c97d Fix bugs of Fitbit mutation scripts 2021-06-11 18:18:33 -04:00
kirtirajk 4b8698a4c6 adding app_episode with the changes as mentioned in the comments 2021-06-10 14:17:56 -04:00
Weiyu 65d5cb7bd4 Bug fixed: countscansmostuniquedevice stays the same for all time segments 2021-06-10 10:49:22 -04:00
JulioV e123a14082 Improve aware_csv msg when CSV files don't exist 2021-06-01 10:57:17 -04:00
Meng Li 9687081fbe Refactor the rule phone_locations_add_doryab_extra_columns 2021-05-28 09:48:36 -04:00
Meng Li 0d6f51be8b Refactor location features from Doryab provider & add a new strategy to infer home location & fix bugs 2021-05-26 17:36:52 -04:00
JulioV 32472461ec - Fix bug when no phone data yield is needed to process location data
- Remove location rows with the same timestamp based on their accuracy
2021-05-26 14:04:29 -04:00
Nikunj Goel 9b21196f35
Fixed `expected_minutes` to account for different time segments. (#136) 2021-05-26 11:44:48 -04:00
Meng Li edf71e055d Add the EXCLUDE_SLEEP module for steps intraday features 2021-05-21 15:23:21 -04:00
Nikunj Goel 5e451f99b0
Added phone keyboard features including docs/tests (#134) 2021-05-21 11:45:27 -04:00
JulioV e9cd9c94d7 Fix PID matching when joining data from participants 2021-05-11 16:49:04 -04:00
JulioV 32818a4802 Fix parse of pids with more than 1 devices 2021-05-11 16:42:20 -04:00
Meng Li 809845143f Test & fix bugs of sleep intraday features 2021-04-27 14:40:14 -04:00
Meng Li 7c7f34ec45 Test & fix bugs of sleep summary features 2021-04-27 14:40:14 -04:00
Meng Li 50fe09cfac Update data streams mutation of fitbit data 2021-04-27 14:40:14 -04:00
Meng Li 66d9a9d640 Update params & docs of sleep features 2021-04-27 14:33:19 -04:00
JulioV 4beafd233d Fix crash when scraping data for an app that does not exist 2021-04-22 14:28:52 -04:00
JulioV ea8094e028 Fix length of periodic segments on days with DLS 2021-04-22 11:32:10 -04:00
JulioV 9c56422529 Add calories intraday features 2021-04-20 12:00:38 -04:00
Meng Li 00a3335623 Add device_id column for sleep intraday episodes 2021-04-08 11:21:28 -04:00
JulioV 286d317af4 Fix crash when there are no periodic segments to assign
This includes a simplification of how periodic segments are computed based on all local dates in the data independently of their time zones
2021-04-07 12:03:25 -04:00
JulioV 9551669d47 Fix periodic segments bug when there are no segments to assign 2021-04-06 20:29:30 -04:00
Meng Li 78173c54ab Convert date time object to string in assign_tz_code() function 2021-04-06 23:28:53 +00:00
JulioV 1025e6d9d8 Fix datetime labels of event segments across multiple tzs 2021-04-06 13:58:58 -04:00
Meng Li 8909876cff Add local_segment column for phone data yield features 2021-04-05 21:13:36 +00:00
Meng Li 68125dc1bf Fix the bug of phone data yield features when the input is empty 2021-04-05 20:57:05 +00:00
JulioV 46f5e24814 Fix Fitbit tz inference from phone data 2021-04-05 11:51:57 -04:00
JulioV 636b64c61a Revert "Added more keyboard features."
This reverts commit 94c72e3172.
2021-04-05 11:25:00 -04:00
Meng Li 68e12a2563 Fix bugs of bluetooth feature extraction when number of unique bt_address is 2 2021-04-05 14:09:50 +00:00
Meng Li 8414977331 Fix the bug of utils.py when one participant have multi timezones 2021-04-03 21:20:10 -04:00
nikunjgoel95 94c72e3172 Added more keyboard features. 2021-04-01 20:54:13 -04:00
Meng Li 1ea5b74eff Fix the bug of utils.py when one participant have multi timezones 2021-03-31 19:34:14 -04:00
Meng Li 136dfef56b Fix the bug of Analysis Workflows while parsing targets with updated segments 2021-03-30 16:41:50 -04:00
JulioV 99dae079d5 Add iOS BT and Wifi visible to formats for old devices 2021-03-30 15:32:50 -04:00
JulioV 30ad3cd586 Validate participant files without device ids 2021-03-28 15:29:08 -04:00
JulioV 87fbbbe402 Refactor and simplify time segments 2021-03-28 15:29:07 -04:00
JulioV c48c1c8f24 Optimize Barnett's computation multi-day segments 2021-03-28 15:29:07 -04:00
JulioV d0858f8833 Fix overlapping periodic time segments 2021-03-28 15:29:07 -04:00
Meng Li e177aa6386 Update create participant files section 2021-03-25 11:39:31 -04:00
Meng Li a5eb535126 Update visualizations docs & add time flag for heatmap of overall data yield 2021-03-23 21:40:55 -04:00
nikunjgoel95 cfc5039918 Fixed the empty dataframe case in infer_home_locations.py and added array condition in doryab location 2021-03-19 11:15:57 -04:00
Meng Li 294d84277d Fix bug of sleep intraday PRICE provider when the dataframe of a segment is empty 2021-03-17 15:21:31 -04:00
JulioV 771c14a928 Improve mysql containers error messages 2021-03-16 20:02:44 -04:00
JulioV 6e234f7951 Fix warn instead of stop when there are not device ids 2021-03-16 20:02:16 -04:00
JulioV 4c2f60fffd Fix bugs in readable datetime and screen episodes 2021-03-16 20:01:43 -04:00
JulioV bb737237d0 Fixes for aware_influxdb 2021-03-16 11:26:46 -04:00
JulioV 4528ab3641 Replace SRC LANGUAGE and FOLDER with SCRIPT 2021-03-14 22:14:13 -04:00
JulioV f4b2bd1fb2 Cleanup data/ 2021-03-14 13:36:22 -04:00
JulioV 42cee67664 Add aware_influxdb in beta 2021-03-14 13:33:43 -04:00
JulioV 3d4a04effe Refactor testing 2021-03-14 00:09:08 -05:00
Meng Li 2b6447105a Migrate analysis example to new data stream 2021-03-12 19:52:34 -05:00
JulioV fae0c2ac05 Swap TABLE for CONTAINER 2021-03-12 18:14:49 -05:00
Meng Li d529490999 Migrate fitbit features to new data stream 2021-03-12 12:38:36 -05:00
JulioV 6e898beca5 Add aware_csv 2021-03-11 19:32:11 -05:00
JulioV 13174b0c2a Fix a bug when fitbit data is empty 2021-03-11 14:51:16 -05:00
JulioV 2ee45995f2 Update config docs and create participant files script 2021-03-11 14:40:33 -05:00
JulioV 1e66dad838 Fix bug in empatica_zip container script 2021-03-11 14:39:26 -05:00
JulioV a79997e0ac Add empatica_zip docs 2021-03-11 14:39:26 -05:00
JulioV d48194fc07 Add fitbitparsed_csv 2021-03-11 14:39:26 -05:00
JulioV b97b70e3a1 Add fitbitjson_csv 2021-03-11 14:39:26 -05:00
JulioV 470f4276af Add fitbitparsed_mysql 2021-03-11 14:39:26 -05:00
JulioV 1b0ee4bbf0 Add sleep intraday to fitbitjson_mysql 2021-03-11 14:39:20 -05:00
JulioV a420f5ef92 Add sleep summary to fitbitjson_mysql 2021-03-11 14:37:22 -05:00
Meng Li 93baff9f83 Migrate phone keyboard sensor to new data stream 2021-03-11 14:37:22 -05:00
Meng Li 35968e2fd0 Migrate phone log sensor to new data stream 2021-03-11 14:37:22 -05:00
Meng Li 091f9c048a Migrate phone apps notifications sensor to new data stream 2021-03-11 14:37:22 -05:00
Meng Li b49dab0949 Migrate phone apps crashes sensor to new data stream 2021-03-11 14:37:22 -05:00
JulioV 47e1b33816 Add hr intraday to fitbitjson_mysql 2021-03-11 14:37:22 -05:00
JulioV 47f449555a Add hr summary to fitbitjson_mysql 2021-03-11 14:37:22 -05:00
JulioV 72f6b2d621 Add steps intraday to fitbitjson_mysql 2021-03-11 14:37:22 -05:00
JulioV 9a276c1c66 Add steps summary to jsonfitbit_mysql 2021-03-11 14:37:22 -05:00
Meng Li 13290cd444 Migrate phone wifi visible sensor to new data stream 2021-03-11 14:36:52 -05:00
Meng Li d42c6e9c91 Migrate phone wifi connected sensor to new data stream 2021-03-11 14:35:34 -05:00
Meng Li 4825962361 Migrate phone screen sensor to new data stream 2021-03-11 14:35:34 -05:00
Meng Li bd4f647d37 Migrate phone messages sensor to new data stream 2021-03-11 14:35:34 -05:00
Meng Li 3a65b3864d Migrate phone locations sensor to new data stream 2021-03-11 14:35:34 -05:00
Meng Li 7a50a52a9d Migrate phone light sensor to new data stream 2021-03-11 14:35:34 -05:00