2e553dc9e7
Add tqdm package to environment.yaml
2021-08-16 11:04:03 -04:00
3ac12e7dad
Fix the bug of step intraday features when INCLUDE_ZERO_STEP_ROWS is False
2021-08-11 12:40:40 -04:00
35eebe8a51
Bug fixed: set ratiovalidyielded mins/hours value to the range 0 to 1
2021-08-09 17:56:29 -04:00
3e69966c91
Update error message
2021-08-04 15:33:02 -04:00
4ddb2845a6
Update initialize_params
2021-08-04 15:33:02 -04:00
834bd3b93d
Refactor in Python of Barnett provider
...
Co-authored-by: Shirley Hayati <sahayati@ucdavis.edu>
Co-authored-by: JulioV <JulioV@users.noreply.github.com>
2021-08-04 15:33:02 -04:00
7f1c502ea0
Fixed bug: Added local_segment column if no data left after filtered
2021-08-04 11:06:31 -04:00
b52059b027
Ensure date/time format is maintained
...
Within the 'determine which is home' for loop, 'xx' is the midpoint of two datetime objects. When the midpoint is calculated to be midnight, only the date is returned. This can be replicated with:
mydates <- as.POSIXct("2018-01-01 00:00:00", tz = "UTC")
mydates
[1] "2018-01-01 UTC"
This results in 'hourofday' being NA as an hour cannot be found. By adding the suggested format wrapper, the time is maintained and 'hourofday' can be determined. It can then successfully be applied to the embedded if-statement within the loop.
mydates <- format(as.POSIXct("2018-01-01 00:00:00", tz = "UTC"), "%Y-%m-%d %H:%M:%S")
mydates
[1] "01-01-2018 00:00:00"
2021-07-23 10:12:11 -04:00
5a465873c4
Tested fitbit heartrate intraday feature
2021-07-21 10:24:02 -04:00
6fa1875bf3
Add app foreground episode count
2021-07-01 16:20:16 -04:00
bc5c0c9a4f
Fix app episode length bug
2021-07-01 16:20:16 -04:00
065a926a87
Change own to custom categories name
2021-07-01 16:20:16 -04:00
e74c745f86
Add own categories to app foreground features
2021-07-01 16:20:16 -04:00
5892b6d838
Fix create_participants_files.R to handle numeric PIDs
2021-07-01 16:20:16 -04:00
97ef8a8368
Set color range and avoid SettingWithCopyWarning
2021-06-29 09:50:19 -04:00
1c57320ab3
Update segment labels and fix the bug when we do not have any labels for event segments
2021-06-29 09:49:24 -04:00
cefcb0635b
Update heatmap of recorded phone sensors
2021-06-29 09:49:24 -04:00
bc06477d89
Update heatmap of sensor row count
2021-06-29 09:49:24 -04:00
e98a8ff7ca
Update histogram of phone data yield
2021-06-29 09:49:24 -04:00
f436f1f530
Update heatmap of correlation matrix
2021-06-29 09:49:23 -04:00
4d37696158
Update heatmaps of overall data yield
2021-06-29 09:48:30 -04:00
f374c67bd5
Bug fixed: Added unknown activity case
2021-06-23 19:04:55 -04:00
3e4d167adc
Bug fixed: sort bt_address alphabetically before picking the most frequent bt_address
2021-06-22 17:40:00 -04:00
f248b6c97d
Fix bugs of Fitbit mutation scripts
2021-06-11 18:18:33 -04:00
4b8698a4c6
adding app_episode with the changes as mentioned in the comments
2021-06-10 14:17:56 -04:00
65d5cb7bd4
Bug fixed: countscansmostuniquedevice stays the same for all time segments
2021-06-10 10:49:22 -04:00
e123a14082
Improve aware_csv msg when CSV files don't exist
2021-06-01 10:57:17 -04:00
9687081fbe
Refactor the rule phone_locations_add_doryab_extra_columns
2021-05-28 09:48:36 -04:00
0d6f51be8b
Refactor location features from Doryab provider & add a new strategy to infer home location & fix bugs
2021-05-26 17:36:52 -04:00
32472461ec
- Fix bug when no phone data yield is needed to process location data
...
- Remove location rows with the same timestamp based on their accuracy
2021-05-26 14:04:29 -04:00
9b21196f35
Fixed `expected_minutes` to account for different time segments. ( #136 )
2021-05-26 11:44:48 -04:00
edf71e055d
Add the EXCLUDE_SLEEP module for steps intraday features
2021-05-21 15:23:21 -04:00
5e451f99b0
Added phone keyboard features including docs/tests ( #134 )
2021-05-21 11:45:27 -04:00
e9cd9c94d7
Fix PID matching when joining data from participants
2021-05-11 16:49:04 -04:00
32818a4802
Fix parse of pids with more than 1 devices
2021-05-11 16:42:20 -04:00
809845143f
Test & fix bugs of sleep intraday features
2021-04-27 14:40:14 -04:00
7c7f34ec45
Test & fix bugs of sleep summary features
2021-04-27 14:40:14 -04:00
50fe09cfac
Update data streams mutation of fitbit data
2021-04-27 14:40:14 -04:00
66d9a9d640
Update params & docs of sleep features
2021-04-27 14:33:19 -04:00
4beafd233d
Fix crash when scraping data for an app that does not exist
2021-04-22 14:28:52 -04:00
ea8094e028
Fix length of periodic segments on days with DLS
2021-04-22 11:32:10 -04:00
9c56422529
Add calories intraday features
2021-04-20 12:00:38 -04:00
00a3335623
Add device_id column for sleep intraday episodes
2021-04-08 11:21:28 -04:00
286d317af4
Fix crash when there are no periodic segments to assign
...
This includes a simplification of how periodic segments are computed based on all local dates in the data independently of their time zones
2021-04-07 12:03:25 -04:00
9551669d47
Fix periodic segments bug when there are no segments to assign
2021-04-06 20:29:30 -04:00
78173c54ab
Convert date time object to string in assign_tz_code() function
2021-04-06 23:28:53 +00:00
1025e6d9d8
Fix datetime labels of event segments across multiple tzs
2021-04-06 13:58:58 -04:00
8909876cff
Add local_segment column for phone data yield features
2021-04-05 21:13:36 +00:00
68125dc1bf
Fix the bug of phone data yield features when the input is empty
2021-04-05 20:57:05 +00:00
46f5e24814
Fix Fitbit tz inference from phone data
2021-04-05 11:51:57 -04:00
636b64c61a
Revert "Added more keyboard features."
...
This reverts commit 94c72e3172
2021-04-05 11:25:00 -04:00
68e12a2563
Fix bugs of bluetooth feature extraction when number of unique bt_address is 2
2021-04-05 14:09:50 +00:00
8414977331
Fix the bug of utils.py when one participant have multi timezones
2021-04-03 21:20:10 -04:00
94c72e3172
Added more keyboard features.
2021-04-01 20:54:13 -04:00
1ea5b74eff
Fix the bug of utils.py when one participant have multi timezones
2021-03-31 19:34:14 -04:00
136dfef56b
Fix the bug of Analysis Workflows while parsing targets with updated segments
2021-03-30 16:41:50 -04:00
99dae079d5
Add iOS BT and Wifi visible to formats for old devices
2021-03-30 15:32:50 -04:00
30ad3cd586
Validate participant files without device ids
2021-03-28 15:29:08 -04:00
87fbbbe402
Refactor and simplify time segments
2021-03-28 15:29:07 -04:00
c48c1c8f24
Optimize Barnett's computation multi-day segments
2021-03-28 15:29:07 -04:00
d0858f8833
Fix overlapping periodic time segments
2021-03-28 15:29:07 -04:00
e177aa6386
Update create participant files section
2021-03-25 11:39:31 -04:00
a5eb535126
Update visualizations docs & add time flag for heatmap of overall data yield
2021-03-23 21:40:55 -04:00
cfc5039918
Fixed the empty dataframe case in infer_home_locations.py and added array condition in doryab location
2021-03-19 11:15:57 -04:00
294d84277d
Fix bug of sleep intraday PRICE provider when the dataframe of a segment is empty
2021-03-17 15:21:31 -04:00
771c14a928
Improve mysql containers error messages
2021-03-16 20:02:44 -04:00
6e234f7951
Fix warn instead of stop when there are not device ids
2021-03-16 20:02:16 -04:00
4c2f60fffd
Fix bugs in readable datetime and screen episodes
2021-03-16 20:01:43 -04:00
bb737237d0
Fixes for aware_influxdb
2021-03-16 11:26:46 -04:00
4528ab3641
Replace SRC LANGUAGE and FOLDER with SCRIPT
2021-03-14 22:14:13 -04:00
f4b2bd1fb2
Cleanup data/
2021-03-14 13:36:22 -04:00
42cee67664
Add aware_influxdb in beta
2021-03-14 13:33:43 -04:00
3d4a04effe
Refactor testing
2021-03-14 00:09:08 -05:00
2b6447105a
Migrate analysis example to new data stream
2021-03-12 19:52:34 -05:00
fae0c2ac05
Swap TABLE for CONTAINER
2021-03-12 18:14:49 -05:00
d529490999
Migrate fitbit features to new data stream
2021-03-12 12:38:36 -05:00
6e898beca5
Add aware_csv
2021-03-11 19:32:11 -05:00
13174b0c2a
Fix a bug when fitbit data is empty
2021-03-11 14:51:16 -05:00
2ee45995f2
Update config docs and create participant files script
2021-03-11 14:40:33 -05:00
1e66dad838
Fix bug in empatica_zip container script
2021-03-11 14:39:26 -05:00
a79997e0ac
Add empatica_zip docs
2021-03-11 14:39:26 -05:00
d48194fc07
Add fitbitparsed_csv
2021-03-11 14:39:26 -05:00
b97b70e3a1
Add fitbitjson_csv
2021-03-11 14:39:26 -05:00
470f4276af
Add fitbitparsed_mysql
2021-03-11 14:39:26 -05:00
1b0ee4bbf0
Add sleep intraday to fitbitjson_mysql
2021-03-11 14:39:20 -05:00
a420f5ef92
Add sleep summary to fitbitjson_mysql
2021-03-11 14:37:22 -05:00
93baff9f83
Migrate phone keyboard sensor to new data stream
2021-03-11 14:37:22 -05:00
35968e2fd0
Migrate phone log sensor to new data stream
2021-03-11 14:37:22 -05:00
091f9c048a
Migrate phone apps notifications sensor to new data stream
2021-03-11 14:37:22 -05:00
b49dab0949
Migrate phone apps crashes sensor to new data stream
2021-03-11 14:37:22 -05:00
47e1b33816
Add hr intraday to fitbitjson_mysql
2021-03-11 14:37:22 -05:00
47f449555a
Add hr summary to fitbitjson_mysql
2021-03-11 14:37:22 -05:00
72f6b2d621
Add steps intraday to fitbitjson_mysql
2021-03-11 14:37:22 -05:00
9a276c1c66
Add steps summary to jsonfitbit_mysql
2021-03-11 14:37:22 -05:00
13290cd444
Migrate phone wifi visible sensor to new data stream
2021-03-11 14:36:52 -05:00
d42c6e9c91
Migrate phone wifi connected sensor to new data stream
2021-03-11 14:35:34 -05:00
4825962361
Migrate phone screen sensor to new data stream
2021-03-11 14:35:34 -05:00
bd4f647d37
Migrate phone messages sensor to new data stream
2021-03-11 14:35:34 -05:00
3a65b3864d
Migrate phone locations sensor to new data stream
2021-03-11 14:35:34 -05:00
7a50a52a9d
Migrate phone light sensor to new data stream
2021-03-11 14:35:34 -05:00
c1682d8cd3
Migrate phone calls sensor to new data stream
2021-03-11 14:35:34 -05:00
0e96f39599
Migrate phone bluetooth sensor to new data stream
2021-03-11 14:35:34 -05:00
6d06d2b1eb
Update MUTATION structure code & docs for AR, Apps foreground, and battery sensors
2021-03-11 14:35:34 -05:00
6970954358
Change MUTATION structure
2021-03-11 14:35:34 -05:00
58ef276179
Add stream_parameters arg to phone and empatica mutation scripts
2021-03-11 14:35:34 -05:00
1063b4ca65
Add steps summary to fitbitjson_mysql
2021-03-11 14:35:34 -05:00
f7cf316133
Migrate phone battery sensor to new data stream
2021-03-11 14:35:34 -05:00
824523e32c
Migrate phone apps foreground sensor to new data stream
2021-03-11 14:35:34 -05:00
6b13c80e40
Fix bug while checking OS in stream_schema
2021-03-11 14:35:34 -05:00
8c79cfc56f
Fix OS specific dependcies
2021-03-11 14:35:34 -05:00
7372fca0dd
Fix bug when sensor is not available for an OS
2021-03-11 14:35:34 -05:00
41711fcdb7
Rename download_data add support for py containers
2021-03-11 14:35:34 -05:00
4b33ee43ba
Replace .env with credentials.yaml
2021-03-11 14:35:34 -05:00
f65e3c8b1a
Migrate empatica sensors to new data stream
2021-03-11 14:35:34 -05:00
2eae84ff05
Add docs of AR & conversation sensors
2021-03-11 14:35:34 -05:00
fb054b539f
Add support for multiple time zones
2021-03-11 14:35:34 -05:00
6c51c6c239
Move phone_conversation to aware_mysql stream
2021-03-11 14:35:34 -05:00
ab1a3dbf79
Move phone_activity_recognition to aware_mysql stream
2021-03-11 14:35:34 -05:00
dc11cb593d
Add support for smartphone sources and schemas.
...
Initial support for accelerometer
Update docs for automatically create participants
Update docs for initial multiple time zones
2021-03-11 14:35:33 -05:00
aac87311e8
Update socialjetlag feature of sleep intraday: replace bedtime with midpoint sleep
2021-03-04 15:49:25 -05:00
7b4598357d
Update the PRICE provider's example in sleep intraday docs
2021-02-26 17:47:01 -05:00
716ff3c592
Fix PRICE provider's bug when input is an empty dataframe
2021-02-26 17:47:01 -05:00
d74196cab4
Add sleep intraday features with PRICE provider
2021-02-26 17:47:01 -05:00
bedf7106e5
Fix sleep episodes bug
2021-02-26 17:47:01 -05:00
8377c12efb
Add sleep intraday features with RAPIDS provider
2021-02-26 17:47:01 -05:00
724027e383
Small fixes to timeathome docs, add config validation
2021-02-24 17:49:22 -05:00
3d6caea6c4
Added the timeathome feature using infer_home_location.py as interim file.
2021-02-24 16:57:25 -05:00
9668dfac7a
Update docs to support Empatica
2021-02-21 17:32:41 -05:00
faefca8b9a
Fix extra index column when dataset is empty
2021-02-21 17:32:41 -05:00
2e46f56111
Empatica zips must be placed in pid folder and small fixes
2021-02-21 17:32:41 -05:00
a26a44819a
Add stats features for empatica bvp, eda, ibi, temp
...
Fix Snakefile file indentention
2021-02-21 17:32:41 -05:00
4469cfd6bb
add stats features for bvp, eda, ibi, temp
2021-02-21 17:32:41 -05:00
c6dc7e675a
Add stats features for empatica heartrate
...
Turn off all empatica compute features
2021-02-21 17:32:41 -05:00
3bb0230bac
Add statistic features for empatica accelerometer
2021-02-21 17:32:41 -05:00
5f5f19866f
implement extract_empatica_data script
...
add support for all data types
Fix name comparison of zipped files
2021-02-21 17:32:41 -05:00
4b9857562b
Add support for zip input files
2021-02-21 17:32:41 -05:00
8c726f5d4f
Start empatica support
2021-02-21 17:32:41 -05:00
d8813e2d04
Fix bug when any of the rows from any sensor do not belong a time segment
2021-02-09 14:51:54 -05:00
b67f990816
Add new `FITBIT_DATA_YIELD` `RAPIDS` provider
2021-02-02 18:30:21 -05:00
9b248c449d
Fixing and adding MAXIMUM_ROW_DURATION.
2021-02-02 11:38:13 -05:00
e7fc8f44f2
Removing Sampling Frequency and fixing ROG, location entropy and normalized location entropy.
2021-02-02 11:38:08 -05:00
0bbf15f52e
Fixed the features dependent on time duration.
2021-02-02 11:36:14 -05:00
3d0d062491
Fix HR summary bug: do not consider rows with restinghr=0
2021-02-01 17:29:30 -05:00
f83c5a585e
Fix HR intraday bug: minutesonZONE features are 0
2021-02-01 13:57:12 -05:00
aefc794274
Fix location processing when certain columns don't exist
2021-02-01 11:49:22 -05:00
dbf57f43f3
Updated battery feature and its testing for periodic
2021-01-31 20:24:43 -05:00
25a3492eba
Drop rows without "assigned_segments" column before feature extraction
2021-01-21 19:41:17 -05:00
797de54b34
Fix merge bug of fetch_provider_features() function
2021-01-21 14:58:31 -05:00
5f60aac5c8
Fix KeyError bug of parsing steps data
2021-01-20 11:26:28 -05:00
d0fe4d4c28
Add ALL_RESAMPLED flag and accuracy limit
2021-01-14 14:34:25 -05:00
38fadbf202
Feature/doryab location clustering ( #111 )
...
* Added OPTICS - lightweight clustering algorithm.
* Changed the error message for inconsistent parameters in CONFIG
* Removing hardcoded values and changing default EPS value in the clustering algorithm.
* Added Observation in Doryab Feature docs.
Co-authored-by: nikunjgoel95 <nikunjgoel2009@gmail.com>
2021-01-14 14:22:51 -05:00
b7ba3c6407
Feature/location doryab fix ( #109 )
...
* Fixing the doryab location features for context of clustering.
* Fixed the wrong shifting while calculating the distance.
* Refractoring the haversine function
* Removed comments.
* Cleaning parts of the code.
* Updated the documentation for CLUSTER_ON parameter.
Co-authored-by: nikunjgoel95 <nikunjgoel2009@gmail.com>
2021-01-07 16:20:46 -05:00
4926497ae2
Fix bugs in Fitbit data parsing
...
- Fix the script that was breaking with an empty file
- Fix the script that was breaking when start/end dates were empty
- Ambigous and nonexistent DST times are handled now
- Remove unnecessary else clause
2021-01-06 11:43:01 -05:00
5203aa60d1
Fix bugs in create participants files script
...
- The PHONE and FITBIT flags were mixed up
- The start/end dates from the CSV file weren't being parsed correctly
2021-01-06 11:14:15 -05:00
3a80f93771
Fix segment error when device ids is empty
2021-01-06 11:12:10 -05:00
51c7739bfc
Revert "Start support for phone_keyboard"
...
This reverts commit dd95b4f941
2020-12-20 16:29:40 -05:00
dd95b4f941
Start support for phone_keyboard
2020-12-14 13:42:22 -05:00
63202c62cc
Fix BT doryab bug and update change log
2020-12-12 17:11:48 -05:00
05627296f4
Fix filter_data_by_segment bug
2020-12-12 17:10:59 -05:00
5bd1bfe856
Add new bluetooth doryab features and deprecate rapids provider
2020-12-12 17:01:46 -05:00
266dd28d02
Add bluetooth doryab features
2020-12-11 12:03:22 -05:00
ae21c076cd
Fix conversation data bug
2020-12-04 15:23:08 -05:00
9a02e93b25
Swap RMySQL for RMariaDB
2020-12-04 14:57:13 -05:00
9a0e57301b
Update 4 plots for time_segments
2020-12-03 21:00:32 -05:00
614e759551
Refactor day segments to time segments
2020-12-02 18:41:03 -05:00
c24cba8679
Event segments' device id can be one of any assigned to a participant
2020-12-02 17:17:55 -05:00
0bd43c139c
Update plot#1: histogram_phone_data_yield
2020-12-01 14:21:33 -05:00
016bdbfe8c
Update Python feature scripts to add sensor and provider names automatically
2020-11-30 14:42:19 -05:00
70991d6667
Add dates filter while parsing fitbit data
2020-11-30 12:34:14 -05:00
5178be585d
Rename modeling.py to modelling.py & Update example_config.yaml
2020-11-25 22:35:38 -05:00
5f51c94ac6
Separate device data configuration and update docs
2020-11-25 19:42:11 -05:00
b4a512faf3
Add analysis example workflow
2020-11-25 16:34:05 -05:00
ced3305ddb
Update R feature scripts to add sensor and provider names automatically
2020-11-25 14:49:42 -05:00
f02ca2624d
Add phone yield sensor
2020-11-24 19:12:16 -05:00
555811211d
Fix column check in day segment compute
2020-11-24 14:51:06 -05:00
767e838c87
Fix HR&Sleep bugs
2020-11-24 13:43:24 -05:00
deba6b9e4f
Update sleep summary features for segments
2020-11-23 12:01:00 -05:00
10384204a1
Update Fitbit HR feature name
2020-11-20 11:31:16 -05:00
d3241c79f1
Update filter_data_by_segment() function: call chunk_episodes() inside the filter function
2020-11-19 17:27:53 -05:00
895b70719e
Update minimal example and fix create participants bug
2020-11-17 17:27:48 -05:00
a71efd6b85
Split FITBIT_STEPS into FITBIT_STEPS_SUMMARY and FITBIT_STEPS_INTRADAY
2020-11-11 21:16:48 -05:00
9fc36f67e2
Split FITBIT_HEARTRATE into FITBIT_HEARTRATE_SUMMARY and FITBIT_HEARTRATE_INTRADAY
2020-11-11 17:27:46 -05:00
39632c66f5
Update steps features for segments
2020-10-30 18:57:04 -04:00
7f7eac0769
Merge branch 'ds2' into day_segments
2020-10-28 17:53:00 -04:00
c5197b6c5f
Merge branch 'master' into ds2
2020-10-28 17:50:37 -04:00
93055dacee
Update HR main.py
2020-10-28 17:02:21 -04:00
f36a7052b3
Add config update file to create participants file
2020-10-28 14:11:41 -04:00
84068ab33c
Update fitbit_parse_steps.py: add summary data
2020-10-28 13:07:35 -04:00
ed7585c2bf
Update heartrate features for segments
2020-10-27 17:55:57 -04:00
d5931c75d8
Refactor script to create participants files
2020-10-27 17:13:16 -04:00
25e1f1fbb5
Update Python chunk_episodes
2020-10-26 18:47:57 -04:00
b986599226
Add support to read fitbit data from csv files
2020-10-26 17:17:53 -04:00
f0a551164a
Fix AR names
2020-10-26 15:44:21 -04:00
e10e030baf
Unify AR iOS walking and running conversion
2020-10-26 15:44:09 -04:00
234588c94d
Expose episode threshold between rows parameter
2020-10-26 15:28:19 -04:00
2fe5a0d822
Update R chunk_episodes
2020-10-26 15:10:32 -04:00
c78ccfced7
Remove microseconds from chuncked date times
2020-10-26 15:03:31 -04:00
2b7bd0ae6e
Modify output format of chunk_episodes() function
2020-10-26 13:00:53 -04:00
1d04aa6807
Add check for non-overlapping event day segments
2020-10-23 12:15:26 -04:00
86509207ac
Turn off warnings for tidyverse and dplyr
2020-10-23 10:41:00 -04:00
JulioV
Meng Li
Weiyu
Shirley
Hannah Roberts
kirtirajk
Nikunj Goel
Joe Kim
Juseong Kim
abhineethreddyk