Phone Data Quality¶
-Phone Valid Sensed Bins¶
-A valid bin is any period of BIN_SIZE minutes starting from midnight with at least 1 row from any phone sensor. PHONE_VALID_SENSED_BINS are used to compute PHONE_VALID_SENSED_DAYS, to resample fused location data and to compute some screen features.
Phone Data Yield¶
+This is a combinatorial sensor which means that we use the data from multiple sensors to extract data yield features. Data yield features can be used to remove rows (day segments) that do not contain enough data. You should decide what is your “enough” threshold depending on the type of sensors you collected (frequency vs event based, e.g. acceleroemter vs calls), the length of your study, and the rates of missing data that your analysis could handle.
-
-Hint
-PHONE_VALID_SENSED_DAYS are an approximation to the time the phone was sensing data so add as many sensors as you have to [PHONE_VALID_SENSED_BINS][PHONE_SENSORS]
Why is data yield important?
+Imagine that you want to extract PHONE_CALL features on daily segments (00:00 to 23:59). Let’s say that on day 1 the phone logged 10 calls and 23 hours of data from other sensors and on day 2 the phone logged 10 calls and only 2 hours of data from other sensors. It’s more likely that other calls were placed on the 22 hours of data that you didn’t log on day 2 than on the 1 hour of data you didn’t log on day 1, and so including day 2 in your analysis could bias your results.
Parameters description for PHONE_VALID_SENSED_BINS:
Sensor parameters description for [PHONE_DATA_YIELD]:
| Key | +Description | +
|---|---|
[SENSORS] |
+One or more phone sensor config keys (e.g. PHONE_MESSAGE). The more keys you include the more accurately RAPIDS can approximate the time an smartphone was sensing data. The supported phone sensors you can include in this list are outlined below (do NOT include Fitbit sensors). |
+
+
+
+Supported phone sensors for [PHONE_DATA_YIELD][SENSORS]
PHONE_ACCELEROMETER
+PHONE_ACTIVITY_RECOGNITION
+PHONE_APPLICATIONS_FOREGROUND
+PHONE_BATTERY
+PHONE_BLUETOOTH
+PHONE_CALLS
+PHONE_CONVERSATION
+PHONE_MESSAGES
+PHONE_LIGHT
+PHONE_LOCATIONS
+PHONE_SCREEN
+PHONE_WIFI_VISIBLE
+PHONE_WIFI_CONNECTED
+RAPIDS provider¶
+Before explaining the data yield features, let’s define the following relevant concepts:
+-
+
- A valid minute is any 60 second window when any phone sensor logged at least 1 row of data +
- A valid hour is any 60 minute window with at least X valid minutes. The X or threshold is given by
[MINUTE_RATIO_THRESHOLD_FOR_VALID_YIELDED_HOURS]
+
The timestamps of all sensors are concatenated and then grouped per day segment. Minute and hour windows are created from the beginning of each day segment instance and these windows are marked as valid based on the definitions above. The duration of each day segment is taken into account to compute the features described below.
+
+
+Available day segments and platforms
+-
+
- Available for all day segments +
- Available for Android and iOS +
+
+
+File Sequence
+- data/raw/{pid}/{sensor}_raw.csv # one for every [PHONE_DATA_YIELD][SENSORS]
+- data/interim/{pid}/phone_yielded_timestamps.csv
+- data/interim/{pid}/phone_yielded_timestamps_with_datetime.csv
+- data/interim/{pid}/phone_data_yield_features/phone_data_yield_{language}_{provider_key}.csv
+- data/processed/features/{pid}/phone_data_yield.csv
+Parameters description for [PHONE_DATA_YIELD][PROVIDERS][RAPIDS]:
[COMPUTE] |
-Set to True to compute |
+Set to True to extract PHONE_DATA_YIELD features from the RAPIDS provider |
|
[BIN_SIZE] |
-Size of each bin in minutes | +[FEATURES] |
+Features to be computed, see table below |
[PHONE_SENSORS] |
-One or more sensor config keys (e.g. PHONE_MESSAGE) to be used to flag a bin as valid or not (whether or not a bin contains at least one row from any sensor) |
+[MINUTE_RATIO_THRESHOLD_FOR_VALID_YIELDED_HOURS] |
+The proportion [0.0 ,1.0] of valid minutes in a 60-minute window necessary to flag that window as valid. |
-
-
-Possible values for [PHONE_VALID_SENSED_BINS][PHONE_SENSORS]
PHONE_MESSAGES
-PHONE_CALLS
-PHONE_LOCATIONS
-PHONE_BLUETOOTH
-PHONE_ACTIVITY_RECOGNITION
-PHONE_BATTERY
-PHONE_SCREEN
-PHONE_LIGHT
-PHONE_ACCELEROMETER
-PHONE_APPLICATIONS_FOREGROUND
-PHONE_WIFI_VISIBLE
-PHONE_WIFI_CONNECTED
-PHONE_CONVERSATION
-Phone Valid Sensed Days¶
-On any given day, a phone could have sensed data only for a few minutes or for 24 hours. Features should considered more reliable the more hours the phone was logging data, for example, 10 calls logged on a day when only 1 hour of data was recorded is a less reliable feature compared to 10 calls on a day when 23 hours of data were recorded.
-Therefore, we define a valid hour as those that contain a minimum number of valid bins (see above). We mark an hour as valid when contains at least MIN_VALID_BINS_PER_HOUR (out of 60min/BIN_SIZE bins). In turn, we mark a day as valid if it has at least MIN_VALID_HOURS_PER_DAY. You can use PHONE_VALID_SENSED_DAYS to manually discard days when not enough data was collected after your features are computed.
Parameters description for PHONE_VALID_SENSED_DAYS:
Features description for [PHONE_DATA_YIELD][PROVIDERS][RAPIDS]:
| Key | +Feature | +Units | Description | |
|---|---|---|---|---|
[COMPUTE] |
-Set to True to compute |
+ratiovalidyieldedminutes | +rows | +The ratio between the number of valid minutes and the duration in minutes of a day segment. |
[MIN_VALID_BINS_PER_HOUR] |
-An array of integer values, 6 by default. Minimum number of valid bins to mark an hour as valid | -|||
[MIN_VALID_HOURS_PER_DAY] |
-An array of integer values, 16 by default. Minimum number of valid hours to mark a day as valid | +ratiovalidyieldedhours | +lux | +The ratio between the number of valid hours and the duration in hours of a day segment. If the day segment is shorter than 1 hour this feature will always be 1. |
+
diff --git a/0.1/features/phone-light/index.html b/0.1/features/phone-light/index.html
index c139d08d..385fe58b 100644
--- a/0.1/features/phone-light/index.html
+++ b/0.1/features/phone-light/index.html
@@ -412,8 +412,8 @@
Assumptions/Observations
+-
+
-
+
We recommend using
+ratiovalidyieldedminuteson day segments that are shorter than two or three hours andratiovalidyieldedhoursfor longer segments. This is because relying on yielded minutes only can be misleading when a big chunk of those missing minutes are clustered together.For example, let’s assume we are working with a 24-hour day segment that is missing 12 hours of data. Two extreme cases can occur:
+-
+
- the 12 missing hours are from the beginning of the segment or +
- 30 minutes could be missing from every hour (24 * 30 minutes = 12 hours). +
+ratiovalidyieldedminuteswould be 0.5 for bothaandb(hinting the missing circumstances are similar). However,ratiovalidyieldedhourswould be 0.5 foraand 1.0 forbif[MINUTE_RATIO_THRESHOLD_FOR_VALID_YIELDED_HOURS]is between [0.0 and 0.49] (hinting that the missing circumstances might be more favorable forb. In other words, sensed data forbis more evenly spread compared toa.
+
-
+
Skip to content
@@ -95,7 +95,7 @@
- Phone Data Quality
+ Phone Data Yield
@@ -421,12 +421,12 @@
- Phone Data Quality
+ Phone Data Yield
@@ -443,15 +443,8 @@
- - - Phone Valid Sensed Bins - - - - -
- - - Phone Valid Sensed Days + + RAPIDS provider @@ -910,15 +903,8 @@
- - - Phone Valid Sensed Bins - - - - -
- - - Phone Valid Sensed Days + + RAPIDS provider @@ -935,19 +921,73 @@
- A valid minute is any 60 second window when any phone sensor logged at least 1 row of data +
- A valid hour is any 60 minute window with at least X valid minutes. The X or threshold is given by
[MINUTE_RATIO_THRESHOLD_FOR_VALID_YIELDED_HOURS]
+ - Available for all day segments +
- Available for Android and iOS +
-
+
We recommend using
+ratiovalidyieldedminuteson day segments that are shorter than two or three hours andratiovalidyieldedhoursfor longer segments. This is because relying on yielded minutes only can be misleading when a big chunk of those missing minutes are clustered together.For example, let’s assume we are working with a 24-hour day segment that is missing 12 hours of data. Two extreme cases can occur:
+-
+
- the 12 missing hours are from the beginning of the segment or +
- 30 minutes could be missing from every hour (24 * 30 minutes = 12 hours). +
+ratiovalidyieldedminuteswould be 0.5 for bothaandb(hinting the missing circumstances are similar). However,ratiovalidyieldedhourswould be 0.5 foraand 1.0 forbif[MINUTE_RATIO_THRESHOLD_FOR_VALID_YIELDED_HOURS]is between [0.0 and 0.49] (hinting that the missing circumstances might be more favorable forb. In other words, sensed data forbis more evenly spread compared toa.
+ - - - Phone Data Quality + + Phone Data Yield diff --git a/latest/features/phone-locations/index.html b/latest/features/phone-locations/index.html index 67cd0d4c..0edf307d 100644 --- a/latest/features/phone-locations/index.html +++ b/latest/features/phone-locations/index.html @@ -412,8 +412,8 @@
- - - Phone Data Quality + + Phone Data Yield diff --git a/latest/features/phone-messages/index.html b/latest/features/phone-messages/index.html index 2f256649..29b365e1 100644 --- a/latest/features/phone-messages/index.html +++ b/latest/features/phone-messages/index.html @@ -412,8 +412,8 @@
- - - Phone Data Quality + + Phone Data Yield diff --git a/latest/features/phone-screen/index.html b/latest/features/phone-screen/index.html index ac3c4db5..6ee49a2d 100644 --- a/latest/features/phone-screen/index.html +++ b/latest/features/phone-screen/index.html @@ -412,8 +412,8 @@
- - - Phone Data Quality + + Phone Data Yield diff --git a/latest/features/phone-wifi-connected/index.html b/latest/features/phone-wifi-connected/index.html index 8a3da383..c8b0626b 100644 --- a/latest/features/phone-wifi-connected/index.html +++ b/latest/features/phone-wifi-connected/index.html @@ -412,8 +412,8 @@
- - - Phone Data Quality + + Phone Data Yield diff --git a/latest/features/phone-wifi-visible/index.html b/latest/features/phone-wifi-visible/index.html index bd3708b8..1fff189e 100644 --- a/latest/features/phone-wifi-visible/index.html +++ b/latest/features/phone-wifi-visible/index.html @@ -412,8 +412,8 @@
- - - Phone Data Quality + + Phone Data Yield diff --git a/latest/file-structure/index.html b/latest/file-structure/index.html index 1628b4d1..55bb8bb7 100644 --- a/latest/file-structure/index.html +++ b/latest/file-structure/index.html @@ -419,8 +419,8 @@
- - - Phone Data Quality + + Phone Data Yield diff --git a/latest/index.html b/latest/index.html index 88b36b62..53bbca05 100644 --- a/latest/index.html +++ b/latest/index.html @@ -460,8 +460,8 @@
- - - Phone Data Quality + + Phone Data Yield diff --git a/latest/search/search_index.json b/latest/search/search_index.json index fde82d31..4ea44349 100644 --- a/latest/search/search_index.json +++ b/latest/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Welcome to RAPIDS documentation \u00b6 Warning The functionality outlined in these docs is implemented in the branch day_segments which we will merge to master soon as release 0.1 . The previous (first) release of RAPIDS along with the old docs will be labeled beta . If you landed on this page feel free to look around, just have in mind that we are polishing the last rough patches before we advertise 0.1 (Nov 16, 2020) Reproducible Analysis Pipeline for Data Streams (RAPIDS) allows you to process smartphone and wearable data to extract behavioral features (a.k.a. digital biomarkers/phenotypes). RAPIDS is open source, documented, modular, tested, and reproducible. At the moment we support smartphone data collected with AWARE and wearable data from Fitbit devices. Questions or feedback can be posted on #rapids in AWARE Framework's slack . Bugs should be reported on Github . Join our discussions on our algorithms and assumptions for feature processing . Ready to start? Go to Installation and then to Initial Configuration How does it work? \u00b6 RAPIDS is formed by R and Python scripts orchestrated by Snakemake . We suggest you read Snakemake\u2019s docs but in short: every link in the analysis chain is atomic and has files as input and output. Behavioral features are processed per sensor and per participant. What are the benefits of using RAPIDS? \u00b6 Consistent analysis . Every participant sensor dataset is analyzed in the exact same way and isolated from each other. Efficient analysis . Every analysis step is executed only once. Whenever your data or configuration changes only the affected files are updated. Parallel execution . Thanks to Snakemake, your analysis can be executed over multiple cores without changing your code. Extensible code . You can easily add your own behavioral features in R or Python and keep authorship and citations. Timezone aware . Your data is adjusted to the specified timezone (multiple timezones suport coming soon ). Flexible day segments . You can extract behavioral features on time windows of any length (e.g. 5 minutes, 3 hours, 2 days), on every day or particular days (e.g. weekends, Mondays, the 1 st of each month, etc.) or around events of interest (e.g. surveys or clinical relapses). Tested code . We are constantly adding tests to make sure our behavioral features are correct. Reproducible code . You can be sure your code will run in other computers as intended thanks to R and Python virtual environments. You can share your analysis code along your publications without any overhead. Private . All your data is processed locally. How is it organized? \u00b6 In broad terms the config.yaml , .env files, participant files, and day segment files are the only ones that you will have to modify. All data is stored in data/ and all scripts are stored in src/ . For more information see RAPIDS\u2019 File Structure .","title":"Home"},{"location":"#welcome-to-rapids-documentation","text":"Warning The functionality outlined in these docs is implemented in the branch day_segments which we will merge to master soon as release 0.1 . The previous (first) release of RAPIDS along with the old docs will be labeled beta . If you landed on this page feel free to look around, just have in mind that we are polishing the last rough patches before we advertise 0.1 (Nov 16, 2020) Reproducible Analysis Pipeline for Data Streams (RAPIDS) allows you to process smartphone and wearable data to extract behavioral features (a.k.a. digital biomarkers/phenotypes). RAPIDS is open source, documented, modular, tested, and reproducible. At the moment we support smartphone data collected with AWARE and wearable data from Fitbit devices. Questions or feedback can be posted on #rapids in AWARE Framework's slack . Bugs should be reported on Github . Join our discussions on our algorithms and assumptions for feature processing . Ready to start? Go to Installation and then to Initial Configuration","title":"Welcome to RAPIDS documentation"},{"location":"#how-does-it-work","text":"RAPIDS is formed by R and Python scripts orchestrated by Snakemake . We suggest you read Snakemake\u2019s docs but in short: every link in the analysis chain is atomic and has files as input and output. Behavioral features are processed per sensor and per participant.","title":"How does it work?"},{"location":"#what-are-the-benefits-of-using-rapids","text":"Consistent analysis . Every participant sensor dataset is analyzed in the exact same way and isolated from each other. Efficient analysis . Every analysis step is executed only once. Whenever your data or configuration changes only the affected files are updated. Parallel execution . Thanks to Snakemake, your analysis can be executed over multiple cores without changing your code. Extensible code . You can easily add your own behavioral features in R or Python and keep authorship and citations. Timezone aware . Your data is adjusted to the specified timezone (multiple timezones suport coming soon ). Flexible day segments . You can extract behavioral features on time windows of any length (e.g. 5 minutes, 3 hours, 2 days), on every day or particular days (e.g. weekends, Mondays, the 1 st of each month, etc.) or around events of interest (e.g. surveys or clinical relapses). Tested code . We are constantly adding tests to make sure our behavioral features are correct. Reproducible code . You can be sure your code will run in other computers as intended thanks to R and Python virtual environments. You can share your analysis code along your publications without any overhead. Private . All your data is processed locally.","title":"What are the benefits of using RAPIDS?"},{"location":"#how-is-it-organized","text":"In broad terms the config.yaml , .env files, participant files, and day segment files are the only ones that you will have to modify. All data is stored in data/ and all scripts are stored in src/ . For more information see RAPIDS\u2019 File Structure .","title":"How is it organized?"},{"location":"citation/","text":"Cite RAPIDS and providers \u00b6 RAPIDS and the community RAPIDS is a community effort and as such we want to continue recognizing the contributions from other researchers. Besides citing RAPIDS, we ask you to cite any of the authors listed below if you used those sensor providers in your analysis, thank you! RAPIDS \u00b6 If you used RAPIDS, please cite this paper . RAPIDS et al. citation Vega J, Li M, Aguillera K, Goel N, Joshi E, Durica KC, Kunta AR, Low CA RAPIDS: Reproducible Analysis Pipeline for Data Streams Collected with Mobile Devices JMIR Preprints. 18/08/2020:23246 DOI: 10.2196/preprints.23246 URL: https://preprints.jmir.org/preprint/23246 Panda (accelerometer) \u00b6 If you computed accelerometer features using the provider [PHONE_ACCLEROMETER][PANDA] cite this paper in addition to RAPIDS. Panda et al. citation Panda N, Solsky I, Huang EJ, Lipsitz S, Pradarelli JC, Delisle M, Cusack JC, Gadd MA, Lubitz CC, Mullen JT, Qadan M, Smith BL, Specht M, Stephen AE, Tanabe KK, Gawande AA, Onnela JP, Haynes AB. Using Smartphones to Capture Novel Recovery Metrics After Cancer Surgery. JAMA Surg. 2020 Feb 1;155(2):123-129. doi: 10.1001/jamasurg.2019.4702. PMID: 31657854; PMCID: PMC6820047. Stachl (applications foreground) \u00b6 If you computed applications foreground features using the app category (genre) catalogue in [PHONE_APPLICATIONS_FOREGROUND][RAPIDS] cite this paper in addition to RAPIDS. Stachl et al. citation Clemens Stachl, Quay Au, Ramona Schoedel, Samuel D. Gosling, Gabriella M. Harari, Daniel Buschek, Sarah Theres V\u00f6lkel, Tobias Schuwerk, Michelle Oldemeier, Theresa Ullmann, Heinrich Hussmann, Bernd Bischl, Markus B\u00fchner. Proceedings of the National Academy of Sciences Jul 2020, 117 (30) 17680-17687; DOI: 10.1073/pnas.1920484117 Barnett (locations) \u00b6 If you computed locations features using the provider [PHONE_LOCATIONS][BARNETT] cite this paper and this paper in addition to RAPIDS. Barnett et al. citation Ian Barnett, Jukka-Pekka Onnela, Inferring mobility measures from GPS traces with missing data, Biostatistics, Volume 21, Issue 2, April 2020, Pages e98\u2013e112, https://doi.org/10.1093/biostatistics/kxy059 Canzian et al. citation Luca Canzian and Mirco Musolesi. 2015. Trajectories of depression: unobtrusive monitoring of depressive states by means of smartphone mobility traces analysis. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp \u201815). Association for Computing Machinery, New York, NY, USA, 1293\u20131304. DOI: https://doi.org/10.1145/2750858.2805845 Doryab (locations) \u00b6 If you computed locations features using the provider [PHONE_LOCATIONS][DORYAB] cite this paper and this paper in addition to RAPIDS. Doryab et al. citation Doryab, A., Chikarsel, P., Liu, X., & Dey, A. K. (2019). Extraction of Behavioral Features from Smartphone and Wearable Data. ArXiv:1812.10394 [Cs, Stat]. http://arxiv.org/abs/1812.10394 Canzian et al. citation Luca Canzian and Mirco Musolesi. 2015. Trajectories of depression: unobtrusive monitoring of depressive states by means of smartphone mobility traces analysis. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp \u201815). Association for Computing Machinery, New York, NY, USA, 1293\u20131304. DOI: https://doi.org/10.1145/2750858.2805845","title":"Citation"},{"location":"citation/#cite-rapids-and-providers","text":"RAPIDS and the community RAPIDS is a community effort and as such we want to continue recognizing the contributions from other researchers. Besides citing RAPIDS, we ask you to cite any of the authors listed below if you used those sensor providers in your analysis, thank you!","title":"Cite RAPIDS and providers"},{"location":"citation/#rapids","text":"If you used RAPIDS, please cite this paper . RAPIDS et al. citation Vega J, Li M, Aguillera K, Goel N, Joshi E, Durica KC, Kunta AR, Low CA RAPIDS: Reproducible Analysis Pipeline for Data Streams Collected with Mobile Devices JMIR Preprints. 18/08/2020:23246 DOI: 10.2196/preprints.23246 URL: https://preprints.jmir.org/preprint/23246","title":"RAPIDS"},{"location":"citation/#panda-accelerometer","text":"If you computed accelerometer features using the provider [PHONE_ACCLEROMETER][PANDA] cite this paper in addition to RAPIDS. Panda et al. citation Panda N, Solsky I, Huang EJ, Lipsitz S, Pradarelli JC, Delisle M, Cusack JC, Gadd MA, Lubitz CC, Mullen JT, Qadan M, Smith BL, Specht M, Stephen AE, Tanabe KK, Gawande AA, Onnela JP, Haynes AB. Using Smartphones to Capture Novel Recovery Metrics After Cancer Surgery. JAMA Surg. 2020 Feb 1;155(2):123-129. doi: 10.1001/jamasurg.2019.4702. PMID: 31657854; PMCID: PMC6820047.","title":"Panda (accelerometer)"},{"location":"citation/#stachl-applications-foreground","text":"If you computed applications foreground features using the app category (genre) catalogue in [PHONE_APPLICATIONS_FOREGROUND][RAPIDS] cite this paper in addition to RAPIDS. Stachl et al. citation Clemens Stachl, Quay Au, Ramona Schoedel, Samuel D. Gosling, Gabriella M. Harari, Daniel Buschek, Sarah Theres V\u00f6lkel, Tobias Schuwerk, Michelle Oldemeier, Theresa Ullmann, Heinrich Hussmann, Bernd Bischl, Markus B\u00fchner. Proceedings of the National Academy of Sciences Jul 2020, 117 (30) 17680-17687; DOI: 10.1073/pnas.1920484117","title":"Stachl (applications foreground)"},{"location":"citation/#barnett-locations","text":"If you computed locations features using the provider [PHONE_LOCATIONS][BARNETT] cite this paper and this paper in addition to RAPIDS. Barnett et al. citation Ian Barnett, Jukka-Pekka Onnela, Inferring mobility measures from GPS traces with missing data, Biostatistics, Volume 21, Issue 2, April 2020, Pages e98\u2013e112, https://doi.org/10.1093/biostatistics/kxy059 Canzian et al. citation Luca Canzian and Mirco Musolesi. 2015. Trajectories of depression: unobtrusive monitoring of depressive states by means of smartphone mobility traces analysis. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp \u201815). Association for Computing Machinery, New York, NY, USA, 1293\u20131304. DOI: https://doi.org/10.1145/2750858.2805845","title":"Barnett (locations)"},{"location":"citation/#doryab-locations","text":"If you computed locations features using the provider [PHONE_LOCATIONS][DORYAB] cite this paper and this paper in addition to RAPIDS. Doryab et al. citation Doryab, A., Chikarsel, P., Liu, X., & Dey, A. K. (2019). Extraction of Behavioral Features from Smartphone and Wearable Data. ArXiv:1812.10394 [Cs, Stat]. http://arxiv.org/abs/1812.10394 Canzian et al. citation Luca Canzian and Mirco Musolesi. 2015. Trajectories of depression: unobtrusive monitoring of depressive states by means of smartphone mobility traces analysis. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp \u201815). Association for Computing Machinery, New York, NY, USA, 1293\u20131304. DOI: https://doi.org/10.1145/2750858.2805845","title":"Doryab (locations)"},{"location":"faq/","text":"Frequently Asked Questions \u00b6 Cannot connect to your MySQL server \u00b6 Problem **Error in .local ( drv, \\. .. ) :** **Failed to connect to database: Error: Can \\' t initialize character set unknown ( path: compiled \\_ in ) ** : Calls: dbConnect -> dbConnect -> .local -> .Call Execution halted [ Tue Mar 10 19 :40:15 2020 ] Error in rule download_dataset: jobid: 531 output: data/raw/p60/locations_raw.csv RuleException: CalledProcessError in line 20 of /home/ubuntu/rapids/rules/preprocessing.snakefile: Command 'set -euo pipefail; Rscript --vanilla /home/ubuntu/rapids/.snakemake/scripts/tmp_2jnvqs7.download_dataset.R' returned non-zero exit status 1 . File \"/home/ubuntu/rapids/rules/preprocessing.snakefile\" , line 20 , in __rule_download_dataset File \"/home/ubuntu/anaconda3/envs/moshi-env/lib/python3.7/concurrent/futures/thread.py\" , line 57 , in run Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Solution Please make sure the DATABASE_GROUP in config.yaml matches your DB credentials group in .env . Cannot start mysql in linux via brew services start mysql \u00b6 Problem Cannot start mysql in linux via brew services start mysql Solution Use mysql.server start Every time I run force the download_dataset rule all rules are executed \u00b6 Problem When running snakemake -j1 -R download_phone_data or ./rapids -j1 -R download_phone_data all the rules and files are re-computed Solution This is expected behavior. The advantage of using snakemake under the hood is that every time a file containing data is modified every rule that depends on that file will be re-executed to update their results. In this case, since download_dataset updates all the raw data, and you are forcing the rule with the flag -R every single rule that depends on those raw files will be executed. Error Table XXX doesn't exist while running the download_phone_data or download_fitbit_data rule. \u00b6 Problem Error in .local ( conn, statement, ... ) : could not run statement: Table 'db_name.table_name' doesn ' t exist Calls: colnames ... .local -> dbSendQuery -> dbSendQuery -> .local -> .Call Execution halted Solution Please make sure the sensors listed in [PHONE_VALID_SENSED_BINS][PHONE_SENSORS] and the [TABLE] of each sensor you activated in config.yaml match your database tables. How do I install RAPIDS on Ubuntu 16.04 \u00b6 Solution Install dependencies (Homebrew - if not installed): sudo apt-get install libmariadb-client-lgpl-dev libxml2-dev libssl-dev Install brew for linux and add the following line to ~/.bashrc : export PATH=$HOME/.linuxbrew/bin:$PATH source ~/.bashrc Install MySQL brew install mysql brew services start mysql Install R, pandoc and rmarkdown: brew install r brew install gcc@6 (needed due to this bug ) HOMEBREW_CC=gcc-6 brew install pandoc Install miniconda using these instructions Clone our repo: git clone https://github.com/carissalow/rapids Create a python virtual environment: cd rapids conda env create -f environment.yml -n MY_ENV_NAME conda activate MY_ENV_NAME Install R packages and virtual environment: snakemake renv_install snakemake renv_init snakemake renv_restore This step could take several minutes to complete. Please be patient and let it run until completion. mysql.h cannot be found \u00b6 Problem -------------------------- [ ERROR MESSAGE ] ----------------------------
- - - Phone Data Quality + + Phone Data Yield diff --git a/latest/setup/execution/index.html b/latest/setup/execution/index.html index 29e79be9..e511cfc7 100644 --- a/latest/setup/execution/index.html +++ b/latest/setup/execution/index.html @@ -419,8 +419,8 @@
- - - Phone Data Quality + + Phone Data Yield diff --git a/latest/setup/installation/index.html b/latest/setup/installation/index.html index 4ead2cf5..625caeaf 100644 --- a/latest/setup/installation/index.html +++ b/latest/setup/installation/index.html @@ -419,8 +419,8 @@
- - - Phone Data Quality + + Phone Data Yield diff --git a/latest/sitemap.xml.gz b/latest/sitemap.xml.gz index 3ed45ce1..f0c66fa4 100644 Binary files a/latest/sitemap.xml.gz and b/latest/sitemap.xml.gz differ diff --git a/latest/team/index.html b/latest/team/index.html index 72f3e5af..bfa41fe8 100644 --- a/latest/team/index.html +++ b/latest/team/index.html @@ -408,8 +408,8 @@
- - - Phone Data Quality + + Phone Data Yield diff --git a/latest/workflow-examples/minimal/index.html b/latest/workflow-examples/minimal/index.html index 287bb6de..1c96aa3e 100644 --- a/latest/workflow-examples/minimal/index.html +++ b/latest/workflow-examples/minimal/index.html @@ -419,8 +419,8 @@
- - - Phone Data Quality + + Phone Data Yield
Phone Data Quality¶
-Phone Valid Sensed Bins¶
-A valid bin is any period of BIN_SIZE minutes starting from midnight with at least 1 row from any phone sensor. PHONE_VALID_SENSED_BINS are used to compute PHONE_VALID_SENSED_DAYS, to resample fused location data and to compute some screen features.
Phone Data Yield¶
+This is a combinatorial sensor which means that we use the data from multiple sensors to extract data yield features. Data yield features can be used to remove rows (day segments) that do not contain enough data. You should decide what is your “enough” threshold depending on the type of sensors you collected (frequency vs event based, e.g. acceleroemter vs calls), the length of your study, and the rates of missing data that your analysis could handle.
-
-Hint
-PHONE_VALID_SENSED_DAYS are an approximation to the time the phone was sensing data so add as many sensors as you have to [PHONE_VALID_SENSED_BINS][PHONE_SENSORS]
Why is data yield important?
+Imagine that you want to extract PHONE_CALL features on daily segments (00:00 to 23:59). Let’s say that on day 1 the phone logged 10 calls and 23 hours of data from other sensors and on day 2 the phone logged 10 calls and only 2 hours of data from other sensors. It’s more likely that other calls were placed on the 22 hours of data that you didn’t log on day 2 than on the 1 hour of data you didn’t log on day 1, and so including day 2 in your analysis could bias your results.
Parameters description for PHONE_VALID_SENSED_BINS:
Sensor parameters description for [PHONE_DATA_YIELD]:
| Key | +Description | +
|---|---|
[SENSORS] |
+One or more phone sensor config keys (e.g. PHONE_MESSAGE). The more keys you include the more accurately RAPIDS can approximate the time an smartphone was sensing data. The supported phone sensors you can include in this list are outlined below (do NOT include Fitbit sensors). |
+
+
+
+Supported phone sensors for [PHONE_DATA_YIELD][SENSORS]
PHONE_ACCELEROMETER
+PHONE_ACTIVITY_RECOGNITION
+PHONE_APPLICATIONS_FOREGROUND
+PHONE_BATTERY
+PHONE_BLUETOOTH
+PHONE_CALLS
+PHONE_CONVERSATION
+PHONE_MESSAGES
+PHONE_LIGHT
+PHONE_LOCATIONS
+PHONE_SCREEN
+PHONE_WIFI_VISIBLE
+PHONE_WIFI_CONNECTED
+RAPIDS provider¶
+Before explaining the data yield features, let’s define the following relevant concepts:
+-
+
The timestamps of all sensors are concatenated and then grouped per day segment. Minute and hour windows are created from the beginning of each day segment instance and these windows are marked as valid based on the definitions above. The duration of each day segment is taken into account to compute the features described below.
+
+
+Available day segments and platforms
+-
+
+
+
+File Sequence
+- data/raw/{pid}/{sensor}_raw.csv # one for every [PHONE_DATA_YIELD][SENSORS]
+- data/interim/{pid}/phone_yielded_timestamps.csv
+- data/interim/{pid}/phone_yielded_timestamps_with_datetime.csv
+- data/interim/{pid}/phone_data_yield_features/phone_data_yield_{language}_{provider_key}.csv
+- data/processed/features/{pid}/phone_data_yield.csv
+Parameters description for [PHONE_DATA_YIELD][PROVIDERS][RAPIDS]:
[COMPUTE] |
-Set to True to compute |
+Set to True to extract PHONE_DATA_YIELD features from the RAPIDS provider |
|
[BIN_SIZE] |
-Size of each bin in minutes | +[FEATURES] |
+Features to be computed, see table below |
[PHONE_SENSORS] |
-One or more sensor config keys (e.g. PHONE_MESSAGE) to be used to flag a bin as valid or not (whether or not a bin contains at least one row from any sensor) |
+[MINUTE_RATIO_THRESHOLD_FOR_VALID_YIELDED_HOURS] |
+The proportion [0.0 ,1.0] of valid minutes in a 60-minute window necessary to flag that window as valid. |
-
-
-Possible values for [PHONE_VALID_SENSED_BINS][PHONE_SENSORS]
PHONE_MESSAGES
-PHONE_CALLS
-PHONE_LOCATIONS
-PHONE_BLUETOOTH
-PHONE_ACTIVITY_RECOGNITION
-PHONE_BATTERY
-PHONE_SCREEN
-PHONE_LIGHT
-PHONE_ACCELEROMETER
-PHONE_APPLICATIONS_FOREGROUND
-PHONE_WIFI_VISIBLE
-PHONE_WIFI_CONNECTED
-PHONE_CONVERSATION
-Phone Valid Sensed Days¶
-On any given day, a phone could have sensed data only for a few minutes or for 24 hours. Features should considered more reliable the more hours the phone was logging data, for example, 10 calls logged on a day when only 1 hour of data was recorded is a less reliable feature compared to 10 calls on a day when 23 hours of data were recorded.
-Therefore, we define a valid hour as those that contain a minimum number of valid bins (see above). We mark an hour as valid when contains at least MIN_VALID_BINS_PER_HOUR (out of 60min/BIN_SIZE bins). In turn, we mark a day as valid if it has at least MIN_VALID_HOURS_PER_DAY. You can use PHONE_VALID_SENSED_DAYS to manually discard days when not enough data was collected after your features are computed.
Parameters description for PHONE_VALID_SENSED_DAYS:
Features description for [PHONE_DATA_YIELD][PROVIDERS][RAPIDS]:
| Key | +Feature | +Units | Description | |
|---|---|---|---|---|
[COMPUTE] |
-Set to True to compute |
+ratiovalidyieldedminutes | +rows | +The ratio between the number of valid minutes and the duration in minutes of a day segment. |
[MIN_VALID_BINS_PER_HOUR] |
-An array of integer values, 6 by default. Minimum number of valid bins to mark an hour as valid | -|||
[MIN_VALID_HOURS_PER_DAY] |
-An array of integer values, 16 by default. Minimum number of valid hours to mark a day as valid | +ratiovalidyieldedhours | +lux | +The ratio between the number of valid hours and the duration in hours of a day segment. If the day segment is shorter than 1 hour this feature will always be 1. |
+
diff --git a/latest/features/phone-light/index.html b/latest/features/phone-light/index.html
index c139d08d..385fe58b 100644
--- a/latest/features/phone-light/index.html
+++ b/latest/features/phone-light/index.html
@@ -412,8 +412,8 @@
Assumptions/Observations
+-
+