Update docs: add data_exploration.rst
Meng Li
parent
ea27d46a05
commit
fa74f4012a
|
|
@ -33,11 +33,7 @@ Contents:
|
|||
:maxdepth: 2
|
||||
:caption: Visualization
|
||||
|
||||
visualization/histogram_of_valid_sensed_hours
|
||||
visualization/heatmap_of_phone_sensed_bins
|
||||
visualization/heatmap_of_days_by_sensors
|
||||
visualization/overall_compliance_heatmap
|
||||
visualization/heatmap_of_correlation_matrix_between_features
|
||||
visualization/data_exploration
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
|
|
|||
|
|
@ -0,0 +1,216 @@
|
|||
.. _data_exploration:
|
||||
|
||||
Data Exploration
|
||||
================
|
||||
|
||||
These plots are in beta, if you get an error while computing them please let us know.
|
||||
|
||||
.. _histogram-of-valid-sensed-hours:
|
||||
|
||||
Histogram of valid sensed hours
|
||||
"""""""""""""""""""""""""""""""
|
||||
|
||||
See `Histogram of Valid Sensed Hours Config Code`_
|
||||
|
||||
**Rule Chain:**
|
||||
|
||||
- Rule: ``rules/preprocessing.smk/download_dataset``
|
||||
- Rule: ``rules/preprocessing.smk/readable_datetime``
|
||||
- Rule: ``rules/preprocessing.smk/phone_sensed_bins``
|
||||
- Rule: ``rules/preprocessing.smk/phone_valid_sensed_days``
|
||||
- Rule: ``rules/reports.smk/histogram_valid_sensed_hours``
|
||||
|
||||
.. _figure1-parameters:
|
||||
|
||||
**Parameters of histogram_valid_sensed_hours Rule:**
|
||||
|
||||
======================= =======================
|
||||
Name Description
|
||||
======================= =======================
|
||||
plot Whether the rule is executed or not. The available options are ``True`` and ``False``.
|
||||
min_valid_bins_per_hour The minimum valid bins an hour should have to be considered valid. A valid bin has at least 1 row of data. It modifies the way we compute phone valid days. Read :ref:`PHONE_VALID_SENSED_BINS<phone-valid-sensed-bins>` for more information.
|
||||
min_valid_hours_per_day The minimum valid hours a day should have to be considered valid. It modifies the way we compute phone valid days. Read :ref:`PHONE_VALID_SENSED_DAYS<phone-valid-sensed-days>` for more information.
|
||||
======================= =======================
|
||||
|
||||
**Observations:**
|
||||
|
||||
This histogram shows the valid sensed hours of all participants processed in RAPIDS (See valid sensed :ref:`bins<phone-valid-sensed-bins>` and :ref:`days<phone-valid-sensed-days>` sections). It can be used as a rough indication of the AWARE client monitoring coverage during a study for all participants. See Figure 1.
|
||||
|
||||
.. figure:: figures/Figure1.png
|
||||
:scale: 90 %
|
||||
:align: center
|
||||
|
||||
Figure 1 Histogram of valid sensed hours for all participants
|
||||
|
||||
|
||||
.. _heatmap-of-phone-sensed-bins:
|
||||
|
||||
Heatmap of phone sensed bins
|
||||
""""""""""""""""""""""""""""
|
||||
|
||||
See `Heatmap of Phone Sensed Bins Config Code`_
|
||||
|
||||
**Rule Chain:**
|
||||
|
||||
- Rule: ``rules/preprocessing.smk/download_dataset``
|
||||
- Rule: ``rules/preprocessing.smk/readable_datetime``
|
||||
- Rule: ``rules/preprocessing.smk/phone_sensed_bins``
|
||||
- Rule: ``rules/reports.smk/heatmap_sensed_bins``
|
||||
|
||||
.. _figure2-parameters:
|
||||
|
||||
**Parameters of heatmap_sensed_bins Rule:**
|
||||
|
||||
======================= =======================
|
||||
Name Description
|
||||
======================= =======================
|
||||
plot Whether the rule is executed or not. The available options are ``True`` and ``False``.
|
||||
bin_size Every hour is divided into N bins of size ``BIN_SIZE`` (in minutes). It modifies the way we compute ``data/interim/pXX/phone_sensed_bins.csv`` file.
|
||||
======================= =======================
|
||||
|
||||
**Observations:**
|
||||
|
||||
In this heatmap rows are dates, columns are sensed bins for a participant, and cells’ color shows the number of mobile sensors that logged at least one row of data during that bin. This plot shows the periods of time without data for a participant and can be used as a rough indication of whether time-based sensors were following their sensing schedule (e.g. if location was being sensed every 2 minutes). See Figure 2.
|
||||
|
||||
.. figure:: figures/Figure2.png
|
||||
:scale: 90 %
|
||||
:align: center
|
||||
|
||||
Figure 2 Heatmap of phone sensed bins for a single participant
|
||||
|
||||
|
||||
.. _heatmap-of-days-by-sensors
|
||||
|
||||
Heatmap of days by sensors
|
||||
""""""""""""""""""""""""""
|
||||
|
||||
See `Heatmap of Days by Sensors Config Code`_
|
||||
|
||||
**Rule Chain:**
|
||||
|
||||
- Rule: ``rules/preprocessing.smk/download_dataset``
|
||||
- Rule: ``rules/preprocessing.smk/readable_datetime``
|
||||
- Rule: ``rules/preprocessing.smk/phone_sensed_bins``
|
||||
- Rule: ``rules/preprocessing.smk/phone_valid_sensed_days``
|
||||
- Rule: ``rules/reports.smk/heatmap_days_by_sensors``
|
||||
|
||||
.. _figure3-parameters:
|
||||
|
||||
**Parameters of heatmap_days_by_sensors Rule:**
|
||||
|
||||
======================= =======================
|
||||
Name Description
|
||||
======================= =======================
|
||||
plot Whether the rule is executed or not. The available options are ``True`` and ``False``.
|
||||
min_valid_bins_per_hour The minimum valid bins an hour should have to be considered valid. A valid bin has at least 1 row of data. It modifies the way we compute phone valid days. Read :ref:`PHONE_VALID_SENSED_BINS<phone-valid-sensed-bins>` for more information.
|
||||
min_valid_hours_per_day The minimum valid hours a day should have to be considered valid. It modifies the way we compute phone valid days. Read :ref:`PHONE_VALID_SENSED_DAYS<phone-valid-sensed-days>` for more information.
|
||||
expected_num_of_days The number of days of data to show starting from the first day of each participant.
|
||||
db_tables List of sensor tables to compute valid bins & hours.
|
||||
======================= =======================
|
||||
|
||||
**Observations:**
|
||||
|
||||
In this heatmap rows are sensors, columns are days and cells’ color shows the normalized (0 to 1) number of valid sensed hours (See valid sensed :ref:`bins<phone-valid-sensed-bins>` and :ref:`days<phone-valid-sensed-days>` sections) collected by a sensor during a day for a participant. The user can decide how many days of data to show starting from the first day of each participant. This plot can used to judge missing data on a per participant, per sensor basis as well as the number of valid sensed hours (usable data) for each day. See Figure 3.
|
||||
|
||||
.. figure:: figures/Figure3.png
|
||||
:scale: 90 %
|
||||
:align: center
|
||||
|
||||
Figure 3 Heatmap of days by sensors for a participant
|
||||
|
||||
|
||||
.. _overall-compliance-heatmap
|
||||
|
||||
Overall compliance heatmap
|
||||
""""""""""""""""""""""""""
|
||||
|
||||
See `Overall Compliance Heatmap Config Code`_
|
||||
|
||||
**Rule Chain:**
|
||||
|
||||
- Rule: ``rules/preprocessing.smk/download_dataset``
|
||||
- Rule: ``rules/preprocessing.smk/readable_datetime``
|
||||
- Rule: ``rules/preprocessing.smk/phone_sensed_bins``
|
||||
- Rule: ``rules/preprocessing.smk/phone_valid_sensed_days``
|
||||
- Rule: ``rules/reports.smk/overall_compliance_heatmap``
|
||||
|
||||
.. _figure4-parameters:
|
||||
|
||||
**Parameters of overall_compliance_heatmap Rule:**
|
||||
|
||||
======================= =======================
|
||||
Name Description
|
||||
======================= =======================
|
||||
plot Whether the rule is executed or not. The available options are ``True`` and ``False``.
|
||||
only_show_valid_days Whether the plot only shows valid days or not. The available options are ``True`` and ``False``.
|
||||
expected_num_of_days The number of days to show before today.
|
||||
bin_size Every hour is divided into N bins of size ``BIN_SIZE`` (in minutes). It modifies the way we compute ``data/interim/pXX/phone_sensed_bins.csv`` file.
|
||||
min_valid_bins_per_hour The minimum valid bins an hour should have to be considered valid. A valid bin has at least 1 row of data. It modifies the way we compute phone valid days. Read :ref:`PHONE_VALID_SENSED_BINS<phone-valid-sensed-bins>` for more information.
|
||||
min_valid_hours_per_day The minimum valid hours a day should have to be considered valid. It modifies the way we compute phone valid days. Read :ref:`PHONE_VALID_SENSED_DAYS<phone-valid-sensed-days>` for more information.
|
||||
======================= =======================
|
||||
|
||||
**Observations:**
|
||||
|
||||
In this heatmap rows are participants, columns are days and cells’ color shows the valid sensed hours for a participant during a day (See valid sensed :ref:`bins<phone-valid-sensed-bins>` and :ref:`days<phone-valid-sensed-days>` sections). This plot can be configured to show a certain number of days before today using the ``EXPECTED_NUM_OF_DAYS`` parameter (by default -1 showing all days for every participant). As different participants might join the study on different dates, the x-axis has a day index instead of a date. This plot gives the user a quick overview of the amount of data collected per person and is complementary to the histogram of valid sensed hours as it is broken down per participant and per day. See Figure 4.
|
||||
|
||||
.. figure:: figures/Figure4.png
|
||||
:scale: 90 %
|
||||
:align: center
|
||||
|
||||
Figure 4 Overall compliance heatmap for all participants
|
||||
|
||||
|
||||
.. _heatmap-of-correlation-matrix-between-features
|
||||
|
||||
Heatmap of correlation matrix between features
|
||||
""""""""""""""""""""""""""""""""""""""""""""""
|
||||
|
||||
See `Heatmap of Correlation Matrix Config Code`_
|
||||
|
||||
**Rule Chain:**
|
||||
|
||||
- Rules to extract features
|
||||
- Rule: ``rules/preprocessing.smk/download_dataset``
|
||||
- Rule: ``rules/preprocessing.smk/readable_datetime``
|
||||
- Rule: ``rules/preprocessing.smk/phone_sensed_bins``
|
||||
- Rule: ``rules/preprocessing.smk/phone_valid_sensed_days``
|
||||
- Rule: ``rules/reports.smk/heatmap_features_correlations``
|
||||
|
||||
.. _figure5-parameters:
|
||||
|
||||
**Parameters of heatmap_features_correlations Rule:**
|
||||
|
||||
======================= ==============
|
||||
Name Description
|
||||
======================= ==============
|
||||
plot Whether the rule is executed or not. The available options are ``True`` and ``False``.
|
||||
min_valid_bins_per_hour The minimum valid bins an hour should have to be considered valid. A valid bin has at least 1 row of data. It modifies the way we compute phone valid days. Read :ref:`PHONE_VALID_SENSED_BINS<phone-valid-sensed-bins>` for more information.
|
||||
min_valid_hours_per_day The minimum valid hours a day should have to be considered valid. It modifies the way we compute phone valid days. Read :ref:`PHONE_VALID_SENSED_DAYS<phone-valid-sensed-days>` for more information.
|
||||
corr_method Method of correlation. The available options are ``pearson``, ``kendall`` and ``spearman``.
|
||||
min_rows_ratio Minimum number of observations required per pair of columns to have a valid correlation coefient. Currently, only available for ``pearson`` and ``spearman`` correlation.
|
||||
phone_features The list of phone features.
|
||||
fitbit_features The list of Fitbit features.
|
||||
corr_threshold Only correlation coefficients larger than ``corr_threshold`` can be shown in the heatmap.
|
||||
======================= ==============
|
||||
|
||||
**Observations:**
|
||||
|
||||
Columns and rows are features computed in RAPIDS, cells’ color represents the correlation coefficient between all days of data for every pair of feature of all participants. The user can specify a minimum number of observations required to compute the correlation between two features using the ``MIN_ROWS_RATIO`` parameter (0.5 by default). In addition, this plot can be configured to only display correlation coefficients above a threshold using the ``CORR_THRESHOLD`` parameter (0.1 by default). See Figure 5.
|
||||
|
||||
.. figure:: figures/Figure5.png
|
||||
:scale: 90 %
|
||||
:align: center
|
||||
|
||||
Figure 5 Correlation matrix heatmap for all the data of all participants
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
.. _`Histogram of Valid Sensed Hours Config Code`: https://github.com/carissalow/rapids/blob/master/config.yaml#L221
|
||||
.. _`Heatmap of Phone Sensed Bins Config Code`: https://github.com/carissalow/rapids/blob/master/config.yaml#L233
|
||||
.. _`Heatmap of Days by Sensors Config Code`: https://github.com/carissalow/rapids/blob/master/config.yaml#L226
|
||||
.. _`Overall Compliance Heatmap Config Code`: https://github.com/carissalow/rapids/blob/master/config.yaml#L237
|
||||
.. _`Heatmap of Correlation Matrix Config Code`: https://github.com/carissalow/rapids/blob/master/config.yaml#L211
|
||||
|
|
@ -1,45 +0,0 @@
|
|||
.. _heatmap-of-correlation-matrix-between-features
|
||||
|
||||
Heatmap of correlation matrix between features
|
||||
==============================================
|
||||
|
||||
See `Heatmap of Correlation Matrix Config Code`_
|
||||
|
||||
**Rule Chain:**
|
||||
|
||||
- Rules to extract features
|
||||
- Rule: ``rules/preprocessing.smk/download_dataset``
|
||||
- Rule: ``rules/preprocessing.smk/readable_datetime``
|
||||
- Rule: ``rules/preprocessing.smk/phone_sensed_bins``
|
||||
- Rule: ``rules/preprocessing.smk/phone_valid_sensed_days``
|
||||
- Rule: ``rules/reports.smk/heatmap_features_correlations``
|
||||
|
||||
.. _figure5-parameters:
|
||||
|
||||
**Parameters of heatmap_features_correlations Rule:**
|
||||
|
||||
======================= ==============
|
||||
Name Description
|
||||
======================= ==============
|
||||
plot Whether the rule is executed or not. The available options are ``True`` and ``False``.
|
||||
min_valid_hours_per_day The minimum valid hours per day.
|
||||
min_valid_bins_per_hour The minimum valid bins per hour.
|
||||
corr_method Method of correlation. The available options are ``pearson``, ``kendall`` and ``spearman``.
|
||||
min_rows_ratio Minimum number of observations required per pair of columns to have a valid correlation coefient. Currently, only available for ``pearson`` and ``spearman`` correlation.
|
||||
phone_features The list of phone features.
|
||||
fitbit_features The list of Fitbit features.
|
||||
corr_threshold Only correlation coefficients larger than ``corr_threshold`` can be shown in the heatmap.
|
||||
======================= ==============
|
||||
|
||||
**Observations:**
|
||||
|
||||
Columns and rows are features computed in RAPIDS, cells’ color represents the correlation coefficient between all days of data for every pair of feature of all participants. The user can specify a minimum number of observations required to compute the correlation between two features using the ``MIN_ROWS_RATIO`` parameter (0.5 by default). In addition, this plot can be configured to only display correlation coefficients above a threshold using the ``CORR_THRESHOLD`` parameter (0.1 by default). See Figure 5.
|
||||
|
||||
.. figure:: figures/Figure5.png
|
||||
:scale: 90 %
|
||||
:align: center
|
||||
|
||||
Figure 5 Correlation matrix heatmap for all the data of all participants
|
||||
|
||||
|
||||
.. _`Heatmap of Correlation Matrix Config Code`: https://github.com/carissalow/rapids/blob/master/config.yaml#L211
|
||||
|
|
@ -1,41 +0,0 @@
|
|||
.. _heatmap-of-days-by-sensors
|
||||
|
||||
Heatmap of days by sensors
|
||||
==========================
|
||||
|
||||
See `Heatmap of Days by Sensors Config Code`_
|
||||
|
||||
**Rule Chain:**
|
||||
|
||||
- Rule: ``rules/preprocessing.smk/download_dataset``
|
||||
- Rule: ``rules/preprocessing.smk/readable_datetime``
|
||||
- Rule: ``rules/preprocessing.smk/phone_sensed_bins``
|
||||
- Rule: ``rules/preprocessing.smk/phone_valid_sensed_days``
|
||||
- Rule: ``rules/reports.smk/heatmap_days_by_sensors``
|
||||
|
||||
.. _figure3-parameters:
|
||||
|
||||
**Parameters of heatmap_days_by_sensors Rule:**
|
||||
|
||||
======================= =======================
|
||||
Name Description
|
||||
======================= =======================
|
||||
plot Whether the rule is executed or not. The available options are ``True`` and ``False``.
|
||||
min_valid_hours_per_day The minimum valid hours per day.
|
||||
min_valid_bins_per_hour The minimum valid bins per hour.
|
||||
expected_num_of_days The number of days of data to show starting from the first day of each participant.
|
||||
db_tables List of sensor tables to compute valid bins & hours.
|
||||
======================= =======================
|
||||
|
||||
**Observations:**
|
||||
|
||||
In this heatmap rows are sensors, columns are days and cells’ color shows the normalized (0 to 1) number of valid sensed hours (See valid sensed :ref:`bins<phone-valid-sensed-bins>` and :ref:`days<phone-valid-sensed-days>` sections) collected by a sensor during a day for a participant. The user can decide how many days of data to show starting from the first day of each participant. This plot can used to judge missing data on a per participant, per sensor basis as well as the number of valid sensed hours (usable data) for each day. See Figure 3.
|
||||
|
||||
.. figure:: figures/Figure3.png
|
||||
:scale: 90 %
|
||||
:align: center
|
||||
|
||||
Figure 3 Heatmap of days by sensors for a participant
|
||||
|
||||
|
||||
.. _`Heatmap of Days by Sensors Config Code`: https://github.com/carissalow/rapids/blob/master/config.yaml#L226
|
||||
|
|
@ -1,37 +0,0 @@
|
|||
.. _heatmap-of-phone-sensed-bins:
|
||||
|
||||
Heatmap of phone sensed bins
|
||||
============================
|
||||
|
||||
See `Heatmap of Phone Sensed Bins Config Code`_
|
||||
|
||||
**Rule Chain:**
|
||||
|
||||
- Rule: ``rules/preprocessing.smk/download_dataset``
|
||||
- Rule: ``rules/preprocessing.smk/readable_datetime``
|
||||
- Rule: ``rules/preprocessing.smk/phone_sensed_bins``
|
||||
- Rule: ``rules/reports.smk/heatmap_sensed_bins``
|
||||
|
||||
.. _figure2-parameters:
|
||||
|
||||
**Parameters of heatmap_sensed_bins Rule:**
|
||||
|
||||
======================= =======================
|
||||
Name Description
|
||||
======================= =======================
|
||||
plot Whether the rule is executed or not. The available options are ``True`` and ``False``.
|
||||
bin_size Every hour is divided into N bins of size ``BIN_SIZE`` (in minutes).
|
||||
======================= =======================
|
||||
|
||||
**Observations:**
|
||||
|
||||
In this heatmap rows are dates, columns are sensed bins for a participant, and cells’ color shows the number of mobile sensors that logged at least one row of data during that bin. This plot shows the periods of time without data for a participant and can be used as a rough indication of whether time-based sensors were following their sensing schedule (e.g. if location was being sensed every 2 minutes). See Figure 2.
|
||||
|
||||
.. figure:: figures/Figure2.png
|
||||
:scale: 90 %
|
||||
:align: center
|
||||
|
||||
Figure 2 Heatmap of phone sensed bins for a single participant
|
||||
|
||||
|
||||
.. _`Heatmap of Phone Sensed Bins Config Code`: https://github.com/carissalow/rapids/blob/master/config.yaml#L233
|
||||
|
|
@ -1,39 +0,0 @@
|
|||
.. _histogram-of-valid-sensed-hours:
|
||||
|
||||
Histogram of valid sensed hours
|
||||
===============================
|
||||
|
||||
See `Histogram of Valid Sensed Hours Config Code`_
|
||||
|
||||
**Rule Chain:**
|
||||
|
||||
- Rule: ``rules/preprocessing.smk/download_dataset``
|
||||
- Rule: ``rules/preprocessing.smk/readable_datetime``
|
||||
- Rule: ``rules/preprocessing.smk/phone_sensed_bins``
|
||||
- Rule: ``rules/preprocessing.smk/phone_valid_sensed_days``
|
||||
- Rule: ``rules/reports.smk/histogram_valid_sensed_hours``
|
||||
|
||||
.. _figure1-parameters:
|
||||
|
||||
**Parameters of histogram_valid_sensed_hours Rule:**
|
||||
|
||||
======================= =======================
|
||||
Name Description
|
||||
======================= =======================
|
||||
plot Whether the rule is executed or not. The available options are ``True`` and ``False``.
|
||||
min_valid_hours_per_day The minimum valid hours per day.
|
||||
min_valid_bins_per_hour The minimum valid bins per hour.
|
||||
======================= =======================
|
||||
|
||||
**Observations:**
|
||||
|
||||
This histogram shows the valid sensed hours of all participants processed in RAPIDS (See valid sensed :ref:`bins<phone-valid-sensed-bins>` and :ref:`days<phone-valid-sensed-days>` sections). It can be used as a rough indication of the AWARE client monitoring coverage during a study for all participants. See Figure 1.
|
||||
|
||||
.. figure:: figures/Figure1.png
|
||||
:scale: 90 %
|
||||
:align: center
|
||||
|
||||
Figure 1 Histogram of valid sensed hours for all participants
|
||||
|
||||
|
||||
.. _`Histogram of Valid Sensed Hours Config Code`: https://github.com/carissalow/rapids/blob/master/config.yaml#L221
|
||||
|
|
@ -1,42 +0,0 @@
|
|||
.. _overall-compliance-heatmap
|
||||
|
||||
Overall compliance heatmap
|
||||
==========================
|
||||
|
||||
See `Overall Compliance Heatmap Config Code`_
|
||||
|
||||
**Rule Chain:**
|
||||
|
||||
- Rule: ``rules/preprocessing.smk/download_dataset``
|
||||
- Rule: ``rules/preprocessing.smk/readable_datetime``
|
||||
- Rule: ``rules/preprocessing.smk/phone_sensed_bins``
|
||||
- Rule: ``rules/preprocessing.smk/phone_valid_sensed_days``
|
||||
- Rule: ``rules/reports.smk/overall_compliance_heatmap``
|
||||
|
||||
.. _figure4-parameters:
|
||||
|
||||
**Parameters of overall_compliance_heatmap Rule:**
|
||||
|
||||
======================= =======================
|
||||
Name Description
|
||||
======================= =======================
|
||||
plot Whether the rule is executed or not. The available options are ``True`` and ``False``.
|
||||
only_show_valid_days Whether the plot only shows valid days or not. The available options are ``True`` and ``False``.
|
||||
expected_num_of_days The number of days to show before today.
|
||||
bin_size Every hour is divided into N bins of size ``BIN_SIZE`` (in minutes).
|
||||
min_valid_hours_per_day The minimum valid hours per day.
|
||||
min_valid_bins_per_hour The minimum valid bins per hour.
|
||||
======================= =======================
|
||||
|
||||
**Observations:**
|
||||
|
||||
In this heatmap rows are participants, columns are days and cells’ color shows the valid sensed hours for a participant during a day (See valid sensed :ref:`bins<phone-valid-sensed-bins>` and :ref:`days<phone-valid-sensed-days>` sections). This plot can be configured to show a certain number of days before today using the ``EXPECTED_NUM_OF_DAYS`` parameter (by default -1 showing all days for every participant). As different participants might join the study on different dates, the x-axis has a day index instead of a date. This plot gives the user a quick overview of the amount of data collected per person and is complementary to the histogram of valid sensed hours as it is broken down per participant and per day. See Figure 4.
|
||||
|
||||
.. figure:: figures/Figure4.png
|
||||
:scale: 90 %
|
||||
:align: center
|
||||
|
||||
Figure 4 Overall compliance heatmap for all participants
|
||||
|
||||
|
||||
.. _`Overall Compliance Heatmap Config Code`: https://github.com/carissalow/rapids/blob/master/config.yaml#L237
|
||||
Loading…
Reference in New Issue