junos
/
rapids
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Add fitbitparsed_mysql

pull/128/head
JulioV 2021-03-11 09:54:12 -05:00
parent 1b0ee4bbf0
commit 470f4276af
8 changed files with 369 additions and 23 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
config.yaml
View File

@ -341,6 +341,11 @@ FITBIT_DATA_STREAMS:
fitbitjson_mysql: fitbitjson_mysql:
DATABASE_GROUP: MY_GROUP DATABASE_GROUP: MY_GROUP
SLEEP_SUMMARY_EPISODE_DAY_ANCHOR: end # summary sleep episodes are considered as events based on either the start timestamp or end timestamp. SLEEP_SUMMARY_EPISODE_DAY_ANCHOR: end # summary sleep episodes are considered as events based on either the start timestamp or end timestamp.
fitbitparsed_mysql:
DATABASE_GROUP: MY_GROUP
SLEEP_SUMMARY_EPISODE_DAY_ANCHOR: end # summary sleep episodes are considered as events based on either the start timestamp or end timestamp.
# Sensors ------ # Sensors ------

2
docs/datastreams/data-streams-introduction.md
View File

@ -9,7 +9,7 @@ If you want to process a data stream using RAPIDS, make sure that your data is s
If RAPIDS doesn't support your data stream yet (e.g. Beiwe data stored in PostgreSQL, or AWARE data stored in InfluxDB), you can always [implement a new data stream](../add-new-data-streams). If it's something you think other people might be interested on, we will be happy to include your new data stream in RAPIDS, so get in touch!. If RAPIDS doesn't support your data stream yet (e.g. Beiwe data stored in PostgreSQL, or AWARE data stored in InfluxDB), you can always [implement a new data stream](../add-new-data-streams). If it's something you think other people might be interested on, we will be happy to include your new data stream in RAPIDS, so get in touch!.
!!! hint !!! hint
You can only add new data streams for Smartphone or Fitbit data. If you need RAPIDS to process data from **different devices**, like Oura Rings or Actigraph wearables, get in touch. It is a more complex process that could take a few days to implement for someone familiar with R or Python but that we would be happy to work on together. Currently, you can add new data streams for smartphones, Fitbit, and Empatica devices. If you need RAPIDS to process data from **other devices**, like Oura Rings or Actigraph wearables, get in touch. It is a more complicated process that could take a couple of days to implement for someone familiar with R or Python, but we would be happy to work on it together.
For reference, these are the data streams we currently support: For reference, these are the data streams we currently support:

8
docs/datastreams/fitbitjson-mysql.md
View File

@ -2,7 +2,7 @@
This [data stream](../../datastreams/data-streams-introduction) handles Fitbit sensor data downloaded using the [Fitbit Web API](https://dev.fitbit.com/build/reference/web-api/) and stored in a MySQL database. Please note that RAPIDS cannot query the API directly; you need to use other available tools or implement your own. Once you have your sensor data in a MySQL database, RAPIDS can process it. This [data stream](../../datastreams/data-streams-introduction) handles Fitbit sensor data downloaded using the [Fitbit Web API](https://dev.fitbit.com/build/reference/web-api/) and stored in a MySQL database. Please note that RAPIDS cannot query the API directly; you need to use other available tools or implement your own. Once you have your sensor data in a MySQL database, RAPIDS can process it.
## Container ## Container
The container should be a MySQL database with a table per sensor, each containing the data for all participants. The container should be a MySQL database with a table per sensor, each containing all participants' data.
The script to connect and download data from this container is at: The script to connect and download data from this container is at:
```bash ```bash
@ -17,7 +17,7 @@ The `format.yaml` maps and transforms columns in your raw data stream to the [ma
src/data/streams/fitbitjson_mysql/format.yaml src/data/streams/fitbitjson_mysql/format.yaml
``` ```
If you want RAPIDS to process Fitbit sensor data using this stream, you will need to replace `[RAPIDS_COLUMN_MAPPINGS]`/`[MUTATION][COLUMN_MAPPINGS]` inside **each sensor** section in `format.yaml` to match your raw data column names: If you want RAPIDS to process Fitbit sensor data using this stream, you will need to map `DEVICE_ID` and `JSON_FITBIT_COLUMN` to your own raw data columns inside **each sensor** section in `format.yaml`.
??? info "FITBIT_HEARTRATE_SUMMARY" ??? info "FITBIT_HEARTRATE_SUMMARY"
@ -52,7 +52,7 @@ If you want RAPIDS to process Fitbit sensor data using this stream, you will nee
All columns except `DEVICE_ID` are parsed from `JSON_FITBIT_COLUMN`. `JSON_FITBIT_COLUMN` is a string column containing the JSON objects returned by Fitbit's API. See an example of the raw data RAPIDS expects for this data stream: All columns except `DEVICE_ID` are parsed from `JSON_FITBIT_COLUMN`. `JSON_FITBIT_COLUMN` is a string column containing the JSON objects returned by Fitbit's API. See an example of the raw data RAPIDS expects for this data stream:
??? "Example of the raw data RAPIDS expects for this data stream" ??? example "Example of the raw data RAPIDS expects for this data stream"
|device_id |fitbit_data | |device_id |fitbit_data |
|---------------------------------------- |--------------------------------------------------------- | |---------------------------------------- |--------------------------------------------------------- |
@ -90,7 +90,7 @@ If you want RAPIDS to process Fitbit sensor data using this stream, you will nee
All columns except `DEVICE_ID` are parsed from `JSON_FITBIT_COLUMN`. `JSON_FITBIT_COLUMN` is a string column containing the JSON objects returned by Fitbit's API. See an example of the raw data RAPIDS expects for this data stream: All columns except `DEVICE_ID` are parsed from `JSON_FITBIT_COLUMN`. `JSON_FITBIT_COLUMN` is a string column containing the JSON objects returned by Fitbit's API. See an example of the raw data RAPIDS expects for this data stream:
??? "Example of the raw data RAPIDS expects for this data stream" ??? example "Example of the raw data RAPIDS expects for this data stream"
|device_id |fitbit_data | |device_id |fitbit_data |
|---------------------------------------- |--------------------------------------------------------- | |---------------------------------------- |--------------------------------------------------------- |

232
docs/datastreams/fitbitparsed-mysql.md
View File

@ -1,15 +1,15 @@
# `fitbitparsed_mysql` # `fitbitparsed_mysql`
This [data stream](../../datastreams/data-streams-introduction) handles Fitbit sensor data downloaded using the [Fitbit Web API](https://dev.fitbit.com/build/reference/web-api/), **parsed**, and stored in a MySQL database. Please note that RAPIDS cannot query the API directly, you need to use other available tools or implement your own. Once you have your sensor data in a MySQL database, RAPIDS can process it. This [data stream](../../datastreams/data-streams-introduction) handles Fitbit sensor data downloaded using the [Fitbit Web API](https://dev.fitbit.com/build/reference/web-api/), **parsed**, and stored in a MySQL database. Please note that RAPIDS cannot query the API directly; you need to use other available tools or implement your own. Once you have your parsed sensor data in a MySQL database, RAPIDS can process it.
!!! info "What is the difference between JSON and plain data streams" !!! info "What is the difference between JSON and plain data streams"
Most people will only need `fitbitjson_mysql` because they downloaded and stored their data directly from Fitbit's API. However, if for some reason you don't have access to that JSON data and instead only have the parsed data (columns and rows) you can use this data stream. Most people will only need `fitbitjson_mysql` because they downloaded and stored their data directly from Fitbit's API. However, if, for some reason, you don't have access to that JSON data and instead only have the parsed data (columns and rows), you can use this data stream.
## Container ## Container
A MySQL database with a table per sensor, each containing the data for all participants. The container should be a MySQL database with a table per sensor, each containing all participants' data.
The script to connect and download data from this container is at: The script to connect and download data from this container is at:
```bash ```bash
src/data/streams/fitbitjson_mysql/container.R src/data/streams/fitbitparsed_mysql/container.R
``` ```
## Format ## Format
@ -20,33 +20,229 @@ The `format.yaml` maps and transforms columns in your raw data stream to the [ma
src/data/streams/fitbitparsed_mysql/format.yaml src/data/streams/fitbitparsed_mysql/format.yaml
``` ```
If you want RAPIDS to process Fitbit sensor data using this stream, you will need to replace any `RAPIDS_COLUMN_MAPPINGS` inside **each sensor** section in `format.yaml` to match your raw data column names. If you want to use this stream with your data, modify every sensor in `format.yaml` to map all columns except `TIMESTAMP` in `[RAPIDS_COLUMN_MAPPINGS]` to your raw data column names.
All columns are mandatory, however, all except `device_id` and `local_date_time` can be empty if you don't have that data. Just have in mind that some features will be empty if some of these columns are empty. All columns are mandatory; however, all except `device_id` and `local_date_time` can be empty if you don't have that data. Just have in mind that some features will be empty if some of these columns are empty.
??? info "FITBIT_HEARTRATE_SUMMARY section"
??? info "FITBIT_HEARTRATE_SUMMARY"
**RAPIDS_COLUMN_MAPPINGS** **RAPIDS_COLUMN_MAPPINGS**
| RAPIDS column | Stream column | | RAPIDS column | Stream column |
|-----------------|-----------------| |-----------------|-----------------|
| LOCAL_DATE_TIME | local_date_time | | TIMESTAMP| FLAG_TO_MUTATE |
| DEVICE_ID | device_id | | LOCAL_DATE_TIME | local_date_time |
| DEVICE_ID | device_id |
| HEARTRATE_DAILY_RESTINGHR | heartrate_daily_restinghr | | HEARTRATE_DAILY_RESTINGHR | heartrate_daily_restinghr |
| HEARTRATE_DAILY_CALORIESOUTOFRANGE | heartrate_daily_caloriesoutofrange | | HEARTRATE_DAILY_CALORIESOUTOFRANGE | heartrate_daily_caloriesoutofrange |
| HEARTRATE_DAILY_CALORIESFATBURN | heartrate_daily_caloriesfatburn | | HEARTRATE_DAILY_CALORIESFATBURN | heartrate_daily_caloriesfatburn |
| HEARTRATE_DAILY_CALORIESCARDIO | heartrate_daily_caloriescardio | | HEARTRATE_DAILY_CALORIESCARDIO | heartrate_daily_caloriescardio |
| HEARTRATE_DAILY_CALORIESPEAK | heartrate_daily_caloriespeak | | HEARTRATE_DAILY_CALORIESPEAK | heartrate_daily_caloriespeak |
**MUTATION_SCRIPTS**
TODO list our parsing script **MUTATION**
??? "Example of the raw data RAPIDS expects for this data stream" - **COLUMN_MAPPINGS** (None)
|device_id |local_date_time |heartrate_daily_restinghr |heartrate_daily_caloriesoutofrange |heartrate_daily_caloriesfatburn |heartrate_daily_caloriescardio |heartrate_daily_caloriespeak | - **SCRIPTS**
|-------------------------------------- |----------------- |------- |-------------- |------------- |------------ |-------|
|a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-07 |72 |1200.6102 |760.3020 |15.2048 |0 | ```bash
|a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-08 |70 |1100.1120 |660.0012 |23.7088 |0 | src/data/streams/mutations/fitbit/add_zero_timestamp.py
|a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-09 |69 |750.3615 |734.1516 |131.8579 |0 | ```
!!! note
`add_zero_timestamp` adds an all-zero column called `timestamp` that will be filled in later in the pipeline by `readable_time.R` converting LOCAL_DATE_TIME to a unix timestamp taking into account single or multiple time zones.
??? example "Example of the raw data RAPIDS expects for this data stream"
|device_id |local_date_time |heartrate_daily_restinghr |heartrate_daily_caloriesoutofrange |heartrate_daily_caloriesfatburn |heartrate_daily_caloriescardio |heartrate_daily_caloriespeak |
|-------------------------------------- |----------------- |------- |-------------- |------------- |------------ |-------|
|a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-07 |72 |1200.6102 |760.3020 |15.2048 |0 |
|a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-08 |70 |1100.1120 |660.0012 |23.7088 |0 |
|a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-09 |69 |750.3615 |734.1516 |131.8579 |0 |
??? info "FITBIT_HEARTRATE_INTRADAY"
**RAPIDS_COLUMN_MAPPINGS**
| RAPIDS column | Stream column |
|-----------------|-----------------|
| TIMESTAMP| FLAG_TO_MUTATE |
| LOCAL_DATE_TIME | local_date_time |
| DEVICE_ID | device_id |
| HEARTRATE | heartrate |
| HEARTRATE_ZONE | heartrate_zone |
**MUTATION**
- **COLUMN_MAPPINGS** (None)
- **SCRIPTS**
```bash
src/data/streams/mutations/fitbit/add_zero_timestamp.py
```
!!! note
`add_zero_timestamp` adds an all-zero column called `timestamp` that will be filled in later in the pipeline by `readable_time.R` converting LOCAL_DATE_TIME to a unix timestamp taking into account single or multiple time zones.
??? example "Example of the raw data RAPIDS expects for this data stream"
|device_id |local_date_time |heartrate |heartrate_zone |
|-------------------------------------- |---------------------- |--------- |--------------- |
|a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-07 00:00:00 |68 |outofrange |
|a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-07 00:01:00 |67 |outofrange |
|a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-07 00:02:00 |67 |outofrange |
??? info "FITBIT_SLEEP_SUMMARY"
**RAPIDS_COLUMN_MAPPINGS**
| RAPIDS column | Stream column |
|-----------------|-----------------|
| TIMESTAMP| FLAG_TO_MUTATE |
| LOCAL_DATE_TIME| local_date_time |
| LOCAL_START_DATE_TIME| local_start_date_time |
| LOCAL_END_DATE_TIME| local_end_date_time |
| DEVICE_ID| device_id |
| EFFICIENCY| efficiency |
| MINUTES_AFTER_WAKEUP| minutes_after_wakeup |
| MINUTES_ASLEEP| minutes_asleep |
| MINUTES_AWAKE| minutes_awake |
| MINUTES_TO_FALL_ASLEEP| minutes_to_fall_asleep |
| MINUTES_IN_BED| minutes_in_bed |
| IS_MAIN_SLEEP| is_main_sleep |
| TYPE| type |
**MUTATION**
- **COLUMN_MAPPINGS** (None)
- **SCRIPTS**
```bash
src/data/streams/mutations/fitbit/add_zero_timestamp.py
```
!!! note
`add_zero_timestamp` adds an all-zero column called `timestamp` that will be filled in later in the pipeline by `readable_time.R` converting LOCAL_DATE_TIME to a unix timestamp taking into account single or multiple time zones.
Fitbit API has two versions for sleep data, v1 and v1.2. We support both but ignore v1's `count_awake`, `duration_awake`, and `count_awakenings`, `count_restless`, `duration_restless` columns.
??? example "Example of the expected raw data"
|device_id |local_start_date_time |local_end_date_time |efficiency |minutes_after_wakeup |minutes_asleep |minutes_awake |minutes_to_fall_asleep |minutes_in_bed |is_main_sleep |type |
|-------------------------------------- |---------------------- |---------------------- |----------- |--------------------- |--------------- |-------------- |----------------------- |--------------- |-------------- |-------- |
|a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-10 15:36:30 |2020-10-10 16:37:00 |92 |0 |55 |5 |0 |60 |0 |classic |
|a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-10 01:46:30 |2020-10-10 08:10:00 |88 |0 |318 |65 |0 |383 |1 |stages |
|a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-11 00:12:30 |2020-10-11 11:47:00 |89 |1 |562 |132 |0 |694 |1 |stages |
|a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-12 01:31:00 |2020-10-12 09:34:30 |93 |0 |415 |68 |0 |483 |1 |stages |
??? info "FITBIT_SLEEP_INTRADAY"
**RAPIDS_COLUMN_MAPPINGS**
| RAPIDS column | Stream column |
|-----------------|-----------------|
| TIMESTAMP | FLAG_TO_MUTATE |
| LOCAL_DATE_TIME | local_date_time |
| DEVICE_ID | device_id |
| TYPE_EPISODE_ID | type_episode_id |
| DURATION | duration |
| IS_MAIN_SLEEP | is_main_sleep |
| TYPE | type |
| LEVEL | level |
**MUTATION**
- **COLUMN_MAPPINGS** (None)
- **SCRIPTS**
```bash
src/data/streams/mutations/fitbit/add_zero_timestamp.py
```
!!! note
`add_zero_timestamp` adds an all-zero column called `timestamp` that will be filled in later in the pipeline by `readable_time.R` converting LOCAL_DATE_TIME to a unix timestamp taking into account single or multiple time zones.
Fitbit API has two versions for sleep data, v1 and v1.2, we support both.
??? example "Example of the expected raw data"
|device_id |type_episode_id |local_date_time |duration |level |is_main_sleep |type |
|------------------------------------ |---------------- |------------------- |--------- |---------- |-------------- |-------------- |
|a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |0 |2020-10-10 15:36:30 |60 |restless |0 |classic |
|a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |0 |2020-10-10 15:37:30 |660 |asleep |0 |classic |
|a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |0 |2020-10-10 15:48:30 |60 |restless |0 |classic |
|a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |... |... |... |... |... |... |
|a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |1 |2020-10-10 01:46:30 |420 |light |1 |stages |
|a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |1 |2020-10-10 01:53:30 |1230 |deep |1 |stages |
??? info "FITBIT_STEPS_SUMMARY"
**RAPIDS_COLUMN_MAPPINGS**
| RAPIDS column | Stream column |
|-----------------|-----------------|
| TIMESTAMP | FLAG_TO_MUTATE |
| DEVICE_ID | device_id |
| LOCAL_DATE_TIME | local_date_time |
| STEPS | steps |
**MUTATION**
- **COLUMN_MAPPINGS** (None)
- **SCRIPTS**
```bash
src/data/streams/mutations/fitbit/add_zero_timestamp.py
```
!!! note
`add_zero_timestamp` adds an all-zero column called `timestamp` that will be filled in later in the pipeline by `readable_time.R` converting LOCAL_DATE_TIME to a unix timestamp taking into account single or multiple time zones.
??? example "Example of the expected raw data"
|device_id |local_date_time |steps |
|-------------------------------------- |---------------------- |--------- |
|a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-07 |1775 |
|a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-08 |3201 |
|a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-09 |998 |
??? info "FITBIT_STEPS_INTRADAY"
**RAPIDS_COLUMN_MAPPINGS**
| RAPIDS column | Stream column |
|-----------------|-----------------|
| TIMESTAMP | FLAG_TO_MUTATE |
| DEVICE_ID | device_id |
| LOCAL_DATE_TIME | local_date_time |
| STEPS | steps |
**MUTATION**
- **COLUMN_MAPPINGS** (None)
- **SCRIPTS**
```bash
src/data/streams/mutations/fitbit/add_zero_timestamp.py
```
!!! note
`add_zero_timestamp` adds an all-zero column called `timestamp` that will be filled in later in the pipeline by `readable_time.R` converting LOCAL_DATE_TIME to a unix timestamp taking into account single or multiple time zones.
??? example "Example of the expected raw data"
|device_id |local_date_time |steps |
|-------------------------------------- |---------------------- |--------- |
|a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-07 00:00:00 |5 |
|a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-07 00:01:00 |3 |
|a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-07 00:02:00 |0 |

47
src/data/streams/fitbitparsed_mysql/container.R 100644
View File

@ -0,0 +1,47 @@
# if you need a new package, you should add it with renv::install(package) so your renv venv is updated
library(RMariaDB)
library(yaml)
#' @description
#' Auxiliary function to parse the connection credentials from a specifc group in ./credentials.yaml
#' You can reause most of this function if you are connection to a DB or Web API.
#' It's OK to delete this function if you don't need credentials, e.g., you are pulling data from a CSV for example.
#' @param group the yaml key containing the credentials to connect to a database
#' @preturn dbEngine a database engine (connection) ready to perform queries
get_db_engine <- function(group){
# The working dir is aways RAPIDS root folder, so your credentials file is always /credentials.yaml
credentials <- read_yaml("./credentials.yaml")
if(!group %in% names(credentials))
stop(paste("The credentials group",group, "does not exist in ./credentials.yaml. The only groups that exist in that file are:", paste(names(credentials), collapse = ",")))
dbEngine <- dbConnect(MariaDB(), db = credentials[[group]][["database"]],
username = credentials[[group]][["user"]],
password = credentials[[group]][["password"]],
host = credentials[[group]][["host"]],
port = credentials[[group]][["port"]])
return(dbEngine)
}
#' @description
#' Gets the sensor data for a specific device id from a database table, file or whatever source you want to query
#'
#' @param stream_parameters The PHONE_STREAM_PARAMETERS key in config.yaml. If you need specific parameters add them there.
#' @param device A device ID string
#' @param sensor_container database table or file containing the sensor data for all participants. This is the PHONE_SENSOR[TABLE] key in config.yaml
#' @param columns the columns needed from this sensor (we recommend to only return these columns instead of every column in sensor_container)
#' @return A dataframe with the sensor data for device
pull_data <- function(stream_parameters, device, sensor, sensor_container, columns){
dbEngine <- get_db_engine(stream_parameters$DATABASE_GROUP)
query <- paste0("SELECT ", paste(columns, collapse = ",")," FROM ", sensor_container, " WHERE ",columns$DEVICE_ID," = '", device,"'")
# Letting the user know what we are doing
message(paste0("Executing the following query to download data: ", query))
sensor_data <- dbGetQuery(dbEngine, query)
dbDisconnect(dbEngine)
if(nrow(sensor_data) == 0)
warning(paste("The device '", device,"' did not have data in ", sensor_container))
return(sensor_data)
}

83
src/data/streams/fitbitparsed_mysql/format.yaml 100644
View File

@ -0,0 +1,83 @@
FITBIT_HEARTRATE_SUMMARY:
RAPIDS_COLUMN_MAPPINGS:
TIMESTAMP: FLAG_TO_MUTATE
DEVICE_ID: device_id
LOCAL_DATE_TIME: local_date_time
HEARTRATE_DAILY_RESTINGHR: heartrate_daily_restinghr
HEARTRATE_DAILY_CALORIESOUTOFRANGE: heartrate_daily_caloriesoutofrange
HEARTRATE_DAILY_CALORIESFATBURN: heartrate_daily_caloriesfatburn
HEARTRATE_DAILY_CALORIESCARDIO: heartrate_daily_caloriescardio
HEARTRATE_DAILY_CALORIESPEAK: heartrate_daily_caloriespeak
MUTATION:
COLUMN_MAPPINGS:
SCRIPTS: # List any python or r scripts that mutate your raw data
- src/data/streams/mutations/fitbit/add_zero_timestamp.py
FITBIT_HEARTRATE_INTRADAY:
RAPIDS_COLUMN_MAPPINGS:
TIMESTAMP: FLAG_TO_MUTATE
DEVICE_ID: device_id
LOCAL_DATE_TIME: local_date_time
HEARTRATE: heartrate
HEARTRATE_ZONE: heartrate_zone
MUTATION:
COLUMN_MAPPINGS:
SCRIPTS: # List any python or r scripts that mutate your raw data
- src/data/streams/mutations/fitbit/add_zero_timestamp.py
FITBIT_SLEEP_SUMMARY:
RAPIDS_COLUMN_MAPPINGS:
TIMESTAMP: FLAG_TO_MUTATE
DEVICE_ID: device_id
LOCAL_DATE_TIME: local_date_time
LOCAL_START_DATE_TIME: local_start_date_time
LOCAL_END_DATE_TIME: local_end_date_time
EFFICIENCY: efficiency
MINUTES_AFTER_WAKEUP: minutes_after_wakeup
MINUTES_ASLEEP: minutes_asleep
MINUTES_AWAKE: minutes_awake
MINUTES_TO_FALL_ASLEEP: minutes_to_fall_asleep
MINUTES_IN_BED: minutes_in_bed
IS_MAIN_SLEEP: is_main_sleep
TYPE: type
MUTATION:
COLUMN_MAPPINGS:
SCRIPTS: # List any python or r scripts that mutate your raw data
- src/data/streams/mutations/fitbit/add_zero_timestamp.py
FITBIT_SLEEP_INTRADAY:
RAPIDS_COLUMN_MAPPINGS:
TIMESTAMP: FLAG_TO_MUTATE
DEVICE_ID: device_id
LOCAL_DATE_TIME: local_date_time
TYPE_EPISODE_ID: type_episode_id
DURATION: duration
IS_MAIN_SLEEP: is_main_sleep
TYPE: type
LEVEL: level
MUTATION:
COLUMN_MAPPINGS:
SCRIPTS: # List any python or r scripts that mutate your raw data
- src/data/streams/mutations/fitbit/add_zero_timestamp.py
FITBIT_STEPS_SUMMARY:
RAPIDS_COLUMN_MAPPINGS:
TIMESTAMP: FLAG_TO_MUTATE
DEVICE_ID: device_id
LOCAL_DATE_TIME: local_date_time
STEPS: steps
MUTATION:
COLUMN_MAPPINGS:
SCRIPTS: # List any python or r scripts that mutate your raw data
- src/data/streams/mutations/fitbit/add_zero_timestamp.py
FITBIT_STEPS_INTRADAY:
RAPIDS_COLUMN_MAPPINGS:
TIMESTAMP: FLAG_TO_MUTATE
DEVICE_ID: device_id
LOCAL_DATE_TIME: local_date_time
STEPS: steps
MUTATION:
COLUMN_MAPPINGS:
SCRIPTS: # List any python or r scripts that mutate your raw data
- src/data/streams/mutations/fitbit/add_zero_timestamp.py

6
src/data/streams/mutations/fitbit/add_zero_timestamp.py 100644
View File

@ -0,0 +1,6 @@
import pandas as pd
def main(parsed_data, stream_parameters):
parsed_data["timestamp"] = 0 # this column is added at readable_datetime.R because we neeed to take into account multiple timezones
# parsed_data['local_date_time'] = parsed_data['local_date_time'].dt.strftime('%Y-%m-%d %H:%M:%S')
return(parsed_data)

9
tools/config.schema.yaml
View File

@ -839,6 +839,15 @@ properties:
SLEEP_SUMMARY_EPISODE_DAY_ANCHOR: SLEEP_SUMMARY_EPISODE_DAY_ANCHOR:
type: string type: string
enum: ["start", "end"] enum: ["start", "end"]
fitbitparsed_mysql:
type: object
required: [DATABASE_GROUP, SLEEP_SUMMARY_EPISODE_DAY_ANCHOR]
properties:
DATABASE_GROUP:
type: string
SLEEP_SUMMARY_EPISODE_DAY_ANCHOR:
type: string
enum: ["start", "end"]
FITBIT_DATA_YIELD: FITBIT_DATA_YIELD:
type: object type: object