diff --git a/config.yaml b/config.yaml index 65a6d52a..09ed16aa 100644 --- a/config.yaml +++ b/config.yaml @@ -341,6 +341,11 @@ FITBIT_DATA_STREAMS: fitbitjson_mysql: DATABASE_GROUP: MY_GROUP SLEEP_SUMMARY_EPISODE_DAY_ANCHOR: end # summary sleep episodes are considered as events based on either the start timestamp or end timestamp. + + fitbitparsed_mysql: + DATABASE_GROUP: MY_GROUP + SLEEP_SUMMARY_EPISODE_DAY_ANCHOR: end # summary sleep episodes are considered as events based on either the start timestamp or end timestamp. + # Sensors ------ diff --git a/docs/datastreams/data-streams-introduction.md b/docs/datastreams/data-streams-introduction.md index 7ab75461..36b69d2a 100644 --- a/docs/datastreams/data-streams-introduction.md +++ b/docs/datastreams/data-streams-introduction.md @@ -9,7 +9,7 @@ If you want to process a data stream using RAPIDS, make sure that your data is s If RAPIDS doesn't support your data stream yet (e.g. Beiwe data stored in PostgreSQL, or AWARE data stored in InfluxDB), you can always [implement a new data stream](../add-new-data-streams). If it's something you think other people might be interested on, we will be happy to include your new data stream in RAPIDS, so get in touch!. !!! hint - You can only add new data streams for Smartphone or Fitbit data. If you need RAPIDS to process data from **different devices**, like Oura Rings or Actigraph wearables, get in touch. It is a more complex process that could take a few days to implement for someone familiar with R or Python but that we would be happy to work on together. + Currently, you can add new data streams for smartphones, Fitbit, and Empatica devices. If you need RAPIDS to process data from **other devices**, like Oura Rings or Actigraph wearables, get in touch. It is a more complicated process that could take a couple of days to implement for someone familiar with R or Python, but we would be happy to work on it together. For reference, these are the data streams we currently support: diff --git a/docs/datastreams/fitbitjson-mysql.md b/docs/datastreams/fitbitjson-mysql.md index e0e0ad78..86dc7ab0 100644 --- a/docs/datastreams/fitbitjson-mysql.md +++ b/docs/datastreams/fitbitjson-mysql.md @@ -2,7 +2,7 @@ This [data stream](../../datastreams/data-streams-introduction) handles Fitbit sensor data downloaded using the [Fitbit Web API](https://dev.fitbit.com/build/reference/web-api/) and stored in a MySQL database. Please note that RAPIDS cannot query the API directly; you need to use other available tools or implement your own. Once you have your sensor data in a MySQL database, RAPIDS can process it. ## Container -The container should be a MySQL database with a table per sensor, each containing the data for all participants. +The container should be a MySQL database with a table per sensor, each containing all participants' data. The script to connect and download data from this container is at: ```bash @@ -17,7 +17,7 @@ The `format.yaml` maps and transforms columns in your raw data stream to the [ma src/data/streams/fitbitjson_mysql/format.yaml ``` -If you want RAPIDS to process Fitbit sensor data using this stream, you will need to replace `[RAPIDS_COLUMN_MAPPINGS]`/`[MUTATION][COLUMN_MAPPINGS]` inside **each sensor** section in `format.yaml` to match your raw data column names: +If you want RAPIDS to process Fitbit sensor data using this stream, you will need to map `DEVICE_ID` and `JSON_FITBIT_COLUMN` to your own raw data columns inside **each sensor** section in `format.yaml`. ??? info "FITBIT_HEARTRATE_SUMMARY" @@ -52,7 +52,7 @@ If you want RAPIDS to process Fitbit sensor data using this stream, you will nee All columns except `DEVICE_ID` are parsed from `JSON_FITBIT_COLUMN`. `JSON_FITBIT_COLUMN` is a string column containing the JSON objects returned by Fitbit's API. See an example of the raw data RAPIDS expects for this data stream: - ??? "Example of the raw data RAPIDS expects for this data stream" + ??? example "Example of the raw data RAPIDS expects for this data stream" |device_id |fitbit_data | |---------------------------------------- |--------------------------------------------------------- | @@ -90,7 +90,7 @@ If you want RAPIDS to process Fitbit sensor data using this stream, you will nee All columns except `DEVICE_ID` are parsed from `JSON_FITBIT_COLUMN`. `JSON_FITBIT_COLUMN` is a string column containing the JSON objects returned by Fitbit's API. See an example of the raw data RAPIDS expects for this data stream: - ??? "Example of the raw data RAPIDS expects for this data stream" + ??? example "Example of the raw data RAPIDS expects for this data stream" |device_id |fitbit_data | |---------------------------------------- |--------------------------------------------------------- | diff --git a/docs/datastreams/fitbitparsed-mysql.md b/docs/datastreams/fitbitparsed-mysql.md index 86646404..a7fa94dc 100644 --- a/docs/datastreams/fitbitparsed-mysql.md +++ b/docs/datastreams/fitbitparsed-mysql.md @@ -1,15 +1,15 @@ # `fitbitparsed_mysql` -This [data stream](../../datastreams/data-streams-introduction) handles Fitbit sensor data downloaded using the [Fitbit Web API](https://dev.fitbit.com/build/reference/web-api/), **parsed**, and stored in a MySQL database. Please note that RAPIDS cannot query the API directly, you need to use other available tools or implement your own. Once you have your sensor data in a MySQL database, RAPIDS can process it. +This [data stream](../../datastreams/data-streams-introduction) handles Fitbit sensor data downloaded using the [Fitbit Web API](https://dev.fitbit.com/build/reference/web-api/), **parsed**, and stored in a MySQL database. Please note that RAPIDS cannot query the API directly; you need to use other available tools or implement your own. Once you have your parsed sensor data in a MySQL database, RAPIDS can process it. !!! info "What is the difference between JSON and plain data streams" - Most people will only need `fitbitjson_mysql` because they downloaded and stored their data directly from Fitbit's API. However, if for some reason you don't have access to that JSON data and instead only have the parsed data (columns and rows) you can use this data stream. + Most people will only need `fitbitjson_mysql` because they downloaded and stored their data directly from Fitbit's API. However, if, for some reason, you don't have access to that JSON data and instead only have the parsed data (columns and rows), you can use this data stream. ## Container -A MySQL database with a table per sensor, each containing the data for all participants. +The container should be a MySQL database with a table per sensor, each containing all participants' data. The script to connect and download data from this container is at: ```bash -src/data/streams/fitbitjson_mysql/container.R +src/data/streams/fitbitparsed_mysql/container.R ``` ## Format @@ -20,33 +20,229 @@ The `format.yaml` maps and transforms columns in your raw data stream to the [ma src/data/streams/fitbitparsed_mysql/format.yaml ``` -If you want RAPIDS to process Fitbit sensor data using this stream, you will need to replace any `RAPIDS_COLUMN_MAPPINGS` inside **each sensor** section in `format.yaml` to match your raw data column names. +If you want to use this stream with your data, modify every sensor in `format.yaml` to map all columns except `TIMESTAMP` in `[RAPIDS_COLUMN_MAPPINGS]` to your raw data column names. -All columns are mandatory, however, all except `device_id` and `local_date_time` can be empty if you don't have that data. Just have in mind that some features will be empty if some of these columns are empty. +All columns are mandatory; however, all except `device_id` and `local_date_time` can be empty if you don't have that data. Just have in mind that some features will be empty if some of these columns are empty. -??? info "FITBIT_HEARTRATE_SUMMARY section" - +??? info "FITBIT_HEARTRATE_SUMMARY" + **RAPIDS_COLUMN_MAPPINGS** | RAPIDS column | Stream column | |-----------------|-----------------| - | LOCAL_DATE_TIME | local_date_time | - | DEVICE_ID | device_id | + | TIMESTAMP| FLAG_TO_MUTATE | + | LOCAL_DATE_TIME | local_date_time | + | DEVICE_ID | device_id | | HEARTRATE_DAILY_RESTINGHR | heartrate_daily_restinghr | | HEARTRATE_DAILY_CALORIESOUTOFRANGE | heartrate_daily_caloriesoutofrange | | HEARTRATE_DAILY_CALORIESFATBURN | heartrate_daily_caloriesfatburn | | HEARTRATE_DAILY_CALORIESCARDIO | heartrate_daily_caloriescardio | | HEARTRATE_DAILY_CALORIESPEAK | heartrate_daily_caloriespeak | - **MUTATION_SCRIPTS** - TODO list our parsing script + **MUTATION** - ??? "Example of the raw data RAPIDS expects for this data stream" + - **COLUMN_MAPPINGS** (None) - |device_id |local_date_time |heartrate_daily_restinghr |heartrate_daily_caloriesoutofrange |heartrate_daily_caloriesfatburn |heartrate_daily_caloriescardio |heartrate_daily_caloriespeak | - |-------------------------------------- |----------------- |------- |-------------- |------------- |------------ |-------| - |a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-07 |72 |1200.6102 |760.3020 |15.2048 |0 | - |a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-08 |70 |1100.1120 |660.0012 |23.7088 |0 | - |a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-09 |69 |750.3615 |734.1516 |131.8579 |0 | + - **SCRIPTS** + + ```bash + src/data/streams/mutations/fitbit/add_zero_timestamp.py + ``` + + !!! note + `add_zero_timestamp` adds an all-zero column called `timestamp` that will be filled in later in the pipeline by `readable_time.R` converting LOCAL_DATE_TIME to a unix timestamp taking into account single or multiple time zones. + + ??? example "Example of the raw data RAPIDS expects for this data stream" + + |device_id |local_date_time |heartrate_daily_restinghr |heartrate_daily_caloriesoutofrange |heartrate_daily_caloriesfatburn |heartrate_daily_caloriescardio |heartrate_daily_caloriespeak | + |-------------------------------------- |----------------- |------- |-------------- |------------- |------------ |-------| + |a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-07 |72 |1200.6102 |760.3020 |15.2048 |0 | + |a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-08 |70 |1100.1120 |660.0012 |23.7088 |0 | + |a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-09 |69 |750.3615 |734.1516 |131.8579 |0 | + +??? info "FITBIT_HEARTRATE_INTRADAY" + + **RAPIDS_COLUMN_MAPPINGS** + + | RAPIDS column | Stream column | + |-----------------|-----------------| + | TIMESTAMP| FLAG_TO_MUTATE | + | LOCAL_DATE_TIME | local_date_time | + | DEVICE_ID | device_id | + | HEARTRATE | heartrate | + | HEARTRATE_ZONE | heartrate_zone | + + + **MUTATION** + + - **COLUMN_MAPPINGS** (None) + + - **SCRIPTS** + + ```bash + src/data/streams/mutations/fitbit/add_zero_timestamp.py + ``` + + !!! note + `add_zero_timestamp` adds an all-zero column called `timestamp` that will be filled in later in the pipeline by `readable_time.R` converting LOCAL_DATE_TIME to a unix timestamp taking into account single or multiple time zones. + + ??? example "Example of the raw data RAPIDS expects for this data stream" + + |device_id |local_date_time |heartrate |heartrate_zone | + |-------------------------------------- |---------------------- |--------- |--------------- | + |a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-07 00:00:00 |68 |outofrange | + |a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-07 00:01:00 |67 |outofrange | + |a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-07 00:02:00 |67 |outofrange | + +??? info "FITBIT_SLEEP_SUMMARY" + + **RAPIDS_COLUMN_MAPPINGS** + + | RAPIDS column | Stream column | + |-----------------|-----------------| + | TIMESTAMP| FLAG_TO_MUTATE | + | LOCAL_DATE_TIME| local_date_time | + | LOCAL_START_DATE_TIME| local_start_date_time | + | LOCAL_END_DATE_TIME| local_end_date_time | + | DEVICE_ID| device_id | + | EFFICIENCY| efficiency | + | MINUTES_AFTER_WAKEUP| minutes_after_wakeup | + | MINUTES_ASLEEP| minutes_asleep | + | MINUTES_AWAKE| minutes_awake | + | MINUTES_TO_FALL_ASLEEP| minutes_to_fall_asleep | + | MINUTES_IN_BED| minutes_in_bed | + | IS_MAIN_SLEEP| is_main_sleep | + | TYPE| type | + + **MUTATION** + + - **COLUMN_MAPPINGS** (None) + + - **SCRIPTS** + + ```bash + src/data/streams/mutations/fitbit/add_zero_timestamp.py + ``` + + !!! note + `add_zero_timestamp` adds an all-zero column called `timestamp` that will be filled in later in the pipeline by `readable_time.R` converting LOCAL_DATE_TIME to a unix timestamp taking into account single or multiple time zones. + + Fitbit API has two versions for sleep data, v1 and v1.2. We support both but ignore v1's `count_awake`, `duration_awake`, and `count_awakenings`, `count_restless`, `duration_restless` columns. + + ??? example "Example of the expected raw data" + + |device_id |local_start_date_time |local_end_date_time |efficiency |minutes_after_wakeup |minutes_asleep |minutes_awake |minutes_to_fall_asleep |minutes_in_bed |is_main_sleep |type | + |-------------------------------------- |---------------------- |---------------------- |----------- |--------------------- |--------------- |-------------- |----------------------- |--------------- |-------------- |-------- | + |a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-10 15:36:30 |2020-10-10 16:37:00 |92 |0 |55 |5 |0 |60 |0 |classic | + |a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-10 01:46:30 |2020-10-10 08:10:00 |88 |0 |318 |65 |0 |383 |1 |stages | + |a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-11 00:12:30 |2020-10-11 11:47:00 |89 |1 |562 |132 |0 |694 |1 |stages | + |a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-12 01:31:00 |2020-10-12 09:34:30 |93 |0 |415 |68 |0 |483 |1 |stages | + + +??? info "FITBIT_SLEEP_INTRADAY" + + **RAPIDS_COLUMN_MAPPINGS** + + | RAPIDS column | Stream column | + |-----------------|-----------------| + | TIMESTAMP | FLAG_TO_MUTATE | + | LOCAL_DATE_TIME | local_date_time | + | DEVICE_ID | device_id | + | TYPE_EPISODE_ID | type_episode_id | + | DURATION | duration | + | IS_MAIN_SLEEP | is_main_sleep | + | TYPE | type | + | LEVEL | level | + + **MUTATION** + + - **COLUMN_MAPPINGS** (None) + + - **SCRIPTS** + + ```bash + src/data/streams/mutations/fitbit/add_zero_timestamp.py + ``` + + !!! note + `add_zero_timestamp` adds an all-zero column called `timestamp` that will be filled in later in the pipeline by `readable_time.R` converting LOCAL_DATE_TIME to a unix timestamp taking into account single or multiple time zones. + + Fitbit API has two versions for sleep data, v1 and v1.2, we support both. + + ??? example "Example of the expected raw data" + + |device_id |type_episode_id |local_date_time |duration |level |is_main_sleep |type | + |------------------------------------ |---------------- |------------------- |--------- |---------- |-------------- |-------------- | + |a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |0 |2020-10-10 15:36:30 |60 |restless |0 |classic | + |a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |0 |2020-10-10 15:37:30 |660 |asleep |0 |classic | + |a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |0 |2020-10-10 15:48:30 |60 |restless |0 |classic | + |a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |... |... |... |... |... |... | + |a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |1 |2020-10-10 01:46:30 |420 |light |1 |stages | + |a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |1 |2020-10-10 01:53:30 |1230 |deep |1 |stages | + +??? info "FITBIT_STEPS_SUMMARY" + + **RAPIDS_COLUMN_MAPPINGS** + + | RAPIDS column | Stream column | + |-----------------|-----------------| + | TIMESTAMP | FLAG_TO_MUTATE | + | DEVICE_ID | device_id | + | LOCAL_DATE_TIME | local_date_time | + | STEPS | steps | + + **MUTATION** + + - **COLUMN_MAPPINGS** (None) + + - **SCRIPTS** + + ```bash + src/data/streams/mutations/fitbit/add_zero_timestamp.py + ``` + + !!! note + `add_zero_timestamp` adds an all-zero column called `timestamp` that will be filled in later in the pipeline by `readable_time.R` converting LOCAL_DATE_TIME to a unix timestamp taking into account single or multiple time zones. + + ??? example "Example of the expected raw data" + + |device_id |local_date_time |steps | + |-------------------------------------- |---------------------- |--------- | + |a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-07 |1775 | + |a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-08 |3201 | + |a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-09 |998 | + +??? info "FITBIT_STEPS_INTRADAY" + + **RAPIDS_COLUMN_MAPPINGS** + + | RAPIDS column | Stream column | + |-----------------|-----------------| + | TIMESTAMP | FLAG_TO_MUTATE | + | DEVICE_ID | device_id | + | LOCAL_DATE_TIME | local_date_time | + | STEPS | steps | + + **MUTATION** + + - **COLUMN_MAPPINGS** (None) + + - **SCRIPTS** + + ```bash + src/data/streams/mutations/fitbit/add_zero_timestamp.py + ``` + + !!! note + `add_zero_timestamp` adds an all-zero column called `timestamp` that will be filled in later in the pipeline by `readable_time.R` converting LOCAL_DATE_TIME to a unix timestamp taking into account single or multiple time zones. + + ??? example "Example of the expected raw data" + + |device_id |local_date_time |steps | + |-------------------------------------- |---------------------- |--------- | + |a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-07 00:00:00 |5 | + |a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-07 00:01:00 |3 | + |a748ee1a-1d0b-4ae9-9074-279a2b6ba524 |2020-10-07 00:02:00 |0 | + diff --git a/src/data/streams/fitbitparsed_mysql/container.R b/src/data/streams/fitbitparsed_mysql/container.R new file mode 100644 index 00000000..98f8ccb5 --- /dev/null +++ b/src/data/streams/fitbitparsed_mysql/container.R @@ -0,0 +1,47 @@ +# if you need a new package, you should add it with renv::install(package) so your renv venv is updated +library(RMariaDB) +library(yaml) + +#' @description +#' Auxiliary function to parse the connection credentials from a specifc group in ./credentials.yaml +#' You can reause most of this function if you are connection to a DB or Web API. +#' It's OK to delete this function if you don't need credentials, e.g., you are pulling data from a CSV for example. +#' @param group the yaml key containing the credentials to connect to a database +#' @preturn dbEngine a database engine (connection) ready to perform queries +get_db_engine <- function(group){ + # The working dir is aways RAPIDS root folder, so your credentials file is always /credentials.yaml + credentials <- read_yaml("./credentials.yaml") + if(!group %in% names(credentials)) + stop(paste("The credentials group",group, "does not exist in ./credentials.yaml. The only groups that exist in that file are:", paste(names(credentials), collapse = ","))) + dbEngine <- dbConnect(MariaDB(), db = credentials[[group]][["database"]], + username = credentials[[group]][["user"]], + password = credentials[[group]][["password"]], + host = credentials[[group]][["host"]], + port = credentials[[group]][["port"]]) + return(dbEngine) +} + +#' @description +#' Gets the sensor data for a specific device id from a database table, file or whatever source you want to query +#' +#' @param stream_parameters The PHONE_STREAM_PARAMETERS key in config.yaml. If you need specific parameters add them there. +#' @param device A device ID string +#' @param sensor_container database table or file containing the sensor data for all participants. This is the PHONE_SENSOR[TABLE] key in config.yaml +#' @param columns the columns needed from this sensor (we recommend to only return these columns instead of every column in sensor_container) +#' @return A dataframe with the sensor data for device + +pull_data <- function(stream_parameters, device, sensor, sensor_container, columns){ + dbEngine <- get_db_engine(stream_parameters$DATABASE_GROUP) + query <- paste0("SELECT ", paste(columns, collapse = ",")," FROM ", sensor_container, " WHERE ",columns$DEVICE_ID," = '", device,"'") + # Letting the user know what we are doing + message(paste0("Executing the following query to download data: ", query)) + sensor_data <- dbGetQuery(dbEngine, query) + + dbDisconnect(dbEngine) + + if(nrow(sensor_data) == 0) + warning(paste("The device '", device,"' did not have data in ", sensor_container)) + + return(sensor_data) +} + diff --git a/src/data/streams/fitbitparsed_mysql/format.yaml b/src/data/streams/fitbitparsed_mysql/format.yaml new file mode 100644 index 00000000..ecaaab11 --- /dev/null +++ b/src/data/streams/fitbitparsed_mysql/format.yaml @@ -0,0 +1,83 @@ +FITBIT_HEARTRATE_SUMMARY: + RAPIDS_COLUMN_MAPPINGS: + TIMESTAMP: FLAG_TO_MUTATE + DEVICE_ID: device_id + LOCAL_DATE_TIME: local_date_time + HEARTRATE_DAILY_RESTINGHR: heartrate_daily_restinghr + HEARTRATE_DAILY_CALORIESOUTOFRANGE: heartrate_daily_caloriesoutofrange + HEARTRATE_DAILY_CALORIESFATBURN: heartrate_daily_caloriesfatburn + HEARTRATE_DAILY_CALORIESCARDIO: heartrate_daily_caloriescardio + HEARTRATE_DAILY_CALORIESPEAK: heartrate_daily_caloriespeak + MUTATION: + COLUMN_MAPPINGS: + SCRIPTS: # List any python or r scripts that mutate your raw data + - src/data/streams/mutations/fitbit/add_zero_timestamp.py + +FITBIT_HEARTRATE_INTRADAY: + RAPIDS_COLUMN_MAPPINGS: + TIMESTAMP: FLAG_TO_MUTATE + DEVICE_ID: device_id + LOCAL_DATE_TIME: local_date_time + HEARTRATE: heartrate + HEARTRATE_ZONE: heartrate_zone + MUTATION: + COLUMN_MAPPINGS: + SCRIPTS: # List any python or r scripts that mutate your raw data + - src/data/streams/mutations/fitbit/add_zero_timestamp.py + +FITBIT_SLEEP_SUMMARY: + RAPIDS_COLUMN_MAPPINGS: + TIMESTAMP: FLAG_TO_MUTATE + DEVICE_ID: device_id + LOCAL_DATE_TIME: local_date_time + LOCAL_START_DATE_TIME: local_start_date_time + LOCAL_END_DATE_TIME: local_end_date_time + EFFICIENCY: efficiency + MINUTES_AFTER_WAKEUP: minutes_after_wakeup + MINUTES_ASLEEP: minutes_asleep + MINUTES_AWAKE: minutes_awake + MINUTES_TO_FALL_ASLEEP: minutes_to_fall_asleep + MINUTES_IN_BED: minutes_in_bed + IS_MAIN_SLEEP: is_main_sleep + TYPE: type + MUTATION: + COLUMN_MAPPINGS: + SCRIPTS: # List any python or r scripts that mutate your raw data + - src/data/streams/mutations/fitbit/add_zero_timestamp.py + +FITBIT_SLEEP_INTRADAY: + RAPIDS_COLUMN_MAPPINGS: + TIMESTAMP: FLAG_TO_MUTATE + DEVICE_ID: device_id + LOCAL_DATE_TIME: local_date_time + TYPE_EPISODE_ID: type_episode_id + DURATION: duration + IS_MAIN_SLEEP: is_main_sleep + TYPE: type + LEVEL: level + MUTATION: + COLUMN_MAPPINGS: + SCRIPTS: # List any python or r scripts that mutate your raw data + - src/data/streams/mutations/fitbit/add_zero_timestamp.py + +FITBIT_STEPS_SUMMARY: + RAPIDS_COLUMN_MAPPINGS: + TIMESTAMP: FLAG_TO_MUTATE + DEVICE_ID: device_id + LOCAL_DATE_TIME: local_date_time + STEPS: steps + MUTATION: + COLUMN_MAPPINGS: + SCRIPTS: # List any python or r scripts that mutate your raw data + - src/data/streams/mutations/fitbit/add_zero_timestamp.py + +FITBIT_STEPS_INTRADAY: + RAPIDS_COLUMN_MAPPINGS: + TIMESTAMP: FLAG_TO_MUTATE + DEVICE_ID: device_id + LOCAL_DATE_TIME: local_date_time + STEPS: steps + MUTATION: + COLUMN_MAPPINGS: + SCRIPTS: # List any python or r scripts that mutate your raw data + - src/data/streams/mutations/fitbit/add_zero_timestamp.py diff --git a/src/data/streams/mutations/fitbit/add_zero_timestamp.py b/src/data/streams/mutations/fitbit/add_zero_timestamp.py new file mode 100644 index 00000000..5d4a4490 --- /dev/null +++ b/src/data/streams/mutations/fitbit/add_zero_timestamp.py @@ -0,0 +1,6 @@ +import pandas as pd + +def main(parsed_data, stream_parameters): + parsed_data["timestamp"] = 0 # this column is added at readable_datetime.R because we neeed to take into account multiple timezones + # parsed_data['local_date_time'] = parsed_data['local_date_time'].dt.strftime('%Y-%m-%d %H:%M:%S') + return(parsed_data) diff --git a/tools/config.schema.yaml b/tools/config.schema.yaml index b6dedbc5..dadcd0db 100644 --- a/tools/config.schema.yaml +++ b/tools/config.schema.yaml @@ -839,6 +839,15 @@ properties: SLEEP_SUMMARY_EPISODE_DAY_ANCHOR: type: string enum: ["start", "end"] + fitbitparsed_mysql: + type: object + required: [DATABASE_GROUP, SLEEP_SUMMARY_EPISODE_DAY_ANCHOR] + properties: + DATABASE_GROUP: + type: string + SLEEP_SUMMARY_EPISODE_DAY_ANCHOR: + type: string + enum: ["start", "end"] FITBIT_DATA_YIELD: type: object