diff --git a/docs/setup/configuration.md b/docs/setup/configuration.md index 366befc6..60a52e2a 100644 --- a/docs/setup/configuration.md +++ b/docs/setup/configuration.md @@ -454,25 +454,7 @@ Modify the following keys in your `config.yaml` depending on the [data stream](. |---------------------|----------------------------------------------------------------------------------------------------------------------------| | `[DATABASE_GROUP]` | A database credentials group. Read the instructions below to set it up | - ??? info "Setting up a DATABASE_GROUP and its connection credentials" - - 1. If you haven't done so, create an empty file called `#!bash .env` in your RAPIDS root directory: `./.env` - 2. Add the following lines to `./.env` and replace your database-specific credentials (user, password, host, and database): - 1. Note that the label `[MY_GROUP]` is arbitrary but it has to match `[PHONE_DATA_STREAMS][aware_mysql] [DATABASE_GROUP]` - - ``` yaml - [MY_GROUP] - user=MY_USER - password=MY_PASSWORD - host=MY_HOST - port=3306 - database=MY_DATABASE - ``` - - ??? hint "Connecting to localhost (host machine) from inside our docker container" - If you are using RAPIDS' docker container and Docker-for-mac or Docker-for-Windows 18.03+, you can connect to a MySQL database in your host machine using `host.docker.internal` instead of `127.0.0.1` or `localhost`. In a Linux host you need to run our docker container using `docker run --network="host" -d moshiresearch/rapids:latest` and then `127.0.0.1` will point to your host machine. - --- - + --8<---- "docs/snippets/database.md" === "aware_csv" @@ -522,24 +504,7 @@ Modify the following keys in your `config.yaml` depending on the [data stream](. | `[DATABASE_GROUP]` | A database credentials group. Read the instructions below to set it up | | `[COLUMN_MAPPINGS_READY]` | Set this to `True` after you have modified this stream's `format.yaml` column mappings to match your raw data column names: [`fitbitjson_mysql`](../../datastreams/fitbitjson-mysql#format) | - ??? info "Setting up a DATABASE_GROUP and its connection credentials" - - 1. If you haven't done so, create an empty file called `#!bash .env` in your RAPIDS root directory: `./.env` - 2. Add the following lines to `./.env` and replace your database-specific credentials (user, password, host, and database): - 1. Note that the label `[MY_GROUP]` is arbitrary but it has to match `[FITBIT_DATA_STREAMS][fitbitjson_mysql] [DATABASE_GROUP]` - - ``` yaml - [MY_GROUP] - user=MY_USER - password=MY_PASSWORD - host=MY_HOST - port=3306 - database=MY_DATABASE - ``` - - ??? hint "Connecting to localhost (host machine) from inside our docker container" - If you are using RAPIDS' docker container and Docker-for-mac or Docker-for-Windows 18.03+, you can connect to a MySQL database in your host machine using `host.docker.internal` instead of `127.0.0.1` or `localhost`. In a Linux host you need to run our docker container using `docker run --network="host" -d moshiresearch/rapids:latest` and then `127.0.0.1` will point to your host machine. - --- + --8<---- "docs/snippets/database.md" === "fitbitjson_csv" @@ -561,24 +526,7 @@ Modify the following keys in your `config.yaml` depending on the [data stream](. | `[DATABASE_GROUP]` | A database credentials group. Read the instructions below to set it up | | `[COLUMN_MAPPINGS_READY]` | Set this to `True` after you have modified this stream's `format.yaml` column mappings to match your raw data column names: [`fitbitparsed_mysql`](../../datastreams/fitbitparsed-mysql#format) | - ??? info "Setting up a DATABASE_GROUP and its connection credentials" - - 1. If you haven't done so, create an empty file called `#!bash .env` in your RAPIDS root directory: `./.env` - 2. Add the following lines to `./.env` and replace your database-specific credentials (user, password, host, and database): - 1. Note that the label `[MY_GROUP]` is arbitrary but it has to match `[FITBIT_DATA_STREAMS][fitbitparsed_mysql] [DATABASE_GROUP]` - - ``` yaml - [MY_GROUP] - user=MY_USER - password=MY_PASSWORD - host=MY_HOST - port=3306 - database=MY_DATABASE - ``` - - ??? hint "Connecting to localhost (host machine) from inside our docker container" - If you are using RAPIDS' docker container and Docker-for-mac or Docker-for-Windows 18.03+, you can connect to a MySQL database in your host machine using `host.docker.internal` instead of `127.0.0.1` or `localhost`. In a Linux host you need to run our docker container using `docker run --network="host" -d moshiresearch/rapids:latest` and then `127.0.0.1` will point to your host machine. - --- + --8<---- "docs/snippets/database.md" === "fitbitparsed_csv" diff --git a/docs/snippets/database.md b/docs/snippets/database.md new file mode 100644 index 00000000..e35d6915 --- /dev/null +++ b/docs/snippets/database.md @@ -0,0 +1,58 @@ +??? info "Setting up a DATABASE_GROUP and its connection credentials." + + 1. If you haven't done so, create an empty file called `#!bash credentials.yaml` in your RAPIDS root directory: + + 2. Add the following lines to `credentials.yaml` and replace your database-specific credentials (user, password, host, and database): + + ``` yaml + MY_GROUP: + database: MY_DATABASE + host: MY_HOST + password: MY_PASSWORD + port: 3306 + user: MY_USER + ``` + + 1. Notes + + 1. The label `[MY_GROUP]` is arbitrary but it has to match the `[DATABASE_GROUP]` attribute of the data stream you choose to use. + + 2. Indentation matters + + 3. You can have more than one credentials group in `credentials.yaml` + + ??? hint "Upgrading from `./.env` from RAPIDS 0.x" + In RAPIDS versions 0.x, database credentials were stored in a `./.env` file. If you are migrating from that type of file, you have two options: + + 1. Migrate your credentials by hand: + + === "change .env format" + + ``` yaml + [MY_GROUP] + user=MY_USER + password=MY_PASSWORD + host=MY_HOST + port=3306 + database=MY_DATABASE + ``` + + === "to credentials.yaml format" + + ``` yaml + MY_GROUP: + user: MY_USER + password: MY_PASSWORD + host: MY_HOST + port: 3306 + database: MY_DATABASE + ``` + + 2. Use the migration script we provide (make sure your conda environment is active): + + ```python + python tools/update_format_env.py + ``` + + ??? hint "Connecting to localhost (host machine) from inside our docker container." + If you are using RAPIDS' docker container and Docker-for-mac or Docker-for-Windows 18.03+, you can connect to a MySQL database in your host machine using `host.docker.internal` instead of `127.0.0.1` or `localhost`. In a Linux host, you need to run our docker container using `docker run --network="host" -d moshiresearch/rapids:latest` and then `127.0.0.1` will point to your host machine. \ No newline at end of file diff --git a/src/data/streams/aware_mysql/container.R b/src/data/streams/aware_mysql/container.R index 83942d20..b635a6b6 100644 --- a/src/data/streams/aware_mysql/container.R +++ b/src/data/streams/aware_mysql/container.R @@ -1,5 +1,25 @@ # if you need a new package, you should add it with renv::install(package) so your renv venv is updated library(RMariaDB) +library(yaml) + +#' @description +#' Auxiliary function to parse the connection credentials from a specifc group in ./credentials.yaml +#' You can reause most of this function if you are connection to a DB or Web API. +#' It's OK to delete this function if you don't need credentials, e.g., you are pulling data from a CSV for example. +#' @param group the yaml key containing the credentials to connect to a database +#' @preturn dbEngine a database engine (connection) ready to perform queries +get_db_engine <- function(group){ + # The working dir is aways RAPIDS root folder, so your credentials file is always /credentials.yaml + credentials <- read_yaml("./credentials.yaml") + if(!group %in% names(credentials)) + stop(paste("The credentials group",group, "does not exist in ./credentials.yaml. The only groups that exist in that file are:", paste(names(credentials), collapse = ","))) + dbEngine <- dbConnect(MariaDB(), db = credentials[[group]][["database"]], + username = credentials[[group]][["user"]], + password = credentials[[group]][["password"]], + host = credentials[[group]][["host"]], + port = credentials[[group]][["port"]]) + return(dbEngine) +} # This file gets executed for each PHONE_SENSOR of each participant # If you are connecting to a database the env file containing its credentials is available at "./.env" @@ -18,9 +38,7 @@ library(RMariaDB) #' @return The OS the device ran, "android" or "ios" infer_device_os <- function(data_configuration, device){ - group <- data_configuration$SOURCE$DATABASE_GROUP # specified DB credentials group in config.yaml - - dbEngine <- dbConnect(MariaDB(), default.file = "./.env", group = group) + dbEngine <- get_db_engine(data_configuration$SOURCE$DATABASE_GROUP) query <- paste0("SELECT device_id,brand FROM aware_device WHERE device_id = '", device, "'") message(paste0("Executing the following query to infer phone OS: ", query)) os <- dbGetQuery(dbEngine, query) @@ -44,10 +62,7 @@ infer_device_os <- function(data_configuration, device){ #' @return A dataframe with the sensor data for device download_data <- function(data_configuration, device, sensor_container, columns){ - group <- data_configuration$SOURCE$DATABASE_GROUP - dbEngine <- dbConnect(MariaDB(), default.file = "./.env", group = group) - - + dbEngine <- get_db_engine(data_configuration$SOURCE$DATABASE_GROUP) query <- paste0("SELECT ", paste(columns, collapse = ",")," FROM ", sensor_container, " WHERE device_id = '", device,"'") # Letting the user know what we are doing message(paste0("Executing the following query to download data: ", query)) diff --git a/tools/update_format_env.py b/tools/update_format_env.py new file mode 100644 index 00000000..66668d2b --- /dev/null +++ b/tools/update_format_env.py @@ -0,0 +1,20 @@ +import configparser +import yaml +from pathlib import Path + +if not Path('./.env').exists(): + print("There is no .env file in RAPIDS root directory, nothing to migrate") +else: + config = configparser.ConfigParser() + config.read('./.env') + + credentials = dict() + for section in config.sections(): + credentials[section] = dict() + for attribute in config[section]: + credentials[section][attribute] = config[section][attribute] + + with open('./credentials.yaml', 'w') as f: + data = yaml.dump(credentials, f) + + print("Migration complete. Your credentials stored in .env should now be in credentials.yaml") \ No newline at end of file