Replace .env with credentials.yaml

pull/128/head
JulioV 2021-03-06 23:19:32 -05:00
parent f65e3c8b1a
commit 4b33ee43ba
4 changed files with 103 additions and 62 deletions

View File

@ -454,25 +454,7 @@ Modify the following keys in your `config.yaml` depending on the [data stream](.
|---------------------|----------------------------------------------------------------------------------------------------------------------------| |---------------------|----------------------------------------------------------------------------------------------------------------------------|
| `[DATABASE_GROUP]` | A database credentials group. Read the instructions below to set it up | | `[DATABASE_GROUP]` | A database credentials group. Read the instructions below to set it up |
??? info "Setting up a DATABASE_GROUP and its connection credentials" --8<---- "docs/snippets/database.md"
1. If you haven't done so, create an empty file called `#!bash .env` in your RAPIDS root directory: `./.env`
2. Add the following lines to `./.env` and replace your database-specific credentials (user, password, host, and database):
1. Note that the label `[MY_GROUP]` is arbitrary but it has to match `[PHONE_DATA_STREAMS][aware_mysql] [DATABASE_GROUP]`
``` yaml
[MY_GROUP]
user=MY_USER
password=MY_PASSWORD
host=MY_HOST
port=3306
database=MY_DATABASE
```
??? hint "Connecting to localhost (host machine) from inside our docker container"
If you are using RAPIDS' docker container and Docker-for-mac or Docker-for-Windows 18.03+, you can connect to a MySQL database in your host machine using `host.docker.internal` instead of `127.0.0.1` or `localhost`. In a Linux host you need to run our docker container using `docker run --network="host" -d moshiresearch/rapids:latest` and then `127.0.0.1` will point to your host machine.
---
=== "aware_csv" === "aware_csv"
@ -522,24 +504,7 @@ Modify the following keys in your `config.yaml` depending on the [data stream](.
| `[DATABASE_GROUP]` | A database credentials group. Read the instructions below to set it up | | `[DATABASE_GROUP]` | A database credentials group. Read the instructions below to set it up |
| `[COLUMN_MAPPINGS_READY]` | Set this to `True` after you have modified this stream's `format.yaml` column mappings to match your raw data column names: [`fitbitjson_mysql`](../../datastreams/fitbitjson-mysql#format) | | `[COLUMN_MAPPINGS_READY]` | Set this to `True` after you have modified this stream's `format.yaml` column mappings to match your raw data column names: [`fitbitjson_mysql`](../../datastreams/fitbitjson-mysql#format) |
??? info "Setting up a DATABASE_GROUP and its connection credentials" --8<---- "docs/snippets/database.md"
1. If you haven't done so, create an empty file called `#!bash .env` in your RAPIDS root directory: `./.env`
2. Add the following lines to `./.env` and replace your database-specific credentials (user, password, host, and database):
1. Note that the label `[MY_GROUP]` is arbitrary but it has to match `[FITBIT_DATA_STREAMS][fitbitjson_mysql] [DATABASE_GROUP]`
``` yaml
[MY_GROUP]
user=MY_USER
password=MY_PASSWORD
host=MY_HOST
port=3306
database=MY_DATABASE
```
??? hint "Connecting to localhost (host machine) from inside our docker container"
If you are using RAPIDS' docker container and Docker-for-mac or Docker-for-Windows 18.03+, you can connect to a MySQL database in your host machine using `host.docker.internal` instead of `127.0.0.1` or `localhost`. In a Linux host you need to run our docker container using `docker run --network="host" -d moshiresearch/rapids:latest` and then `127.0.0.1` will point to your host machine.
---
=== "fitbitjson_csv" === "fitbitjson_csv"
@ -561,24 +526,7 @@ Modify the following keys in your `config.yaml` depending on the [data stream](.
| `[DATABASE_GROUP]` | A database credentials group. Read the instructions below to set it up | | `[DATABASE_GROUP]` | A database credentials group. Read the instructions below to set it up |
| `[COLUMN_MAPPINGS_READY]` | Set this to `True` after you have modified this stream's `format.yaml` column mappings to match your raw data column names: [`fitbitparsed_mysql`](../../datastreams/fitbitparsed-mysql#format) | | `[COLUMN_MAPPINGS_READY]` | Set this to `True` after you have modified this stream's `format.yaml` column mappings to match your raw data column names: [`fitbitparsed_mysql`](../../datastreams/fitbitparsed-mysql#format) |
??? info "Setting up a DATABASE_GROUP and its connection credentials" --8<---- "docs/snippets/database.md"
1. If you haven't done so, create an empty file called `#!bash .env` in your RAPIDS root directory: `./.env`
2. Add the following lines to `./.env` and replace your database-specific credentials (user, password, host, and database):
1. Note that the label `[MY_GROUP]` is arbitrary but it has to match `[FITBIT_DATA_STREAMS][fitbitparsed_mysql] [DATABASE_GROUP]`
``` yaml
[MY_GROUP]
user=MY_USER
password=MY_PASSWORD
host=MY_HOST
port=3306
database=MY_DATABASE
```
??? hint "Connecting to localhost (host machine) from inside our docker container"
If you are using RAPIDS' docker container and Docker-for-mac or Docker-for-Windows 18.03+, you can connect to a MySQL database in your host machine using `host.docker.internal` instead of `127.0.0.1` or `localhost`. In a Linux host you need to run our docker container using `docker run --network="host" -d moshiresearch/rapids:latest` and then `127.0.0.1` will point to your host machine.
---
=== "fitbitparsed_csv" === "fitbitparsed_csv"

View File

@ -0,0 +1,58 @@
??? info "Setting up a DATABASE_GROUP and its connection credentials."
1. If you haven't done so, create an empty file called `#!bash credentials.yaml` in your RAPIDS root directory:
2. Add the following lines to `credentials.yaml` and replace your database-specific credentials (user, password, host, and database):
``` yaml
MY_GROUP:
database: MY_DATABASE
host: MY_HOST
password: MY_PASSWORD
port: 3306
user: MY_USER
```
1. Notes
1. The label `[MY_GROUP]` is arbitrary but it has to match the `[DATABASE_GROUP]` attribute of the data stream you choose to use.
2. Indentation matters
3. You can have more than one credentials group in `credentials.yaml`
??? hint "Upgrading from `./.env` from RAPIDS 0.x"
In RAPIDS versions 0.x, database credentials were stored in a `./.env` file. If you are migrating from that type of file, you have two options:
1. Migrate your credentials by hand:
=== "change .env format"
``` yaml
[MY_GROUP]
user=MY_USER
password=MY_PASSWORD
host=MY_HOST
port=3306
database=MY_DATABASE
```
=== "to credentials.yaml format"
``` yaml
MY_GROUP:
user: MY_USER
password: MY_PASSWORD
host: MY_HOST
port: 3306
database: MY_DATABASE
```
2. Use the migration script we provide (make sure your conda environment is active):
```python
python tools/update_format_env.py
```
??? hint "Connecting to localhost (host machine) from inside our docker container."
If you are using RAPIDS' docker container and Docker-for-mac or Docker-for-Windows 18.03+, you can connect to a MySQL database in your host machine using `host.docker.internal` instead of `127.0.0.1` or `localhost`. In a Linux host, you need to run our docker container using `docker run --network="host" -d moshiresearch/rapids:latest` and then `127.0.0.1` will point to your host machine.

View File

@ -1,5 +1,25 @@
# if you need a new package, you should add it with renv::install(package) so your renv venv is updated # if you need a new package, you should add it with renv::install(package) so your renv venv is updated
library(RMariaDB) library(RMariaDB)
library(yaml)
#' @description
#' Auxiliary function to parse the connection credentials from a specifc group in ./credentials.yaml
#' You can reause most of this function if you are connection to a DB or Web API.
#' It's OK to delete this function if you don't need credentials, e.g., you are pulling data from a CSV for example.
#' @param group the yaml key containing the credentials to connect to a database
#' @preturn dbEngine a database engine (connection) ready to perform queries
get_db_engine <- function(group){
# The working dir is aways RAPIDS root folder, so your credentials file is always /credentials.yaml
credentials <- read_yaml("./credentials.yaml")
if(!group %in% names(credentials))
stop(paste("The credentials group",group, "does not exist in ./credentials.yaml. The only groups that exist in that file are:", paste(names(credentials), collapse = ",")))
dbEngine <- dbConnect(MariaDB(), db = credentials[[group]][["database"]],
username = credentials[[group]][["user"]],
password = credentials[[group]][["password"]],
host = credentials[[group]][["host"]],
port = credentials[[group]][["port"]])
return(dbEngine)
}
# This file gets executed for each PHONE_SENSOR of each participant # This file gets executed for each PHONE_SENSOR of each participant
# If you are connecting to a database the env file containing its credentials is available at "./.env" # If you are connecting to a database the env file containing its credentials is available at "./.env"
@ -18,9 +38,7 @@ library(RMariaDB)
#' @return The OS the device ran, "android" or "ios" #' @return The OS the device ran, "android" or "ios"
infer_device_os <- function(data_configuration, device){ infer_device_os <- function(data_configuration, device){
group <- data_configuration$SOURCE$DATABASE_GROUP # specified DB credentials group in config.yaml dbEngine <- get_db_engine(data_configuration$SOURCE$DATABASE_GROUP)
dbEngine <- dbConnect(MariaDB(), default.file = "./.env", group = group)
query <- paste0("SELECT device_id,brand FROM aware_device WHERE device_id = '", device, "'") query <- paste0("SELECT device_id,brand FROM aware_device WHERE device_id = '", device, "'")
message(paste0("Executing the following query to infer phone OS: ", query)) message(paste0("Executing the following query to infer phone OS: ", query))
os <- dbGetQuery(dbEngine, query) os <- dbGetQuery(dbEngine, query)
@ -44,10 +62,7 @@ infer_device_os <- function(data_configuration, device){
#' @return A dataframe with the sensor data for device #' @return A dataframe with the sensor data for device
download_data <- function(data_configuration, device, sensor_container, columns){ download_data <- function(data_configuration, device, sensor_container, columns){
group <- data_configuration$SOURCE$DATABASE_GROUP dbEngine <- get_db_engine(data_configuration$SOURCE$DATABASE_GROUP)
dbEngine <- dbConnect(MariaDB(), default.file = "./.env", group = group)
query <- paste0("SELECT ", paste(columns, collapse = ",")," FROM ", sensor_container, " WHERE device_id = '", device,"'") query <- paste0("SELECT ", paste(columns, collapse = ",")," FROM ", sensor_container, " WHERE device_id = '", device,"'")
# Letting the user know what we are doing # Letting the user know what we are doing
message(paste0("Executing the following query to download data: ", query)) message(paste0("Executing the following query to download data: ", query))

View File

@ -0,0 +1,20 @@
import configparser
import yaml
from pathlib import Path
if not Path('./.env').exists():
print("There is no .env file in RAPIDS root directory, nothing to migrate")
else:
config = configparser.ConfigParser()
config.read('./.env')
credentials = dict()
for section in config.sections():
credentials[section] = dict()
for attribute in config[section]:
credentials[section][attribute] = config[section][attribute]
with open('./credentials.yaml', 'w') as f:
data = yaml.dump(credentials, f)
print("Migration complete. Your credentials stored in .env should now be in credentials.yaml")