Replace .env with credentials.yaml
parent
f65e3c8b1a
commit
4b33ee43ba
|
@ -454,25 +454,7 @@ Modify the following keys in your `config.yaml` depending on the [data stream](.
|
||||||
|---------------------|----------------------------------------------------------------------------------------------------------------------------|
|
|---------------------|----------------------------------------------------------------------------------------------------------------------------|
|
||||||
| `[DATABASE_GROUP]` | A database credentials group. Read the instructions below to set it up |
|
| `[DATABASE_GROUP]` | A database credentials group. Read the instructions below to set it up |
|
||||||
|
|
||||||
??? info "Setting up a DATABASE_GROUP and its connection credentials"
|
--8<---- "docs/snippets/database.md"
|
||||||
|
|
||||||
1. If you haven't done so, create an empty file called `#!bash .env` in your RAPIDS root directory: `./.env`
|
|
||||||
2. Add the following lines to `./.env` and replace your database-specific credentials (user, password, host, and database):
|
|
||||||
1. Note that the label `[MY_GROUP]` is arbitrary but it has to match `[PHONE_DATA_STREAMS][aware_mysql] [DATABASE_GROUP]`
|
|
||||||
|
|
||||||
``` yaml
|
|
||||||
[MY_GROUP]
|
|
||||||
user=MY_USER
|
|
||||||
password=MY_PASSWORD
|
|
||||||
host=MY_HOST
|
|
||||||
port=3306
|
|
||||||
database=MY_DATABASE
|
|
||||||
```
|
|
||||||
|
|
||||||
??? hint "Connecting to localhost (host machine) from inside our docker container"
|
|
||||||
If you are using RAPIDS' docker container and Docker-for-mac or Docker-for-Windows 18.03+, you can connect to a MySQL database in your host machine using `host.docker.internal` instead of `127.0.0.1` or `localhost`. In a Linux host you need to run our docker container using `docker run --network="host" -d moshiresearch/rapids:latest` and then `127.0.0.1` will point to your host machine.
|
|
||||||
---
|
|
||||||
|
|
||||||
|
|
||||||
=== "aware_csv"
|
=== "aware_csv"
|
||||||
|
|
||||||
|
@ -522,24 +504,7 @@ Modify the following keys in your `config.yaml` depending on the [data stream](.
|
||||||
| `[DATABASE_GROUP]` | A database credentials group. Read the instructions below to set it up |
|
| `[DATABASE_GROUP]` | A database credentials group. Read the instructions below to set it up |
|
||||||
| `[COLUMN_MAPPINGS_READY]` | Set this to `True` after you have modified this stream's `format.yaml` column mappings to match your raw data column names: [`fitbitjson_mysql`](../../datastreams/fitbitjson-mysql#format) |
|
| `[COLUMN_MAPPINGS_READY]` | Set this to `True` after you have modified this stream's `format.yaml` column mappings to match your raw data column names: [`fitbitjson_mysql`](../../datastreams/fitbitjson-mysql#format) |
|
||||||
|
|
||||||
??? info "Setting up a DATABASE_GROUP and its connection credentials"
|
--8<---- "docs/snippets/database.md"
|
||||||
|
|
||||||
1. If you haven't done so, create an empty file called `#!bash .env` in your RAPIDS root directory: `./.env`
|
|
||||||
2. Add the following lines to `./.env` and replace your database-specific credentials (user, password, host, and database):
|
|
||||||
1. Note that the label `[MY_GROUP]` is arbitrary but it has to match `[FITBIT_DATA_STREAMS][fitbitjson_mysql] [DATABASE_GROUP]`
|
|
||||||
|
|
||||||
``` yaml
|
|
||||||
[MY_GROUP]
|
|
||||||
user=MY_USER
|
|
||||||
password=MY_PASSWORD
|
|
||||||
host=MY_HOST
|
|
||||||
port=3306
|
|
||||||
database=MY_DATABASE
|
|
||||||
```
|
|
||||||
|
|
||||||
??? hint "Connecting to localhost (host machine) from inside our docker container"
|
|
||||||
If you are using RAPIDS' docker container and Docker-for-mac or Docker-for-Windows 18.03+, you can connect to a MySQL database in your host machine using `host.docker.internal` instead of `127.0.0.1` or `localhost`. In a Linux host you need to run our docker container using `docker run --network="host" -d moshiresearch/rapids:latest` and then `127.0.0.1` will point to your host machine.
|
|
||||||
---
|
|
||||||
|
|
||||||
=== "fitbitjson_csv"
|
=== "fitbitjson_csv"
|
||||||
|
|
||||||
|
@ -561,24 +526,7 @@ Modify the following keys in your `config.yaml` depending on the [data stream](.
|
||||||
| `[DATABASE_GROUP]` | A database credentials group. Read the instructions below to set it up |
|
| `[DATABASE_GROUP]` | A database credentials group. Read the instructions below to set it up |
|
||||||
| `[COLUMN_MAPPINGS_READY]` | Set this to `True` after you have modified this stream's `format.yaml` column mappings to match your raw data column names: [`fitbitparsed_mysql`](../../datastreams/fitbitparsed-mysql#format) |
|
| `[COLUMN_MAPPINGS_READY]` | Set this to `True` after you have modified this stream's `format.yaml` column mappings to match your raw data column names: [`fitbitparsed_mysql`](../../datastreams/fitbitparsed-mysql#format) |
|
||||||
|
|
||||||
??? info "Setting up a DATABASE_GROUP and its connection credentials"
|
--8<---- "docs/snippets/database.md"
|
||||||
|
|
||||||
1. If you haven't done so, create an empty file called `#!bash .env` in your RAPIDS root directory: `./.env`
|
|
||||||
2. Add the following lines to `./.env` and replace your database-specific credentials (user, password, host, and database):
|
|
||||||
1. Note that the label `[MY_GROUP]` is arbitrary but it has to match `[FITBIT_DATA_STREAMS][fitbitparsed_mysql] [DATABASE_GROUP]`
|
|
||||||
|
|
||||||
``` yaml
|
|
||||||
[MY_GROUP]
|
|
||||||
user=MY_USER
|
|
||||||
password=MY_PASSWORD
|
|
||||||
host=MY_HOST
|
|
||||||
port=3306
|
|
||||||
database=MY_DATABASE
|
|
||||||
```
|
|
||||||
|
|
||||||
??? hint "Connecting to localhost (host machine) from inside our docker container"
|
|
||||||
If you are using RAPIDS' docker container and Docker-for-mac or Docker-for-Windows 18.03+, you can connect to a MySQL database in your host machine using `host.docker.internal` instead of `127.0.0.1` or `localhost`. In a Linux host you need to run our docker container using `docker run --network="host" -d moshiresearch/rapids:latest` and then `127.0.0.1` will point to your host machine.
|
|
||||||
---
|
|
||||||
|
|
||||||
=== "fitbitparsed_csv"
|
=== "fitbitparsed_csv"
|
||||||
|
|
||||||
|
|
|
@ -0,0 +1,58 @@
|
||||||
|
??? info "Setting up a DATABASE_GROUP and its connection credentials."
|
||||||
|
|
||||||
|
1. If you haven't done so, create an empty file called `#!bash credentials.yaml` in your RAPIDS root directory:
|
||||||
|
|
||||||
|
2. Add the following lines to `credentials.yaml` and replace your database-specific credentials (user, password, host, and database):
|
||||||
|
|
||||||
|
``` yaml
|
||||||
|
MY_GROUP:
|
||||||
|
database: MY_DATABASE
|
||||||
|
host: MY_HOST
|
||||||
|
password: MY_PASSWORD
|
||||||
|
port: 3306
|
||||||
|
user: MY_USER
|
||||||
|
```
|
||||||
|
|
||||||
|
1. Notes
|
||||||
|
|
||||||
|
1. The label `[MY_GROUP]` is arbitrary but it has to match the `[DATABASE_GROUP]` attribute of the data stream you choose to use.
|
||||||
|
|
||||||
|
2. Indentation matters
|
||||||
|
|
||||||
|
3. You can have more than one credentials group in `credentials.yaml`
|
||||||
|
|
||||||
|
??? hint "Upgrading from `./.env` from RAPIDS 0.x"
|
||||||
|
In RAPIDS versions 0.x, database credentials were stored in a `./.env` file. If you are migrating from that type of file, you have two options:
|
||||||
|
|
||||||
|
1. Migrate your credentials by hand:
|
||||||
|
|
||||||
|
=== "change .env format"
|
||||||
|
|
||||||
|
``` yaml
|
||||||
|
[MY_GROUP]
|
||||||
|
user=MY_USER
|
||||||
|
password=MY_PASSWORD
|
||||||
|
host=MY_HOST
|
||||||
|
port=3306
|
||||||
|
database=MY_DATABASE
|
||||||
|
```
|
||||||
|
|
||||||
|
=== "to credentials.yaml format"
|
||||||
|
|
||||||
|
``` yaml
|
||||||
|
MY_GROUP:
|
||||||
|
user: MY_USER
|
||||||
|
password: MY_PASSWORD
|
||||||
|
host: MY_HOST
|
||||||
|
port: 3306
|
||||||
|
database: MY_DATABASE
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Use the migration script we provide (make sure your conda environment is active):
|
||||||
|
|
||||||
|
```python
|
||||||
|
python tools/update_format_env.py
|
||||||
|
```
|
||||||
|
|
||||||
|
??? hint "Connecting to localhost (host machine) from inside our docker container."
|
||||||
|
If you are using RAPIDS' docker container and Docker-for-mac or Docker-for-Windows 18.03+, you can connect to a MySQL database in your host machine using `host.docker.internal` instead of `127.0.0.1` or `localhost`. In a Linux host, you need to run our docker container using `docker run --network="host" -d moshiresearch/rapids:latest` and then `127.0.0.1` will point to your host machine.
|
|
@ -1,5 +1,25 @@
|
||||||
# if you need a new package, you should add it with renv::install(package) so your renv venv is updated
|
# if you need a new package, you should add it with renv::install(package) so your renv venv is updated
|
||||||
library(RMariaDB)
|
library(RMariaDB)
|
||||||
|
library(yaml)
|
||||||
|
|
||||||
|
#' @description
|
||||||
|
#' Auxiliary function to parse the connection credentials from a specifc group in ./credentials.yaml
|
||||||
|
#' You can reause most of this function if you are connection to a DB or Web API.
|
||||||
|
#' It's OK to delete this function if you don't need credentials, e.g., you are pulling data from a CSV for example.
|
||||||
|
#' @param group the yaml key containing the credentials to connect to a database
|
||||||
|
#' @preturn dbEngine a database engine (connection) ready to perform queries
|
||||||
|
get_db_engine <- function(group){
|
||||||
|
# The working dir is aways RAPIDS root folder, so your credentials file is always /credentials.yaml
|
||||||
|
credentials <- read_yaml("./credentials.yaml")
|
||||||
|
if(!group %in% names(credentials))
|
||||||
|
stop(paste("The credentials group",group, "does not exist in ./credentials.yaml. The only groups that exist in that file are:", paste(names(credentials), collapse = ",")))
|
||||||
|
dbEngine <- dbConnect(MariaDB(), db = credentials[[group]][["database"]],
|
||||||
|
username = credentials[[group]][["user"]],
|
||||||
|
password = credentials[[group]][["password"]],
|
||||||
|
host = credentials[[group]][["host"]],
|
||||||
|
port = credentials[[group]][["port"]])
|
||||||
|
return(dbEngine)
|
||||||
|
}
|
||||||
|
|
||||||
# This file gets executed for each PHONE_SENSOR of each participant
|
# This file gets executed for each PHONE_SENSOR of each participant
|
||||||
# If you are connecting to a database the env file containing its credentials is available at "./.env"
|
# If you are connecting to a database the env file containing its credentials is available at "./.env"
|
||||||
|
@ -18,9 +38,7 @@ library(RMariaDB)
|
||||||
#' @return The OS the device ran, "android" or "ios"
|
#' @return The OS the device ran, "android" or "ios"
|
||||||
|
|
||||||
infer_device_os <- function(data_configuration, device){
|
infer_device_os <- function(data_configuration, device){
|
||||||
group <- data_configuration$SOURCE$DATABASE_GROUP # specified DB credentials group in config.yaml
|
dbEngine <- get_db_engine(data_configuration$SOURCE$DATABASE_GROUP)
|
||||||
|
|
||||||
dbEngine <- dbConnect(MariaDB(), default.file = "./.env", group = group)
|
|
||||||
query <- paste0("SELECT device_id,brand FROM aware_device WHERE device_id = '", device, "'")
|
query <- paste0("SELECT device_id,brand FROM aware_device WHERE device_id = '", device, "'")
|
||||||
message(paste0("Executing the following query to infer phone OS: ", query))
|
message(paste0("Executing the following query to infer phone OS: ", query))
|
||||||
os <- dbGetQuery(dbEngine, query)
|
os <- dbGetQuery(dbEngine, query)
|
||||||
|
@ -44,10 +62,7 @@ infer_device_os <- function(data_configuration, device){
|
||||||
#' @return A dataframe with the sensor data for device
|
#' @return A dataframe with the sensor data for device
|
||||||
|
|
||||||
download_data <- function(data_configuration, device, sensor_container, columns){
|
download_data <- function(data_configuration, device, sensor_container, columns){
|
||||||
group <- data_configuration$SOURCE$DATABASE_GROUP
|
dbEngine <- get_db_engine(data_configuration$SOURCE$DATABASE_GROUP)
|
||||||
dbEngine <- dbConnect(MariaDB(), default.file = "./.env", group = group)
|
|
||||||
|
|
||||||
|
|
||||||
query <- paste0("SELECT ", paste(columns, collapse = ",")," FROM ", sensor_container, " WHERE device_id = '", device,"'")
|
query <- paste0("SELECT ", paste(columns, collapse = ",")," FROM ", sensor_container, " WHERE device_id = '", device,"'")
|
||||||
# Letting the user know what we are doing
|
# Letting the user know what we are doing
|
||||||
message(paste0("Executing the following query to download data: ", query))
|
message(paste0("Executing the following query to download data: ", query))
|
||||||
|
|
|
@ -0,0 +1,20 @@
|
||||||
|
import configparser
|
||||||
|
import yaml
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
if not Path('./.env').exists():
|
||||||
|
print("There is no .env file in RAPIDS root directory, nothing to migrate")
|
||||||
|
else:
|
||||||
|
config = configparser.ConfigParser()
|
||||||
|
config.read('./.env')
|
||||||
|
|
||||||
|
credentials = dict()
|
||||||
|
for section in config.sections():
|
||||||
|
credentials[section] = dict()
|
||||||
|
for attribute in config[section]:
|
||||||
|
credentials[section][attribute] = config[section][attribute]
|
||||||
|
|
||||||
|
with open('./credentials.yaml', 'w') as f:
|
||||||
|
data = yaml.dump(credentials, f)
|
||||||
|
|
||||||
|
print("Migration complete. Your credentials stored in .env should now be in credentials.yaml")
|
Loading…
Reference in New Issue