Replace .env with credentials.yaml
JulioV
parent
f65e3c8b1a
commit
4b33ee43ba
|
|
@ -454,25 +454,7 @@ Modify the following keys in your `config.yaml` depending on the [data stream](.
|
|||
|---------------------|----------------------------------------------------------------------------------------------------------------------------|
|
||||
| `[DATABASE_GROUP]` | A database credentials group. Read the instructions below to set it up |
|
||||
|
||||
??? info "Setting up a DATABASE_GROUP and its connection credentials"
|
||||
|
||||
1. If you haven't done so, create an empty file called `#!bash .env` in your RAPIDS root directory: `./.env`
|
||||
2. Add the following lines to `./.env` and replace your database-specific credentials (user, password, host, and database):
|
||||
1. Note that the label `[MY_GROUP]` is arbitrary but it has to match `[PHONE_DATA_STREAMS][aware_mysql] [DATABASE_GROUP]`
|
||||
|
||||
``` yaml
|
||||
[MY_GROUP]
|
||||
user=MY_USER
|
||||
password=MY_PASSWORD
|
||||
host=MY_HOST
|
||||
port=3306
|
||||
database=MY_DATABASE
|
||||
```
|
||||
|
||||
??? hint "Connecting to localhost (host machine) from inside our docker container"
|
||||
If you are using RAPIDS' docker container and Docker-for-mac or Docker-for-Windows 18.03+, you can connect to a MySQL database in your host machine using `host.docker.internal` instead of `127.0.0.1` or `localhost`. In a Linux host you need to run our docker container using `docker run --network="host" -d moshiresearch/rapids:latest` and then `127.0.0.1` will point to your host machine.
|
||||
---
|
||||
|
||||
--8<---- "docs/snippets/database.md"
|
||||
|
||||
=== "aware_csv"
|
||||
|
||||
|
|
@ -522,24 +504,7 @@ Modify the following keys in your `config.yaml` depending on the [data stream](.
|
|||
| `[DATABASE_GROUP]` | A database credentials group. Read the instructions below to set it up |
|
||||
| `[COLUMN_MAPPINGS_READY]` | Set this to `True` after you have modified this stream's `format.yaml` column mappings to match your raw data column names: [`fitbitjson_mysql`](../../datastreams/fitbitjson-mysql#format) |
|
||||
|
||||
??? info "Setting up a DATABASE_GROUP and its connection credentials"
|
||||
|
||||
1. If you haven't done so, create an empty file called `#!bash .env` in your RAPIDS root directory: `./.env`
|
||||
2. Add the following lines to `./.env` and replace your database-specific credentials (user, password, host, and database):
|
||||
1. Note that the label `[MY_GROUP]` is arbitrary but it has to match `[FITBIT_DATA_STREAMS][fitbitjson_mysql] [DATABASE_GROUP]`
|
||||
|
||||
``` yaml
|
||||
[MY_GROUP]
|
||||
user=MY_USER
|
||||
password=MY_PASSWORD
|
||||
host=MY_HOST
|
||||
port=3306
|
||||
database=MY_DATABASE
|
||||
```
|
||||
|
||||
??? hint "Connecting to localhost (host machine) from inside our docker container"
|
||||
If you are using RAPIDS' docker container and Docker-for-mac or Docker-for-Windows 18.03+, you can connect to a MySQL database in your host machine using `host.docker.internal` instead of `127.0.0.1` or `localhost`. In a Linux host you need to run our docker container using `docker run --network="host" -d moshiresearch/rapids:latest` and then `127.0.0.1` will point to your host machine.
|
||||
---
|
||||
--8<---- "docs/snippets/database.md"
|
||||
|
||||
=== "fitbitjson_csv"
|
||||
|
||||
|
|
@ -561,24 +526,7 @@ Modify the following keys in your `config.yaml` depending on the [data stream](.
|
|||
| `[DATABASE_GROUP]` | A database credentials group. Read the instructions below to set it up |
|
||||
| `[COLUMN_MAPPINGS_READY]` | Set this to `True` after you have modified this stream's `format.yaml` column mappings to match your raw data column names: [`fitbitparsed_mysql`](../../datastreams/fitbitparsed-mysql#format) |
|
||||
|
||||
??? info "Setting up a DATABASE_GROUP and its connection credentials"
|
||||
|
||||
1. If you haven't done so, create an empty file called `#!bash .env` in your RAPIDS root directory: `./.env`
|
||||
2. Add the following lines to `./.env` and replace your database-specific credentials (user, password, host, and database):
|
||||
1. Note that the label `[MY_GROUP]` is arbitrary but it has to match `[FITBIT_DATA_STREAMS][fitbitparsed_mysql] [DATABASE_GROUP]`
|
||||
|
||||
``` yaml
|
||||
[MY_GROUP]
|
||||
user=MY_USER
|
||||
password=MY_PASSWORD
|
||||
host=MY_HOST
|
||||
port=3306
|
||||
database=MY_DATABASE
|
||||
```
|
||||
|
||||
??? hint "Connecting to localhost (host machine) from inside our docker container"
|
||||
If you are using RAPIDS' docker container and Docker-for-mac or Docker-for-Windows 18.03+, you can connect to a MySQL database in your host machine using `host.docker.internal` instead of `127.0.0.1` or `localhost`. In a Linux host you need to run our docker container using `docker run --network="host" -d moshiresearch/rapids:latest` and then `127.0.0.1` will point to your host machine.
|
||||
---
|
||||
--8<---- "docs/snippets/database.md"
|
||||
|
||||
=== "fitbitparsed_csv"
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,58 @@
|
|||
??? info "Setting up a DATABASE_GROUP and its connection credentials."
|
||||
|
||||
1. If you haven't done so, create an empty file called `#!bash credentials.yaml` in your RAPIDS root directory:
|
||||
|
||||
2. Add the following lines to `credentials.yaml` and replace your database-specific credentials (user, password, host, and database):
|
||||
|
||||
``` yaml
|
||||
MY_GROUP:
|
||||
database: MY_DATABASE
|
||||
host: MY_HOST
|
||||
password: MY_PASSWORD
|
||||
port: 3306
|
||||
user: MY_USER
|
||||
```
|
||||
|
||||
1. Notes
|
||||
|
||||
1. The label `[MY_GROUP]` is arbitrary but it has to match the `[DATABASE_GROUP]` attribute of the data stream you choose to use.
|
||||
|
||||
2. Indentation matters
|
||||
|
||||
3. You can have more than one credentials group in `credentials.yaml`
|
||||
|
||||
??? hint "Upgrading from `./.env` from RAPIDS 0.x"
|
||||
In RAPIDS versions 0.x, database credentials were stored in a `./.env` file. If you are migrating from that type of file, you have two options:
|
||||
|
||||
1. Migrate your credentials by hand:
|
||||
|
||||
=== "change .env format"
|
||||
|
||||
``` yaml
|
||||
[MY_GROUP]
|
||||
user=MY_USER
|
||||
password=MY_PASSWORD
|
||||
host=MY_HOST
|
||||
port=3306
|
||||
database=MY_DATABASE
|
||||
```
|
||||
|
||||
=== "to credentials.yaml format"
|
||||
|
||||
``` yaml
|
||||
MY_GROUP:
|
||||
user: MY_USER
|
||||
password: MY_PASSWORD
|
||||
host: MY_HOST
|
||||
port: 3306
|
||||
database: MY_DATABASE
|
||||
```
|
||||
|
||||
2. Use the migration script we provide (make sure your conda environment is active):
|
||||
|
||||
```python
|
||||
python tools/update_format_env.py
|
||||
```
|
||||
|
||||
??? hint "Connecting to localhost (host machine) from inside our docker container."
|
||||
If you are using RAPIDS' docker container and Docker-for-mac or Docker-for-Windows 18.03+, you can connect to a MySQL database in your host machine using `host.docker.internal` instead of `127.0.0.1` or `localhost`. In a Linux host, you need to run our docker container using `docker run --network="host" -d moshiresearch/rapids:latest` and then `127.0.0.1` will point to your host machine.
|
||||
|
|
@ -1,5 +1,25 @@
|
|||
# if you need a new package, you should add it with renv::install(package) so your renv venv is updated
|
||||
library(RMariaDB)
|
||||
library(yaml)
|
||||
|
||||
#' @description
|
||||
#' Auxiliary function to parse the connection credentials from a specifc group in ./credentials.yaml
|
||||
#' You can reause most of this function if you are connection to a DB or Web API.
|
||||
#' It's OK to delete this function if you don't need credentials, e.g., you are pulling data from a CSV for example.
|
||||
#' @param group the yaml key containing the credentials to connect to a database
|
||||
#' @preturn dbEngine a database engine (connection) ready to perform queries
|
||||
get_db_engine <- function(group){
|
||||
# The working dir is aways RAPIDS root folder, so your credentials file is always /credentials.yaml
|
||||
credentials <- read_yaml("./credentials.yaml")
|
||||
if(!group %in% names(credentials))
|
||||
stop(paste("The credentials group",group, "does not exist in ./credentials.yaml. The only groups that exist in that file are:", paste(names(credentials), collapse = ",")))
|
||||
dbEngine <- dbConnect(MariaDB(), db = credentials[[group]][["database"]],
|
||||
username = credentials[[group]][["user"]],
|
||||
password = credentials[[group]][["password"]],
|
||||
host = credentials[[group]][["host"]],
|
||||
port = credentials[[group]][["port"]])
|
||||
return(dbEngine)
|
||||
}
|
||||
|
||||
# This file gets executed for each PHONE_SENSOR of each participant
|
||||
# If you are connecting to a database the env file containing its credentials is available at "./.env"
|
||||
|
|
@ -18,9 +38,7 @@ library(RMariaDB)
|
|||
#' @return The OS the device ran, "android" or "ios"
|
||||
|
||||
infer_device_os <- function(data_configuration, device){
|
||||
group <- data_configuration$SOURCE$DATABASE_GROUP # specified DB credentials group in config.yaml
|
||||
|
||||
dbEngine <- dbConnect(MariaDB(), default.file = "./.env", group = group)
|
||||
dbEngine <- get_db_engine(data_configuration$SOURCE$DATABASE_GROUP)
|
||||
query <- paste0("SELECT device_id,brand FROM aware_device WHERE device_id = '", device, "'")
|
||||
message(paste0("Executing the following query to infer phone OS: ", query))
|
||||
os <- dbGetQuery(dbEngine, query)
|
||||
|
|
@ -44,10 +62,7 @@ infer_device_os <- function(data_configuration, device){
|
|||
#' @return A dataframe with the sensor data for device
|
||||
|
||||
download_data <- function(data_configuration, device, sensor_container, columns){
|
||||
group <- data_configuration$SOURCE$DATABASE_GROUP
|
||||
dbEngine <- dbConnect(MariaDB(), default.file = "./.env", group = group)
|
||||
|
||||
|
||||
dbEngine <- get_db_engine(data_configuration$SOURCE$DATABASE_GROUP)
|
||||
query <- paste0("SELECT ", paste(columns, collapse = ",")," FROM ", sensor_container, " WHERE device_id = '", device,"'")
|
||||
# Letting the user know what we are doing
|
||||
message(paste0("Executing the following query to download data: ", query))
|
||||
|
|
|
|||
|
|
@ -0,0 +1,20 @@
|
|||
import configparser
|
||||
import yaml
|
||||
from pathlib import Path
|
||||
|
||||
if not Path('./.env').exists():
|
||||
print("There is no .env file in RAPIDS root directory, nothing to migrate")
|
||||
else:
|
||||
config = configparser.ConfigParser()
|
||||
config.read('./.env')
|
||||
|
||||
credentials = dict()
|
||||
for section in config.sections():
|
||||
credentials[section] = dict()
|
||||
for attribute in config[section]:
|
||||
credentials[section][attribute] = config[section][attribute]
|
||||
|
||||
with open('./credentials.yaml', 'w') as f:
|
||||
data = yaml.dump(credentials, f)
|
||||
|
||||
print("Migration complete. Your credentials stored in .env should now be in credentials.yaml")
|
||||
Loading…
Reference in New Issue