Go to file
junos cf6af7c9a4 Add a TODO. 2023-04-18 16:11:30 +02:00
.github Update docs with contributing guide 2021-05-10 15:48:14 -04:00
data/external Squashed commit of the following: 2022-12-08 16:04:39 +00:00
docs Update change-log for v1.8.0 2022-02-10 15:05:55 -05:00
example_profile Make place for STRAW models. 2022-02-04 17:25:24 +01:00
models First commit 2019-10-22 13:11:01 -04:00
notebooks First commit 2019-10-22 13:11:01 -04:00
problems Keep track of warning messages. 2021-12-15 16:19:29 +01:00
references First commit 2019-10-22 13:11:01 -04:00
renv Small misc changes 2022-06-02 06:41:53 +00:00
reports Add restore_sql_file rule; notsummarised module; diff platforms for heatmap_days_by_sensors 2020-08-03 13:09:16 -04:00
rules Integrate phone_speech into rapids pipeline. 2023-01-17 14:00:14 +00:00
src Drop all window count related features in cleaning script. 2023-02-15 14:15:56 +00:00
tests Squashed commit of the following: 2022-12-08 16:04:39 +00:00
tools Data cleaning (#166) 2021-11-19 10:34:36 -05:00
.Rprofile Add Travis CI 2020-05-18 19:24:34 -04:00
.gitattributes Force Unix style end of line. 2021-11-17 19:12:40 +01:00
.gitignore Squashed commit of the following: 2022-07-20 13:51:22 +00:00
Dockerfile Add libglpk40 dependency 2021-01-05 19:24:49 -05:00
LICENSE Update LICENSE 2020-03-10 16:52:23 -04:00
README.md Update README 2022-12-09 16:04:11 +00:00
Snakefile Integrate phone_speech into rapids pipeline. 2023-01-17 14:00:14 +00:00
__init__.py Add folder structure for CF testing and EDA test. 2022-03-21 10:40:18 +00:00
automl_test.py Squashed commit of the following: 2022-12-08 16:04:39 +00:00
code_of_conduct.md Add code of conduct 2020-11-30 13:51:05 -05:00
config.yaml Add a TODO. 2023-04-18 16:11:30 +02:00
donotmakechanges.py Bug fixed (mixed dtype warning). 2022-12-12 11:29:59 +00:00
environment.yml Add a missing pip dependency. 2023-04-11 19:06:29 +02:00
mkdocs.yml Add data stream for AWARE Micro server (#173) 2022-01-27 10:47:50 -05:00
rapids remove .py ext from rapids.py 2020-10-28 16:59:19 -04:00
renv.lock Update more R packages. 2023-04-11 15:31:44 +02:00

README.md

GitHub release (latest SemVer) Snakemake Documentation Status tests Contributor Covenant

RAPIDS

Reproducible Analysis Pipeline for Data Streams

For more information refer to our documentation

By MoSHI, University of Pittsburgh

Installation

For RAPIDS installation refer to to the documentation

For the installation of the Docker version

  1. Follow the instructions to setup RAPIDS via Docker (from scratch).

  2. Delete current contents in /rapids/ folder when in a container session.

    cd ..
    rm -rf rapids/{*,.*}
    cd rapids
    
  3. Clone RAPIDS workspace from Git and checkout a specific branch.

    git clone "https://repo.ijs.si/junoslukan/rapids.git" .
    git checkout <branch_name>
    
  4. Install missing “libpq-dev” dependency with bash.

    apt-get update -y
    apt-get install -y libpq-dev
    
  5. Restore R venv. Type R to go to the interactive R session and then:

    renv::restore()
    
  6. Install cr-features module From: https://repo.ijs.si/matjazbostic/calculatingfeatures.git -> branch master. Then follow the "cr-features module" section below.

  7. Install all required packages from environment.yml, prune also deletes conda packages not present in environment file.

    conda env update --file environment.yml prune
    
  8. If you wish to update your R or Python venvs.

    R in interactive session:
    renv::snapshot()
    Python: 
    conda env export --no-builds | sed 's/^.*libgfortran.*$/  - libgfortran/' | sed 's/^.*mkl=.*$/  - mkl/' >  environment.yml
    

cr-features module

This RAPIDS extension uses cr-features library accessible here.

To use cr-features library:

  • Follow the installation instructions in the README.md.

  • Copy built calculatingfeatures folder into the RAPIDS workspace.

  • Install the cr-features package by:

    pip install path/to/the/calculatingfeatures/folder
    e.g. pip install ./calculatingfeatures if the folder is copied to main parent directory
    cr-features package has to be built and installed everytime to get the newest version. 
    Or an the newest version of the docker image must be used.   
    

Updating RAPIDS

To update RAPIDS, first pull and merge origin, such as with:

git fetch --progress "origin" refs/heads/master
git merge --no-ff origin/master

Next, update the conda and R virtual environment.

R -e 'renv::restore(repos = c(CRAN = "https://packagemanager.rstudio.com/all/__linux__/focal/latest"))'

Custom configuration

Credentials

As mentioned under Database in RAPIDS documentation, a credentials.yaml file is needed to connect to a database. It should contain:

PSQL_STRAW:
  database: staw
  host: 212.235.208.113
  password: password
  port: 5432
  user: staw_db

wherepassword needs to be specified as well.

Possible installation issues

Missing dependencies for RPostgres

To install RPostgres R package (used to connect to the PostgreSQL database), an error might occur:

------------------------- ANTICONF ERROR ---------------------------
Configuration failed because libpq was not found. Try installing:
   * deb: libpq-dev (Debian, Ubuntu, etc)
   * rpm: postgresql-devel (Fedora, EPEL)
   * rpm: postgreql8-devel, psstgresql92-devel, postgresql93-devel, or postgresql94-devel (Amazon Linux)
   * csw: postgresql_dev (Solaris)
   * brew: libpq (OSX)
If libpq is already installed, check that either:
  (i)  'pkg-config' is in your PATH AND PKG_CONFIG_PATH contains a libpq.pc file; or
  (ii) 'pg_config' is in your PATH.
If neither can detect , you can set INCLUDE_DIR
and LIB_DIR manually via:
  R CMD INSTALL --configure-vars='INCLUDE_DIR=... LIB_DIR=...'
--------------------------[ ERROR MESSAGE ]----------------------------
  <stdin>:1:10: fatal error: libpq-fe.h: No such file or directory
compilation terminated.

The library requires libpq for compiling from source, so install accordingly.

Timezone environment variable for tidyverse (relevant for WSL2)

One of the R packages, tidyverse might need access to the TZ environment variable during the installation. On Ubuntu 20.04 on WSL2 this triggers the following error:

> install.packages('tidyverse')

ERROR: configuration failed for package xml2
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to create bus connection: Host is down
Warning in system("timedatectl", intern = TRUE) :
  running command 'timedatectl' had status 1
Error in loadNamespace(j <- i[[1L]], c(lib.loc, .libPaths()), versionCheck = vI[[j]]) :
  namespace xml2 1.3.1 is already loaded, but >= 1.3.2 is required
Calls: <Anonymous> ... namespaceImportFrom -> asNamespace -> loadNamespace
Execution halted
ERROR: lazy loading failed for package tidyverse

This happens because WSL2 does not use the timedatectl service, which provides this variable.

~$ timedatectl
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to create bus connection: Host is down

and later

Warning message:
In system("timedatectl", intern = TRUE) :
  running command 'timedatectl' had status 1
Execution halted

This can be amended by setting the environment variable manually before attempting to install tidyverse:

export TZ='Europe/Ljubljana'

Note: if this is needed to avoid runtime issues, you need to either define this environment variable in each new terminal window or (better) define it in your ~/.bashrc or ~/.bash_profile.

Possible runtime issues

Unix end of line characters

Upon running rapids, an error might occur:

/usr/bin/env: python3\r: No such file or directory

This is due to Windows style end of line characters. To amend this, I added a .gitattributes files to force git to checkout rapids using Unix EOL characters. If this still fails, dos2unix can be used to change them.

System has not been booted with systemd as init system (PID 1)

See the installation issue above.