junos
/
rapids
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Migrate phone log sensor to new data stream

pull/128/head
Meng Li 2021-03-10 16:26:50 -05:00 committed by JulioV
parent 091f9c048a
commit 35968e2fd0
14 changed files with 124 additions and 58 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
Snakefile
View File

@ -187,13 +187,13 @@ if isinstance(config["PHONE_KEYBOARD"]["PROVIDERS"], dict):
files_to_compute.extend(expand("data/processed/features/{pid}/all_sensor_features.csv", pid=config["PIDS"]))
files_to_compute.append("data/processed/features/all_participants/all_sensor_features.csv")
if isinstance(config["PHONE_AWARE_LOG"]["PROVIDERS"], dict):
for provider in config["PHONE_AWARE_LOG"]["PROVIDERS"].keys():
if config["PHONE_AWARE_LOG"]["PROVIDERS"][provider]["COMPUTE"]:
files_to_compute.extend(expand("data/raw/{pid}/phone_aware_log_raw.csv", pid=config["PIDS"]))
files_to_compute.extend(expand("data/raw/{pid}/phone_aware_log_with_datetime.csv", pid=config["PIDS"]))
files_to_compute.extend(expand("data/interim/{pid}/phone_aware_log_features/phone_aware_log_{language}_{provider_key}.csv", pid=config["PIDS"], language=config["PHONE_AWARE_LOG"]["PROVIDERS"][provider]["SRC_LANGUAGE"].lower(), provider_key=provider.lower()))
files_to_compute.extend(expand("data/processed/features/{pid}/phone_aware_log.csv", pid=config["PIDS"]))
if isinstance(config["PHONE_LOG"]["PROVIDERS"], dict):
for provider in config["PHONE_LOG"]["PROVIDERS"].keys():
if config["PHONE_LOG"]["PROVIDERS"][provider]["COMPUTE"]:
files_to_compute.extend(expand("data/raw/{pid}/phone_log_raw.csv", pid=config["PIDS"]))
files_to_compute.extend(expand("data/raw/{pid}/phone_log_with_datetime.csv", pid=config["PIDS"]))
files_to_compute.extend(expand("data/interim/{pid}/phone_log_features/phone_log_{language}_{provider_key}.csv", pid=config["PIDS"], language=config["PHONE_LOG"]["PROVIDERS"][provider]["SRC_LANGUAGE"].lower(), provider_key=provider.lower()))
files_to_compute.extend(expand("data/processed/features/{pid}/phone_log.csv", pid=config["PIDS"]))
files_to_compute.extend(expand("data/processed/features/{pid}/all_sensor_features.csv", pid=config["PIDS"]))
files_to_compute.append("data/processed/features/all_participants/all_sensor_features.csv")

12
config.yaml
View File

@ -138,11 +138,6 @@ PHONE_APPLICATIONS_NOTIFICATIONS:
SCRAPE_MISSING_CATEGORIES: False # whether or not to scrape missing genres, only effective if CATALOGUE_SOURCE is equal to FILE. If CATALOGUE_SOURCE is equal to GOOGLE, all genres are scraped anyway
PROVIDERS: # None implemented yet but this sensor can be used in PHONE_DATA_YIELD
# See https://www.rapids.science/latest/features/phone-aware-log/
PHONE_AWARE_LOG:
TABLE: aware_log
PROVIDERS: # None implemented yet but this sensor can be used in PHONE_DATA_YIELD
# See https://www.rapids.science/latest/features/phone-battery/
PHONE_BATTERY:
TABLE: battery
@ -278,6 +273,13 @@ PHONE_LOCATIONS:
SRC_FOLDER: "barnett" # inside src/features/phone_locations
SRC_LANGUAGE: "r"
# See https://www.rapids.science/latest/features/phone-log/
PHONE_LOG:
TABLE:
ANDROID: aware_log
IOS: ios_aware_log
PROVIDERS: # None implemented yet but this sensor can be used in PHONE_DATA_YIELD
# See https://www.rapids.science/latest/features/phone-messages/
PHONE_MESSAGES:
TABLE: messages

22
docs/datastreams/aware-mysql.md
View File

@ -451,6 +451,28 @@ Stream columns named `FLAG_TO_MUTATE` means they are extracted based on the `MUT
Same as ANDROID
??? info "PHONE_LOG"
=== "ANDROID"
**RAPIDS_COLUMN_MAPPINGS**
| RAPIDS column | Stream column |
|----------------------|---------------------|
| TIMESTAMP | timestamp |
| DEVICE_ID | device_id |
| LOG_MESSAGE | log_message |
**MUTATION**
- **COLUMN_MAPPINGS** (None)
- **SCRIPTS** (None)
=== "IOS"
Same as ANDROID
??? info "PHONE_MESSAGES"
=== "ANDROID"

9
docs/datastreams/mandatory-phone-format.md
View File

@ -135,6 +135,15 @@ This is a description of the format RAPIDS needs to process data for the followi
| ACCURACY | The estimated location accuracy |
??? info "PHONE_LOG"
| RAPIDS column | Description |
|--------------------|---------------------------------------------------------------------------|
| TIMESTAMP | An UNIX timestamp (13 digits) when a row of data was logged |
| DEVICE_ID | A string that uniquely identifies a device |
| LOG_MESSAGE | A string that denotes log message |
??? info "PHONE_MESSAGES"
| RAPIDS column | Description |

2
docs/features/phone-data-yield.md
View File

@ -18,7 +18,6 @@ Sensor parameters description for `[PHONE_DATA_YIELD]`:
PHONE_APPLICATIONS_CRASHES
PHONE_APPLICATIONS_FOREGROUND
PHONE_APPLICATIONS_NOTIFICATIONS
PHONE_AWARE_LOG
PHONE_BATTERY
PHONE_BLUETOOTH
PHONE_CALLS
@ -27,6 +26,7 @@ Sensor parameters description for `[PHONE_DATA_YIELD]`:
PHONE_KEYBOARD
PHONE_LIGHT
PHONE_LOCATIONS
PHONE_LOG
PHONE_SCREEN
PHONE_WIFI_VISIBLE
PHONE_WIFI_CONNECTED

6
docs/features/phone-aware-log.md → docs/features/phone-log.md
View File

@ -1,10 +1,10 @@
# Phone Aware
# Phone Log
Sensor parameters description for `[PHONE_AWARE_LOG]`:
Sensor parameters description for `[PHONE_LOG]`:
|Key                              | Description |
|----------------|-----------------------------------------------------------------------------------------------------------------------------------
|`[TABLE]`| Database table where the aware data is stored
!!! note
No feature providers have been implemented for this sensor yet, however you can use its key (`PHONE_AWARE_LOG`) to improve [`PHONE_DATA_YIELD`](../phone-data-yield) or you can [implement your own features](../add-new-features).
No feature providers have been implemented for this sensor yet, however you can use its key (`PHONE_LOG`) to improve [`PHONE_DATA_YIELD`](../phone-data-yield) or you can [implement your own features](../add-new-features).

2
mkdocs.yml
View File

@ -101,7 +101,6 @@ nav:
- Phone Applications Crashes: features/phone-applications-crashes.md
- Phone Applications Foreground: features/phone-applications-foreground.md
- Phone Applications Notifications: features/phone-applications-notifications.md
- Phone Aware Log: features/phone-aware-log.md
- Phone Battery: features/phone-battery.md
- Phone Bluetooth: features/phone-bluetooth.md
- Phone Calls: features/phone-calls.md
@ -110,6 +109,7 @@ nav:
- Phone Keyboard: features/phone-keyboard.md
- Phone Light: features/phone-light.md
- Phone Locations: features/phone-locations.md
- Phone Log: features/phone-log.md
- Phone Messages: features/phone-messages.md
- Phone Screen: features/phone-screen.md
- Phone WiFI Connected: features/phone-wifi-connected.md

20
rules/features.smk
View File

@ -174,29 +174,29 @@ rule phone_applications_notifications_r_features:
script:
"../src/features/entry.R"
rule phone_aware_log_python_features:
rule phone_log_python_features:
input:
sensor_data = "data/raw/{pid}/phone_aware_log_with_datetime.csv",
sensor_data = "data/raw/{pid}/phone_log_with_datetime.csv",
time_segments_labels = "data/interim/time_segments/{pid}_time_segments_labels.csv"
params:
provider = lambda wildcards: config["PHONE_AWARE_LOG"]["PROVIDERS"][wildcards.provider_key.upper()],
provider = lambda wildcards: config["PHONE_LOG"]["PROVIDERS"][wildcards.provider_key.upper()],
provider_key = "{provider_key}",
sensor_key = "phone_aware_log"
sensor_key = "phone_log"
output:
"data/interim/{pid}/phone_aware_log_features/phone_aware_log_python_{provider_key}.csv"
"data/interim/{pid}/phone_log_features/phone_log_python_{provider_key}.csv"
script:
"../src/features/entry.py"
rule phone_aware_log_r_features:
rule phone_log_r_features:
input:
sensor_data = "data/raw/{pid}/phone_aware_log_with_datetime.csv",
sensor_data = "data/raw/{pid}/phone_log_with_datetime.csv",
time_segments_labels = "data/interim/time_segments/{pid}_time_segments_labels.csv"
params:
provider = lambda wildcards: config["PHONE_AWARE_LOG"]["PROVIDERS"][wildcards.provider_key.upper()],
provider = lambda wildcards: config["PHONE_LOG"]["PROVIDERS"][wildcards.provider_key.upper()],
provider_key = "{provider_key}",
sensor_key = "phone_aware_log"
sensor_key = "phone_log"
output:
"data/interim/{pid}/phone_aware_log_features/phone_aware_log_r_{provider_key}.csv"
"data/interim/{pid}/phone_log_features/phone_log_r_{provider_key}.csv"
script:
"../src/features/entry.R"

18
src/data/streams/aware_mysql/format.yaml
View File

@ -215,6 +215,24 @@ PHONE_LOCATIONS:
COLUMN_MAPPINGS:
SCRIPTS: # List any python or r scripts that mutate your raw data
PHONE_LOG:
ANDROID:
RAPIDS_COLUMN_MAPPINGS:
TIMESTAMP: timestamp
DEVICE_ID: device_id
LOG_MESSAGE: log_message
MUTATION:
COLUMN_MAPPINGS:
SCRIPTS: # List any python or r scripts that mutate your raw data
IOS:
RAPIDS_COLUMN_MAPPINGS:
TIMESTAMP: timestamp
DEVICE_ID: device_id
LOG_MESSAGE: log_message
MUTATION:
COLUMN_MAPPINGS:
SCRIPTS: # List any python or r scripts that mutate your raw data
PHONE_MESSAGES:
ANDROID:
RAPIDS_COLUMN_MAPPINGS:

5
src/data/streams/rapids_columns.yaml
View File

@ -87,6 +87,11 @@ PHONE_LOCATIONS:
- PROVIDER
- ACCURACY
PHONE_LOG:
- TIMESTAMP
- DEVICE_ID
- LOG_MESSAGE
PHONE_MESSAGES:
- TIMESTAMP
- DEVICE_ID

14
tests/Snakefile
View File

@ -187,13 +187,13 @@ if isinstance(config["PHONE_KEYBOARD"]["PROVIDERS"], dict):
files_to_compute.extend(expand("data/processed/features/{pid}/all_sensor_features.csv", pid=config["PIDS"]))
files_to_compute.append("data/processed/features/all_participants/all_sensor_features.csv")
if isinstance(config["PHONE_AWARE_LOG"]["PROVIDERS"], dict):
for provider in config["PHONE_AWARE_LOG"]["PROVIDERS"].keys():
if config["PHONE_AWARE_LOG"]["PROVIDERS"][provider]["COMPUTE"]:
files_to_compute.extend(expand("data/raw/{pid}/phone_aware_log_raw.csv", pid=config["PIDS"]))
files_to_compute.extend(expand("data/raw/{pid}/phone_aware_log_with_datetime.csv", pid=config["PIDS"]))
files_to_compute.extend(expand("data/interim/{pid}/phone_aware_log_features/phone_aware_log_{language}_{provider_key}.csv", pid=config["PIDS"], language=config["PHONE_AWARE_LOG"]["PROVIDERS"][provider]["SRC_LANGUAGE"].lower(), provider_key=provider.lower()))
files_to_compute.extend(expand("data/processed/features/{pid}/phone_aware_log.csv", pid=config["PIDS"]))
if isinstance(config["PHONE_LOG"]["PROVIDERS"], dict):
for provider in config["PHONE_LOG"]["PROVIDERS"].keys():
if config["PHONE_LOG"]["PROVIDERS"][provider]["COMPUTE"]:
files_to_compute.extend(expand("data/raw/{pid}/phone_log_raw.csv", pid=config["PIDS"]))
files_to_compute.extend(expand("data/raw/{pid}/phone_log_with_datetime.csv", pid=config["PIDS"]))
files_to_compute.extend(expand("data/interim/{pid}/phone_log_features/phone_log_{language}_{provider_key}.csv", pid=config["PIDS"], language=config["PHONE_LOG"]["PROVIDERS"][provider]["SRC_LANGUAGE"].lower(), provider_key=provider.lower()))
files_to_compute.extend(expand("data/processed/features/{pid}/phone_log.csv", pid=config["PIDS"]))
files_to_compute.extend(expand("data/processed/features/{pid}/all_sensor_features.csv", pid=config["PIDS"]))
files_to_compute.append("data/processed/features/all_participants/all_sensor_features.csv")

12
tests/settings/frequency/testing_config.yaml
View File

@ -127,11 +127,6 @@ PHONE_APPLICATIONS_NOTIFICATIONS:
SCRAPE_MISSING_CATEGORIES: False # whether or not to scrape missing genres, only effective if CATALOGUE_SOURCE is equal to FILE. If CATALOGUE_SOURCE is equal to GOOGLE, all genres are scraped anyway
PROVIDERS: # None implemented yet but this sensor can be used in PHONE_DATA_YIELD
# See https://www.rapids.science/latest/features/phone-aware-log/
PHONE_AWARE_LOG:
TABLE: aware_log
PROVIDERS: # None implemented yet but this sensor can be used in PHONE_DATA_YIELD
# See https://www.rapids.science/latest/features/phone-battery/
PHONE_BATTERY:
TABLE: battery
@ -260,6 +255,13 @@ PHONE_LOCATIONS:
SRC_FOLDER: "barnett" # inside src/features/phone_locations
SRC_LANGUAGE: "r"
# See https://www.rapids.science/latest/features/phone-log/
PHONE_LOG:
TABLE:
ANDROID: aware_log
IOS: ios_aware_log
PROVIDERS: # None implemented yet but this sensor can be used in PHONE_DATA_YIELD
# See https://www.rapids.science/latest/features/phone-messages/
PHONE_MESSAGES:
TABLE: messages

12
tests/settings/periodic/testing_config.yaml
View File

@ -127,11 +127,6 @@ PHONE_APPLICATIONS_NOTIFICATIONS:
SCRAPE_MISSING_CATEGORIES: False # whether or not to scrape missing genres, only effective if CATALOGUE_SOURCE is equal to FILE. If CATALOGUE_SOURCE is equal to GOOGLE, all genres are scraped anyway
PROVIDERS: # None implemented yet but this sensor can be used in PHONE_DATA_YIELD
# See https://www.rapids.science/latest/features/phone-aware-log/
PHONE_AWARE_LOG:
TABLE: aware_log
PROVIDERS: # None implemented yet but this sensor can be used in PHONE_DATA_YIELD
# See https://www.rapids.science/latest/features/phone-battery/
PHONE_BATTERY:
TABLE: battery
@ -260,6 +255,13 @@ PHONE_LOCATIONS:
SRC_FOLDER: "barnett" # inside src/features/phone_locations
SRC_LANGUAGE: "r"
# See https://www.rapids.science/latest/features/phone-log/
PHONE_LOG:
TABLE:
ANDROID: aware_log
IOS: ios_aware_log
PROVIDERS: # None implemented yet but this sensor can be used in PHONE_DATA_YIELD
# See https://www.rapids.science/latest/features/phone-messages/
PHONE_MESSAGES:
TABLE: messages

34
tools/config.schema.yaml
View File

@ -12,7 +12,6 @@ required:
- PHONE_APPLICATIONS_CRASHES
- PHONE_APPLICATIONS_FOREGROUND
- PHONE_APPLICATIONS_NOTIFICATIONS
- PHONE_AWARE_LOG
- PHONE_BATTERY
- PHONE_BLUETOOTH
- PHONE_CALLS
@ -20,6 +19,7 @@ required:
- PHONE_DATA_YIELD
- PHONE_KEYBOARD
- PHONE_LOCATIONS
- PHONE_LOG
- PHONE_MESSAGES
- PHONE_SCREEN
- PHONE_WIFI_CONNECTED
@ -392,17 +392,6 @@ properties:
type: ["null", object]
additionalProperties:
$ref: "#/definitions/PROVIDER"
PHONE_AWARE_LOG:
type: object
required: [TABLE, PROVIDERS]
properties:
TABLE:
type: string
PROVIDERS:
type: ["null", object]
additionalProperties:
$ref: "#/definitions/PROVIDER"
PHONE_BATTERY:
type: object
@ -561,7 +550,7 @@ properties:
type: ["null", array]
items:
type: string
enum: ["PHONE_ACCELEROMETER", "PHONE_ACTIVITY_RECOGNITION", "PHONE_APPLICATIONS_CRASHES", "PHONE_APPLICATIONS_FOREGROUND", "PHONE_APPLICATIONS_NOTIFICATIONS", "PHONE_AWARE_LOG", "PHONE_BATTERY", "PHONE_BLUETOOTH", "PHONE_CALLS", "PHONE_CONVERSATION", "PHONE_KEYBOARD", "PHONE_LIGHT", "PHONE_LOCATIONS", "PHONE_MESSAGES", "PHONE_SCREEN", "PHONE_WIFI_CONNECTED", "PHONE_WIFI_VISIBLE"]
enum: ["PHONE_ACCELEROMETER", "PHONE_ACTIVITY_RECOGNITION", "PHONE_APPLICATIONS_CRASHES", "PHONE_APPLICATIONS_FOREGROUND", "PHONE_APPLICATIONS_NOTIFICATIONS", "PHONE_LOG", "PHONE_BATTERY", "PHONE_BLUETOOTH", "PHONE_CALLS", "PHONE_CONVERSATION", "PHONE_KEYBOARD", "PHONE_LIGHT", "PHONE_LOCATIONS", "PHONE_MESSAGES", "PHONE_SCREEN", "PHONE_WIFI_CONNECTED", "PHONE_WIFI_VISIBLE"]
PROVIDERS:
type: ["null", object]
properties:
@ -704,7 +693,24 @@ properties:
type: boolean
additionalProperties:
$ref: "#/definitions/PROVIDER"
PHONE_LOG:
type: object
required: [TABLE, PROVIDERS]
properties:
TABLE:
type: object
required: [ANDROID, IOS]
properties:
ANDROID:
type: string
IOS:
type: string
PROVIDERS:
type: ["null", object]
additionalProperties:
$ref: "#/definitions/PROVIDER"
PHONE_MESSAGES:
type: object
required: [TABLE, PROVIDERS]