junos
/
rapids
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Update MUTATION structure code & docs for AR, Apps foreground, and battery sensors

pull/128/head
Meng Li 2021-03-09 21:52:38 -05:00 committed by JulioV
parent 6970954358
commit 6d06d2b1eb
7 changed files with 154 additions and 56 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

17
docs/datastreams/add-new-data-streams.md
View File

@ -264,9 +264,9 @@ There is a special case for a complex mapping scenario for smartphone data strea
In case you didn't notice, the examples we have used so far are grouped under an `ANDROID` key, which means they will be applied to data collected by Android phones. Additionally, each sensor has an `IOS` key for a similar purpose. We use the complex mapping described above to transform iOS data into an Android format (it's always iOS to Android and any new phone data stream must do the same).
For example, this is the `format.yaml` key for `PHONE_ACTVITY_RECOGNITION`. Note that the `ANDROID` mapping is simple (one-to-one) but the `IOS` mapping is complex with two `FLAG_TO_MUTATE` columns, one `[MUTATE][COLUMN_MAPPINGS]` mapping, and one `[MUTATION][SCRIPT]`.
For example, this is the `format.yaml` key for `PHONE_ACTVITY_RECOGNITION`. Note that the `ANDROID` mapping is simple (one-to-one) but the `IOS` mapping is complex with three `FLAG_TO_MUTATE` columns, two `[MUTATE][COLUMN_MAPPINGS]` mappings, and one `[MUTATION][SCRIPT]`.
```yaml hl_lines="16 17 21 23"
```yaml hl_lines="16 17 18 21 22 24"
PHONE_ACTIVITY_RECOGNITION:
ANDROID:
RAPIDS_COLUMN_MAPPINGS:
@ -275,7 +275,7 @@ PHONE_ACTIVITY_RECOGNITION:
ACTIVITY_TYPE: activity_type
ACTIVITY_NAME: activity_name
CONFIDENCE: confidence
MUTATE:
MUTATION:
COLUMN_MAPPINGS:
SCRIPTS:
IOS:
@ -284,10 +284,11 @@ PHONE_ACTIVITY_RECOGNITION:
DEVICE_ID: device_id
ACTIVITY_TYPE: FLAG_TO_MUTATE
ACTIVITY_NAME: FLAG_TO_MUTATE
CONFIDENCE: confidence
MUTATE:
CONFIDENCE: FLAG_TO_MUTATE
MUTATION:
COLUMN_MAPPINGS:
ACTIVITIES: activities
CONFIDENCE: confidence
SCRIPTS:
- "src/data/streams/mutations/phone/aware/activity_recogniton_ios_unification.R"
```
@ -342,8 +343,8 @@ PHONE_ACTIVITY_RECOGNITION:
activities == "stationary" ~ 3,
activities == "unknown" ~ 4),
confidence = case_when(confidence == 0 ~ 0,
confidence == 1 ~ 50,
confidence == 2 ~ 100)
confidence == 1 ~ 50,
confidence == 2 ~ 100)
) %>%
select(-activities)
@ -351,6 +352,6 @@ PHONE_ACTIVITY_RECOGNITION:
}
main <- function(data, stream_parameters){
return(unify_ios_activity_recognition(data))
return(unify_ios_activity_recognition(data, stream_parameters))
}
```

135
docs/datastreams/aware-mysql.md
View File

@ -20,7 +20,7 @@ The yaml file that describes the format of this data stream is at:
src/data/streams/aware_mysql/format.yaml
```
Stream columns named `FLAG_TO_MUTATE` means they are extracted from the `FLAG_AS_EXTRA` RAPIDS column. You can refer to [OS complex mapping](../../datastreams/add-new-data-streams/#os-complex-mapping) section for detailed information.
Stream columns named `FLAG_TO_MUTATE` means they are extracted based on the `MUTATION` section. You can refer to [OS complex mapping](../../datastreams/add-new-data-streams/#os-complex-mapping) for detailed information.
!!! hint
The mappings in this stream (RAPIDS/Stream) are the same names because AWARE data was the first stream RAPIDS supported, meaning that it considers AWARE column names the default.
@ -39,9 +39,10 @@ Stream columns named `FLAG_TO_MUTATE` means they are extracted from the `FLAG_AS
| DOUBLE_VALUES_1 | double_values_1 |
| DOUBLE_VALUES_2 | double_values_2 |
**MUTATION_SCRIPTS**
**MUTATION**
None
- **COLUMN_MAPPINGS** (None)
- **SCRIPTS** (None)
=== "IOS"
@ -57,13 +58,14 @@ Stream columns named `FLAG_TO_MUTATE` means they are extracted from the `FLAG_AS
|-----------------|-----------------|
| TIMESTAMP | timestamp |
| DEVICE_ID | device_id |
| ACTIVITY_TYPE | activity_type |
| ACTIVITY_NAME | activity_name |
| ACTIVITY_TYPE | activity_type |
| CONFIDENCE | confidence |
**MUTATION_SCRIPTS**
**MUTATION**
None
- **COLUMN_MAPPINGS** (None)
- **SCRIPTS** (None)
=== "IOS"
@ -73,22 +75,49 @@ Stream columns named `FLAG_TO_MUTATE` means they are extracted from the `FLAG_AS
|-----------------|-----------------|
| TIMESTAMP | timestamp |
| DEVICE_ID | device_id |
| ACTIVITY_TYPE | FLAG_TO_MUTATE |
| ACTIVITY_NAME | FLAG_TO_MUTATE |
| CONFIDENCE | confidence |
| FLAG_AS_EXTRA | activities |
| ACTIVITY_TYPE | FLAG_TO_MUTATE |
| CONFIDENCE | FLAG_TO_MUTATE |
**MUTATION_SCRIPTS**
**MUTATION**
- **COLUMN_MAPPINGS**
| Script column | Stream column |
|-----------------|-----------------|
| ACTIVITIES | activities |
| CONFIDENCE | confidence |
- **SCRIPTS**
```bash
src/data/streams/mutations/phone/aware/activity_recogniton_ios_unification.R
```
!!! note
For RAPIDS columns of `ACTIVITY_NAME` and `ACTIVITY_TYPE`:
- if stream's `activities` field is automotive, set `ACTIVITY_NAME` = in_vehicle and `ACTIVITY_TYPE` = 0
- if stream's `activities` field is cycling, set `ACTIVITY_NAME` = on_bicycle and `ACTIVITY_TYPE` = 1
- if stream's `activities` field is walking, set `ACTIVITY_NAME` = walking and `ACTIVITY_TYPE` = 7
- if stream's `activities` field is running, set `ACTIVITY_NAME` = running and `ACTIVITY_TYPE` = 8
- if stream's `activities` field is stationary, set `ACTIVITY_NAME` = still and `ACTIVITY_TYPE` = 3
- if stream's `activities` field is unknown, set `ACTIVITY_NAME` = unknown and `ACTIVITY_TYPE` = 4
For RAPIDS `CONFIDENCE` column:
- if stream's `confidence` field is 0, set `CONFIDENCE` = 0
- if stream's `confidence` field is 1, set `CONFIDENCE` = 50
- if stream's `confidence` field is 2, set `CONFIDENCE` = 100
??? info "PHONE_APPLICATIONS_FOREGROUND"
=== "ANDROID"
**COLUMN_MAPPINGS**
**RAPIDS_COLUMN_MAPPINGS**
| RAPIDS column | Stream column |
|--------------------|--------------------|
@ -98,15 +127,20 @@ Stream columns named `FLAG_TO_MUTATE` means they are extracted from the `FLAG_AS
| APPLICATION_NAME | application_name |
| IS_SYSTEM_APP | is_system_app |
**MUTATION_SCRIPTS**
**MUTATION**
None
- **COLUMN_MAPPINGS** (None)
- **SCRIPTS** (None)
=== "IOS"
This sensor is not supported by iOS devices.
??? info "PHONE_BATTERY"
=== "ANDROID"
**COLUMN_MAPPINGS**
**RAPIDS_COLUMN_MAPPINGS**
| RAPIDS column | Stream column |
|----------------------|---------------------|
@ -114,23 +148,49 @@ Stream columns named `FLAG_TO_MUTATE` means they are extracted from the `FLAG_AS
| DEVICE_ID | device_id |
| BATTERY_STATUS | battery_status |
| BATTERY_LEVEL | battery_level |
| BATTERY_SCALE | battery_scale |
**MUTATION_SCRIPTS**
**MUTATION**
None
- **COLUMN_MAPPINGS** (None)
- **SCRIPTS** (None)
=== "IOS"
=== "IOS Client V1"
**COLUMN_MAPPINGS**
**RAPIDS_COLUMN_MAPPINGS**
Same as ANDROID
| RAPIDS column | Stream column |
|----------------------|---------------------|
| TIMESTAMP | timestamp |
| DEVICE_ID | device_id |
| BATTERY_STATUS | FLAG_TO_MUTATE |
| BATTERY_LEVEL | battery_level |
| BATTERY_SCALE | battery_scale |
**MUTATION_SCRIPTS**
**MUTATION**
- **COLUMN_MAPPINGS**
| Script column | Stream column |
|----------------------|---------------------|
| BATTERY_STATUS | battery_status |
- **SCRIPTS**
```bash
src/data/streams/mutations/phone/aware/battery_ios_unification.R
```
!!! note
For RAPIDS `BATTERY_STATUS` column:
- if stream's `battery_status` field is 3, set `BATTERY_STATUS` = 5 (full status)
- if stream's `battery_status` field is 1, set `BATTERY_STATUS` = 3 (discharge)
=== "IOS Client V2"
Same as ANDROID
??? info "PHONE_CONVERSATION"
@ -147,18 +207,45 @@ Stream columns named `FLAG_TO_MUTATE` means they are extracted from the `FLAG_AS
| DOUBLE_CONVO_START | double_convo_start |
| DOUBLE_CONVO_END | double_convo_end |
**MUTATION_SCRIPTS**
**MUTATION**
None
- **COLUMN_MAPPINGS** (None)
- **SCRIPTS** (None)
=== "IOS"
**RAPIDS_COLUMN_MAPPINGS**
Same as ANDROID
| RAPIDS column | Stream column |
|----------------------|---------------------|
| TIMESTAMP | timestamp |
| DEVICE_ID | device_id |
| DOUBLE_ENERGY | double_energy |
| INFERENCE | inference |
| DOUBLE_CONVO_START | FLAG_TO_MUTATE |
| DOUBLE_CONVO_END | FLAG_TO_MUTATE |
**MUTATION_SCRIPTS**
**MUTATION**
- **COLUMN_MAPPINGS**
| Script column | Stream column |
|----------------------|---------------------|
| DOUBLE_CONVO_START | double_convo_start |
| DOUBLE_CONVO_END | double_convo_end |
- **SCRIPTS**
```bash
src/data/streams/mutations/phone/aware/conversation_ios_timestamp.R
```
!!! note
For RAPIDS columns of `DOUBLE_CONVO_START` and `DOUBLE_CONVO_END`:
- if stream's `double_convo_start` field is smaller than 9999999999, it is in seconds instead of milliseconds. Set `DOUBLE_CONVO_START` = 1000 * `double_convo_start`.
- if stream's `double_convo_end` field is smaller than 9999999999, it is in seconds instead of milliseconds. Set `DOUBLE_CONVO_END` = 1000 * `double_convo_end`.

15
docs/datastreams/mandatory-phone-format.md
View File

@ -19,8 +19,8 @@ This is a description of the format RAPIDS needs to process data for the followi
|-----------------|---------------------------------------------------------------------------|
| TIMESTAMP | An UNIX timestamp (13 digits) when a row of data was logged |
| DEVICE_ID | A string that uniquely identifies a device |
| ACTIVITY_TYPE | An integer (ranged from 0 to 8) that denotes current activity type |
| ACTIVITY_NAME | An string that denotes current activity name: `in_vehicle`, `on_bicycle`, `on_foot`, `still`, `unknown`, `tilting`, `walking` or `running` |
| ACTIVITY_TYPE | An integer (ranged from 0 to 8) that denotes current activity type |
| CONFIDENCE | An integer (ranged from 0 to 100) that denotes the prediction accuracy |
@ -37,12 +37,13 @@ This is a description of the format RAPIDS needs to process data for the followi
??? info "PHONE_BATTERY"
| RAPIDS column | Description |
|--------------------|---------------------------------------------------------------------------|
| TIMESTAMP | An UNIX timestamp (13 digits) when a row of data was logged |
| DEVICE_ID | A string that uniquely identifies a device |
| BATTERY_STATUS | An integer that denotes battery status: 0 or 1 = unknown, 2 = charging, 3 = discharging, 4 = not charging, 5 = full |
| BATTERY_LEVEL | An integer that denotes battery level |
| RAPIDS column | Description |
|----------------------|------------------------------------------------------------------------------------------------------------------------|
| TIMESTAMP | An UNIX timestamp (13 digits) when a row of data was logged |
| DEVICE_ID | A string that uniquely identifies a device |
| BATTERY_STATUS | An integer that denotes battery status: 0 or 1 = unknown, 2 = charging, 3 = discharging, 4 = not charging, 5 = full |
| BATTERY_LEVEL | An integer that denotes battery level, between 0 and `BATTERY_SCALE` |
| BATTERY_SCALE | An integer that denotes the maximum battery level |
??? info "PHONE_CONVERSATION"

32
src/data/streams/aware_mysql/format.yaml
View File

@ -25,8 +25,8 @@ PHONE_ACTIVITY_RECOGNITION:
RAPIDS_COLUMN_MAPPINGS:
TIMESTAMP: timestamp
DEVICE_ID: device_id
ACTIVITY_TYPE: activity_type
ACTIVITY_NAME: activity_name
ACTIVITY_TYPE: activity_type
CONFIDENCE: confidence
MUTATION:
COLUMN_MAPPINGS:
@ -35,41 +35,51 @@ PHONE_ACTIVITY_RECOGNITION:
RAPIDS_COLUMN_MAPPINGS:
TIMESTAMP: timestamp
DEVICE_ID: device_id
ACTIVITY_TYPE: FLAG_TO_MUTATE
ACTIVITY_NAME: FLAG_TO_MUTATE
CONFIDENCE: confidence
ACTIVITY_TYPE: FLAG_TO_MUTATE
CONFIDENCE: FLAG_TO_MUTATE
MUTATION:
COLUMN_MAPPINGS:
ACTIVITIES: activities
CONFIDENCE: confidence
SCRIPTS: # List any python or r scripts that mutate your raw data
- "src/data/streams/mutations/phone/aware/activity_recogniton_ios_unification.R"
PHONE_APPLICATIONS_FOREGROUND:
ANDROID:
COLUMN_MAPPINGS:
RAPIDS_COLUMN_MAPPINGS:
TIMESTAMP: timestamp
DEVICE_ID: device_id
PACKAGE_NAME: package_name
APPLICATION_NAME: application_name
IS_SYSTEM_APP: is_system_app
MUTATION_SCRIPTS: # List any python or r scripts that mutate your raw data
MUTATION:
COLUMN_MAPPINGS:
SCRIPTS: # List any python or r scripts that mutate your raw data
PHONE_BATTERY:
ANDROID:
COLUMN_MAPPINGS:
RAPIDS_COLUMN_MAPPINGS:
TIMESTAMP: timestamp
DEVICE_ID: device_id
BATTERY_STATUS: battery_status
BATTERY_LEVEL: battery_level
MUTATION_SCRIPTS: # List any python or r scripts that mutate your raw data
BATTERY_SCALE: battery_scale
MUTATION:
COLUMN_MAPPINGS:
SCRIPTS: # List any python or r scripts that mutate your raw data
IOS:
COLUMN_MAPPINGS:
RAPIDS_COLUMN_MAPPINGS:
TIMESTAMP: timestamp
DEVICE_ID: device_id
BATTERY_STATUS: battery_status
BATTERY_STATUS: FLAG_TO_MUTATE
BATTERY_LEVEL: battery_level
MUTATION_SCRIPTS:
- "src/data/streams/mutations/phone/aware/battery_ios_unification.R"
BATTERY_SCALE: battery_scale
MUTATION:
COLUMN_MAPPINGS:
BATTERY_STATUS: battery_status
SCRIPTS:
- "src/data/streams/mutations/phone/aware/battery_ios_unification.R"
PHONE_CONVERSATION:
ANDROID:

2
src/data/streams/mutations/phone/aware/battery_ios_unification.R
View File

@ -11,6 +11,6 @@ unify_ios_battery <- function(ios_battery){
return(ios_battery)
}
main <- function(data){
main <- function(data, stream_parameters){
return(unify_ios_battery(data))
}

6
src/data/streams/pull_phone_data.R
View File

@ -65,20 +65,18 @@ validate_expected_columns_mapping <- function(schema, rapids_schema, sensor, rap
if("ANDROID" %in% schema[[sensor]]){
android_columns <- names(schema[[sensor]][["ANDROID"]][["RAPIDS_COLUMN_MAPPINGS"]])
android_columns <- android_columns[(android_columns != "FLAG_AS_EXTRA")]
if(length(setdiff(rapids_columns, android_columns)) > 0)
stop(paste(sensor," mappings are missing one or more mandatory columns for ANDROID. The missing column mappings are for ", paste(setdiff(rapids_columns, android_columns), collapse=","),"in", stream_format, " (the mappings are case sensitive)"))
if(length(setdiff(android_columns, rapids_columns)) > 0)
stop(paste(sensor," mappings have one or more columns than required for ANDROID, add them as FLAG_AS_EXTRA instead. The extra column mappings are for ", paste(setdiff(android_columns, rapids_columns), collapse=","),"in", stream_format, " (the mappings are case sensitive)"))
stop(paste(sensor," mappings have one or more columns than required for ANDROID. The extra column mappings are for ", paste(setdiff(android_columns, rapids_columns), collapse=","),"in", stream_format, " (the mappings are case sensitive)"))
}
if("IOS" %in% schema[[sensor]]){
ios_columns <- names(schema[[sensor]][["IOS"]][["RAPIDS_COLUMN_MAPPINGS"]])
ios_columns <- ios_columns[(ios_columns != "FLAG_AS_EXTRA")]
if(length(setdiff(rapids_columns, ios_columns)) > 0)
stop(paste(sensor," mappings are missing one or more mandatory columns for IOS. The missing column mappings are for ", paste(setdiff(rapids_columns, ios_columns), collapse=","),"in", stream_format, " (the mappings are case sensitive)"))
if(length(setdiff(ios_columns, rapids_columns)) > 0)
stop(paste(sensor," mappings have one or more columns than required for IOS, add them as FLAG_AS_EXTRA instead. The extra column mappings are for ", paste(setdiff(ios_columns, rapids_columns), collapse=","),"in", stream_format, " (the mappings are case sensitive)"))
stop(paste(sensor," mappings have one or more columns than required for IOS. The extra column mappings are for ", paste(setdiff(ios_columns, rapids_columns), collapse=","),"in", stream_format, " (the mappings are case sensitive)"))
}
}

3
src/data/streams/rapids_columns.yaml
View File

@ -8,8 +8,8 @@ PHONE_ACCELEROMETER:
PHONE_ACTIVITY_RECOGNITION:
- TIMESTAMP
- DEVICE_ID
- ACTIVITY_TYPE
- ACTIVITY_NAME
- ACTIVITY_TYPE
- CONFIDENCE
PHONE_APPLICATIONS_FOREGROUND:
@ -24,6 +24,7 @@ PHONE_BATTERY:
- DEVICE_ID
- BATTERY_STATUS
- BATTERY_LEVEL
- BATTERY_SCALE
PHONE_CONVERSATION:
- TIMESTAMP