Update MUTATION structure code & docs for AR, Apps foreground, and battery sensors

docs/datastreams/add-new-data-streams.md

@@ -264,9 +264,9 @@ There is a special case for a complex mapping scenario for smartphone data strea
 
 In case you didn't notice, the examples we have used so far are grouped under an `ANDROID` key, which means they will be applied to data collected by Android phones. Additionally, each sensor has an `IOS` key for a similar purpose. We use the complex mapping described above to transform iOS data into an Android format (it's always iOS to Android and any new phone data stream must do the same).
 
-For example, this is the `format.yaml` key for `PHONE_ACTVITY_RECOGNITION`. Note that the `ANDROID` mapping is simple (one-to-one) but the `IOS` mapping is complex with two `FLAG_TO_MUTATE` columns, one `[MUTATE][COLUMN_MAPPINGS]` mapping, and one `[MUTATION][SCRIPT]`.
+For example, this is the `format.yaml` key for `PHONE_ACTVITY_RECOGNITION`. Note that the `ANDROID` mapping is simple (one-to-one) but the `IOS` mapping is complex with three `FLAG_TO_MUTATE` columns, two `[MUTATE][COLUMN_MAPPINGS]` mappings, and one `[MUTATION][SCRIPT]`.
 
-```yaml hl_lines="16 17 21 23"
+```yaml hl_lines="16 17 18 21 22 24"
 PHONE_ACTIVITY_RECOGNITION:
   ANDROID:
     RAPIDS_COLUMN_MAPPINGS:
@@ -275,7 +275,7 @@ PHONE_ACTIVITY_RECOGNITION:
       ACTIVITY_TYPE: activity_type
       ACTIVITY_NAME: activity_name
       CONFIDENCE: confidence
-    MUTATE:
+    MUTATION:
       COLUMN_MAPPINGS:
       SCRIPTS:
   IOS:
@@ -284,10 +284,11 @@ PHONE_ACTIVITY_RECOGNITION:
       DEVICE_ID: device_id
       ACTIVITY_TYPE: FLAG_TO_MUTATE
       ACTIVITY_NAME: FLAG_TO_MUTATE
-      CONFIDENCE: confidence
+      CONFIDENCE: FLAG_TO_MUTATE
-    MUTATE:
+    MUTATION:
       COLUMN_MAPPINGS:
         ACTIVITIES: activities
+        CONFIDENCE: confidence
       SCRIPTS:
         - "src/data/streams/mutations/phone/aware/activity_recogniton_ios_unification.R"
 ```
@@ -351,6 +352,6 @@ PHONE_ACTIVITY_RECOGNITION:
     }
 
     main <- function(data, stream_parameters){
-        return(unify_ios_activity_recognition(data))
+        return(unify_ios_activity_recognition(data, stream_parameters))
     }
     ```

docs/datastreams/aware-mysql.md

@@ -20,7 +20,7 @@ The yaml file that describes the format of this data stream is at:
 src/data/streams/aware_mysql/format.yaml
 ```
 
-Stream columns named `FLAG_TO_MUTATE` means they are extracted from the `FLAG_AS_EXTRA` RAPIDS column. You can refer to [OS complex mapping](../../datastreams/add-new-data-streams/#os-complex-mapping) section for detailed information.
+Stream columns named `FLAG_TO_MUTATE` means they are extracted based on the `MUTATION` section. You can refer to [OS complex mapping](../../datastreams/add-new-data-streams/#os-complex-mapping) for detailed information.
 
 !!! hint
     The mappings in this stream (RAPIDS/Stream) are the same names because AWARE data was the first stream RAPIDS supported, meaning that it considers AWARE column names the default.
@@ -39,9 +39,10 @@ Stream columns named `FLAG_TO_MUTATE` means they are extracted from the `FLAG_AS
         | DOUBLE_VALUES_1 | double_values_1 |
         | DOUBLE_VALUES_2 | double_values_2 |
 
-        **MUTATION_SCRIPTS**
+        **MUTATION**
 
-        None
+        - **COLUMN_MAPPINGS** (None)
+        - **SCRIPTS** (None)
 
     === "IOS"
     
@@ -57,13 +58,14 @@ Stream columns named `FLAG_TO_MUTATE` means they are extracted from the `FLAG_AS
         |-----------------|-----------------|
         | TIMESTAMP       | timestamp       |
         | DEVICE_ID       | device_id       |
-        | ACTIVITY_TYPE   | activity_type   |
         | ACTIVITY_NAME   | activity_name   |
+        | ACTIVITY_TYPE   | activity_type   |
         | CONFIDENCE      | confidence      |
 
-        **MUTATION_SCRIPTS**
+        **MUTATION**
 
-        None
+        - **COLUMN_MAPPINGS** (None)
+        - **SCRIPTS** (None)
 
     === "IOS"
 
@@ -73,22 +75,49 @@ Stream columns named `FLAG_TO_MUTATE` means they are extracted from the `FLAG_AS
         |-----------------|-----------------|
         | TIMESTAMP       | timestamp       |
         | DEVICE_ID       | device_id       |
-        | ACTIVITY_TYPE   | FLAG_TO_MUTATE  |
         | ACTIVITY_NAME   | FLAG_TO_MUTATE  |
-        | CONFIDENCE      | confidence      |
+        | ACTIVITY_TYPE   | FLAG_TO_MUTATE  |
-        | FLAG_AS_EXTRA   | activities      |
+        | CONFIDENCE      | FLAG_TO_MUTATE  |
 
-        **MUTATION_SCRIPTS**
+        **MUTATION**
+
+        - **COLUMN_MAPPINGS**
+
+        | Script column   | Stream column   |
+        |-----------------|-----------------|
+        | ACTIVITIES      | activities      |
+        | CONFIDENCE      | confidence      |
+
+        - **SCRIPTS**
         
         ```bash
         src/data/streams/mutations/phone/aware/activity_recogniton_ios_unification.R
         ```
 
+
+        !!! note
+            For RAPIDS columns of `ACTIVITY_NAME` and `ACTIVITY_TYPE`:
+
+            - if stream's `activities` field is automotive, set `ACTIVITY_NAME` = in_vehicle and `ACTIVITY_TYPE` = 0
+            - if stream's `activities` field is cycling, set `ACTIVITY_NAME` = on_bicycle and `ACTIVITY_TYPE` = 1
+            - if stream's `activities` field is walking, set `ACTIVITY_NAME` = walking and `ACTIVITY_TYPE` = 7
+            - if stream's `activities` field is running, set `ACTIVITY_NAME` = running and `ACTIVITY_TYPE` = 8
+            - if stream's `activities` field is stationary, set `ACTIVITY_NAME` = still and `ACTIVITY_TYPE` = 3
+            - if stream's `activities` field is unknown, set `ACTIVITY_NAME` = unknown and  `ACTIVITY_TYPE` = 4
+
+            For RAPIDS `CONFIDENCE` column:
+            
+            - if stream's `confidence` field is 0, set `CONFIDENCE` = 0
+            - if stream's `confidence` field is 1, set `CONFIDENCE` = 50
+            - if stream's `confidence` field is 2, set `CONFIDENCE` = 100
+
+
+
 ??? info "PHONE_APPLICATIONS_FOREGROUND"
 
     === "ANDROID"
     
-        **COLUMN_MAPPINGS**
+         **RAPIDS_COLUMN_MAPPINGS**
 
         | RAPIDS column      | Stream column      |
         |--------------------|--------------------|
@@ -98,15 +127,20 @@ Stream columns named `FLAG_TO_MUTATE` means they are extracted from the `FLAG_AS
         | APPLICATION_NAME   | application_name   |
         | IS_SYSTEM_APP      | is_system_app      |
 
-        **MUTATION_SCRIPTS**
+        **MUTATION**
 
-        None
+        - **COLUMN_MAPPINGS** (None)
+        - **SCRIPTS** (None)
+    
+    === "IOS"
+
+        This sensor is not supported by iOS devices.
 
 ??? info "PHONE_BATTERY"
 
     === "ANDROID"
     
-        **COLUMN_MAPPINGS**
+        **RAPIDS_COLUMN_MAPPINGS**
 
         | RAPIDS column        | Stream column       |
         |----------------------|---------------------|
@@ -114,23 +148,49 @@ Stream columns named `FLAG_TO_MUTATE` means they are extracted from the `FLAG_AS
         | DEVICE_ID            | device_id           |
         | BATTERY_STATUS       | battery_status      |
         | BATTERY_LEVEL        | battery_level       |
+        | BATTERY_SCALE        | battery_scale       |
 
-        **MUTATION_SCRIPTS**
+        **MUTATION**
 
-        None
+        - **COLUMN_MAPPINGS** (None)
+        - **SCRIPTS** (None)
 
-    === "IOS"
+    === "IOS Client V1"
 
-        **COLUMN_MAPPINGS**
+        **RAPIDS_COLUMN_MAPPINGS**
 
-        Same as ANDROID
+        | RAPIDS column        | Stream column       |
+        |----------------------|---------------------|
+        | TIMESTAMP            | timestamp           |
+        | DEVICE_ID            | device_id           |
+        | BATTERY_STATUS       | FLAG_TO_MUTATE      |
+        | BATTERY_LEVEL        | battery_level       |
+        | BATTERY_SCALE        | battery_scale       |
 
-        **MUTATION_SCRIPTS**
+        **MUTATION**
+
+        - **COLUMN_MAPPINGS**
+
+        | Script column        | Stream column       |
+        |----------------------|---------------------|
+        | BATTERY_STATUS       | battery_status      |
+        
+        - **SCRIPTS**
 
         ```bash
         src/data/streams/mutations/phone/aware/battery_ios_unification.R
         ```
 
+        !!! note
+            For RAPIDS `BATTERY_STATUS` column:
+
+            - if stream's `battery_status` field is 3, set `BATTERY_STATUS` = 5 (full status)
+            - if stream's `battery_status` field is 1, set `BATTERY_STATUS` = 3 (discharge)
+    
+    === "IOS Client V2"
+
+        Same as ANDROID
+
 
 ??? info "PHONE_CONVERSATION"
 
@@ -147,18 +207,45 @@ Stream columns named `FLAG_TO_MUTATE` means they are extracted from the `FLAG_AS
         | DOUBLE_CONVO_START   | double_convo_start  |
         | DOUBLE_CONVO_END     | double_convo_end    |
 
-        **MUTATION_SCRIPTS**
+        **MUTATION**
 
-        None
+        - **COLUMN_MAPPINGS** (None)
+        - **SCRIPTS** (None)
 
     === "IOS"
 
         **RAPIDS_COLUMN_MAPPINGS**
 
-        Same as ANDROID
+        | RAPIDS column        | Stream column       |
+        |----------------------|---------------------|
+        | TIMESTAMP            | timestamp           |
+        | DEVICE_ID            | device_id           |
+        | DOUBLE_ENERGY        | double_energy       |
+        | INFERENCE            | inference           |
+        | DOUBLE_CONVO_START   | FLAG_TO_MUTATE      |
+        | DOUBLE_CONVO_END     | FLAG_TO_MUTATE      |        
 
-        **MUTATION_SCRIPTS**
+        **MUTATION**
+
+        - **COLUMN_MAPPINGS**
+
+        | Script column        | Stream column       |
+        |----------------------|---------------------|
+        | DOUBLE_CONVO_START   | double_convo_start  |
+        | DOUBLE_CONVO_END     | double_convo_end    |
+
+        - **SCRIPTS**
         
         ```bash
         src/data/streams/mutations/phone/aware/conversation_ios_timestamp.R
         ```
+
+        !!! note
+            For RAPIDS columns of `DOUBLE_CONVO_START` and `DOUBLE_CONVO_END`:
+
+            - if stream's `double_convo_start` field is smaller than 9999999999, it is in seconds instead of milliseconds. Set `DOUBLE_CONVO_START` = 1000 * `double_convo_start`.
+            - if stream's `double_convo_end` field is smaller than 9999999999, it is in seconds instead of milliseconds. Set `DOUBLE_CONVO_END` = 1000 * `double_convo_end`.
+                
+
+
+

docs/datastreams/mandatory-phone-format.md

@@ -19,8 +19,8 @@ This is a description of the format RAPIDS needs to process data for the followi
     |-----------------|---------------------------------------------------------------------------|
     | TIMESTAMP       | An UNIX timestamp (13 digits) when a row of data was logged               |
     | DEVICE_ID       | A string that uniquely identifies a device                                |
-    | ACTIVITY_TYPE   | An integer (ranged from 0 to 8) that denotes current activity type        |
     | ACTIVITY_NAME   | An string that denotes current activity name: `in_vehicle`, `on_bicycle`, `on_foot`, `still`, `unknown`, `tilting`, `walking` or `running`   |
+    | ACTIVITY_TYPE   | An integer (ranged from 0 to 8) that denotes current activity type        |
     | CONFIDENCE      | An integer (ranged from 0 to 100) that denotes the prediction accuracy    |
 
 
@@ -38,11 +38,12 @@ This is a description of the format RAPIDS needs to process data for the followi
 ??? info "PHONE_BATTERY"
 
     | RAPIDS column        | Description                                                                                                            |
-    |--------------------|---------------------------------------------------------------------------|
+    |----------------------|------------------------------------------------------------------------------------------------------------------------|
     | TIMESTAMP            | An UNIX timestamp (13 digits) when a row of data was logged                                                            |
     | DEVICE_ID            | A string that uniquely identifies a device                                                                             |
     | BATTERY_STATUS       | An integer that denotes battery status: 0 or 1 = unknown, 2 = charging, 3 = discharging, 4 = not charging, 5 = full    |
-    | BATTERY_LEVEL      | An integer that denotes battery level                                     |
+    | BATTERY_LEVEL        | An integer that denotes battery level, between 0 and `BATTERY_SCALE`                                                   |
+    | BATTERY_SCALE        | An integer that denotes the maximum battery level                                                                      |
 
 
 ??? info "PHONE_CONVERSATION"

src/data/streams/aware_mysql/format.yaml

@@ -25,8 +25,8 @@ PHONE_ACTIVITY_RECOGNITION:
     RAPIDS_COLUMN_MAPPINGS:
       TIMESTAMP: timestamp
       DEVICE_ID: device_id
-      ACTIVITY_TYPE: activity_type
       ACTIVITY_NAME: activity_name
+      ACTIVITY_TYPE: activity_type
       CONFIDENCE: confidence
     MUTATION:
       COLUMN_MAPPINGS:
@@ -35,40 +35,50 @@ PHONE_ACTIVITY_RECOGNITION:
     RAPIDS_COLUMN_MAPPINGS:
       TIMESTAMP: timestamp
       DEVICE_ID: device_id
-      ACTIVITY_TYPE: FLAG_TO_MUTATE
       ACTIVITY_NAME: FLAG_TO_MUTATE
-      CONFIDENCE: confidence
+      ACTIVITY_TYPE: FLAG_TO_MUTATE
+      CONFIDENCE: FLAG_TO_MUTATE
     MUTATION:
       COLUMN_MAPPINGS:
         ACTIVITIES: activities
+        CONFIDENCE: confidence
       SCRIPTS: # List any python or r scripts that mutate your raw data
         - "src/data/streams/mutations/phone/aware/activity_recogniton_ios_unification.R"
 
 PHONE_APPLICATIONS_FOREGROUND:
   ANDROID:
-    COLUMN_MAPPINGS:
+    RAPIDS_COLUMN_MAPPINGS:
       TIMESTAMP: timestamp
       DEVICE_ID: device_id
       PACKAGE_NAME: package_name
       APPLICATION_NAME: application_name
       IS_SYSTEM_APP: is_system_app
-    MUTATION_SCRIPTS: # List any python or r scripts that mutate your raw data
+    MUTATION:
+      COLUMN_MAPPINGS:
+      SCRIPTS: # List any python or r scripts that mutate your raw data
 
 PHONE_BATTERY:
   ANDROID:
-    COLUMN_MAPPINGS:
+    RAPIDS_COLUMN_MAPPINGS:
       TIMESTAMP: timestamp
       DEVICE_ID: device_id
       BATTERY_STATUS: battery_status
       BATTERY_LEVEL: battery_level
-    MUTATION_SCRIPTS: # List any python or r scripts that mutate your raw data
+      BATTERY_SCALE: battery_scale
+    MUTATION:
+      COLUMN_MAPPINGS:
+      SCRIPTS: # List any python or r scripts that mutate your raw data
   IOS:
-    COLUMN_MAPPINGS:
+    RAPIDS_COLUMN_MAPPINGS:
       TIMESTAMP: timestamp
       DEVICE_ID: device_id
-      BATTERY_STATUS: battery_status
+      BATTERY_STATUS: FLAG_TO_MUTATE
       BATTERY_LEVEL: battery_level
-    MUTATION_SCRIPTS:
+      BATTERY_SCALE: battery_scale
+    MUTATION:
+      COLUMN_MAPPINGS:
+        BATTERY_STATUS: battery_status
+      SCRIPTS:
         - "src/data/streams/mutations/phone/aware/battery_ios_unification.R"
 
 PHONE_CONVERSATION:

src/data/streams/mutations/phone/aware/battery_ios_unification.R

@@ -11,6 +11,6 @@ unify_ios_battery <- function(ios_battery){
     return(ios_battery)
 }
 
-main <- function(data){
+main <- function(data, stream_parameters){
     return(unify_ios_battery(data))
 }

src/data/streams/pull_phone_data.R

@@ -65,20 +65,18 @@ validate_expected_columns_mapping <- function(schema, rapids_schema, sensor, rap
   
   if("ANDROID" %in% schema[[sensor]]){
     android_columns <- names(schema[[sensor]][["ANDROID"]][["RAPIDS_COLUMN_MAPPINGS"]])
-    android_columns <- android_columns[(android_columns != "FLAG_AS_EXTRA")]
     if(length(setdiff(rapids_columns, android_columns)) > 0)
       stop(paste(sensor," mappings are missing one or more mandatory columns for ANDROID. The missing column mappings are for ", paste(setdiff(rapids_columns, android_columns), collapse=","),"in", stream_format, " (the mappings are case sensitive)"))
     if(length(setdiff(android_columns, rapids_columns)) > 0)
-      stop(paste(sensor," mappings have one or more columns than required for ANDROID, add them as FLAG_AS_EXTRA instead. The extra column mappings are for ", paste(setdiff(android_columns, rapids_columns), collapse=","),"in", stream_format, " (the mappings are case sensitive)"))
+      stop(paste(sensor," mappings have one or more columns than required for ANDROID. The extra column mappings are for ", paste(setdiff(android_columns, rapids_columns), collapse=","),"in", stream_format, " (the mappings are case sensitive)"))
   }
 
   if("IOS" %in% schema[[sensor]]){
     ios_columns <- names(schema[[sensor]][["IOS"]][["RAPIDS_COLUMN_MAPPINGS"]])
-    ios_columns <- ios_columns[(ios_columns != "FLAG_AS_EXTRA")]
     if(length(setdiff(rapids_columns, ios_columns)) > 0)
       stop(paste(sensor," mappings are missing one or more mandatory columns for IOS. The missing column mappings are for ", paste(setdiff(rapids_columns, ios_columns), collapse=","),"in", stream_format, " (the mappings are case sensitive)"))
     if(length(setdiff(ios_columns, rapids_columns)) > 0)
-      stop(paste(sensor," mappings have one or more columns than required for IOS, add them as FLAG_AS_EXTRA instead. The extra column mappings are for ", paste(setdiff(ios_columns, rapids_columns), collapse=","),"in", stream_format, " (the mappings are case sensitive)"))
+      stop(paste(sensor," mappings have one or more columns than required for IOS. The extra column mappings are for ", paste(setdiff(ios_columns, rapids_columns), collapse=","),"in", stream_format, " (the mappings are case sensitive)"))
   }
 }
 

src/data/streams/rapids_columns.yaml

@@ -8,8 +8,8 @@ PHONE_ACCELEROMETER:
 PHONE_ACTIVITY_RECOGNITION:
   - TIMESTAMP
   - DEVICE_ID
-  - ACTIVITY_TYPE
   - ACTIVITY_NAME
+  - ACTIVITY_TYPE
   - CONFIDENCE
 
 PHONE_APPLICATIONS_FOREGROUND:
@@ -24,6 +24,7 @@ PHONE_BATTERY:
   - DEVICE_ID
   - BATTERY_STATUS
   - BATTERY_LEVEL
+  - BATTERY_SCALE
 
 PHONE_CONVERSATION:
   - TIMESTAMP