Fixed bluetooth bug and added feeature testing

docs/develop/test_cases.rst

@@ -18,7 +18,7 @@ Messages (SMS)
     - The second day's data contains 6 records for each of only 2 ``epoch`` (currently ``morning`` and ``evening``)
     - The raw message data contains records for both ``message_types`` (i.e. ``recieved`` and ``sent``) in both days in all epochs. The number records with each ``message_types`` per epoch is randomly distributed There is at least one records with each ``message_types`` per epoch.
     - There is one raw message data file each, as described above, for testing both iOS and Android data. 
-    - There is also an additional for both android and iOS that contains no raw input data for testing empty data files
+    - There is also an additional empty data file for both android and iOS for testing empty data files
 
 Calls
 """""""
@@ -30,7 +30,7 @@ Calls
     - The second day's data contains 6 records for each of only 2 ``epoch`` (currently ``morning`` and ``evening``)
     - The call data contains records for all ``call_types`` (i.e. ``incoming``, ``outgoing`` and ``missed``) in both days in all epochs. The number records with each of the ``call_types`` per epoch is randomly distributed. There is at least one records with each ``call_types`` per epoch.
     - There is one call data file each, as described above, for testing both iOS and Android data. 
-    - There is also an additional for both android and iOS that contains no raw input data for testing empty data files
+    - There is also an additional empty data file for both android and iOS for testing empty data files
 
 Screen
 """"""""
@@ -42,7 +42,7 @@ Screen
     - The screen delta data contains 1 record to represent an ``unlock`` episode that falls across the boundary of 2 epochs. Namely the ``unlock`` episode starts in one epoch and ends in the next, thus there is a record for ``unlock`` episodes that fall across ``night`` to ``morning``, ``morning`` to ``afternoon`` and finally ``afternoon`` to ``night``
     - The testing is done for ``unlock`` episode_type.
     - There is one screen data file each for testing both iOS and Android data formats.
-    - There is also an additional for both android and iOS that contains no raw input data for testing empty data files
+    - There is also an additional empty data file for both android and iOS for testing empty data files
 
 Battery
 """""""""
@@ -53,5 +53,13 @@ Battery
     - The battery delta data contains 1 record each for a ``charging`` and ``discharging`` episode that falls within an ``epoch`` for every ``epoch``. Thus for the ``daily`` epoch there would be multiple ``charging`` and ``discharging`` episodes
     - Since either a ``charging`` episode or a ``discharging`` episode and not both can occur across epochs, in order to test epsiodes that occur across epochs alternating episodes of ``charging`` and ``discharging`` episodes that fall across ``night`` to ``morning``, ``morning`` to ``afternoon`` and finally ``afternoon`` to ``night`` are present in the battery delta data. This starts with a ``discharging`` episode that begins in ``night`` and end in ``morning``.
     - There is one battery data file each, for testing both iOS and Android data formats.
-    - There is also an additional for both android and iOS that contains no raw input data for testing empty data files
+    - There is also an additional empty data file for both android and iOS for testing empty data files
 
+Bluetooth
+""""""""""
+
+    - The raw bluetooth data file contains data for 1 day. 
+    - The raw bluetooth data contains at least 2 records for each ``epoch``.
+    - An option of 5 bluetooth devices are randomly distributed throughout the data records.
+    - There is one raw bluetooth data file each, for testing both iOS and Android data formats.
+    - There is also an additional empty data file for both android and iOS for testing empty data files

src/features/bluetooth/bluetooth_base.R

@@ -9,21 +9,26 @@ filter_by_day_segment <- function(data, day_segment) {
 }
 
 compute_bluetooth_feature <- function(data, feature, day_segment){
+  data <- data %>% filter_by_day_segment(day_segment)
   if(feature %in% c("countscans", "uniquedevices")){
-    data <- data %>% filter_by_day_segment(day_segment)
     data <- switch(feature,
               "countscans" = data %>% summarise(!!paste("bluetooth", day_segment, feature, sep = "_") := n()),
               "uniquedevices" = data %>% summarise(!!paste("bluetooth", day_segment, feature, sep = "_") := n_distinct(bt_address)))
     return(data)
    } else if(feature == "countscansmostuniquedevice"){
      # Get the most scanned device
-    data <- data %>% group_by(bt_address) %>% 
+    mostuniquedevice <- data %>% 
+      group_by(bt_address) %>% 
       mutate(N=n()) %>% 
       ungroup() %>%
-      filter(N == max(N))
+      filter(N == max(N)) %>% 
+      head(1) %>% # if there are multiple device with the same amount of scans pick the first one only
+      pull(bt_address)
     return(data %>% 
-             filter_by_day_segment(day_segment) %>%
-             summarise(!!paste("bluetooth", day_segment, feature, sep = "_") := n()))
+             filter(bt_address == mostuniquedevice) %>%
+             group_by(local_date) %>% 
+             summarise(!!paste("bluetooth", day_segment, feature, sep = "_") := n()) %>%
+             replace(is.na(.), 0))
   }
 }
 

tests/Snakefile

@@ -37,6 +37,11 @@ if config["BATTERY"]["COMPUTE"]:
     files_to_compute.extend(expand("data/processed/{pid}/battery_deltas.csv", pid=config["PIDS"]))
     files_to_compute.extend(expand("data/processed/{pid}/battery_{day_segment}.csv", pid = config["PIDS"], day_segment = config["BATTERY"]["DAY_SEGMENTS"]))
 
+if config["BLUETOOTH"]["COMPUTE"]:
+    files_to_compute.extend(expand("data/raw/{pid}/{sensor}_raw.csv", pid=config["PIDS"], sensor=config["BLUETOOTH"]["DB_TABLE"]))
+    files_to_compute.extend(expand("data/raw/{pid}/{sensor}_with_datetime.csv", pid=config["PIDS"], sensor=config["BLUETOOTH"]["DB_TABLE"]))
+    files_to_compute.extend(expand("data/processed/{pid}/bluetooth_{segment}.csv", pid=config["PIDS"], segment = config["BLUETOOTH"]["DAY_SEGMENTS"]))
+
 rule all:
     input:
         files_to_compute

tests/data/processed/test01/bluetooth_afternoon.csv

@@ -0,0 +1,2 @@
+"local_date","bluetooth_afternoon_countscans","bluetooth_afternoon_uniquedevices","bluetooth_afternoon_countscansmostuniquedevice"
+"2020-07-02",2,2,1

tests/data/processed/test01/bluetooth_daily.csv

@@ -0,0 +1,2 @@
+"local_date","bluetooth_daily_countscans","bluetooth_daily_uniquedevices","bluetooth_daily_countscansmostuniquedevice"
+"2020-07-02",14,5,4

tests/data/processed/test01/bluetooth_evening.csv

@@ -0,0 +1,2 @@
+"local_date","bluetooth_evening_countscans","bluetooth_evening_uniquedevices","bluetooth_evening_countscansmostuniquedevice"
+"2020-07-02",5,4,2

tests/data/processed/test01/bluetooth_morning.csv

@@ -0,0 +1,2 @@
+"local_date","bluetooth_morning_countscans","bluetooth_morning_uniquedevices","bluetooth_morning_countscansmostuniquedevice"
+"2020-07-02",3,2,2

tests/data/processed/test01/bluetooth_night.csv

@@ -0,0 +1,2 @@
+"local_date","bluetooth_night_countscans","bluetooth_night_uniquedevices","bluetooth_night_countscansmostuniquedevice"
+"2020-07-02",4,4,1

tests/data/processed/test02/bluetooth_afternoon.csv

@@ -0,0 +1,2 @@
+"local_date","bluetooth_afternoon_countscans","bluetooth_afternoon_uniquedevices","bluetooth_afternoon_countscansmostuniquedevice"
+"2020-07-02",2,2,1

tests/data/processed/test02/bluetooth_daily.csv

@@ -0,0 +1,2 @@
+"local_date","bluetooth_daily_countscans","bluetooth_daily_uniquedevices","bluetooth_daily_countscansmostuniquedevice"
+"2020-07-02",14,5,5

tests/data/processed/test02/bluetooth_evening.csv

@@ -0,0 +1,2 @@
+"local_date","bluetooth_evening_countscans","bluetooth_evening_uniquedevices","bluetooth_evening_countscansmostuniquedevice"
+"2020-07-02",5,3,3

tests/data/processed/test02/bluetooth_morning.csv

@@ -0,0 +1,2 @@
+"local_date","bluetooth_morning_countscans","bluetooth_morning_uniquedevices","bluetooth_morning_countscansmostuniquedevice"
+"2020-07-02",3,3,1

tests/data/processed/test02/bluetooth_night.csv

@@ -0,0 +1,2 @@
+"local_date","bluetooth_night_countscans","bluetooth_night_uniquedevices","bluetooth_night_countscansmostuniquedevice"
+"2020-07-02",4,2,2

tests/data/processed/test03/bluetooth_afternoon.csv

@@ -0,0 +1 @@
+"local_date","bluetooth_afternoon_countscans","bluetooth_afternoon_uniquedevices","bluetooth_afternoon_countscansmostuniquedevice"

tests/data/processed/test03/bluetooth_daily.csv

@@ -0,0 +1 @@
+"local_date","bluetooth_daily_countscans","bluetooth_daily_uniquedevices","bluetooth_daily_countscansmostuniquedevice"

tests/data/processed/test03/bluetooth_evening.csv

@@ -0,0 +1 @@
+"local_date","bluetooth_evening_countscans","bluetooth_evening_uniquedevices","bluetooth_evening_countscansmostuniquedevice"

tests/data/processed/test03/bluetooth_morning.csv

@@ -0,0 +1 @@
+"local_date","bluetooth_morning_countscans","bluetooth_morning_uniquedevices","bluetooth_morning_countscansmostuniquedevice"

tests/data/processed/test03/bluetooth_night.csv

@@ -0,0 +1 @@
+"local_date","bluetooth_night_countscans","bluetooth_night_uniquedevices","bluetooth_night_countscansmostuniquedevice"

tests/data/processed/test04/bluetooth_afternoon.csv

@@ -0,0 +1 @@
+"local_date","bluetooth_afternoon_countscans","bluetooth_afternoon_uniquedevices","bluetooth_afternoon_countscansmostuniquedevice"

tests/data/processed/test04/bluetooth_daily.csv

@@ -0,0 +1 @@
+"local_date","bluetooth_daily_countscans","bluetooth_daily_uniquedevices","bluetooth_daily_countscansmostuniquedevice"

tests/data/processed/test04/bluetooth_evening.csv

@@ -0,0 +1 @@
+"local_date","bluetooth_evening_countscans","bluetooth_evening_uniquedevices","bluetooth_evening_countscansmostuniquedevice"

tests/data/processed/test04/bluetooth_morning.csv

@@ -0,0 +1 @@
+"local_date","bluetooth_morning_countscans","bluetooth_morning_uniquedevices","bluetooth_morning_countscansmostuniquedevice"

tests/data/processed/test04/bluetooth_night.csv

@@ -0,0 +1 @@
+"local_date","bluetooth_night_countscans","bluetooth_night_uniquedevices","bluetooth_night_countscansmostuniquedevice"

tests/data/raw/test01/bluetooth_raw.csv

@@ -0,0 +1,15 @@
+timestamp,device_id,bt_address,bt_name,bt_rssi,label
+1593684000123,wYESbVwI-4GfR-G5I6-7iKL-tOmCKs02MBun,A6:93:AF:8C:36,Phone,-92,1593684000123
+1593705599321,wYESbVwI-4GfR-G5I6-7iKL-tOmCKs02MBun,A6:93:AF:8C:36,Phone,-79,1593684000123
+1593687638456,wYESbVwI-4GfR-G5I6-7iKL-tOmCKs02MBun,7E:D1:4E:80:2B,,-96,1593684000123
+1593705600654,wYESbVwI-4GfR-G5I6-7iKL-tOmCKs02MBun,7E:D1:4E:80:2B,,-92,1593705600654
+1593727199546,wYESbVwI-4GfR-G5I6-7iKL-tOmCKs02MBun,4B:2C:AA:88:1B,Smart TV,-95,1593705600654
+1593727200564,wYESbVwI-4GfR-G5I6-7iKL-tOmCKs02MBun,F5:F6:EC:8D:6C,Laptop,-88,1593727200564
+1593748799465,wYESbVwI-4GfR-G5I6-7iKL-tOmCKs02MBun,A6:93:AF:8C:36,Phone,-91,1593727200564
+1593738619645,wYESbVwI-4GfR-G5I6-7iKL-tOmCKs02MBun,10:6F:61:E4:A9,Speakers,-90,1593727200564
+1593747525789,wYESbVwI-4GfR-G5I6-7iKL-tOmCKs02MBun,7E:D1:4E:80:2B,,-84,1593727200564
+1593733518987,wYESbVwI-4GfR-G5I6-7iKL-tOmCKs02MBun,A6:93:AF:8C:36,Phone,-78,1593727200564
+1593662400798,wYESbVwI-4GfR-G5I6-7iKL-tOmCKs02MBun,7E:D1:4E:80:2B,,-76,1593662400798
+1593683999978,wYESbVwI-4GfR-G5I6-7iKL-tOmCKs02MBun,F5:F6:EC:8D:6C,Laptop,-97,1593662400798
+1593675299879,wYESbVwI-4GfR-G5I6-7iKL-tOmCKs02MBun,10:6F:61:E4:A9,Speakers,-93,1593662400798
+1593665344132,wYESbVwI-4GfR-G5I6-7iKL-tOmCKs02MBun,4B:2C:AA:88:1B,Smart TV,-94,1593662400798

tests/data/raw/test02/bluetooth_raw.csv

@@ -0,0 +1,15 @@
+timestamp,device_id,bt_address,bt_name,bt_rssi,label
+1593684000123,7yKzcQm4-xKTC-0bhC-PZXC-3jAbRIXOsf5w,qaRo3mcQ-m0aJ-awAa-ACT9-6HcxDD26tsxe,Phone,-99,1593684000123
+1593705599132,7yKzcQm4-xKTC-0bhC-PZXC-3jAbRIXOsf5w,ed0moSsY-5Qzi-4kDS-tAyj-eiTUK3LmEG9S,Laptop,-87,1593684000123
+1593687638321,7yKzcQm4-xKTC-0bhC-PZXC-3jAbRIXOsf5w,R3cS8NPr-f43b-sIKF-fEjr-oRHjMuwzYX1V,Smart TV,-89,1593684000123
+1593705600312,7yKzcQm4-xKTC-0bhC-PZXC-3jAbRIXOsf5w,R3cS8NPr-f43b-sIKF-fEjr-oRHjMuwzYX1V,Smart TV,-96,1593705600312
+1593727199456,7yKzcQm4-xKTC-0bhC-PZXC-3jAbRIXOsf5w,ed0moSsY-5Qzi-4kDS-tAyj-eiTUK3LmEG9S,Laptop,-82,1593705600312
+1593727200465,7yKzcQm4-xKTC-0bhC-PZXC-3jAbRIXOsf5w,skYoi2Iy-Hk69-45Ss-mMEc-mdSt0Xi4iwA5,,-79,1593727200465
+1593748799546,7yKzcQm4-xKTC-0bhC-PZXC-3jAbRIXOsf5w,R3cS8NPr-f43b-sIKF-fEjr-oRHjMuwzYX1V,Smart TV,-88,1593662400789
+1593738619564,7yKzcQm4-xKTC-0bhC-PZXC-3jAbRIXOsf5w,R3cS8NPr-f43b-sIKF-fEjr-oRHjMuwzYX1V,Smart TV,-97,1593662400789
+1593747525654,7yKzcQm4-xKTC-0bhC-PZXC-3jAbRIXOsf5w,R3cS8NPr-f43b-sIKF-fEjr-oRHjMuwzYX1V,Smart TV,-82,1593662400789
+1593733518645,7yKzcQm4-xKTC-0bhC-PZXC-3jAbRIXOsf5w,ed0moSsY-5Qzi-4kDS-tAyj-eiTUK3LmEG9S,Laptop,-98,1593662400789
+1593662400789,7yKzcQm4-xKTC-0bhC-PZXC-3jAbRIXOsf5w,skYoi2Iy-Hk69-45Ss-mMEc-mdSt0Xi4iwA5,,-84,1593662400789
+1593683999798,7yKzcQm4-xKTC-0bhC-PZXC-3jAbRIXOsf5w,skYoi2Iy-Hk69-45Ss-mMEc-mdSt0Xi4iwA5,,-99,1593662400789
+1593675299879,7yKzcQm4-xKTC-0bhC-PZXC-3jAbRIXOsf5w,An2oCwIE-LOSJ-teZQ-Bjls-VTgbq1wmTA1e,Speakers,-81,1593662400789
+1593665344987,7yKzcQm4-xKTC-0bhC-PZXC-3jAbRIXOsf5w,An2oCwIE-LOSJ-teZQ-Bjls-VTgbq1wmTA1e,Speakers,-96,1593662400789

tests/data/raw/test03/bluetooth_raw.csv

@@ -0,0 +1 @@
+timestamp,device_id,bt_address,bt_name,bt_rssi,label

tests/data/raw/test04/bluetooth_raw.csv

@@ -0,0 +1 @@
+timestamp,device_id,bt_address,bt_name,bt_rssi,label

tests/settings/config.yaml

@@ -2,4 +2,4 @@ directory: ./
 configfile: ./tests/settings/testing_config.yaml
 snakefile: ./tests/Snakefile
 cores: 1
-forcerun: [messages_features, call_features, screen_features, battery_features]
+forcerun: [messages_features, call_features, screen_features, battery_features, bluetooth_features]

tests/settings/testing_config.yaml

@@ -1,6 +1,6 @@
 # Add as many sensor tables as you have, they all improve the computation of PHONE_SENSED_BINS. 
 # If you are extracting screen or Barnett's location features, screen and locations tables are mandatory.
-TABLES_FOR_SENSED_BINS: [messages, calls, screen, battery]
+TABLES_FOR_SENSED_BINS: [messages, calls, screen, battery, bluetooth]
 
 # Participants to include in the analysis
 # You must create a file for each participant named pXXX containing their device_id. This can be done manually or automatically
@@ -46,3 +46,9 @@ BATTERY:
   DB_TABLE: battery
   DAY_SEGMENTS: *day_segments
   FEATURES: ["countdischarge", "sumdurationdischarge", "countcharge", "sumdurationcharge", "avgconsumptionrate", "maxconsumptionrate"]
+
+BLUETOOTH:
+  COMPUTE: True
+  DB_TABLE: bluetooth
+  DAY_SEGMENTS: *day_segments
+  FEATURES: ["countscans", "uniquedevices", "countscansmostuniquedevice"]