junos
/
rapids
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Refactor call features: replace "metrics" with "features" 

Co-authored-by: Meng Li <AnnieLM1996@gmail.com>
pull/95/head
Mingze Cao 2020-04-08 10:51:18 -05:00
parent e0d19cbc1b
commit d1c38016de
4 changed files with 44 additions and 44 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
config.yaml
View File

@ -42,10 +42,10 @@ SMS:
sent: [count, distinctcontacts, timefirstsms, timelastsms, countmostfrequentcontact] sent: [count, distinctcontacts, timefirstsms, timelastsms, countmostfrequentcontact]
DAY_SEGMENTS: *day_segments DAY_SEGMENTS: *day_segments
# Communication call features config, TYPES and METRICS keys need to match # Communication call features config, TYPES and FEATURES keys need to match
CALLS: CALLS:
TYPES: [missed, incoming, outgoing] TYPES: [missed, incoming, outgoing]
METRICS: FEATURES:
missed: [count, distinctcontacts, timefirstcall, timelastcall, countmostfrequentcontact] missed: [count, distinctcontacts, timefirstcall, timelastcall, countmostfrequentcontact]
incoming: [count, distinctcontacts, meanduration, sumduration, minduration, maxduration, stdduration, modeduration, hubermduration, varqnduration, entropyduration, timefirstcall, timelastcall, countmostfrequentcontact] incoming: [count, distinctcontacts, meanduration, sumduration, minduration, maxduration, stdduration, modeduration, hubermduration, varqnduration, entropyduration, timefirstcall, timelastcall, countmostfrequentcontact]
outgoing: [count, distinctcontacts, meanduration, sumduration, minduration, maxduration, stdduration, modeduration, hubermduration, varqnduration, entropyduration, timefirstcall, timelastcall, countmostfrequentcontact] outgoing: [count, distinctcontacts, meanduration, sumduration, minduration, maxduration, stdduration, modeduration, hubermduration, varqnduration, entropyduration, timefirstcall, timelastcall, countmostfrequentcontact]

36
docs/features/extracted.rst
View File

@ -176,7 +176,7 @@ See `Call Config Code`_
.. - Apply readable datetime to Calls dataset: ``expand("data/raw/{pid}/{sensor}_with_datetime.csv", pid=config["PIDS"], sensor=config["SENSORS"]),`` .. - Apply readable datetime to Calls dataset: ``expand("data/raw/{pid}/{sensor}_with_datetime.csv", pid=config["PIDS"], sensor=config["SENSORS"]),``
- Extract Calls Metrics - Extract Calls Features
| ``expand("data/processed/{pid}/call_{call_type}_{segment}.csv",`` | ``expand("data/processed/{pid}/call_{call_type}_{segment}.csv",``
| ``pid=config["PIDS"],`` | ``pid=config["PIDS"],``
@ -193,9 +193,9 @@ See `Call Config Code`_
- **Script:** ``src/data/readable_datetime.R`` - See the readable_datetime.R_ script. - **Script:** ``src/data/readable_datetime.R`` - See the readable_datetime.R_ script.
- **Rule:** ``rules/features.snakefile/call_metrics`` - See the call_metrics_ rule. - **Rule:** ``rules/features.snakefile/call_features`` - See the call_features_ rule.
- **Script:** ``src/features/call_metrics.R`` - See the call_metrics.R_ script. - **Script:** ``src/features/call_features.R`` - See the call_features.R_ script.
.. _calls-parameters: .. _calls-parameters:
@ -207,14 +207,14 @@ Name Description
============ =================== ============ ===================
call_type The particular ``call_type`` that will be analyzed. The options for this parameter are ``incoming``, ``outgoing`` or ``missed``. call_type The particular ``call_type`` that will be analyzed. The options for this parameter are ``incoming``, ``outgoing`` or ``missed``.
day_segment The particular ``day_segments`` that will be analyzed. The available options are ``daily``, ``morning``, ``afternoon``, ``evening``, ``night`` day_segment The particular ``day_segments`` that will be analyzed. The available options are ``daily``, ``morning``, ``afternoon``, ``evening``, ``night``
metrics The different measures that can be retrieved from the calls dataset. Note that the same metrics are available for both ``incoming`` and ``outgoing`` calls, while ``missed`` calls has its own set of metrics. See :ref:`Available Incoming and Outgoing Call Metrics <available-in-and-out-call-metrics>` Table and :ref:`Available Missed Call Metrics <available-missed-call-metrics>` Table below. features The different measures that can be retrieved from the calls dataset. Note that the same features are available for both ``incoming`` and ``outgoing`` calls, while ``missed`` calls has its own set of features. See :ref:`Available Incoming and Outgoing Call Features <available-in-and-out-call-features>` Table and :ref:`Available Missed Call Features <available-missed-call-features>` Table below.
============ =================== ============ ===================
.. _available-in-and-out-call-metrics: .. _available-in-and-out-call-features:
**Available Incoming and Outgoing Call Metrics** **Available Incoming and Outgoing Call Features**
The following table shows a list of the available metrics for ``incoming`` and ``outgoing`` calls. The following table shows a list of the available features for ``incoming`` and ``outgoing`` calls.
========================= ========= ============= ========================= ========= =============
Name Units Description Name Units Description
@ -235,11 +235,11 @@ timelastcall minutes The time in minutes from 12:00am (Midn
countmostfrequentcontact calls The count of the number of calls of a particular ``call_type`` and ``day_segment`` for the most contacted contact. countmostfrequentcontact calls The count of the number of calls of a particular ``call_type`` and ``day_segment`` for the most contacted contact.
========================= ========= ============= ========================= ========= =============
.. _available-missed-call-metrics: .. _available-missed-call-features:
**Available Missed Call Metrics** **Available Missed Call Features**
The following table shows a list of the available metrics for ``missed`` calls. The following table shows a list of the available features for ``missed`` calls.
========================= ========= ============= ========================= ========= =============
Name Units Description Name Units Description
@ -248,19 +248,19 @@ count calls A count of the number of times a ``mis
distinctcontacts contacts A count of distinct contacts whose calls were ``missed``. distinctcontacts contacts A count of distinct contacts whose calls were ``missed``.
timefirstcall minutes The time in minutes from 12:00am (Midnight) that the first ``missed`` call occurred. timefirstcall minutes The time in minutes from 12:00am (Midnight) that the first ``missed`` call occurred.
timelastcall minutes The time in minutes from 12:00am (Midnight) that the last ``missed`` call occurred. timelastcall minutes The time in minutes from 12:00am (Midnight) that the last ``missed`` call occurred.
countmostfrequentcontact SMS The count of the number of ``missed`` calls for the contact with the most ``missed`` calls. countmostfrequentcontact CALLS The count of the number of ``missed`` calls for the contact with the most ``missed`` calls.
========================= ========= ============= ========================= ========= =============
**Assumptions/Observations:** **Assumptions/Observations:**
#. ``TYPES`` and ``METRICS`` keys need to match. From example:: #. ``TYPES`` and ``FEATURES`` keys need to match. From example::
SMS: CALLS:
TYPES: [missed] TYPES: [missed]
METRICS: FEATURES:
missed: [count, distinctcontacts, timefirstsms, timelastsms, countmostfrequentcontact] missed: [count, distinctcontacts, timefirstcall, timelastcall, countmostfrequentcontact]
In the above config setting code the ``TYPE`` ``missed`` matches the ``METRICS`` key ``missed``. In the above config setting code the ``TYPE`` ``missed`` matches the ``FEATURES`` key ``missed``.
.. _bluetooth-sensor-doc: .. _bluetooth-sensor-doc:
@ -1150,8 +1150,8 @@ stddurationactivebout minutes Std duration active bout: The standard
.. _DAY_SEGMENTS: https://github.com/carissalow/rapids/blob/765bb462636d5029a05f54d4c558487e3786b90b/config.yaml#L13 .. _DAY_SEGMENTS: https://github.com/carissalow/rapids/blob/765bb462636d5029a05f54d4c558487e3786b90b/config.yaml#L13
.. _PHONE_VALID_SENSED_DAYS: https://github.com/carissalow/rapids/blob/765bb462636d5029a05f54d4c558487e3786b90b/config.yaml#L60 .. _PHONE_VALID_SENSED_DAYS: https://github.com/carissalow/rapids/blob/765bb462636d5029a05f54d4c558487e3786b90b/config.yaml#L60
.. _`Call Config Code`: https://github.com/carissalow/rapids/blob/765bb462636d5029a05f54d4c558487e3786b90b/config.yaml#L46 .. _`Call Config Code`: https://github.com/carissalow/rapids/blob/765bb462636d5029a05f54d4c558487e3786b90b/config.yaml#L46
.. _call_metrics: https://github.com/carissalow/rapids/blob/765bb462636d5029a05f54d4c558487e3786b90b/rules/features.snakefile#L13 .. _call_features: https://github.com/carissalow/rapids/blob/765bb462636d5029a05f54d4c558487e3786b90b/rules/features.snakefile#L13
.. _call_metrics.R: https://github.com/carissalow/rapids/blob/master/src/features/call_metrics.R .. _call_features.R: https://github.com/carissalow/rapids/blob/master/src/features/call_features.R
.. _`Bluetooth Config Code`: https://github.com/carissalow/rapids/blob/765bb462636d5029a05f54d4c558487e3786b90b/config.yaml#L76 .. _`Bluetooth Config Code`: https://github.com/carissalow/rapids/blob/765bb462636d5029a05f54d4c558487e3786b90b/config.yaml#L76
.. _bluetooth_feature: https://github.com/carissalow/rapids/blob/765bb462636d5029a05f54d4c558487e3786b90b/rules/features.snakefile#L63 .. _bluetooth_feature: https://github.com/carissalow/rapids/blob/765bb462636d5029a05f54d4c558487e3786b90b/rules/features.snakefile#L63
.. _bluetooth_features.R: https://github.com/carissalow/rapids/blob/765bb462636d5029a05f54d4c558487e3786b90b/src/features/bluetooth_features.R .. _bluetooth_features.R: https://github.com/carissalow/rapids/blob/765bb462636d5029a05f54d4c558487e3786b90b/src/features/bluetooth_features.R

6
rules/features.snakefile
View File

@ -10,17 +10,17 @@ rule sms_metrics:
script: script:
"../src/features/sms_metrics.R" "../src/features/sms_metrics.R"
rule call_metrics: rule call_features:
input: input:
"data/raw/{pid}/calls_with_datetime_unified.csv" "data/raw/{pid}/calls_with_datetime_unified.csv"
params: params:
call_type = "{call_type}", call_type = "{call_type}",
day_segment = "{day_segment}", day_segment = "{day_segment}",
metrics = lambda wildcards: config["CALLS"]["METRICS"][wildcards.call_type] features = lambda wildcards: config["CALLS"]["FEATURES"][wildcards.call_type]
output: output:
"data/processed/{pid}/call_{call_type}_{day_segment}.csv" "data/processed/{pid}/call_{call_type}_{day_segment}.csv"
script: script:
"../src/features/call_metrics.R" "../src/features/call_features.R"
rule battery_deltas: rule battery_deltas:
input: input:

42
src/features/call_metrics.R → src/features/call_features.R
View File

@ -16,8 +16,8 @@ Mode <- function(v) {
uniqv[which.max(tabulate(match(v, uniqv)))] uniqv[which.max(tabulate(match(v, uniqv)))]
} }
compute_call_feature <- function(calls, metric, day_segment){ compute_call_feature <- function(calls, requested_feature, day_segment){
if(metric == "countmostfrequentcontact"){ if(requested_feature == "countmostfrequentcontact"){
# Get the most frequent contact # Get the most frequent contact
calls <- calls %>% group_by(trace) %>% calls <- calls %>% group_by(trace) %>%
mutate(N=n()) %>% mutate(N=n()) %>%
@ -26,41 +26,41 @@ compute_call_feature <- function(calls, metric, day_segment){
return(calls %>% return(calls %>%
filter_by_day_segment(day_segment) %>% filter_by_day_segment(day_segment) %>%
summarise(!!paste("call", type, day_segment, metric, sep = "_") := n())) summarise(!!paste("call", type, day_segment, requested_feature, sep = "_") := n()))
} else { } else {
calls <- calls %>% filter_by_day_segment(day_segment) calls <- calls %>% filter_by_day_segment(day_segment)
feature <- switch(metric, feature <- switch(requested_feature,
"count" = calls %>% summarise(!!paste("call", type, day_segment, metric, sep = "_") := n()), "count" = calls %>% summarise(!!paste("call", type, day_segment, requested_feature, sep = "_") := n()),
"distinctcontacts" = calls %>% summarise(!!paste("call", type, day_segment, metric, sep = "_") := n_distinct(trace)), "distinctcontacts" = calls %>% summarise(!!paste("call", type, day_segment, requested_feature, sep = "_") := n_distinct(trace)),
"meanduration" = calls %>% summarise(!!paste("call", type, day_segment, metric, sep = "_") := mean(call_duration)), "meanduration" = calls %>% summarise(!!paste("call", type, day_segment, requested_feature, sep = "_") := mean(call_duration)),
"sumduration" = calls %>% summarise(!!paste("call", type, day_segment, metric, sep = "_") := sum(call_duration)), "sumduration" = calls %>% summarise(!!paste("call", type, day_segment, requested_feature, sep = "_") := sum(call_duration)),
"minduration" = calls %>% summarise(!!paste("call", type, day_segment, metric, sep = "_") := min(call_duration)), "minduration" = calls %>% summarise(!!paste("call", type, day_segment, requested_feature, sep = "_") := min(call_duration)),
"maxduration" = calls %>% summarise(!!paste("call", type, day_segment, metric, sep = "_") := max(call_duration)), "maxduration" = calls %>% summarise(!!paste("call", type, day_segment, requested_feature, sep = "_") := max(call_duration)),
"stdduration" = calls %>% summarise(!!paste("call", type, day_segment, metric, sep = "_") := sd(call_duration)), "stdduration" = calls %>% summarise(!!paste("call", type, day_segment, requested_feature, sep = "_") := sd(call_duration)),
"modeduration" = calls %>% summarise(!!paste("call", type, day_segment, metric, sep = "_") := Mode(call_duration)), "modeduration" = calls %>% summarise(!!paste("call", type, day_segment, requested_feature, sep = "_") := Mode(call_duration)),
"hubermduration" = calls %>% summarise(!!paste("call", type, day_segment, metric, sep = "_") := huberM(call_duration)$mu), "hubermduration" = calls %>% summarise(!!paste("call", type, day_segment, requested_feature, sep = "_") := huberM(call_duration)$mu),
"varqnduration" = calls %>% summarise(!!paste("call", type, day_segment, metric, sep = "_") := Qn(call_duration)), "varqnduration" = calls %>% summarise(!!paste("call", type, day_segment, requested_feature, sep = "_") := Qn(call_duration)),
"entropyduration" = calls %>% summarise(!!paste("call", type, day_segment, metric, sep = "_") := entropy.MillerMadow(call_duration)), "entropyduration" = calls %>% summarise(!!paste("call", type, day_segment, requested_feature, sep = "_") := entropy.MillerMadow(call_duration)),
"timefirstcall" = calls %>% summarise(!!paste("call", type, day_segment, metric, sep = "_") := first(local_hour) + (first(local_minute)/60)), "timefirstcall" = calls %>% summarise(!!paste("call", type, day_segment, requested_feature, sep = "_") := first(local_hour) + (first(local_minute)/60)),
"timelastcall" = calls %>% summarise(!!paste("call", type, day_segment, metric, sep = "_") := last(local_hour) + (last(local_minute)/60))) "timelastcall" = calls %>% summarise(!!paste("call", type, day_segment, requested_feature, sep = "_") := last(local_hour) + (last(local_minute)/60)))
return(feature) return(feature)
} }
} }
calls <- read.csv(snakemake@input[[1]], stringsAsFactors = FALSE) calls <- read.csv(snakemake@input[[1]], stringsAsFactors = FALSE)
day_segment <- snakemake@params[["day_segment"]] day_segment <- snakemake@params[["day_segment"]]
metrics <- snakemake@params[["metrics"]] requested_features <- snakemake@params[["features"]]
type <- snakemake@params[["call_type"]] type <- snakemake@params[["call_type"]]
features = data.frame(local_date = character(), stringsAsFactors = FALSE) features = data.frame(local_date = character(), stringsAsFactors = FALSE)
calls <- calls %>% filter(call_type == ifelse(type == "incoming", "1", ifelse(type == "outgoing", "2", "3"))) calls <- calls %>% filter(call_type == ifelse(type == "incoming", "1", ifelse(type == "outgoing", "2", "3")))
for(metric in metrics){ for(requested_feature in requested_features){
feature <- compute_call_feature(calls, metric, day_segment) feature <- compute_call_feature(calls, requested_feature, day_segment)
features <- merge(features, feature, by="local_date", all = TRUE) features <- merge(features, feature, by="local_date", all = TRUE)
} }
if("countmostfrequentcontact" %in% metrics) if("countmostfrequentcontact" %in% requested_features)
features <- features %>% mutate_at(vars(contains('countmostfrequentcontact')), funs(ifelse(is.na(.), 0, .))) features <- features %>% mutate_at(vars(contains('countmostfrequentcontact')), funs(ifelse(is.na(.), 0, .)))
write.csv(features, snakemake@output[[1]], row.names = FALSE) write.csv(features, snakemake@output[[1]], row.names = FALSE)