Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor eval_model and integrate evaluation function more deeply with 30_glmnet_model.R #91

Open
geneorama opened this issue Mar 18, 2016 · 0 comments

Comments

@geneorama
Copy link
Member

This is in reference to pull request @cash

When we look at the evaluation we look primarily at the "days difference", which is an attempt to quantify how much faster we would find a critical violation over the test period.

There is a secondary component to the evaluation, which is the basis for many of the graphs shown in the white paper. The second component is a summary of day by day inspection counts, and the number of critical violations found shown side-by-side with the simulated violations.

An example of the second summary is shown below, where N is the inspection count, POS indicates the critical violations found, and SIM indicates the "simulated" results from the experiment. In all cases TOT indicates a running cumulative sum.

          date  N NTOT POS POSTOT POS_SIM POSTOT_SIM
 1: 2014-09-02  1    1   0      0       1          1
 2: 2014-09-03 44   45  10     10      22         23
 3: 2014-09-04 50   95   8     18      24         47
 4: 2014-09-05 46  141  12     30       8         55
 5: 2014-09-08 37  178   6     36       8         63
 6: 2014-09-09 40  218   7     43      12         75
 7: 2014-09-10 33  251   3     46       4         79
 8: 2014-09-11 41  292  10     56      11         90
 9: 2014-09-12 42  334   5     61      11        101
10: 2014-09-15 35  369   6     67       5        106
11: 2014-09-16 51  420   9     76       5        111
12: 2014-09-17 24  444   6     82       5        116
13: 2014-09-18 41  485   5     87       7        123
14: 2014-09-19 45  530   8     95      10        133
15: 2014-09-22 34  564   7    102       4        137
16: 2014-09-23 33  597  10    112       2        139
17: 2014-09-24 29  626   6    118       6        145
18: 2014-09-25 48  674   5    123       7        152
19: 2014-09-26 49  723   8    131      11        163
20: 2014-09-29 55  778   9    140       5        168
21: 2014-09-30 32  810   1    141       8        176
22: 2014-10-01 29  839   4    145       3        179
23: 2014-10-02 39  878   6    151       7        186
24: 2014-10-03 46  924   5    156       7        193
25: 2014-10-06  8  932   1    157       0        193
26: 2014-10-07 35  967   5    162       7        200
27: 2014-10-08 48 1015   6    168       9        209
28: 2014-10-09 40 1055   2    170       5        214
29: 2014-10-10 36 1091   3    173       2        216
30: 2014-10-14 30 1121   6    179       3        219
31: 2014-10-15 31 1152   5    184       4        223
32: 2014-10-16 40 1192   3    187       4        227
33: 2014-10-17 43 1235   4    191       5        232
34: 2014-10-20 28 1263   3    194       3        235
35: 2014-10-21 37 1300   8    202       5        240
36: 2014-10-22 37 1337   7    209       4        244
37: 2014-10-23 45 1382   5    214       3        247
38: 2014-10-24 49 1431  11    225       1        248
39: 2014-10-27 33 1464   4    229       3        251
40: 2014-10-28 51 1515  11    240       2        253
41: 2014-10-29 37 1552   6    246       3        256
42: 2014-10-30 42 1594   5    251       1        257
43: 2014-10-31 43 1637   7    258       1        258

It is possible to calculate the "days saved" (the primary evaluation metric) from the table shown above. However, it will be useful to be able to quickly calculate the "days saved" metric directly, because then it's available if someone wants to use it in an objective function or optimization.

So, the point of this issue is to break out the eavl_model into the first and second components, and remove a lot of unclear and unnecessary code in the 30 model script.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant