Adding frequentist numerical tests + Integrate StatsAPI #67

ShouvikGhosh2048 · 2022-11-13T11:00:33Z

This PR adds frequentist numerical tests, and also integrates StatsAPI into CRRao.

ShouvikGhosh2048 · 2022-11-13T11:06:13Z

@sourish-cmi
Currently this PR tests frequentist models on ordinary formulae and 0 intercept formulae. The tests just check whether the coeftables match.

Should we test on other formulae/datasets?
Should we test on more (or all) functions in src/frequentist/getter.jl?

sourish-cmi · 2022-11-14T08:41:13Z

We should test for predict
Some possible formulas:
i) mpg~hp+hp^2 + wt + wt*hp
ii) log(mpg)~log(hp)+log(wt)
Later we can add few more datasets

ShouvikGhosh2048 · 2022-11-15T10:06:49Z

src/frequentist/getter.jl

@@ -221,7 +221,7 @@ function predict(container::FrequentistRegression{:LogisticRegression}, newdata:
        return Probit_Link.(z)
    elseif (container.link == GLM.CauchitLink)
        return Cauchit_Link.(z)
-    elseif (container.link == GLM.Cloglog)
+    elseif (container.link == GLM.CloglogLink)


I made a change to src/frequentist/getter.jl - I think it should be CloglogLink instead of Cloglog.

okay -- great. thanks.

ShouvikGhosh2048 · 2022-11-15T10:09:27Z

We should test for predict

Some possible formulas:
i) mpg~hp+hp^2 + wt + wt*hp
ii) log(mpg)~log(hp)+log(wt)

Later we can add few more datasets

I added predict and some formulae.

For Logistic, Negative binomial and Poisson should there be an example with log on the left?

sourish-cmi · 2022-11-16T02:53:58Z

We should test for predict

Some possible formulas:
i) mpg~hp+hp^2 + wt + wt*hp
ii) log(mpg)~log(hp)+log(wt)

Later we can add few more datasets

I added predict and some formulae.

For Logistic, Negative binomial and Poisson should there be an example with log on the left?

No @ShouvikGhosh2048 for Logistic, Negative binomial and Poisson, we don't need a log on the left.

sourish-cmi · 2022-11-16T08:45:06Z

Perhaps we should check for AIC and BIC

> attach(mtcars)
> fit = lm(mpg ~ hp + wt + gear,mtcars)
> AIC(fit)
[1] 157.0528
> BIC(fit)
[1] 164.3815

Add it in the code and check if CRRao yields AIC = 157.0528 and BIC=164.3815

ShouvikGhosh2048 · 2022-11-16T13:44:40Z

@sourish-cmi

using RDatasets, GLM, StatsBase

mtcars = dataset("datasets", "mtcars")
glm_model = lm(@formula(MPG ~ HP + HP^2 + WT + WT * HP), mtcars)
print(aic(glm_model))

gives 145.92297178935536

and

attach(mtcars)
fit = lm(mpg ~ hp + hp^2 + wt + wt * hp, mtcars)
AIC(fit)

gives 145.6109.

sourish-cmi · 2022-11-16T15:56:46Z

In your R-code, there is a syntax mistake.

fit = lm(mpg ~ hp + hp^2 + wt + wt * hp, mtcars)
AIC(fit)

The correct formula would be as follows:

> fit = lm(mpg ~ hp + I(hp^2) + wt + wt * hp, mtcars)
> AIC(fit)
[1] 145.923

ShouvikGhosh2048 · 2022-11-20T06:37:09Z

library("Zelig")
data(turnout)
fit = glm(vote ~ age + I(age^2) + race + income * educate, family = binomial(link = "cloglog"), data = turnout)
sprintf("%.15f", AIC(fit))

gives 2022.884413390652753,
while

using GLM, RDatasets
turnout = dataset("Zelig", "turnout")
model = glm(@formula(Vote ~ Age + Age^2 + Race + Income * Educate), turnout, Binomial(), CloglogLink())
GLM.aic(model)

gives 2022.8844886368.

library("Zelig")
library(MASS)
data(sanction)
fit = glm.nb(num ~ target + coop + ncost, data = sanction)
sprintf("%.15f", AIC(fit))

gives 344.879549011806205,
while

using GLM, RDatasets
sanction = dataset("Zelig", "sanction")
model = glm(@formula(Num ~ Target + Coop + NCost), sanction, NegativeBinomial(), LogLink())
GLM.aic(model)

gives 365.8580428654269.

sourish-cmi · 2022-11-20T09:17:55Z

library("Zelig")
data(turnout)
fit = glm(vote ~ age + I(age^2) + race + income * educate, family = binomial(link = "cloglog"), data = turnout)
sprintf("%.15f", AIC(fit))

gives 2022.884413390652753, while

using GLM, RDatasets
turnout = dataset("Zelig", "turnout")
model = glm(@formula(Vote ~ Age + Age^2 + Race + Income * Educate), turnout, Binomial(), CloglogLink())
GLM.aic(model)

gives 2022.8844886368.

library("Zelig")
library(MASS)
data(sanction)
fit = glm.nb(num ~ target + coop + ncost, data = sanction)
sprintf("%.15f", AIC(fit))

gives 344.879549011806205, while

using GLM, RDatasets
sanction = dataset("Zelig", "sanction")
model = glm(@formula(Num ~ Target + Coop + NCost), sanction, NegativeBinomial(), LogLink())
GLM.aic(model)

gives 365.8580428654269.

Okay, it is an issue with GLM.jl. Can you please raise an issue in GLM.jl?

ayushpatnaikgit · 2022-11-21T13:47:22Z

@mousum-github can you please check this?

mousum-github · 2022-11-21T14:45:18Z

mousum-github · 2022-11-21T14:52:41Z

glm.nb estimates the \theta but glm with a Negative Binomial, you have to give theta and the default value is 1.
negbin is the corresponding function where \theta is estimated from the data.

ShouvikGhosh2048 · 2022-11-22T12:02:52Z

@mousum-github

For the Julia program in (2), what will the corresponding R program be?
Do the programs in (1) match in what they should compute? If so, is the AIC difference significant?

mousum-github · 2022-11-22T15:10:57Z

@mousum-github

For the Julia program in (2), what will the corresponding R program be?
Not sure, you may try with negative binomial distribution as a family parameter in R glm function

Do the programs in (1) match in what they should compute? If so, is the AIC difference significant?
model = glm(@formula(Vote ~ Age + Age^2 + Race + Income * Educate), turnout, Binomial(), CloglogLink(); rtol=1.0E-16)
GLM.aic(model)
2022.8844131094252

ShouvikGhosh2048 · 2022-11-23T09:54:05Z

library(Zelig)
library(MASS)
data(sanction)
fit = glm(num ~ target + coop + ncost, family = negative.binomial(theta=1), data = sanction)
sprintf("%.15f", AIC(fit))

gives 363.858042084877013, which is closer to Julia's value, but still significantly different.

Should we remove this specific Cloglog example from tests? We can't set rtol in CRRao.
The formula works for others links, and Cloglog tests with other formulae pass.
We could also try other formulae with square/product terms until we get one which passes tests.

sourish-cmi · 2022-12-02T02:42:42Z

@ShouvikGhosh2048 for now, we can remove this example with cloglog. We can say on the front page that the test for cloglog link function is not completed yet. We have to move ahead with other tests. Particularly with the Bayesian test.

In this issue can you mention the completeness of the tests? or the cases that I have not considered yet !!

ShouvikGhosh2048 · 2022-12-02T10:46:23Z

@ShouvikGhosh2048 for now, we can remove this example with cloglog. We can say on the front page that the test for cloglog link function is not completed yet. We have to move ahead with other tests. Particularly with the Bayesian test.

In this issue can you mention the completeness of the tests? or the cases that I have not considered yet !!

I couldn't construct a R program for NegativeBinomial, so those tests are still incomplete.
I can start working on Bayesian tests parallelly.

sourish-cmi · 2022-12-11T07:04:02Z

@ShouvikGhosh2048 @mousum-github

As discussed in yesterday's meeting we will focus on functional testing.

@ShouvikGhosh2048 will develop the functional testing for the Bayesian module. For a given seed, the Bayesian module should reproduce the result.
@mousum-github will correct the negative binomial model with negbin instead of glm and provide the test with R output

sourish-cmi · 2022-12-20T17:36:26Z

@ShouvikGhosh2048 Mousum da added the negative binomial test, and I approved and merged the test. Perhaps you pull the latest from the main branch. Please check.

sourish-cmi · 2022-12-21T16:27:25Z

@ShouvikGhosh2048 since you created separate PR for Bayesian tests. I believe now we can merge this PR. Please confirm.

ShouvikGhosh2048 · 2022-12-22T01:01:57Z

I've updated the numerical tests for negative binomial. I haven't matched the AIC/BIC values with R - I'm not sure which values to pass to R's negbin to match Julia's negbin.

I tried:

library(aod)
library(Zelig)
data(sanction)
fit = negbin(num ~ 0 + target + coop + ncost, ~1, sanction)
AIC(fit)

This gives 344.896 compared to CRRao's 344.87955007518536.
Otherwise the PR is ready to be merged.

sourish-cmi · 2022-12-22T02:09:20Z

I've updated the numerical tests for negative binomial. I haven't matched the AIC/BIC values with R - I'm not sure which values to pass to R's negbin to match Julia's negbin.

I tried:
library(aod)
library(Zelig)
data(sanction)
fit = negbin(num ~ 0 + target + coop + ncost, ~1, sanction)
AIC(fit)
This gives 344.896 compared to CRRao's 344.87955007518536. Otherwise the PR is ready to be merged.

@ShouvikGhosh2048
okay, let me have a look and get back to you.

sourish-cmi · 2022-12-22T16:30:29Z

I think it is okay - we can ignore it - it might be those denominator issues of AIC. For time being we can ignore it. I think we should go ahead and merge it. Please do let me know your thought.

ShouvikGhosh2048 · 2022-12-24T02:46:52Z

We can merge it for now (to make sure our frequentist models are consistent), but we should keep track of this.

sourish-cmi · 2022-12-28T17:50:55Z

I have filed the issue with the title AIC value for negbin regression does not matches with R. Otherwise, this PR is fine - so I am merging the PR with the main.

sourish-cmi

All codes are checked and passed. Closing the PR - while opening a new issue

ShouvikGhosh2048 added 2 commits November 13, 2022 15:29

Integrate StatsAPI

58bec74

Add simple frequentist numerical tests + 0 intercept tests

be72c35

ShouvikGhosh2048 requested a review from sourish-cmi November 13, 2022 11:00

Added predict to tests + added more formulae

1370a30

ShouvikGhosh2048 commented Nov 15, 2022

View reviewed changes

Add AIC and BIC tests for Linear, Logistic and Poisson models.

2a5f5e4

ShouvikGhosh2048 added 2 commits December 22, 2022 05:06

Merge branch 'xKDR:main' into adding_numerical_tests

0f701af

Add AIC and BIC tests and update negative binomial tests

2c74f60

sourish-cmi added this to the Package Completeness milestone Dec 22, 2022

This was linked to issues Dec 22, 2022

Package completeness #52

Open

Testing strategy, some thoughts #62

Closed

sourish-cmi closed this Dec 28, 2022

sourish-cmi reopened this Dec 28, 2022

sourish-cmi approved these changes Dec 28, 2022

View reviewed changes

Merge branch 'main' into adding_numerical_tests

77db266

sourish-cmi merged commit c51b875 into xKDR:main Dec 29, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding frequentist numerical tests + Integrate StatsAPI #67

Adding frequentist numerical tests + Integrate StatsAPI #67

ShouvikGhosh2048 commented Nov 13, 2022

ShouvikGhosh2048 commented Nov 13, 2022

sourish-cmi commented Nov 14, 2022

ShouvikGhosh2048 Nov 15, 2022

sourish-cmi Nov 16, 2022

ShouvikGhosh2048 commented Nov 15, 2022

sourish-cmi commented Nov 16, 2022

sourish-cmi commented Nov 16, 2022

ShouvikGhosh2048 commented Nov 16, 2022

sourish-cmi commented Nov 16, 2022

ShouvikGhosh2048 commented Nov 20, 2022

sourish-cmi commented Nov 20, 2022

ayushpatnaikgit commented Nov 21, 2022

mousum-github commented Nov 21, 2022

mousum-github commented Nov 21, 2022 •

edited

Loading

ShouvikGhosh2048 commented Nov 22, 2022 •

edited

Loading

mousum-github commented Nov 22, 2022 •

edited

Loading

ShouvikGhosh2048 commented Nov 23, 2022

sourish-cmi commented Dec 2, 2022

ShouvikGhosh2048 commented Dec 2, 2022

sourish-cmi commented Dec 11, 2022

sourish-cmi commented Dec 20, 2022

sourish-cmi commented Dec 21, 2022

ShouvikGhosh2048 commented Dec 22, 2022

sourish-cmi commented Dec 22, 2022

sourish-cmi commented Dec 22, 2022 •

edited

Loading

ShouvikGhosh2048 commented Dec 24, 2022 •

edited

Loading

sourish-cmi commented Dec 28, 2022

sourish-cmi left a comment

Adding frequentist numerical tests + Integrate StatsAPI #67

Adding frequentist numerical tests + Integrate StatsAPI #67

Conversation

ShouvikGhosh2048 commented Nov 13, 2022

ShouvikGhosh2048 commented Nov 13, 2022

sourish-cmi commented Nov 14, 2022

ShouvikGhosh2048 Nov 15, 2022

Choose a reason for hiding this comment

sourish-cmi Nov 16, 2022

Choose a reason for hiding this comment

ShouvikGhosh2048 commented Nov 15, 2022

sourish-cmi commented Nov 16, 2022

sourish-cmi commented Nov 16, 2022

ShouvikGhosh2048 commented Nov 16, 2022

sourish-cmi commented Nov 16, 2022

ShouvikGhosh2048 commented Nov 20, 2022

sourish-cmi commented Nov 20, 2022

ayushpatnaikgit commented Nov 21, 2022

mousum-github commented Nov 21, 2022

mousum-github commented Nov 21, 2022 • edited Loading

ShouvikGhosh2048 commented Nov 22, 2022 • edited Loading

mousum-github commented Nov 22, 2022 • edited Loading

ShouvikGhosh2048 commented Nov 23, 2022

sourish-cmi commented Dec 2, 2022

ShouvikGhosh2048 commented Dec 2, 2022

sourish-cmi commented Dec 11, 2022

sourish-cmi commented Dec 20, 2022

sourish-cmi commented Dec 21, 2022

ShouvikGhosh2048 commented Dec 22, 2022

sourish-cmi commented Dec 22, 2022

sourish-cmi commented Dec 22, 2022 • edited Loading

ShouvikGhosh2048 commented Dec 24, 2022 • edited Loading

sourish-cmi commented Dec 28, 2022

sourish-cmi left a comment

Choose a reason for hiding this comment

mousum-github commented Nov 21, 2022 •

edited

Loading

ShouvikGhosh2048 commented Nov 22, 2022 •

edited

Loading

mousum-github commented Nov 22, 2022 •

edited

Loading

sourish-cmi commented Dec 22, 2022 •

edited

Loading

ShouvikGhosh2048 commented Dec 24, 2022 •

edited

Loading