07-VariationsinDesign.Rmd

# Effect of Varying Designs on Power
```{r  echo=FALSE}
load("data/variation_data.RData")
```
Researchers might consider what the effects on the statistical power of their design is, when they add participants. Participants can be added to an additional condition, or to the existing design. 

In a one-way ANOVA adding a condition means, for example, going from a 1x2 to a 1x3 design. For example, in addition to a control and intensive training condition, we add a light training condition. 

```{r variations_in_design}
string <- "2b"
n <- 50
mu <- c(80, 86) 
sd <- 10
labelnames <- c("Condition", "control", "intensive_training") 
design_result <- ANOVA_design(design = string,
                   n = n, 
                   mu = mu, 
                   sd = sd, 
                   labelnames = labelnames)
# Power for the given N in the design_result
power_oneway_between(design_result)$power
power_oneway_between(design_result)$Cohen_f
power_oneway_between(design_result)$eta_p_2
```
```{r eval=FALSE}
simulation_result <- ANOVA_power(design_result, 
                                 alpha_level = alpha_level, 
                                 nsims = nsims,
                                 verbose = FALSE)
```

```{r echo = FALSE}
knitr::kable(simulation_result_7.1$main_results,
             caption = "Simulated ANOVA Result")%>%
  kable_styling(latex_options = "hold_position")
```


```{r}
exact_result <- ANOVA_exact(design_result,
                            alpha_level = alpha_level,
                            verbose = FALSE)
```


```{r echo = FALSE}
knitr::kable(exact_result$main_results,
             caption = "Exact ANOVA Result")%>%
  kable_styling(latex_options = "hold_position")
```

We now addd a condition. Let's assume the 'light training' condition falls in between the other two means. 

And we can see power across sample sizes

```{r}
# Plot power curve (from 5 to 100)
plot_power(design_result, max_n = 100)
```


```{r}
string <- "3b"
n <- 50
mu <- c(80, 83, 86)
sd <- 10
labelnames <- c("Condition", "control", "light_training", "intensive_training")
design_result <- ANOVA_design(
  design = string,
  n = n,
  mu = mu,
  sd = sd,
  labelnames = labelnames
)
# Power for the given N in the design_result
power_oneway_between(design_result)$power
power_oneway_between(design_result)$Cohen_f
power_oneway_between(design_result)$eta_p_2


exact_result <- ANOVA_exact(design_result,
                            alpha_level = alpha_level,
                            verbose = FALSE)
```

```{r echo = FALSE}
knitr::kable(exact_result$main_results,
             caption = "Exact ANOVA Result") %>%
  kable_styling(latex_options = "hold_position")

```

We see that adding a condition that falls between the other two means reduces our power. Let's instead assume that the 'light training' condition is not different from the control condition. In other words, the mean we add is as extreme as one of the existing means. 

```{r}
# Plot power curve (from 5 to 100)
plot_power(design_result, max_n = 100)
```


```{r}
string <- "3b"
n <- 50
mu <- c(80, 80, 86)
sd <- 10
labelnames <- c("Condition", "control", "light_training", "intensive_training")
design_result <- ANOVA_design(
  design = string,
  n = n,
  mu = mu,
  sd = sd,
  labelnames = labelnames
)

# Power for the given N in the design_result
power_oneway_between(design_result)$power
power_oneway_between(design_result)$Cohen_f
power_oneway_between(design_result)$eta_p_2

```

Now power has increased. This is not always true. The power is a function of many factors in the design, incuding the effect size (Cohen's *f*) and the total sample size (and the degrees of freedom and number of groups). But as we will see below, as we keep adding conditions, the power will reduce, even if initially, the power might increase. 

```{r}
# Plot power curve (from 5 to 100)
plot_power(design_result,max_n = 100)
```


It helps to think of these different designs in terms of either partial eta-squared, or Cohen's *f* (the one can easily be converted into the other). 

```{r}
#Two groups
mu <- c(80, 86)
sd = 10
n <- 50 #sample size per condition
mean_mat <- t(matrix(mu, nrow = 2, ncol = 1)) #Create a mean matrix
# Using the sweep function to remove rowmeans from the matrix
mean_mat_res <- sweep(mean_mat, 2, rowMeans(mean_mat))
mean_mat_res
MS_a <- n * (sum(mean_mat_res ^ 2) / (2 - 1))
MS_a
SS_A <- n * sum(mean_mat_res ^ 2)
SS_A
MS_error <- sd ^ 2
MS_error
SS_error <- MS_error * (n * 2)
SS_error
eta_p_2 <- SS_A / (SS_A + SS_error)
eta_p_2
f_2 <- eta_p_2 / (1 - eta_p_2)
f_2
Cohen_f <- sqrt(f_2)
Cohen_f
#Three groups
mu <- c(80, 83, 86)
sd = 10
n <- 50
mean_mat <- t(matrix(mu, nrow = 3, ncol = 1)) #Create a mean matrix
# Using the sweep function to remove rowmeans from the matrix
mean_mat_res <- sweep(mean_mat, 2, rowMeans(mean_mat))
mean_mat_res
MS_a <- n * (sum(mean_mat_res ^ 2) / (3 - 1))
MS_a
SS_A <- n * sum(mean_mat_res ^ 2)
SS_A
MS_error <- sd ^ 2
MS_error
SS_error <- MS_error * (n * 3)
SS_error
eta_p_2 <- SS_A / (SS_A + SS_error)
eta_p_2
f_2 <- eta_p_2 / (1 - eta_p_2)
f_2
Cohen_f <- sqrt(f_2)
Cohen_f
```

The `SS_A` or the sum of squares for the main effect, is 900 for two groups, and the `SS_error` for the error term is 10000. When we add a group, `SS_A` is 900, and the `SS_error` is 15000. Because the added condition falls exactly on the grand mean (83), the sum of squared for this extra group is 0. In other words, it does nothing to increase the signal that there is a difference between groups. However, the sum of squares for the error, which is a function of the total sample size, is increased, which reduces the effect size. So, adding a condition that falls on the grand mean reduces the power for the main effect of the ANOVA. Obviously, adding such a group has other benefits, such as being able to compare the two means to a new third condition. 

We already saw that adding a condition that has a mean as extreme as one of the existing groups increases the power. Let's again do the calculations step by step when the extra group has a mean as extreme as one of the two original conditions. 

```{r}
#Three groups
mu <- c(80, 80, 86)
sd = 10
n <- 50
mean_mat <- t(matrix(mu, nrow = 3, ncol = 1)) #Create a mean matrix
# Using the sweep function to remove rowmeans from the matrix
mean_mat_res <- sweep(mean_mat, 2, rowMeans(mean_mat))
mean_mat_res
MS_a <- n * (sum(mean_mat_res ^ 2) / (3 - 1))
MS_a
SS_A <- n * sum(mean_mat_res ^ 2)
SS_A
MS_error <- sd ^ 2
MS_error
SS_error <- MS_error * (n * 3)
SS_error
eta_p_2 <- SS_A / (SS_A + SS_error)
eta_p_2
f_2 <- eta_p_2 / (1 - eta_p_2)
f_2
Cohen_f <- sqrt(f_2)
Cohen_f
```

We see the sum of squares of the error stays the same - 15000 - because it is only determined by the standard error and the sample size, but not by the differences in the means. This is an increase of 5000 compared to the 2 group design. The sum of squares (the second component that determines the size of partial eta-squared) increases, which increases Cohen's *f*. 

## Within Designs

Now imagine our design described above was a within design. The means and sd remain the same. We collect 50 participants (instead of 100, or 50 per group, for the between design). Let's first assume the two samples are completely uncorrelated. 

```{r}
string <- "2w"
n <- 50
mu <- c(80, 86)
sd <- 10
labelnames <- c("Condition", "control", "intensive_training") #
design_result <- ANOVA_design(
  design = string,
  n = n,
  mu = mu,
  sd = sd,
  labelnames = labelnames
)

power_oneway_within(design_result)$power
```

```{r}
exact_result <- ANOVA_exact(design_result,
                            alpha_level = alpha_level,
                            verbose = FALSE)
```

```{r echo = FALSE}
knitr::kable(exact_result$main_results,
             caption = "Exact ANOVA Result")%>%
  kable_styling(latex_options = "hold_position")
```

We see power is ever so slightly less than for the between subject design. This is due to the loss in degrees of freedom, which is $2(n-1)$ for between designs, and $n-1$ for within designs. But as the correlation increases, the power advantage of within designs becomes stronger. 

```{r}
string <- "3w"
n <- 50
mu <- c(80, 83, 86)
sd <- 10
labelnames <- c("Condition", "control", "light_training", "intensive_training")
design_result <- ANOVA_design(
  design = string,
  n = n,
  mu = mu,
  sd = sd,
  labelnames = labelnames
)

power_oneway_within(design_result)$power
```

```{r}
exact_result <- ANOVA_exact(design_result,
                            alpha_level = alpha_level,
                            verbose = FALSE)
```

```{r echo=FALSE}
knitr::kable(exact_result$main_results,
             caption = "Exact ANOVA Result")%>%
  kable_styling(latex_options = "hold_position")
```

When we add a a condition in a within design where we expect the mean to be identical to the grand mean, we again see that the power decreases. This similarly shows that adding a condition that equals the grand mean to a within subject design does not come for free, but has a power cost. 

```{r}
n <- 30
sd <- 10
r <- 0.5
string <- "2w"
mu <- c(0, 5) 
design_result <- ANOVA_design(
  design = string,
  n = n,
  mu = mu,
  sd = sd,
  r = r
)

power_oneway_within(design_result)$power
power_oneway_within(design_result)$Cohen_f
power_oneway_within(design_result)$Cohen_f_SPSS
power_oneway_within(design_result)$lambda
power_oneway_within(design_result)$F_critical

string <- "3w"
mu <- c(0, 0, 5)

design_result <- ANOVA_design(
  design = string,
  n = n,
  mu = mu,
  sd = sd,
  r = r
)

power_oneway_within(design_result)$power
power_oneway_within(design_result)$Cohen_f
power_oneway_within(design_result)$Cohen_f_SPSS
power_oneway_within(design_result)$lambda
power_oneway_within(design_result)$F_critical

string <- "4w"
mu <- c(0, 0, 0, 5)
design_result <- ANOVA_design(
  design = string,
  n = n,
  mu = mu,
  sd = sd,
  r = r
)

power_oneway_within(design_result)$power
power_oneway_within(design_result)$Cohen_f
power_oneway_within(design_result)$Cohen_f_SPSS
power_oneway_within(design_result)$lambda
power_oneway_within(design_result)$F_critical

string <- "5w"
mu <- c(0, 0, 0, 0, 5) 

design_result <- ANOVA_design(
  design = string,
  n = n,
  mu = mu,
  sd = sd,
  r = r
)
power_oneway_within(design_result)$power
power_oneway_within(design_result)$Cohen_f
power_oneway_within(design_result)$Cohen_f_SPSS
power_oneway_within(design_result)$lambda
power_oneway_within(design_result)$F_critical

string <- "6w"
mu <- c(0, 0, 0, 0, 0, 5) 
design_result <- ANOVA_design(
  design = string,
  n = n,
  mu = mu,
  sd = sd,
  r = r
)
power_oneway_within(design_result)$power
power_oneway_within(design_result)$Cohen_f
power_oneway_within(design_result)$Cohen_f_SPSS
power_oneway_within(design_result)$lambda
power_oneway_within(design_result)$F_critical

string <- "7w"
mu <- c(0, 0, 0, 0, 0, 0, 5) 
design_result <- ANOVA_design(
  design = string,
  n = n,
  mu = mu,
  sd = sd,
  r = r
)

print(paste0("power: ", power_oneway_within(design_result)$power, "\nCohen's F: ", power_oneway_within(design_result)$Cohen_f,
             "\nCohen's F (SPSS): ", power_oneway_within(design_result)$Cohen_f_SPSS,
             "\nlambda: ", power_oneway_within(design_result)$lambda,
             "\nCritical F: ",
             power_oneway_within(design_result)$F_critical
             ))

# power_oneway_within(design_result)$power
# power_oneway_within(design_result)$Cohen_f
# power_oneway_within(design_result)$Cohen_f_SPSS
# power_oneway_within(design_result)$lambda
# power_oneway_within(design_result)$F_critical
```

This set of designs where we increase the number of conditions demonstrates a common pattern where the power initially increases, but then starts to decrease. 
Again, the exact pattern (and when the power starts to decrease) depends on the effect size and sample size. 
Note also that the effect size (Cohen's f) decreases as we add conditions, but the increased sample size compensates for this when calculating power. 
When using power analysis software such as g\*power [@faul2007g], this is important to realize. 
You can't just power for a medium effect size, and then keep adding conditions under the assumption that the increased power you see in the program will become a reality.
Increasing the number of conditions will reduce the effect size, and therefore, adding conditions will not automatically increase power (and might even decrease it). 

Overall, the effect of adding conditions with an effect close to the grand mean reduces power quite strongly, and adding conditions with means close to the extreme of the current conditions will either slightly increase of decrease power.