Interaction analyses — Power (part 1)

  • This post: How to do a power analysis for an interaction in a linear regression (in R), and what factors effect how much power you have.
  • Part 2: Interpreting the effect-size of an interaction, by connecting it to simple-slopes.
  • Part 3: Determining what sample size is needed for an interaction.

First, some real data

Using publicly available data from https://openpsychometrics.org/ [4], I tested an interaction. Participants completed several surveys, including Depression, Anxiety, and Perceived Stress, as well as a brief personality survey (Big 5), and some demographic questions. They all agreed to make their responses available for research (no one was paid — the surveys are used more like an online personality questionnaire someone might do online for fun). You can see my steps for filtering and quality control in the R doc for this post, but briefly I’ll say that I’m only using responses from respondents in the United States (or with an IP address in the US), who said they were over 18, who didn’t have missing data, and who didn’t fail a pretty basic validity-check.

Call:lm(formula = Stress ~ age + gender + White + Suburban + College + Heterosexual + familysize + Anx + N + Anx:N + (Anx + N):(age gender + White + Suburban + College + Heterosexual +familysize),
data = DASS4)
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.0373975 0.0169867 2.202 0.02783 *
Anx:N 0.0873982 0.0162109 5.391 7.99e-08 ***
Anx 0.5802001 0.0184820 31.393 < 2e-16 ***
N -0.3293825 0.0187527 -17.565 < 2e-16 ***
age 0.0485193 0.0156328 3.104 0.00194 **
gender -0.0101972 0.0152473 -0.669 0.50372
White 0.0004681 0.0149604 0.031 0.97504
Suburban -0.0268078 0.0148041 -1.811 0.07034 .
College -0.0115615 0.0153877 -0.751 0.45255
Heterosexual 0.0163745 0.0148892 1.100 0.27160
familysize -0.0099285 0.0148598 -0.668 0.50413
age:Anx 0.0189089 0.0185720 1.018 0.30876
gender:Anx -0.0015383 0.0183065 -0.084 0.93304
White:Anx -0.0249086 0.0173450 -1.436 0.15117
Suburban:Anx -0.0146885 0.0175529 -0.837 0.40282
College:Anx -0.0029976 0.0182831 -0.164 0.86979
Heterosexual:Anx -0.0095487 0.0165095 -0.578 0.56309
familysize:Anx -0.0022462 0.0179375 -0.125 0.90036
age:N 0.0216759 0.0180634 1.200 0.23031
gender:N -0.0108077 0.0174722 -0.619 0.53629
White:N -0.0087409 0.0173806 -0.503 0.61509
Suburban:N 0.0276818 0.0173809 1.593 0.11143
College:N 0.0163651 0.0179572 0.911 0.36225
Heterosexual:N -0.0043325 0.0176113 -0.246 0.80571
familysize:N -0.0071348 0.0184509 -0.387 0.69903
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

A power analysis

Now that we have this result, we might ask, “How large a sample do I need to replicate this?” Or alternatively, “am I powered to detect this effect in the current analysis?”. Both can be answered with a power analysis. If we run a standard power analysis as if this is a simple regression with an independent variable B=0.087 (the effect size of the above interaction), we would get:

pwr.r.test(r = 0.087,power = 0.8,sig.level = 0.05)
approximate correlation power calculation (arctangh transformation)
n = 1033.84
r = 0.087
sig.level = 0.05
power = 0.8
alternative = two.sided

Effects of interacting-variable effect size and correlation

Our power analysis says N=500 for 80% power, but the power analysis for a simple regression with the same effect size (B=0.087) says N=1,000. What gives? So I decided run some simulations varying either (A) the correlation between our two interacting variables X and M (rXM) or (B) the effect-size of one of our interacting variables (bX).

Final thoughts and next steps

The results here suggest that, all else being equal, we should be better powered (or at least similarly powered) to detect interactions, relative to similarly-sized main effects. The fact that interactions have a fairly poor track-record suggests that something else is going on.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store