[12]{.chapter-number}  [The Importance of Coordination]{.chapter-title}

12 The Importance of Coordination

12.1 Introduction

In large companies, multiple teams often run experiments simultaneously. A common—but risky—assumption is: “We’re both randomizing, so it’s fine.” In reality, uncoordinated experiments can contaminate control groups and obscure true interaction effects. This chapter explains why coordinating experimental designs is crucial for reliable business insights and decisions.

12.2 A Motivating Example

Let’s explore a scenario where two teams are independently trying to improve the same outcome (e.g., conversion rate, customer engagement, sales):

Team A is testing a monetary incentive (e.g., a discount, a gift card)
Team B is testing a non-monetary incentive (e.g., a personalized message, early access to features)

With proper coordination, these teams could run a more informative 2x2 factorial experiment with four arms:

Monetary Incentive Only
Non-Monetary Incentive Only
Both Incentives Combined
Control (No Incentives)

This design enables understanding both individual effects and their interaction: Does combining incentives create synergy (greater than the sum), antagonism (less than the sum), or no interaction?

12.3 The Problem with Uncoordinated Experiments

Without coordination, each team typically runs a two-arm experiment (their incentive vs. control). This creates two major issues:

Each team’s “control” group is contaminated by the other team’s experiment
The true interaction effects between interventions remain unknown

Let’s demonstrate this with synthetic data. We’ll assume:

Monetary Incentive Effect: +2 units
Non-Monetary Incentive Effect: +1 unit
Combined Effect: +2 units (showing a ceiling effect)
Baseline (No Incentives): 0 units

library(tidyverse)

# Set seed for reproducibility
set.seed(42)

# Generate synthetic data
n <- 500  # observations per group

# Create data frame for coordinated experiment
coordinated_data <- data.frame(
  monetary = rep(c(0, 1), each = 2 * n),
  non_monetary = rep(c(0, 1, 0, 1), each = n)
)

# Define treatment effects
monetary_effect <- 2
non_monetary_effect <- 1
interaction_effect <- -1  # Negative interaction

# Generate outcome
coordinated_data$outcome <- with(
  coordinated_data,
  0 +  # Baseline
    monetary * monetary_effect +
    non_monetary * non_monetary_effect +
    monetary * non_monetary * interaction_effect +
    rnorm(4 * n, mean = 0, sd = 1)  # Random noise
)

12.4 Analyzing Uncoordinated Experiments

Let’s see how each team might analyze their data in isolation:

# Team A's regression
lm(outcome ~ monetary, data = coordinated_data) %>% 
  broom::tidy(conf.int = TRUE) %>%
  select(term, estimate, conf.low, conf.high, p.value) %>% 
  knitr::kable(
  caption = "Team A's Analysis (Monetary Incentive)",
  digits = 3
)

Team A’s Analysis (Monetary Incentive)
term	estimate	conf.low	conf.high	p.value
(Intercept)	0.474	0.409	0.540	0
monetary	1.521	1.428	1.613	0

In the absence of coordination, Team A’s analysis would likely lead them to conclude that monetary incentives increase the outcome of interest by 1.5 units compared to no incentives, a statistically significant result. However, their model is misspecified because some of the control units are receiving non-monetary incentives, while some of their treated units are also receiving non-monetary incentives.

The code bellow shows that the same would happen to the non-monetary incentives team:

# Team B's regression
lm(outcome ~ non_monetary, data = coordinated_data) %>% 
  broom::tidy(conf.int = TRUE) %>%
  select(term, estimate, conf.low, conf.high, p.value) %>% 
  knitr::kable(
  caption = "Team B's Analysis (Non-Monetary Incentive)",
  digits = 3
)

Team B’s Analysis (Non-Monetary Incentive)
term	estimate	conf.low	conf.high	p.value
(Intercept)	0.964	0.885	1.043	0
non_monetary	0.540	0.429	0.652	0

Team B would conclude that the impact of non-monetary incentives is 0.5.

12.5 The Power of Coordination

Now let’s analyze the data properly using a coordinated approach:

# Full factorial analysis
lm(outcome ~ monetary * non_monetary, data = coordinated_data) %>% 
  broom::tidy(conf.int = TRUE) %>%
  select(term, estimate, conf.low, conf.high, p.value) %>% 
  knitr::kable(
  caption = "Coordinated Analysis (Full Factorial Design)",
  digits = 3
)

Coordinated Analysis (Full Factorial Design)
term	estimate	conf.low	conf.high	p.value
(Intercept)	-0.030	-0.117	0.057	0.499
monetary	1.989	1.865	2.112	0.000
non_monetary	1.008	0.885	1.132	0.000
monetary:non_monetary	-0.936	-1.111	-0.762	0.000

With this analysis they would be able to correctly communicate what would happen if they only did one of the incentives or both together. Moreover, although the impact of the non monetary incentives is smaller, it is possible that the ROI of doing just the non monetary incentive is higher and therefore that is the right decision for their company.

12.6 The Deceptive Case of Sub-additive Interactions

The dangers of uncoordinated experiments become even more pronounced, and potentially misleading, when interventions exhibit sub-additive interactions. This occurs when the combined effect of multiple incentives is less than the sum of their individual effects. In some unfortunate scenarios, the combined effect can even be less than the effect of the most potent incentive applied in isolation.

Imagine, for instance, that our monetary and non-monetary incentives are email communications. If a customer receives both a discount offer and a personalized message, they might perceive this as excessive or even spammy. Perhaps the user’s spam filter becomes more aggressive, or the sheer volume of communication leads to both emails being overlooked or dismissed. In such cases, the combined impact could be diminished, or even negated, compared to using just one incentive type.

To simulate this sub-additive interaction, let’s adjust our synthetic data. We’ll keep the monetary incentive effect strong and positive, and the non-monetary incentive with a small positive effect in isolation. However, we’ll introduce a negative interaction effect that is large enough to make the combined effect smaller than the sum of the individual effects:

# Create data frame for coordinated experiment
coordinated_data_modified <- data.frame(
  monetary = rep(c(0, 1), each = 2 * n),
  non_monetary = rep(c(0, 1, 0, 1), each = n)
)

# Define treatment effects - SUB-ADDITIVE EFFECTS
monetary_effect <- 2
non_monetary_effect <- 0.5 # Small positive effect
interaction_effect <- -1.5 # Strong negative interaction

# Generate outcome
coordinated_data_modified$outcome <- with(
  coordinated_data_modified,
  0 +  # Baseline
    monetary * monetary_effect +
    non_monetary * non_monetary_effect +
    monetary * non_monetary * interaction_effect +
    rnorm(4 * n, mean = 0, sd = 1)  # Random noise
)

Now, let’s consider what happens when the non-monetary incentive team analyzes their experimental data in isolation, oblivious to the monetary incentive experiment running concurrently. They would perform their standard regression analysis, focusing solely on the non_monetary variable:

# Team B's regression (Non-Monetary Incentive - Sub-additive Interaction)
lm(outcome ~ non_monetary, data = coordinated_data_modified) %>%
  broom::tidy(conf.int = TRUE) %>%
  select(term, estimate, conf.low, conf.high, p.value) %>%
  knitr::kable(
  caption = "Team B's Analysis (Non-Monetary Incentive) - Sub-additive Interaction",
  digits = 3
)

Team B’s Analysis (Non-Monetary Incentive) - Sub-additive Interaction
term	estimate	conf.low	conf.high	p.value
(Intercept)	0.993	0.916	1.070	0
non_monetary	-0.261	-0.370	-0.153	0

Examining Team B’s isolated analysis, we observe a troubling outcome. They would conclude that their intervention has a negative impact when the impact is actually positive.

12.7 Beyond the Simple Case: Real-World Complexities

This example, while illustrative, simplifies reality. Consider these additional complexities:

Sequential Experiments and Carryover: If Team A runs their experiment in Q1 and Team B in Q2, Team A might incorrectly conclude their effect decays over time. This could be due to Team B’s intervention, creating a carryover effect. This is particularly damaging for ROI comparisons. Team A’s intervention might be superior long-term, but this would be masked.
Interference Beyond Shared Outcomes: Even experiments targeting different outcomes can interfere. For example, if both teams use randomized encouragement designs (even for different actions), they are competing for limited user attention and decision-making capacity. Encouraging action A can impact the likelihood of action B, and vice versa. This creates subtle but important dependencies.
The Difficulty of Estimating Interactions: As Andrew Gelman points out, estimating interaction effects often requires much larger sample sizes than estimating main effects. Trying to do too much at once can lead to noisy, inconclusive results. Prioritize the most critical business questions, and be prepared to make trade-offs.

12.8 Key Takeaways: A Checklist for Coordination

Map the Landscape: Before launching any experiment, proactively identify potential interference with other ongoing or planned tests. Create a shared calendar or experiment registry.
Consider All Interaction Types: Think about both direct (shared outcome) and indirect (competing for resources, attention, or carryover) interactions.
Embrace Factorial Designs: Whenever feasible, use factorial designs to disentangle main effects and interactions. This provides a much richer understanding.
Prioritize and Simplify: Balance the desire for complete understanding with the need for statistical power. Focus on the most important interactions first.
Document, Document, Document: Clearly state all assumptions about potential interference in your experimental design and analysis plan. This promotes transparency and facilitates learning.
Communicate: If full coordination isn’t possible, at least communicate with other teams. Sharing experimental designs and timelines can help mitigate some of the risks.

By embracing these principles, business data scientists can avoid the pitfalls of uncoordinated experimentation, leading to more reliable results, better decisions, and ultimately, a greater impact on the business.

--- title: "The Importance of Coordination" share: permalink: "https://book.martinez.fyi/coordination.html" description: "Understanding and managing experimental interference in business settings" linkedin: true email: true mastodon: true --- ## Introduction In large companies, multiple teams often run experiments simultaneously. A common—but risky—assumption is: “We’re both randomizing, so it’s fine.” In reality, uncoordinated experiments can contaminate control groups and obscure true interaction effects. This chapter explains why coordinating experimental designs is crucial for reliable business insights and decisions. ## A Motivating Example Let's explore a scenario where two teams are independently trying to improve the same outcome (e.g., conversion rate, customer engagement, sales): - Team A is testing a monetary incentive (e.g., a discount, a gift card) - Team B is testing a non-monetary incentive (e.g., a personalized message, early access to features) With proper coordination, these teams could run a more informative 2x2 factorial experiment with four arms: 1. Monetary Incentive Only 2. Non-Monetary Incentive Only 3. Both Incentives Combined 4. Control (No Incentives) This design enables understanding both individual effects and their interaction: Does combining incentives create synergy (greater than the sum), antagonism (less than the sum), or no interaction? ## The Problem with Uncoordinated Experiments Without coordination, each team typically runs a two-arm experiment (their incentive vs. control). This creates two major issues: 1. Each team's "control" group is contaminated by the other team's experiment 2. The true interaction effects between interventions remain unknown Let's demonstrate this with synthetic data. We'll assume: - Monetary Incentive Effect: +2 units - Non-Monetary Incentive Effect: +1 unit - Combined Effect: +2 units (showing a ceiling effect) - Baseline (No Incentives): 0 units ```{r} #| label: setup #| message: false library(tidyverse) # Set seed for reproducibility set.seed(42) # Generate synthetic data n <- 500 # observations per group # Create data frame for coordinated experiment coordinated_data <- data.frame( monetary = rep(c(0, 1), each = 2 * n), non_monetary = rep(c(0, 1, 0, 1), each = n) ) # Define treatment effects monetary_effect <- 2 non_monetary_effect <- 1 interaction_effect <- -1 # Negative interaction # Generate outcome coordinated_data$outcome <- with( coordinated_data, 0 + # Baseline monetary * monetary_effect + non_monetary * non_monetary_effect + monetary * non_monetary * interaction_effect + rnorm(4 * n, mean = 0, sd = 1) # Random noise ) ``` ## Analyzing Uncoordinated Experiments Let's see how each team might analyze their data in isolation: ```{r} #| label: team-a-analysis #| warning: false # Team A's regression lm(outcome ~ monetary, data = coordinated_data) %>% broom::tidy(conf.int = TRUE) %>% select(term, estimate, conf.low, conf.high, p.value) %>% knitr::kable( caption = "Team A's Analysis (Monetary Incentive)", digits = 3 ) ``` In the absence of coordination, Team A's analysis would likely lead them to conclude that monetary incentives increase the outcome of interest by 1.5 units compared to no incentives, a statistically significant result. However, their model is misspecified because some of the control units are receiving non-monetary incentives, while some of their treated units are also receiving non-monetary incentives. The code bellow shows that the same would happen to the non-monetary incentives team: ```{r} #| label: team-b-analysis #| warning: false # Team B's regression lm(outcome ~ non_monetary, data = coordinated_data) %>% broom::tidy(conf.int = TRUE) %>% select(term, estimate, conf.low, conf.high, p.value) %>% knitr::kable( caption = "Team B's Analysis (Non-Monetary Incentive)", digits = 3 ) ``` Team B would conclude that the impact of non-monetary incentives is 0.5. ## The Power of Coordination Now let's analyze the data properly using a coordinated approach: ```{r} #| label: coordinated-analysis #| warning: false # Full factorial analysis lm(outcome ~ monetary * non_monetary, data = coordinated_data) %>% broom::tidy(conf.int = TRUE) %>% select(term, estimate, conf.low, conf.high, p.value) %>% knitr::kable( caption = "Coordinated Analysis (Full Factorial Design)", digits = 3 ) ``` With this analysis they would be able to correctly communicate what would happen if they only did one of the incentives or both together. Moreover, although the impact of the non monetary incentives is smaller, it is possible that the ROI of doing just the non monetary incentive is higher and therefore that is the right decision for their company. ## The Deceptive Case of Sub-additive Interactions The dangers of uncoordinated experiments become even more pronounced, and potentially misleading, when interventions exhibit **sub-additive interactions**. This occurs when the combined effect of multiple incentives is *less* than the sum of their individual effects. In some unfortunate scenarios, the combined effect can even be less than the effect of the most potent incentive applied in isolation. Imagine, for instance, that our monetary and non-monetary incentives are email communications. If a customer receives *both* a discount offer and a personalized message, they might perceive this as excessive or even spammy. Perhaps the user's spam filter becomes more aggressive, or the sheer volume of communication leads to both emails being overlooked or dismissed. In such cases, the combined impact could be diminished, or even negated, compared to using just one incentive type. To simulate this sub-additive interaction, let's adjust our synthetic data. We'll keep the monetary incentive effect strong and positive, and the non-monetary incentive with a small positive effect in isolation. However, we'll introduce a *negative* interaction effect that is large enough to make the combined effect smaller than the sum of the individual effects: ```{r} #| label: sub-additive-effects #| message: false #| warning: false # Create data frame for coordinated experiment coordinated_data_modified <- data.frame( monetary = rep(c(0, 1), each = 2 * n), non_monetary = rep(c(0, 1, 0, 1), each = n) ) # Define treatment effects - SUB-ADDITIVE EFFECTS monetary_effect <- 2 non_monetary_effect <- 0.5 # Small positive effect interaction_effect <- -1.5 # Strong negative interaction # Generate outcome coordinated_data_modified$outcome <- with( coordinated_data_modified, 0 + # Baseline monetary * monetary_effect + non_monetary * non_monetary_effect + monetary * non_monetary * interaction_effect + rnorm(4 * n, mean = 0, sd = 1) # Random noise ) ``` Now, let's consider what happens when the non-monetary incentive team analyzes their experimental data in isolation, oblivious to the monetary incentive experiment running concurrently. They would perform their standard regression analysis, focusing solely on the `non_monetary` variable: ```{r} #| label: team-b-analysis-sub-additive #| warning: false # Team B's regression (Non-Monetary Incentive - Sub-additive Interaction) lm(outcome ~ non_monetary, data = coordinated_data_modified) %>% broom::tidy(conf.int = TRUE) %>% select(term, estimate, conf.low, conf.high, p.value) %>% knitr::kable( caption = "Team B's Analysis (Non-Monetary Incentive) - Sub-additive Interaction", digits = 3 ) ``` Examining Team B's isolated analysis, we observe a troubling outcome. They would conclude that their intervention has a negative impact when the impact is actually positive. ## Beyond the Simple Case: Real-World Complexities This example, while illustrative, simplifies reality. Consider these additional complexities: - **Sequential Experiments and Carryover:** If Team A runs their experiment in Q1 and Team B in Q2, Team A might incorrectly conclude their effect decays over time. This could be due to Team B's intervention, creating a carryover effect. This is particularly damaging for ROI comparisons. Team A's intervention might be superior long-term, but this would be masked. - **Interference Beyond Shared Outcomes:** Even experiments targeting different outcomes can interfere. For example, if both teams use [randomized encouragement designs]($https://book.martinez.fyi/iv.html$) (even for different actions), they are competing for limited user attention and decision-making capacity. Encouraging action A can impact the likelihood of action B, and vice versa. This creates subtle but important dependencies. - **The Difficulty of Estimating Interactions:** As [Andrew Gelman points out](https://statmodeling.stat.columbia.edu/2018/03/15/need16/), estimating interaction effects often requires much larger sample sizes than estimating main effects. Trying to do too much at once can lead to noisy, inconclusive results. Prioritize the most critical business questions, and be prepared to make trade-offs. ## Key Takeaways: A Checklist for Coordination 1. **Map the Landscape:** Before launching any experiment, proactively identify potential interference with other ongoing or planned tests. Create a shared calendar or experiment registry. 2. **Consider All Interaction Types:** Think about both direct (shared outcome) and indirect (competing for resources, attention, or carryover) interactions. 3. **Embrace Factorial Designs:** Whenever feasible, use factorial designs to disentangle main effects and interactions. This provides a much richer understanding. 4. **Prioritize and Simplify:** Balance the desire for complete understanding with the need for statistical power. Focus on the most important interactions first. 5. **Document, Document, Document:** Clearly state all assumptions about potential interference in your experimental design and analysis plan. This promotes transparency and facilitates learning. 6. **Communicate:** If full coordination isn't possible, at least communicate with other teams. Sharing experimental designs and timelines can help mitigate some of the risks. By embracing these principles, business data scientists can avoid the pitfalls of uncoordinated experimentation, leading to more reliable results, better decisions, and ultimately, a greater impact on the business.