Tutorial 4

Categories → Tutorial

Does a training programme increase participants' wages?

Introduction

The study explores whether a training program increases participants' wages, given other confounding factors. Later updates to this study added new participants and there was some controversy over whether the right participants were included in the study to draw strong conclusions. But we're not interested in the final result - we would like to show that you can explore counterfactual outcomes using Causal Wizard.

Counterfactual Outcomes - What if?

Remember, a counterfactual outcome is one which didn't happen, but might have. Causal Wizard's notebooks let you understand what would have happened, if different Treatment statuses (Treated, or Control) were applied to various sample sub-groups. Ok, let's get started.

Step 1: Get the data

Download tutorial-4-lalonde.csv, then upload it via the Data page.

Step 2: Create a new Study

1. Click on Studies in the menu bar.
2. Give the new Study any name you like (e.g. Tutorial 4).
3. Select the Dataset you just uploaded, and press Create.

Step 3: Examine the data

Before we go any further, let's examine the data. Click Explore dataset at the top right of the Study page. You'll see the variables available in the data file and their types. In particular, note that Treated and Wage_1978 - our treatment and outcome - are Boolean and Real (continuous numerical) respectively.

Step 4: Specify Treatment and Outcome

In this case study our treatment is the variable Treated (a boolean value, True if the participant completed the training program, and False otherwise). The outcome we're interested in is Wage_1978, 3 years after completing the training program.

Set the Treatment and Outcome variables as described above. Click Treatment groups and confirm False is Control and True is Treated - this is the default for a boolean column.

Step 5: Add Treatment and Outcome to the Causal Diagram

Now let's start to build our Causal Diagram by adding the Treatment (Treated) and Outcome (Wage_1978), then add an edge between them, expressing that Treatment directly affects the Outcome.

Step 6: Complete the Causal Diagram

We should add some additional confounding variables to complete the Causal Diagram. Add nodes for Wage_1974 and Age, then:

• Add an edge from Wage_1974 to Treated, and another from Wage_1974 to Wage_1978. Wage_1974 (earnings before the study) affects both whether someone was selected for training and their later wages - an important confounder, and a good illustration of how an earlier measurement of the outcome can itself be a confounder when values vary over time.
• Add an edge from Age to Wage_1978 only (not to Treated) - just for variety, not reflecting the original study, we'll assume age affects wages but wasn't a factor in who received training.

Click each node in the diagram to change how it's interpreted (Numerical vs Categorical) if you want to experiment - a boolean column can be treated either way (as Numerical, False=0 and True=1).

Step 7: Check and select Model

Press Check to confirm the Study is set up completely. If successful, you'll be offered a choice of models. Select Backdoor: Linear Regression, because this includes counterfactual analysis. Click Download config JSON.

Step 8: Run the notebook

Open the notebooks (Open in Colab, or clone the repository) and run both notebooks top to bottom, providing your config file and data file when asked.

Step 9: Examine counterfactual results

For now, skip the other sections and scroll down to the Counterfactual outcomes table in notebook 2's output. Let's examine the results to see what they say and what insights we can find.

The table has a row per scenario, and count, sum and mean columns.

Each row represents the outcome of a different scenario. For example, one row is "if all samples were treated" i.e. in this study, if all participants received the training program. Scenarios can vary in the number of participants involved - for example, some scenarios include only Control or Treated samples - so the count column tells you how many samples are involved.

The sum and mean columns are summary statistics of each scenario. We can quickly sanity-check that these numbers make sense, for each scenario. Let's focus on mean outcomes.

First, we can see that the mean outcome for all actual controls is approx. 4500 and the mean outcome for all actual treated is about 6300. So we might expect to see the mean outcomes of all scenarios to be between these values.

Sure enough, we can see that if all samples were controls the mean outcome would be close to 4600 (slightly higher than actual controls) and if all samples were treated, the mean outcome would be close to 6200 - slightly lower than the actual treated outcomes (your exact numbers will vary slightly, since the notebook re-fits the model with a random held-out test split each run).

Finally, two additional scenarios swap treatment status for Control and Treated groups separately. These values also lie within the expected ranges.

You can use counterfactual analysis on your own data in the same way, to estimate the effect of treatment on untreated samples, and vice-versa.