The Concept of Validity

Validity can be understood as the approximate truth of an inference. While this may sound abstract, in simpler terms, validity assesses whether the data we have supports a conclusion as being accurate.

It’s essential to remember that validity pertains to the causal relationships between the observed variables in a sample or population. It addresses whether we can believe that A caused B, rather than the observed differences being mere coincidences or influenced by extraneous factors.

To illustrate, consider whether changing the format of a Medium article results in longer reading times. Here, A represents the format change, and B denotes the reading duration. The counterfactual in this scenario is whether the observed increase in reading time (B) would have occurred regardless of the format alteration (A).

Four Key Types of Validity

There are various types of validity, each relevant depending on the context. While we will concentrate on internal and external validity, it's beneficial to be aware of other forms.

Statistical Conclusion Validity ensures that appropriate mathematical and statistical methods are employed to draw accurate conclusions about the relationship between independent and dependent variables.

Construct Validity addresses the concern that the data observed accurately represents the higher-order concepts or variables being measured. This is particularly significant in social sciences and fields of data science focused on human behavior, where many theoretical constructs cannot be directly measured.

External Validity refers to the extent to which the causal relationships identified in a specific study can be generalized to broader populations or different situations. This is crucial when conducting experiments or pilot studies on subsets, as you want to determine if results will be applicable at scale or in different contexts.

Internal Validity pertains to whether changes in the independent variable genuinely lead to changes in the dependent variable within the study population. For instance, can we assert that the observed differences in reading times are due to the formatting changes?

Given its significance, we will delve deeper into internal validity throughout the remainder of this discussion.

Deepening Our Understanding of Internal Validity

Internal validity is paramount for establishing causal relationships between independent variables (like formatting changes) and dependent variables (such as reading time).

It becomes evident that the sequence of effects matters; A must precede B, A must vary with B, and there should be no alternative explanation for B other than the influence of A.

However, what risks must we address to ensure our internal validity is robust?

Risks to Internal Validity:

• Causal Time Order: This risk arises when B occurs before or concurrently with A. For example, if reading times increase before any format changes are implemented, this creates uncertainty.
• Selection Bias: This occurs when the characteristics of study participants differ for reasons unrelated to the treatment or independent variable. For instance, if those who stick to a diet tend to show better weight loss, the true effect of the treatment may be obscured by their inherent traits.
• History: This risk entails that external events might influence B simultaneously with A. In our example, if changes in the Medium app prevent quick scrolling, it could skew reading time results.
• Maturation: Other factors unrelated to the treatment could naturally influence outcomes. For instance, if your followers are becoming devoted fans and thus taking their time with your articles, this could affect reading time.
• Experimental Mortality (Attrition): Over time, participants may drop out of a study, resulting in a sample that doesn't accurately represent the original population.
• Instrumentation Threats: If the measurement tools used are inconsistent or fail to align with study objectives, results may be compromised.
• Contamination: This occurs when the treatment itself alters behavior in a way that affects the results, as seen in the Hawthorne effect, where participants modify their behavior because they know they’re being observed.

Strategies to Mitigate These Risks

Implementing a comparative change design with treatment and control groups can help mitigate risks. This approach ensures that historical events affect both groups equally, allowing researchers to measure treatment impacts effectively.

In addition, centralized randomization in group selection promotes homogeneity between treatment and control cohorts, minimizing selection bias risks. A pretest and post-test design can further elucidate the causal relationships involved.

However, vigilance against potential contamination remains essential, even within a comparative change framework.

Final Thoughts

Your data science team is striving to deliver value, so it’s imperative to establish clear tests to confirm that your initiatives are achieving their intended goals. Increasingly, I hear of projects boasting remarkable improvements, yet many lack the means to discern whether these enhancements were inevitable (the counterfactual) or genuinely stemmed from the data science efforts.

The first video you should check out is about standing out in data science projects. It discusses essential strategies to differentiate your work in the field.

Additionally, gain insights into typical data science projects through this informative presentation, which outlines common practices and outcomes.