A/B Testing Curriculum: To Sneak Peek Or Not?

In our intro tutorial, we explain what programming is, why it's cool, and how you can learn it on Khan Academy. Our goal in that tutorial isn't to teach programming - it's to get them excited about learning programming.

We thought that one way to achieve that goal would be to give students a teaser of what they could make if they kept going - so we added a "Sneak Peek" talk-through, where Sophia creates a cool painting program from scratch, telling students "you won't understand this program yet, but if you keep going, you will!". Since adding that talk-through to the intro, we've had mixed reactions. Some students say "wow, that was so cool, makes me want to keep going!" - but other students leave visibly confused comments. One student even created an animation to encourage those confused studnets to keep. For some, it's a motivator. For others, it's a de-motivator.

So, that leaves us with a dilemma: Do we keep or remove the "Sneak Peek"? What decision will result in more students learning? To help us with this decision, we utilized our A/B testing framework. We created a 50/50 test, where half the students saw "Sneak Peek" in the curriculum and half didn't, and measured various outcomes: how many watched the "Intro to Drawing" talk-through? How many completed the first coding challenge? How many completed the much later variables challenge? We wanted to see both the short-term effect and the more long-term effect, in case there was a difference.

We ran that A/B test for the last week, and already had statistically significant results by the end of that week. More than 24,000 students went through the intro curriculum during that week, and a significantly larger percentage of the "Hide-sneak-peek" students continued on to watch the drawing talk-through and complete the coding challenges. In every measure, hiding the sneak peek resulted in greater learning outcomes:

And now, we can remove the "Sneak Peek" talk-through with a degree of confidence, because we've seen that students are more likely to keep going if they don't see it.

This is one of the reasons that I'm personally excited about teaching on Khan Academy. We have so many students learning on our platform that we can actually do these sort of experiments and get meaningful results. When we combine these data-driven approaches like A/B testing with live user testing, surveys, and feedback, we can learn so much about the most effective ways to teach.