Saturday, March 29, 2014

New way to review what you learn in Intro to JS!

For each of the concepts that we cover in Intro to JS, students learn and practice through an alternating sequence of talk-throughs and challenges, and then finish up with a creative project. We want to give students a way to review what they've learnt in the talk-throughs without having to re-watch them all, to give them another opportunity to solidify everything they've learnt, and to give them one more chance to ask questions about the concepts. So, we've added "Review" articles at the end of each tutorial, which students can read and discuss, before they tackle the project. Here are quick links to check out each one:

We've only had these articles out for a few days, but students love them so far and are asking great questions in the discussion. If you have students learning JS, you may want to point them to the articles to refresh and reenforce what they've learned.

Friday, March 28, 2014

New tool for programming teachers: see your students' progress!

When a student is going through our Intro to JS curriculum, they have a distinct visual goal: make the icons completely green! For example, here's my progress on the Arrays section, where I've made it most of the way through:

As a teacher of students going through the curriculum, how can you know how each of your students is progressing? You can look over their shoulder, as you're wandering through the classroom. But maybe your students are virtual, or they're working on it from home, or you want to check at night when you were wondering how all your students are doing.

To help teachers track individual student progress, we've now added "Intro to JS" as an option in the "Student Progress" report, and we show each student's progress through the curriculum when you select it:

You can see what content they've started, completed, or not started at all, and you can filter to view only the completed content. You can also click on the Videos or Activity tabs to narrow down to what they've done specifically in the last week, month, etc.

We hope this helps give you more insight to how your students are progressing, and of course, we'll continue to work on the teacher experience.

Tuesday, March 25, 2014

New Mastery Badge for Intro to JS!

Now that we have a complete Intro to JS curriculum with more than 30 interactive coding challenges, how can students show to their teachers and the world that they've successfully completed it? As of today, with a brand new shiny "Intro to JS: Drawing & Animation Mastery" badge!

We consider this badge a "challenge patch", like the ones we already reward to students that complete all the "Fraction" exercises or "3rd grade math" mission, so that's where you'll find it on a user's profile. For example, to see it on my profile, visit my badges page and click "Challenge Patches".

Once students earn the badge, they'll get a notification (like with all badges), and they can choose to add it to their showcase, since it's a pretty cool accomplishment they can be proud of.

Monday, March 24, 2014

Watched a programming talk-through? See it in the progress reports!

For both students and especially for teachers, one of the most helpful aspects of Khan Academy is the activity logging: we record when you do videos, exercises, and challenges, and surface that on user profiles, progress viewer, and teacher reports.

After launching the DonorsChoose rewards program a few weeks ago, we got a lot of reports from teachers anxious that they couldn't see the fact that their students had watched the programming videos- that's because the programming "videos" are actually "talk-throughs", these crazy things we invented that sync audio to coding. But, hey, that shouldn't matter, we should still be logging the talk-through watching the same way we log the Youtube video watching everywhere else on the site. We want programming teachers to know as much about their students' progress as math teachers, to have a consistent, great experience.

As of today, we are now logging talk-through progress the same way that we log video progress. You'll see them show up in the activity feed, progress reports, weekly emails, activity graph - anywhere you see videos now:

We hope this helps teachers track their students better, and we look forward to continuing to improve the teacher experience.

What does a computing professional look like? Find out on Khan Academy!

We have thousands of K-12 students learning computer programming on Khan Academy - we don't expect all of them to go on to major in Computer Science and dedicate their career to computing, but well, we certainly hope that some sizable portion of them do make that decision. But do they even know what it means to have a career in computing? As Shuchi Grover described in her article on fixing misperceptions of CS, research shows that most students don't have a "clear understanding of what computer science is and what a computer scientist does."

That's why we've launched our "Meet the Professional" series. We found people working in different fields (like physics, charity, and games), from all over the world (including a few nomads), with a ton of different hobbies, too (dancing, archery, biking, writing) - our goal is to show that diversity comes in many forms!

Here's who we started with:

For each of the professionals, we asked them what they work on, how they learned to program, and what they do for fun. We also finished off by asking each of them "What’s your one piece of advice for new programmers?" and we got some great responses from them, which I want to share a few of here:

"Don’t be afraid to break stuff, it’s all virtual anyway." - Yann Dauphin
"Do not let other people intimidate you. Everyone is continuously learning, and people will probably be using very different tools and languages just a few years from now. This field is extremely broad but extremely young, and we’re not even close to figuring out all the new things we can do, and a few years and a big ego don’t matter as much as being able to continually learn." - Amy Quispe
"Programming is super creative - find a problem you’re interested in solving or something you want to create and it will make the challenges worth solving." - Lauren Haynes
"It’s easy to look at great programmers and feel really inadequate, like you’ll never measure up or be a ‘real’ programmer. But I have had the privilege of meeting some of the greatest programmers on the web today, and they all have had the same experience - they all felt like fakes and impostors when they were starting out, and none of them thought it was easy at first, no matter how easy they make it look now. If you have written one single line of code, then you are a real programmer - and if you keep on coding and learning, you will one day be a master, too." - Bill Mills
"The most important thing for me has been to make things that make me giggle." - Marcos Ojeda

The students have loved the stories so far, expressing their thanks in the comments that they've found people who both inspire them and are like them. A few favorite reactions:

"Thank you so much for your article and for your background story, I can honestly relate to the micro-inequity issues that you mention. However after reading your articles, I am encouraged to persevere in this field that I find so interesting." - a reaction to Philip Guo
"You rock! You're a programmer, you think goofy programs are important, and you have a dog... what could be awesomer? a clue: NOTHING." - a reaction to Marcos Ojeda
"I've been trying to learn on my own while I get ready for a 2nd bachelors in computer science. I feel positive but so nervous about the change--I'm 26 and just learning to code, I'm a girl and not a typical geek, and it feels like I'm leaving the known behind. This was the perfect thing for me to click on! It's encouraging to read your story (down to that we even have the same first names and enjoy dancing). Thanks." - a reaction to Carrie Cai
"I really like how the author mentioned how he failed at code until grad school! For me, it's encouraging to know that it's possible to start later on in life, especially now, where a lot of people start really young and are pros from the start. It really is nice to know that everyone has a chance to become good at coding, regardless on current skill level!" - a reaction to Bill Mills

We look forward to adding more computing professionals to the series, to further expand the range in students' minds of what they can do with computer programming and computer science knowledge. If you have ideas for someone that would be great to highlight, let us know. If you are teaching students, have them read through the series, ask them who their favorite is, and if it's given them new ideas for what they could do. Help us show them the diverse world of computing professionals!

More programming projects for the classroom

Every Wednesday, I'm a volunteer instructor for a local GirlsWhoCode club in San Francisco. GirlsWhoCode happens to use the Khan Academy programming platform for all of the clubs, so it's a great excuse to use my curriculum in the wild, with real live high schoolers.

We've mostly been using the official curriculum and its projects so far, but in the last few weeks, we've been experimenting with new projects, just based on the topics that come up from the students that we think would be interesting to explore.

These projects are better done in a classroom setting, where there's a teacher to facilitate them, so instead of putting them into the official curriculum, I've made a new page in our coach resources section to link to lesson plans for each project.

To start off with, we have lesson plans for a fun multi-day Hangman project (shown below), and a shorter project around the Higher/Lower game and binary search. If you have project ideas from your own classrooms, please leave a comment here.

Tuesday, March 4, 2014

Challenge completion across gender and age

Lately, I've seen the same questions come up in the programming and CS education circles:

  • What age is the best to start to learn programming? What's the minimum? What's the optimal? What type of programming should be learnt?
  • Are both genders equally capable of learning to program? Are there ways of teaching that are more gender-neutral than others?

These are interesting questions, and their answers have big implications for people creating programming curriculum and teaching it to others. Since we're fortunate to have many students learning programming on Khan Academy and we're logging their activity, I wanted to see if we could answer those sort of questions in relation to our programming curriculum. We currently only have gender for 10% of our users and birthdate for 13% of our users (as we introduced them as optional fields a year ago), so our data is far from complete - but given we have such large numbers to start with, it seemed worthwhile to explore what we could find out in that set of users.

Effect of gender

When we look at our entire Khan Academy user base and query the user base with filled out gender information, 48% are female and 52% are male - pretty close to half and half. That may be because we have many classroom students, and classrooms tend to be split evenly across the genders.

Now, if we look specifically at the Khan Academy programming curriculum - by looking at all the users that begin the very first coding challenge and have filled out their gender, then we have a different split: 34% are female and 66% are male. We see here a bias towards male, perhaps because much of CS learning happens outside of classrooms right now - more of our independent learners crowd. For whatever reason (and you may be already thinking of many reasons!), more males are choosing to independently learn programming than females.

We've established that more males are doing our curriculum than females, but how well do each of them do, once they've made the choice to start learning? That's where it gets exciting. If we look at the completion stats for that first coding challenge, 86.2% of males complete it (20762/23940) and 86.7 of females complete it (10770/12494). Those are very close numbers- close enough that the difference could be due to statistical error. That means that males and females are just as likely to complete the coding challenge, as long as they start it. This is very encouraging, and gives us good reason to come up with initiatives to raise the numbers of females that try out the curriculum.

Effect of age

We basically looked at the same stats, but for age. Since the main question is how young programming can be taught (elementary school, middle school, high school, or college), I'll snippet out the data that covers that age range. As it happens, that's also where the data is most interesting.

From ages 8-25, we have challenge data from a total of 40,269 students, with some ages much better represented than others. Here's how many students started the first coding challenge at each age:

8 512
9 1280
10 2660
11 4120
12 4068
13 3116
14 5997
15 5897
16 3207
17 2364
18 1949
19 1460
20 818
21 580
22 474
23 494
24 826
25 447

For each of the age groups, we looked at how many of them completed the challenge, and calculated the challenge completion stats (like we did with gender above). We're attempting to answer the question as to whether someone's age significantly affects the likelihood that they'll complete the challenge. Here are the stats for each age:

8 71.48
9 73.44
10 70.94
11 75.46
12 79.74
13 81.68
14 83.06
15 83.52
16 86.81
17 88.16
18 87.79
19 88.01
20 89.85
21 88.62
22 89.03
23 89.47
24 87.41
25 89.93

As you can see, the completion rate does increase with age, and it appears to be a significant difference. Here's what that looks like in graph form:

I have to admit, I was a little disappointed when I saw the stats. There was a part of me that was hoping that even 8 year olds would be as likely as 12 year olds - when in fact, there's nearly a 10% difference between them. The completion rate gets pretty stable around 16, around 86-89%.

Now, there's a lot of possible reasons why there are seemingly significant completion differences from 8 to 16, and not every reason is "its too young to start to learn programming." Some examples: 1) our programming environment involves a bit of basic math/spatial reasoning, which they may not feel as comfortable in 2) programming requires a degree of patience, which younger kids have less of, 3) programming involves typing skills, more so than other apps that kids may use, 4) our programming is easier on desktop browsers than iPads, and maybe young kids tend to use iPads. Some of these reasons are things we could address - for example, we could teach a non-spatial programming environment, we could teach non-syntactic visual programming (like SCRATCH), we could teach typing skills (something we'd like to do anyway). Some of them may just be the biology of younger kids, and there being an age at which the brain is better prepared for the type of abstract concepts and reasoning involved in programming.

Saying all that, the completion rates are 71% for 8-year-olds and 80% for 12-year-olds - still pretty high rates, high enough that a teacher or parent could feel comfortable introducing their student or kids to programming, especially if they are there to support them.

We look forward to continuing to explore our data, to see what we can learn about teaching programming across different demographics, and we'll be attempting to increase the percentage of our user base that has this demographic data filled out as well, to lend more significance to the results.

Monday, March 3, 2014

A/B Testing Curriculum: To Sneak Peek Or Not?

In our intro tutorial, we explain what programming is, why it's cool, and how you can learn it on Khan Academy. Our goal in that tutorial isn't to teach programming - it's to get them excited about learning programming.

We thought that one way to achieve that goal would be to give students a teaser of what they could make if they kept going - so we added a "Sneak Peek" talk-through, where Sophia creates a cool painting program from scratch, telling students "you won't understand this program yet, but if you keep going, you will!". Since adding that talk-through to the intro, we've had mixed reactions. Some students say "wow, that was so cool, makes me want to keep going!" - but other students leave visibly confused comments. One student even created an animation to encourage those confused studnets to keep. For some, it's a motivator. For others, it's a de-motivator.

So, that leaves us with a dilemma: Do we keep or remove the "Sneak Peek"? What decision will result in more students learning? To help us with this decision, we utilized our A/B testing framework. We created a 50/50 test, where half the students saw "Sneak Peek" in the curriculum and half didn't, and measured various outcomes: how many watched the "Intro to Drawing" talk-through? How many completed the first coding challenge? How many completed the much later variables challenge? We wanted to see both the short-term effect and the more long-term effect, in case there was a difference.

We ran that A/B test for the last week, and already had statistically significant results by the end of that week. More than 24,000 students went through the intro curriculum during that week, and a significantly larger percentage of the "Hide-sneak-peek" students continued on to watch the drawing talk-through and complete the coding challenges. In every measure, hiding the sneak peek resulted in greater learning outcomes:

And now, we can remove the "Sneak Peek" talk-through with a degree of confidence, because we've seen that students are more likely to keep going if they don't see it.

This is one of the reasons that I'm personally excited about teaching on Khan Academy. We have so many students learning on our platform that we can actually do these sort of experiments and get meaningful results. When we combine these data-driven approaches like A/B testing with live user testing, surveys, and feedback, we can learn so much about the most effective ways to teach.