Friday, September 19, 2014

NCWIT Award for Aspirations in Computing: Applications open now!

NCWIT is an organization committed to getting more women in computing, across K-12, college, and industry. Every year, they have NCWIT awards that recognize women for interest in computing and give them an additional community to support them and opportunities for growth. At last year's NCWIT summit, I had the chance to meet a bunch of the award winners, and they were all bursting with enthusiasm for computer science.

If you know high school females with an interest in computing (but not necessarily an expertise!), encourage them to apply here.

In addition, NCWIT is now recognizing educators, so you can nominate any teachers, parents, or coaches that are helping the world get more women in computing.

If you'd like to spread the word, check out NCWIT's sharing kit.

More women in computing = more people in computing!

Tuesday, August 19, 2014

Auto-suggest and live documentation on Khan Academy CS

A new feature familiar to many programmers went live on Khan Academy's programming editor: Auto-suggest, parameter information, and live documentation.

I did a small screencast of that feature when it was introduced as a hackathon project, and before it went live, here.

We expect this feature to:
  • Reduce the number of times you need to interrupt your workflow to scroll down to the documentation.
  • Save time when writing programs by allowing you to click, tab, or navigate to an auto-suggested function call without having to type it all.
  • Make it easier to learn the ProcessingJS library by allowing you to get inline help.
There are 3 main parts to this feature and all of them are included as part of the recently open sourced live-editor:

Inline auto-suggest:

Inline auto-suggest

The ACE editor does provide autocomplete, but we had to modify it to disallow some keywords that we ban (like window), and add the ProcessingJS keywords that we add to the environment.

Parameter help:

Shows you the parameters for the current function call.  The current parameter is displayed in bold and blue.

Parameter help detects when you’re specifying function parameters, and will do a lookup for you.

It will show you the function name, and a list of its parameters.   As you type an argument, it will bold and color the current parameter.

Live documentation:

Highlight over the function name, or parameter to get a description of what the function does.

This allows you to hover over the parameter help and get a description of each function name and parameter relating to the current function.

Click on any function name to go to an example program using that function.

Thursday, August 14, 2014

Peer evaluations: one month retrospective

A few weeks ago we launched a new feature called Project Evaluations in the Computer Programming section on Khan Academy.

This feature allows students to request evaluations on the coding projects that they complete.
Another peer on the site would then see their evaluation request, evaluate them based on a rubric, and if they passed, they'd earn points for the project.

This blog post will analyze the data we’ve gathered since launching the feature, and talk about various changes we’ve made to address challenges that came up.

Uh oh, too many requests!

In the first day, we got a big surge of project evaluation requests. We thought this might have been because it was just released, but it turns out that it would be normal traffic.

Overall only 20% of evaluation requests were being satisfied per day in the first few days. That meant that every day, the average wait queue for getting an evaluation was getting longer.  The challenge was clear: How do we increase the amount of evaluations, without losing quality on the evaluations?

First 4 days since launch (Horizontal axis) vs. New evaluation requests (Purple) per day and evaluations (Pink) per day (Vertical axis)

We overcame this challenge relatively quickly, but only by making dozens of small changes.
See below for a list of all of those.

The same chart over a longer period of time shows the effect of these small changes.  You'll notice that the Evaluations (pink) bars start to dominate the requests (Purple)  bars per day.

Days since launch (Horizontal axis) vs. New evaluation requests (Purple) per day and evaluations (Pink) per day (Vertical axis)

Aggregating all of this data together gives us a pretty good picture of where we're at today.

Overall "Evaluated" (Green) vs "Pending evaluation" (Blue)

How did we increase evaluations?

We started off by thinking about what sort of people might evaluate projects, and there are a few key groups:
  • current students: those that are going through the curriculum themselves,
  • alumni: the students that have completed the curriculum (but may not be particularly active),
  • super users: those that already know how to program (via our curriculum or elsewhere) and are already active in the community
  • coaches of current students

When we launched, we only let the curriculum alumni perform evaluations, so all of the evaluations were coming from students that were both very active (enough to notice the new feature) and had completed the curriculum. They're still some of our most prolific reviewers, but we realized we needed to bring in more types of reviewers from across our student base, and that's where we made a lot of changes--

Lowering the bar overall:
  • We made it so that simply completing any coding challenge would qualify you to be a reviewer (which would enable us to bring in more types of reviewers).
  • To offset the lowered bar for performing evaluations, we added moderation tools and flagging to ensure quality is maintained.
Bring in current students as evaluators:
  • We added a next action after a student submits a project and passes a project, which is a link to evaluate another students' project. Those projects are picked based on the highest difficulty level that the student has passed themselves.
Bringing in more alumni reviewers:
  • We sent out notifications letting thousands of people know about the new feature, and that they were eligible to evaluate. 
  • We added a video that every student will watch at the end of the curriculum, showing them how they can be active in the community in various ways.
  • We added a notification that will get sent to every student when they earn the curriculum badge, encouraging them to be active.
Encouraging more super users:
  • We made the page listing evaluations requests easier to find and linked to it from more places.
  • We enabled the reviewers to be more efficient by presenting them immediately with a link to do another evaluation when they're done with the current one, or to skip the current evaluation if it isn't right for them.
  • We added features to make evaluations more useful (such as allowing for formatted code inside evaluations).
  • We added evaluations to profiles so that reviewers can show off the evaluations they've done.
Bringing in coach reviewers:

  • We improved coach integration so coaches can view which of their students' projects are pending.
  • We added a notification for coaches to find out about a students' project evaluation request.

Besides increasing the number of reviewers, we also reduced the number of requests, by adding messaging to prompt the students requesting evaluation to fix their own problems before asking for an evaluation.

How accurate are the evaluations?

We've seen very few cases of bad evaluations (like students making un-constructive negative comments, or failing projects that should pass), and we're generally very pleased with the quality of evaluations.

We've heard reviewers comment that they've learnt from the requestor's code, and seen reviewers share valuable tidbits of knowledge that wouldn't be gained otherwise.

We do have more work in progress for monitoring and understanding the quality of evaluations at a deeper level.

What's the distribution of evaluations across reviewers?

Initially, there was a very small amount of reviewers with a few outliers.

Since the changes we’ve made, we’ve managed to bring in a lot more people trying reviews. Over 4,000 people have performed at least one review.  2,000 have performed 1-2 evaluations, and around 100 have performed over 60.

The top evaluator has performed a staggering 3400 evaluations on their own.

Shows the number of people (vertical axis) who performed that many evaluates (x-axis)

How many projects pass?

Another interesting metric is how often students actually pass the evaluation.  During the first few days, only 54% of evaluations passed. By adding ways for users to catch their own problems before submitting for review, we’ve helped turn this into 66% passing rate. This lead to less failed evaluations, and in general it feels better when your evaluation passes!

It's worth noting that evaluations are an iterative process. When a project is marked as needing more work, the user will typically make changes and request evaluation. So even without our changes, we'd expect to see a growing number of passing evaluations over time.

"Passing" evaluations (Green) vs. "Needs more work" evaluations (Blue)

Closely monitoring changes in data

We’ve caught up with evaluations vs. requests but we’ll continue to watch this data closely.

From what we’ve gathered so far, it seems like peer evaluations works well and people are getting good evaluations on their projects.  We'll be monitoring and focusing on quality of evaluations even more in the coming weeks.

Wednesday, July 9, 2014

Peer evaluations for coding projects on Khan Academy

In our programming courses on Khan Academy, we have two ways that students practice what they've learnt: challenges and projects.

A tutorial showing both challenges and a project

Challenges give you step by step instructions, and help guide you through learning a new concept - like "here's a black and white ice cream cup, color it in with fill commands." 

Projects are more free-form. They give you a goal to complete, and let you loose to be creative and get practice with programming - like "here's an empty plate, draw something to eat on it, using any of the commands you just learned."

Challenges can be checked for validity automatically using structured.js which parses a student's program into an Abstract Syntax Tree (AST). Each challenge has grading code that matches the user's code against known good structures and outputs messages for known bad structures.

But for projects, the student's code varies so much that they can't be graded as easily automatically - and that's a good thing, we love seeing the variety in the programs that pop up. We still wanted some way to judge projects as being done, however, and to give students feedback on what they'd written.

To do this, we've now introduced the concept of project evaluations: rubric-based code reviews that students can learn from and improve their program based on.

What's it like to be an evaluator?

If a member of our community is eligible to be an evaluator, they can visit our project feedback page to see a list of the projects awaiting evaluation.

For each project, there’s a list of objectives that the reviewer should look out for.  Each objective can be marked as "Passed" or "Needs more work".

Performing an evaluation

When there’s a failed objective, the evaluator can leave additional comments, to give the student ideas of ways they can meet that objective.

The evaluator can additionally leave additional comments at the end, which they often use to leave encouraging remarks or pro tips.

What's it like to be a student?

When a student starts on a project, they are given a summary of what to do and the project is marked as started.  

When they decide they're finished working on a project, they can request an evaluation.

What the person doing the project sees

After some time, the evaluation will happen, and if it's successful, the student  will then get a notification that their project is now marked as completed, or needs more work. If it needs more work, the student will make adjustments and re-request evaluation.  Once the project evaluation passes, the student will get points for their project.

How is eligibility determined for evaluators?

Community members who have earned the "Intro to JS mastery badge" are eligible to do peer evaluations. They can earn this badge by completing the "Intro to JS: Drawing & Animation" computer programming course. 

The eligibility may be tweaked in the future depending on the mean wait time of project evaluation requests and the quality of reviews.

How did the launch go?

So far, the launch has gone amazing. There's an initial surge of project evaluations to be caught up with, but we're hoping to keep the average wait time around a day.  We've immediately seen hundreds of project evaluation requests as well as dozens of awesome and supportive evaluations.  The advanced programmers in our community love the opportunity to help new programmers learn: 

They're also forthcoming with their praise for a well done program - both in terms of functionality and readability.

We've also got a ton of great feedback from the top evaluators, and we'll continue iterating on the system (just like the students are iterating on their projects) to help our community keep learning from each other.

Wednesday, June 25, 2014

Lowering the barriers to learning programming

At Velocity 2014, I gave a talk to the attendees, a room full of web developers and backend ops, about why we need to lower the barriers to programming, and ways we can do that.

Since the goal of the talk was to give everyone an action item to lower a barrier in some way, here's a run-down of the ways you can help:

  1. Donate hardware:
    You can donate your own unused laptops to local schools or organizations (like Boys & Girls Club, YMCA, or coding specific ones).
  2. Fund a project:
    You can search on for projects that are funding laptops to teach CS in their classrooms. You can also filter by topic, poverty level, or location, and you can turn your donations into a gift in the name of a friend or family member.
  3. Contribute to online programming environments:
    You can contribute to existing open-source environments like,, or to the tools that power them like JSHint, ACE editor. You could also build curriculum around existing environments, or create an entirely new environment, like for Arduino, Objective-C, or 3D games.
  4. Make CS count in your K-12 state/district:
    You can visit, select your state, and follow the instructions to sign a petition or email a politician. You could also write letters to your local high school, especially if you have a student there, and offer to explain to the administration why it's important. Perhaps you could even volunteer to teach yourself (like via the TEALS program or mentor a teacher.
  5. Volunteer at a local coding club:
    Search on for local coding clubs, and volunteer as a mentor or teacher. If you can't find one, start one (like a CoderDojo chapter). You can use online self-guided curriculum like Khan Academy, so you don't even have to feel comfortable teaching.
  6. Encourage your kids:
    According to recent Google research, encouragement - especially parental - is very important. Encourage your kids, especially daughters, but do not pressure them.
  7. Clear up misconceptions about CS as a career:
    Many students don't know what CS is like as a career and realize that it can have social impact. Share stories of CS in the real world, like from Khan Academy's Meet the Computing Professional, Computing is Everywhere, Made With Code. Share those with kids you know, and encourage teachers to highlight one a week.

If you're interested in the whole talk, you can watch the video recording, see the slides embedded below or see slides here with speaker notes. Think of other ways that industry professionals can help? Let me know in the comments!

Monday, June 23, 2014

CS4HS @ Cal: Meeting CS high school teachers in person!

This past Friday, on June 20th, I was invited and honored to stop by CS4HS @ Cal, a gathering of local high school Computer Science teachers. As you can see from their group photo, they were a lively bunch:

I presented slides giving an overview of the Khan Academy programming curriculum, community, and challenges, plus our future plans:

It was lovely to speak with teachers that were using our curriculum or thinking of using it, and informative to hear about the struggles (and joys!) they face as CS teachers. If you're a CS teacher, you might be interested in reading through the extensive notes they took during the meeting.

We look forward to more opportunities to connect with CS teachers in the future! Thanks again to the inimitable Dan Garcia for inviting us.

Monday, May 12, 2014

New advanced JS course: Learn to program natural simulations!

Last fall, I discovered an online and print book about programming natural simulations using the Java version of ProcessingJS. I started going through the book myself, porting the programs to JavaScript and trying the suggested exercises. It was so much fun that I thought, hey, let's put this on Khan Academy so that all of our students can have this much fun learning!

Fortunately, author Daniel Shiffman CC-licensed the book, so we were allowed to port over the text, and add our interactive coding challenges and projects on top.

We've just released the new course as "Advanced JS: Natural Simulations," and we think that teachers and students alike will find it a great way to practice JavaScript, learn new ProcessingJS functionality, and brush up on basic math and physics concepts along the way. Here's a run-down of the units, to give you a better idea of what's covered:
  • Randomness: Learn about randomness, probability, and normal distributions, and use those techniques to make all sorts of "random walkers."
  • Noise: Learn about Perlin noise, which you can then use to create animated clouds and mountain ranges.
  • Vectors: Learn how to program with vectors and use them to simulate lightsabers, braking cars, and mouse-following motion.
  • Forces: Use forces to make more realistic simulations, modeling gravity, friction, and fluid resistance.
  • Angular movement: Use angles, polar coordinates, and trigonometry to simulate angular movement like batons spinning, boulders rolling, and cars rotating as they turn.
  • Oscillations: See how you can use the sine function to model simple harmonic motion, how to simulate a pendulum, and how to program Hooke's law for spring forces. Create programs that simulate slinkies, waves, and puppets along the way.
  • Particle Systems: Now that you've learned many ways to program the movement of single objects, you'll create particle systems, the standard way that games and movies simulate movement of groups of particles, like for smoke, wind, fire, and water.

Intrigued? Check out the intro video here, and you'll be on your way!