Peer evaluations for coding projects on Khan Academy

In our programming courses on Khan Academy, we have two ways that students practice what they've learnt: challenges and projects.

A tutorial showing both challenges and a project

Challenges give you step by step instructions, and help guide you through learning a new concept - like "here's a black and white ice cream cup, color it in with fill commands." 

Projects are more free-form. They give you a goal to complete, and let you loose to be creative and get practice with programming - like "here's an empty plate, draw something to eat on it, using any of the commands you just learned."

Challenges can be checked for validity automatically using structured.js which parses a student's program into an Abstract Syntax Tree (AST). Each challenge has grading code that matches the user's code against known good structures and outputs messages for known bad structures.

But for projects, the student's code varies so much that they can't be graded as easily automatically - and that's a good thing, we love seeing the variety in the programs that pop up. We still wanted some way to judge projects as being done, however, and to give students feedback on what they'd written.

To do this, we've now introduced the concept of project evaluations: rubric-based code reviews that students can learn from and improve their program based on.

What's it like to be an evaluator?

If a member of our community is eligible to be an evaluator, they can visit our project feedback page to see a list of the projects awaiting evaluation.

For each project, there’s a list of objectives that the reviewer should look out for.  Each objective can be marked as "Passed" or "Needs more work".

Performing an evaluation

When there’s a failed objective, the evaluator can leave additional comments, to give the student ideas of ways they can meet that objective.

The evaluator can additionally leave additional comments at the end, which they often use to leave encouraging remarks or pro tips.
  

What's it like to be a student?

When a student starts on a project, they are given a summary of what to do and the project is marked as started.  

When they decide they're finished working on a project, they can request an evaluation.

What the person doing the project sees

After some time, the evaluation will happen, and if it's successful, the student  will then get a notification that their project is now marked as completed, or needs more work. If it needs more work, the student will make adjustments and re-request evaluation.  Once the project evaluation passes, the student will get points for their project.
 

How is eligibility determined for evaluators?

Community members who have earned the "Intro to JS mastery badge" are eligible to do peer evaluations. They can earn this badge by completing the "Intro to JS: Drawing & Animation" computer programming course. 

The eligibility may be tweaked in the future depending on the mean wait time of project evaluation requests and the quality of reviews.

How did the launch go?

So far, the launch has gone amazing. There's an initial surge of project evaluations to be caught up with, but we're hoping to keep the average wait time around a day.  We've immediately seen hundreds of project evaluation requests as well as dozens of awesome and supportive evaluations.  The advanced programmers in our community love the opportunity to help new programmers learn: 

They're also forthcoming with their praise for a well done program - both in terms of functionality and readability.

We've also got a ton of great feedback from the top evaluators, and we'll continue iterating on the system (just like the students are iterating on their projects) to help our community keep learning from each other.