MBPP

Note

The Mostly Basic Programming Problems (MBPP)[] dataset contains 974 programming tasks, designed to be solvable by entry-level programmers. We asked crowd-sourcing participants to write a short problem statement, a single self-contained Python function solving the problem specified, and three test cases that check for semantic correctness of the function. Participants also provided a ground-truth solution that passes all three test cases.

We manually inspected, edited, and pruned a subset of the questions, yielding 426 hand-verified questions, which we refer to as the edited dataset. For each question in the edited dataset, we ensured it had a standard Python function signature, that it was unambiguous to a human, and that its test cases accurately reflected the text description.