Autograder Manual
Table of contents
Purpose
The purpose of this document is to outline the functionality, use, and rules of the CS 3210 autograder.
The autograder runs a series of tests to ensure that your implementation satisfies the constraints outlined in the lab specification. The test cases are not public. The autograder will only tell you if you passed or failed a test. It does not provide any additional context into what was tested or exactly why you failed a test. This design is intentional. We want to make sure students pick up the skill of testing and debugging in a non-trivial environment.
The goal of the autograder is to help you ensure you’ve implemented the project correctly, and have a good idea of the credit you will receive. It is not intended for debugging your project.
Usage
Every lab comes with a submit.sh script (located under the scripts/ directory of your xv6 checkout). This script outputs a zipped archive of your source code.
To use the autograder, submit this zip file to the corresponding assignment on Gradescope.
Gradescope will report back results after the autograder is finished running.
Autograder Policies
The autograder is intended to help you throughout this course, and give you some confidence as to where you stand with your lab submissions. However, the autograder doesn’t have infinite resources, and needs to ensure submissions are kept safe. As a result, the autograder has several specific policies it will require you to adhere to.
Feedback Submissions
You may submit to the autograder an unlimited number of times per day, but the autograder will only give feedback for a fixed number of submissions that varies per lab. In addition to this, each lab will have bonus attempts. Bonus attempts are allocated for the entire duration of the lab and do not get refreshed daily. Use them wisely.
| Lab | # of submissions per day | # of bonus attempts |
|---|---|---|
| Lab 1 | 3 | 2 |
| Lab 2 | 5 | 0 |
| Lab 3 | 5 | 0 |
| Lab 4 | 5 | 0 |
If students were to all submit their lab frequently (and last minute), the autograder queue would become very long, potentially causing students to wait many hours before their submission is graded, resulting in poor grades and late submissions. This issue is caused by students attempting to use the autograder as a last-minute debugging tool instead of its true intention – an automatic grading and feedback mechanism. To help alleviate this issue, the autograder will only provide students with a limited amount of feedback per day by limiting the number of submissions it provides feedback for.
To be more precise, when you run the autograder with a feedback-enabled submission, the autograder will tell you what test number it’s running, if you passed that test, then ultimately the total number of tests you’ve passed. This will give you a good idea of how well your submission did. Once you’ve expended all of your available feedback enabled submissions, the autograder will only tell you that it has run your submission, and provide no graded feedback.
Malicious Submissions
All submissions to the autograder will be logged both locally (on the autograder) and remotely (off the autograder), and may be audited. Any unauthorized attempt to subvert the autograder, either by gaining control of the autograder machine, exposing autograder testcases, or generally using the autograder for any purpose other than its intended one (to grade your code), will be considered a violation of Georgia Tech’s honor code and will be severely punished. The autograder has many safety checks and fail-safes in place. Do not attempt to attack the autograder!
Autograder-Specific Requirements
The autograder does place some specific restrictions on how you modify your code and construct your labs. You may add any source files you like to the bootloader, kernel, or user files, but the autograder will not respect any changes to CMakeLists.txt (that would allow students to run arbitrary code). Instead, you must specify any changes in the list within the Sources.cmake directory.
Additionally, the autograder will ignore any changes made to any of the scripts run to create the kernel. The files the autograder will ignore or overwrite includes:
CMakeLists.txtuser/CMakeLists.txtkernel/CMakeLists.txtbootblock/CMakeLists.txttools/CMakeLists.txtscripts/xv6-qemutools/mkfstools/cuthtools/printpcstools/pr.pltools/runofftools/runoff1tools/runoff.listtools/runoff.spectools/show1tools/spinptools/toc.ftrtools/toc.hdrkernel/tools/mksym.shkernel/tools/vectors.pl
Furthermore, the autograder restricts the changes you are allowed to make to your Sources.cmake (kernel/Sources.cmake, user/Sources.cmake, and bootblock/Sources.cmake). You are only allowed to modify the lists present in those files.
You may not add any autograder generated files (e.g. anything in the build directory, or any files generated by cmake to your git repository). Doing so will likely cause an autograder error, wasting your submission.
You may not modify the transition interface between the bootblock and the kernel. Doing so may cause the autograder to crash when booting your kernel.
Finally, when the autograder runs your code, it will do so in Release mode (with the option -DCMAKE_BUILD_TYPE=Release). Ensure your code works in Release mode before submitting it to the autograder!