Tester Jobs Requirements (VIDEO GAME)

Evaluator for algorithm implementations

Evaluator for algorithm implementations

Algorithm tasks evaluator is a system that runs untrusted user process on test data and outputs its correctness.

Test data is a pair of input and output file.

User process reads that input file from standard input and writes the solution to standard output. The task it solves is of an algorithmic nature and must complete in some predefined time and consume no more than some predefined amount of memory (RAM). This means it doesnt have access to disk or network or anything other outside that process, note that this also means it mustnt create any new processes.

There is another component called checker used for determining whether the output produced by user process is a correct output for the given test data. This means it uses three sets of data to determine the correctness of a solution:

1. input file from test data
2. output file from test data
3. solution from user process that had been written to the standard output

The need for checker may not be so obvious, but it is very simple, there may be multiple correct solutions for some tasks. Lets consider a task of calculating ? constant with the precision of 10^-5. This means that the correct answer is everything from the interval of [?-10^-5, ?+10^-5], and this check is the checkers responsibility.


We need to have in mind that the user process and checker can not be trusted, which means they need to run in a limited environment, such as a  sandbox that was discussed in a previous post.

This system will be implemented using python and python lxc library which is available under the latest lxc version.

download file now