Answers.Body: <p>I doubt that there is such a library, but, depending on what you expect from your students there are things you might be able to build.</p>
  <p>First, it is possible to compile and run one Java program from another. Here is a page that describes the how to do it: <a href="https://www.journaldev.com/937/compile-run-java-program-another-java-program" rel="nofollow noreferrer">https://www.journaldev.com/937/compile-run-java-program-another-java-program</a></p>
  <p>Once you can do that, you can, perhaps, read a set of regular expression patterns from a file and match them against what the student program produces using the Pattern class.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-31T15:37:37.780Z
Answers.LastActivityDate: 2020-07-31T15:37:37.780Z
Answers.UserDisplayName: Buffy
Answers.Body: <p>This answer is based on the using features and commands available in a Linux environment. The basic idea is to use file redirection to feed input to the program under test (<code>PUT</code>) and collect output, both <code>stdout</code> and <code>stderr</code>. This describes a small system that we have used at Colorado State University that I wrote. If anyone is interested, I can supply both the scripts and some documentation. Using them would require modifications on the part of the tester to accommodate location of submissions, etc that are particular to CSU.</p>
  <p>The basic idea is to execute a single test case, collect its output and assess the result. The test framework is language agnostic as it is used to test programs written in a variety of languages. It is written in <code>bash</code> and takes advantage of many Linux utilities. A test case consists of a single line of text containing a testName, number of points for the test, and the actual Linux command necessary to run the test. The &quot;language&quot; of the test case has some simple macro capability to make it easy to write multiple test cases easily. For example, the macro <code>$inputFile</code> is expanded to <code>input/$testName</code>, and likewise for <code>$output</code>. The tester creates multiple input files named the same as the testName in the directory <code>input</code>.The required output redirection is done by the framework, so that the actual command is often as simple as <code>java SomeProgram &lt; $inputFile</code> or <code>./myprog &lt; $inputFile</code>.</p>
  <p>Assessment is done using <code>diff</code>, comparing the output of the <code>PUT</code> to that of a master solution. <code>diff</code> has lots of options to ignore case, white space, etc to loosen up the actual comparison. The master solution's output is collected by simply running the framework and renaming its output as the master. The <code>diff</code> is presented in a colored side-by-side format as that is easier for the students to understand than a standard <code>diff</code> output. A student gets no/full points depending on the diff. A typical program has many test cases. Optional post-processing to assess points is possible as is the ability to add in material generated manually.</p>
  <p>In a complete test, the framework takes test cases from a text file, processed them one by one, and collects all output in a single file. At the completion, the file is post-processed to extract the individual test case scores and prepend them to the raw output. The student sees a total score, individual test case scores and finally the results of each test case.</p>
  <p>To test the entire class, the framework processes a list of the student ID's from a text file. Each ID corresponds to a directory containing the submission for that student. The submission consists of a <strong>single</strong> file, though that file is frequently a <code>tar/jar/zip</code>.</p>
  <p>The actual build of the <code>PUT</code> is driven by a <code>Makefile</code>. This may be supplied by the person running the tests or many be a required part of the submission. The framework simple performs a <code>make</code> and runs the resulting code. Part of the specification of the assignment is the name/class of the executable. For testing scripts, the <code>Makefile</code> may simply insure that the <code>execute</code> permission is set.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-31T20:04:35.380Z
Answers.LastActivityDate: 2020-07-31T22:21:47.910Z
Answers.LastEditDate: 2020-07-31T22:21:47.910Z
Answers.UserDisplayName: Fritz Sieker
Answers.Body: <p>There is <code>expect</code>. It is designed to test interactive command-line systems. For non-interactive programs it is easier.</p>
  <p><code>expect</code> is not java, but as pointed out in the comments. The language is irrelevant, as the testing is done from the outside.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-31T16:31:54.567Z
Answers.LastActivityDate: 2020-07-31T16:31:54.567Z
Answers.UserDisplayName: ctrl-alt-delor
Body: <p>JUnit seems to work well for unit tests, but I'm not interested in unit tests, I want to test and entire command line application as a black box.</p>
  <p>I have created a text-based adventure game project for my students and I am trying to build an auto-grader. I do not have any web infrastructure and I would prefer to just creating a local application to test students' submissions. I also would like it to be easy enough to students to use to check their work (JUnit doesn't seem to be easily usable outside of an IDE).</p>
  <p>I am looking to create a program that will attempt to compile a students code and run the game through all the possible paths. I could just build something like this from scratch, but I feel like there should already be some kind of framework to make handling test cases and printing results easy. I haven't found anything outside of web-based services, which isn't what I'm looking for.</p>
  <p>Is there any library to help with this, or am I better off just making my own grader from scratch?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-31T02:38:01.777Z
Comments.Text: I'm virtually positive that there is no such Java library.  Nevertheless, I upvoted since it could certainly be useful to me if it existed.  However, how would such an engine know that it was successful?  If a student uses two spaces between sentences instead of one, wouldn't that make your tests fail?  JUnit could work, or you could just create a macro that types, waits a bit, types, waits a bit, etc, etc, and sit and watch the game play out.
Comments.UserDisplayName: Ben I.
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-31T12:07:18.297Z
Comments.Text: Testing the "entire command line application as a black box" wouldn't require a library, or any specific language. What's going to be needed is a custom built bot, which could be written in any language, including simple shell script. Use of a "library" implies that you would be injecting into the application, not a blcak box approach.
Comments.UserDisplayName: Gypsy Spellweaver
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-31T14:50:21.817Z
Comments.Text: @GypsySpellweaver I'm looking for something that standardizes the process. There would be a compile step where if the student's code doesn't compile that gets logged. And then you can define test cases where certain inputs are applied and the case supplies strings to compare output against. Then you can log those cases in a standard way. Obviously I can just make this myself, but using an existing system seems like it would make it easier to record results and communicate them to students.
Comments.UserDisplayName: TechnoSam
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-31T16:23:24.130Z
Comments.Text: @BenI. Two spaces? `/[[:blank:]]+/` — does java have regexes?
Comments.UserDisplayName: ctrl-alt-delor
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-31T16:26:53.263Z
Comments.Text: You seem to have compleatly missed what @GypsySpellweaver was saying. They were saying that language dose not matter.
Comments.UserDisplayName: ctrl-alt-delor
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-31T16:29:02.283Z
Comments.Text: Is the program interactive?, or can it be run with command-line-arguments and a fixed file for standard in?
Comments.UserDisplayName: ctrl-alt-delor
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-31T19:12:18.550Z
Comments.Text: @ctrl-alt-delor Of course there are regexes, but I was aiming for a trivial example.  Perhaps I was unclear. I was saying that unless the student output is specified, roughly character by character, such an engine would be of no use.  If the student substitutes "You enter the room the the north" with "You go north, entering a room", no regex will save you. Game design with students had to be at least a little open ended, otherwise the assignment wouldn't be fun in the first place.
Comments.UserDisplayName: Ben I.
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-01T01:01:06.647Z
Comments.Text: The key component of the described testing was to treat the "entire command line application as a black box". The system (Windows, MacOS, or Linux) should have scripting capabilities which can compile (with error trapping) the student submission. Outside the pass/fail for compilation, the "black box" concept completely precludes any interactions between the student's application and the testing system except through the application's API. The description of the project suggests that the "API" in this case is the CLI terminal. Testing through such an API is going to require you to make a "bot".
Comments.UserDisplayName: Gypsy Spellweaver
Tags: <java><grading><auto-graders>
Title: Is there a Java library for testing command line applications?
ThreadText: # Is there a Java library for testing command line applications?
  
  (Asked by: username_0 on 2020-07-31)
  
  JUnit seems to work well for unit tests, but I'm not interested in unit tests, I want to test and entire command line application as a black box.
  
  I have created a text-based adventure game project for my students and I am trying to build an auto-grader. I do not have any web infrastructure and I would prefer to just creating a local application to test students' submissions. I also would like it to be easy enough to students to use to check their work (JUnit doesn't seem to be easily usable outside of an IDE).
  
  I am looking to create a program that will attempt to compile a students code and run the game through all the possible paths. I could just build something like this from scratch, but I feel like there should already be some kind of framework to make handling test cases and printing results easy. I haven't found anything outside of web-based services, which isn't what I'm looking for.
  
  Is there any library to help with this, or am I better off just making my own grader from scratch?
  
  **Tags:** java, grading, auto-graders
  
  > **Comments:**
  > **username_1**: I'm virtually positive that there is no such Java library.  Nevertheless, I upvoted since it could certainly be useful to me if it existed.  However, how would such an engine know that it was successful?  If a student uses two spaces between sentences instead of one, wouldn't that make your tests fail?  JUnit could work, or you could just create a macro that types, waits a bit, types, waits a bit, etc, etc, and sit and watch the game play out.  
  > **username_2**: Testing the "entire command line application as a black box" wouldn't require a library, or any specific language. What's going to be needed is a custom built bot, which could be written in any language, including simple shell script. Use of a "library" implies that you would be injecting into the application, not a blcak box approach.  
  > **username_0**: @username_2 I'm looking for something that standardizes the process. There would be a compile step where if the student's code doesn't compile that gets logged. And then you can define test cases where certain inputs are applied and the case supplies strings to compare output against. Then you can log those cases in a standard way. Obviously I can just make this myself, but using an existing system seems like it would make it easier to record results and communicate them to students.  
  > **username_3**: @BenI. Two spaces? `/[[:blank:]]+/` — does java have regexes?  
  > **username_3**: You seem to have compleatly missed what @username_2 was saying. They were saying that language dose not matter.  
  > **username_3**: Is the program interactive?, or can it be run with command-line-arguments and a fixed file for standard in?  
  > **username_1**: @username_3 Of course there are regexes, but I was aiming for a trivial example.  Perhaps I was unclear. I was saying that unless the student output is specified, roughly character by character, such an engine would be of no use.  If the student substitutes "You enter the room the the north" with "You go north, entering a room", no regex will save you. Game design with students had to be at least a little open ended, otherwise the assignment wouldn't be fun in the first place.  
  > **username_2**: The key component of the described testing was to treat the "entire command line application as a black box". The system (Windows, MacOS, or Linux) should have scripting capabilities which can compile (with error trapping) the student submission. Outside the pass/fail for compilation, the "black box" concept completely precludes any interactions between the student's application and the testing system except through the application's API. The description of the project suggests that the "API" in this case is the CLI terminal. Testing through such an API is going to require you to make a "bot".  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2020-07-31. Score: 3
  
  There is `expect`. It is designed to test interactive command-line systems. For non-interactive programs it is easier.
  
  `expect` is not java, but as pointed out in the comments. The language is irrelevant, as the testing is done from the outside.
  
  
  ---
  
  ### Answer by username_5 on 2020-07-31. Score: 1
  
  This answer is based on the using features and commands available in a Linux environment. The basic idea is to use file redirection to feed input to the program under test (`PUT`) and collect output, both `stdout` and `stderr`. This describes a small system that we have used at Colorado State University that I wrote. If anyone is interested, I can supply both the scripts and some documentation. Using them would require modifications on the part of the tester to accommodate location of submissions, etc that are particular to CSU.
  
  The basic idea is to execute a single test case, collect its output and assess the result. The test framework is language agnostic as it is used to test programs written in a variety of languages. It is written in `bash` and takes advantage of many Linux utilities. A test case consists of a single line of text containing a testName, number of points for the test, and the actual Linux command necessary to run the test. The "language" of the test case has some simple macro capability to make it easy to write multiple test cases easily. For example, the macro `$inputFile` is expanded to `input/$testName`, and likewise for `$output`. The tester creates multiple input files named the same as the testName in the directory `input`.The required output redirection is done by the framework, so that the actual command is often as simple as `java SomeProgram < $inputFile` or `./myprog < $inputFile`.
  
  Assessment is done using `diff`, comparing the output of the `PUT` to that of a master solution. `diff` has lots of options to ignore case, white space, etc to loosen up the actual comparison. The master solution's output is collected by simply running the framework and renaming its output as the master. The `diff` is presented in a colored side-by-side format as that is easier for the students to understand than a standard `diff` output. A student gets no/full points depending on the diff. A typical program has many test cases. Optional post-processing to assess points is possible as is the ability to add in material generated manually.
  
  In a complete test, the framework takes test cases from a text file, processed them one by one, and collects all output in a single file. At the completion, the file is post-processed to extract the individual test case scores and prepend them to the raw output. The student sees a total score, individual test case scores and finally the results of each test case.
  
  To test the entire class, the framework processes a list of the student ID's from a text file. Each ID corresponds to a directory containing the submission for that student. The submission consists of a **single** file, though that file is frequently a `tar/jar/zip`.
  
  The actual build of the `PUT` is driven by a `Makefile`. This may be supplied by the person running the tests or many be a required part of the submission. The framework simple performs a `make` and runs the resulting code. Part of the specification of the assignment is the name/class of the executable. For testing scripts, the `Makefile` may simply insure that the `execute` permission is set.
  
  
  ---
  
  ### Answer by username_4 on 2020-07-31. Score: 0
  
  I doubt that there is such a library, but, depending on what you expect from your students there are things you might be able to build.
  
  First, it is possible to compile and run one Java program from another. Here is a page that describes the how to do it: <https://www.journaldev.com/937/compile-run-java-program-another-java-program>
  
  Once you can do that, you can, perhaps, read a set of regular expression patterns from a file and match them against what the student program produces using the Pattern class.
  
  
  ---
  
  