Adding more Unit Tests

[Iaron da Costa Araújo]

I've been trying to automatically generate more Unit Tests using Evosuite, and while I was able to generate them, I wasn't able to execute them using Evosuite inside the project (Evosuite was primarily made to work on Maven, so I am still trying to figure out how to make it work properly on a non-maven project). I managed to transform the generated tests into normal JUnit tests, though, so we could still use them to improve the game's test coverage.

On this week and the next two, I intend to generate a LOT of tests with Evosuite and also write some more detailed tests manually. Since I don't have much time until Gsoc application period closes, and I have to make my first PR on this week or the next one, I would like to know which one would you like me to do first? Also, is there any packages you want me to test first?

I am uploading a converted test generated by Evosuite so you all can see how are the tests generated by this tool.

Thank you all for your time! : )

 

[Iaron da Costa Araújo]

I was able to generate more modified Evosuite tests. In total, there are 346 tests in 18 JUnit classes. Of these 346 tests, 26 failed. I will check these tests that failed to see if they make sense. If they do, I will try to fix the faults.

 

[Cervator]

Generated unit tests? How does that actually work?

I haven't had a chance to check out what Evosuite actually is or does, but would appreciate a quick summary sometime

 

[Iaron da Costa Araújo]

"To find defects in software, one needs test cases that execute the software systematically, and oracles that assess the correctness of the observed behavior when running these test cases. EvoSuite is a tool that automatically generates test cases with assertions for classes written in Java code. To achieve this, EvoSuite applies a novel hybrid approach that generates and optimizes whole test suites towards satisfying a coverage criterion. For the produced test suites, EvoSuite suggests possible oracles by adding small and effective sets of assertions that concisely summarize the current behavior; these assertions allow the developer to detect deviations from expected behavior, and to capture the current behavior in order to protect against future defects breaking this behaviour."

So, basically, EvoSuite can generate a lot of unit tests automatically! We can run it on cmd but right now I am using the IntelliJ plugin they made. With this plugin, I can right click a class or package and run evosuite to generate tests for them!

You can also configure some details of its execution.

The problem I've been facing is that the plugin can generate tests but cannot run them. The tests generated by Evosuite uses some things in Evosuite's standalone library, and they also have a dependency to Maven to some degree. While I know it is possible to make it run without Maven, I decided to modify the tests so they can work only using JUnit for now. While this has some disadvantages, such as being unable to run a few specific tests, I thought it would be more viable to me since it takes less time and I am on the end of my university semester. I intend to work on this matter in about two weeks, though, so we can quickly generate and execute tests for almost every class on Terasology, greatly improving its test coverage! (yay \o/)

I hope this answers your questions. If you still have doubts, let me know and I will do my best to answer them!

EvoSuite link: http://www.evosuite.org/

 

[Iaron da Costa Araújo]

I've looked at the tests and noticed that a few tests didn't make sense. Apparently, EvoSuite can't handle methods that use Random's methods. I made a pull request with the tests that made sense. It is also important to know that Evosuite's tests don't substitute manually written tests since automatically generated tests sometimes can't catch some more complex bugs. I also intend to write some code manually to see if some recently fixed bugs could have been caught with unit tests!

 

[Skaldarnar]

Hi @Iaron da Costa Araújo - thanks for the PR. I just replied over on GitHub and I hope we will get a good discussion going! Don't worry if we are not going to merge this directly - in this case this is somewhat a good sign that you hit a sweet spot

 

[Iaron da Costa Araújo]

Thank you for your reply, @Skaldarnar ! And having worked for a bit more than a year as an undergraduate researcher, I do love these discussions!

Also, I have a question regarding GSoC. On Terasology's GSoC it says we should "Submit at least one pull request to one of our GitHub repos before you submit an application". Does a submission count even if it's not accepted?

 

[Skaldarnar]

I would say "it depends". You can definitely start working on a proposal and submit an early draft. Staying active in the discussion, and maybe coming up with another PR, will proof that you are motivated. No need for being too strict with ourselves here

 

[Iaron da Costa Araújo]

Thank you very much! I am already planning on another PR, but I think I should also begin working on a proposal. When I have its first draft, should I already apply for GSoC or should I show it to you first?

 

[Cervator]

Thank you very much! I am already planning on another PR, but I think I should also begin working on a proposal. When I have its first draft, should I already apply for GSoC or should I show it to you first?

Submit it as a draft and also feel free to poke us directly for feedback when you're around

Drafts in a GDoc are really nice to review and having the entry in the GSOC system allows us to coordinate better between mentors