Tool Competition

This year as well we are pleased to announce the tenth edition of the testing tool competition. The competition has the goal to experiment with testing tools for a diversified set of traditional and emerging systems and domains.

Java tool competition: As for recent years, we invite researchers to participate in the competition with their unit test generation tool for Java. Tools will be evaluated against a benchmark with respect to code coverage and mutation score.

Java test example

Cyber-physical systems (CPS) testing competition: In addition to the traditional Java tool competition, we also organize a CPS testing competition on self-driving cars simulation environments. Specifically, in collaboration with the BeamNG research team, this competition focuses on the generation of scenarios using BeamNG self-driving cars simulator.

Beam-NG Visual 1
Beam-NG Visual 2

For future editions of the SBST Tool competition we foresee the addition of a further "Python testing tool competition" where:

  • an infrastructure will be provided to generate the tools
  • it is conducted in an evaluation setting similar to the one performed for the traditional Java testing tool competition

How to participate in the two competitions?

To participate:

Java Infrastructure:

The infrastructure concerning the Java tool competition is available on GitHub and can also be tried using Docker: It is important to note that some changes are expected to the infrastructure during the end of 2021 and during the beginning of 2022.

CPS Infrastructure:

The infrastructure concerning the CPS tool competition can be found here:

Important Dates

  • Tool submission: Friday January 21
  • Benchmark results communicated to authors: Friday February 25
  • Submission of camera-ready paper: Sunday March 18