In a previous blog post we looked into how to get started with Diffblue Cover. In this post, we will focus on utilizing Diffblue Cover to generate tests for a pre-existing project that contains untested code.

I find the best time to write tests to be as you write the code but, sometimes we discover untested code in our projects. Fully understanding someone else’s code that does not have tests written for it can be difficult and quite time-consuming. Let’s try out Diffblue Cover to see if it can help us.

Controller Test

Let’s first have a look at a simple REST controller that returns a list of users as well as the currentUser. For this example, I will not run Diffblue Cover on the entire controller but on the getUsers method seen below.

    @GetMapping
    public UserList getUsers() {
        try {
            User currentUser = getCurrentUserQuery.getCurrentUser();
            List<User> members = listUsersQuery.listUsers();

            return UserList.builder()
                .currentUser(currentUser)
                .users(members)
                .build();
        } catch (Exception ex) {
            throw ExceptionTranslator.translate("UserController.getUsers", ex);
        }
    }

The below tests are generated:

@ContextConfiguration(classes = {UserController.class})
@RunWith(SpringJUnit4ClassRunner.class)
public class UserControllerTest { (1)
    @MockBean
    private GetCurrentUserQuery getCurrentUserQuery;

    @MockBean
    private ListUsersQuery listUsersQuery;

    @Autowired
    private UserController userController;

    /**
     * Method under test: {@link UserController#getUsers()}
     */
    @Test
    public void testGetUsers() throws Exception {
        when(getCurrentUserQuery.getCurrentUser())
            .thenReturn(new User("Name", "Team Name", "jane.doe@example.org", "Display Name", "Member Name", "Avatar"));
        when(listUsersQuery.listUsers()).thenReturn(new ArrayList<>());
        MockHttpServletRequestBuilder requestBuilder = MockMvcRequestBuilders.get("/api/users");
        MockMvcBuilders.standaloneSetup(userController) (2)
            .build()
            .perform(requestBuilder)
            .andExpect(MockMvcResultMatchers.status().isOk())
            .andExpect(MockMvcResultMatchers.content().contentType("application/json"))
            .andExpect(MockMvcResultMatchers.content()
                .string(
                    "{\"currentUser\":{\"name\":\"Name\",\"teamName\":\"Team Name\",\"email\":\"jane.doe@example.org\",\"displayName\":\"Display"
                        + " Name\",\"memberName\":\"Member Name\",\"avatar\":\"Avatar\"},\"users\":[]}"));
    }

    /**
     * Method under test: {@link UserController#getUsers()}
     */
    @Test
    public void testGetUsers2() throws Exception {
        when(getCurrentUserQuery.getCurrentUser())
            .thenReturn(new User("Name", "Team Name", "jane.doe@example.org", "Display Name", "Member Name", "Avatar"));

        ArrayList<User> userList = new ArrayList<>();
        userList.add(new User("?", "?", "jane.doe@example.org", "?", "?", "?"));
        when(listUsersQuery.listUsers()).thenReturn(userList);
        MockHttpServletRequestBuilder requestBuilder = MockMvcRequestBuilders.get("/api/users");
        MockMvcBuilders.standaloneSetup(userController)
            .build()
            .perform(requestBuilder)
            .andExpect(MockMvcResultMatchers.status().isOk())
            .andExpect(MockMvcResultMatchers.content().contentType("application/json"))
            .andExpect(MockMvcResultMatchers.content()
                .string(
                    "{\"currentUser\":{\"name\":\"Name\",\"teamName\":\"Team Name\",\"email\":\"jane.doe@example.org\",\"displayName\":\"Display"
                        + " Name\",\"memberName\":\"Member Name\",\"avatar\":\"Avatar\"},\"users\":[{\"name\":\"?\",\"teamName\":\"?\",\"email\":\"jane"
                        + ".doe@example.org\",\"displayName\":\"?\",\"memberName\":\"?\",\"avatar\":\"?\"}]}")); (3)
    }

    /**
     * Method under test: {@link UserController#getUsers()}
     */
    @Test
    public void testGetUsers3() throws Exception {
        when(getCurrentUserQuery.getCurrentUser())
            .thenReturn(new User("Name", "Team Name", "jane.doe@example.org", "Display Name", "Member Name", "Avatar"));

        ArrayList<User> userList = new ArrayList<>();
        userList.add(new User("?", "?", "jane.doe@example.org", "?", "?", "?"));
        userList.add(new User("?", "?", "jane.doe@example.org", "?", "?", "?")); (4)
        when(listUsersQuery.listUsers()).thenReturn(userList);
        MockHttpServletRequestBuilder requestBuilder = MockMvcRequestBuilders.get("/api/users");
        MockMvcBuilders.standaloneSetup(userController)
            .build()
            .perform(requestBuilder)
            .andExpect(MockMvcResultMatchers.status().isOk())
            .andExpect(MockMvcResultMatchers.content().contentType("application/json"))
            .andExpect(MockMvcResultMatchers.content()
                .string(
                    "{\"currentUser\":{\"name\":\"Name\",\"teamName\":\"Team Name\",\"email\":\"jane.doe@example.org\",\"displayName\":\"Display"
                        + " Name\",\"memberName\":\"Member Name\",\"avatar\":\"Avatar\"},\"users\":[{\"name\":\"?\",\"teamName\":\"?\",\"email\":\"jane"
                        + ".doe@example.org\",\"displayName\":\"?\",\"memberName\":\"?\",\"avatar\":\"?\"},{\"name\":\"?\",\"teamName\":\"?\",\"email"
                        + "\":\"jane.doe@example.org\",\"displayName\":\"?\",\"memberName\":\"?\",\"avatar\":\"?\"}]}"));
    }

    /**
     * Method under test: {@link UserController#getUsers()}
     */
    @Test
    public void testGetUsers4() throws Exception { (5)
        when(getCurrentUserQuery.getCurrentUser())
            .thenReturn(new User("Name", "Team Name", "jane.doe@example.org", "Display Name", "Member Name", "Avatar"));
        when(listUsersQuery.listUsers()).thenReturn(new ArrayList<>());
        SecurityMockMvcRequestBuilders.FormLoginRequestBuilder requestBuilder = SecurityMockMvcRequestBuilders
            .formLogin();
        ResultActions actualPerformResult = MockMvcBuilders.standaloneSetup(userController)
            .build()
            .perform(requestBuilder);
        actualPerformResult.andExpect(MockMvcResultMatchers.status().isNotFound());
    }
}
1 After adding a all arguments constructor to the User class Diffblue Cover was able to generate 4 tests for us. It was unable to create a test for an exception being thrown and therefor the generated tests have 75% line coverage.
2 I find the tests to be very verbose and that a lot of code is repeated.
3 A string based expectation can be good but can also be difficult to reason about to validate if the tests are behaving as expected.
4 Mock data generation is not different per request which could result in false positives.
5 It even wrote a test for a formLogin expecting a not found response.

I have refactored one of the tests in a way that I feel is easier to reason about and make changes in the future.

    private MockMvc mockMvc;

    @Before
    public void setup() {
        mockMvc = MockMvcBuilders.standaloneSetup(userController).build();
    }

    @Test
    public void testGetUsersWithCurrentUserAndTwoUsersInList() throws Exception {
        when(getCurrentUserQuery.getCurrentUser())
            .thenReturn(new User("Name", "Team Name", "jane.doe1@example.org", "Display Name", "Member Name", "Avatar"));
        when(listUsersQuery.listUsers()).thenReturn(new ArrayList<>() {{
            add(new User("?", "?", "jane.doe2@example.org", "?", "?", "?"));
            add(new User("?", "?", "jane.doe3@example.org", "?", "?", "?"));
        }});
        mockMvc.perform(get("/api/users"))
            .andExpect(status().isOk())
            .andExpect(content().contentType("application/json"))
            .andExpect(jsonPath("$.currentUser.email").value("jane.doe1@example.org"))
            ...
            .andExpect(jsonPath("$.users[0].email").value("jane.doe2@example.org"))
            .andExpect(jsonPath("$.users[1].email").value("jane.doe3@example.org"));
    }

REST Service Test

Let’s run Diffblue Cover on a service that integrates with a REST API using RestTemplate. The method under test expects a response object with a List of emails inside the values field.

    public List<EmailResult> getEmailAddresses() {
        EmailsResult response = restTemplate.getForObject(EMAIL_DETAILS_URL, EmailsResult.class);
        if (response == null) {
            return Collections.emptyList();
        }
        return response.values;
    }

Here is the tests that Diffblue Cover generated:

@ContextConfiguration(classes = {RestClient.class, String.class})
@RunWith(SpringJUnit4ClassRunner.class)
public class RestClientTest {
    @Autowired
    private RestClient restClient;

    @MockBean
    private RestTemplate restTemplate;

    /**
     * Method under test: {@link RestClient#getEmailAddresses()}
     */
    @Test
    public void testGetEmailAddresses() throws RestClientException { (1)
        EmailsResult emailsResult = new EmailsResult();
        ArrayList<EmailResult> emailResultList = new ArrayList<>();
        emailsResult.values = emailResultList;
        when(restTemplate.getForObject(Mockito.<String>any(), Mockito.<Class<EmailsResult>>any(), (Object[]) any()))
            .thenReturn(emailsResult);
        List<EmailResult> actualEmailAddresses = restClient.getEmailAddresses();
        assertSame(emailResultList, actualEmailAddresses);
        assertTrue(actualEmailAddresses.isEmpty());
        verify(restTemplate).getForObject(Mockito.<String>any(), Mockito.<Class<EmailsResult>>any(), (Object[]) any());
    }

    /**
     * Method under test: {@link RestClient#getEmailAddresses()}
     */
    @Test
    @Ignore("TODO: Complete this test") (2)
    public void testGetEmailAddresses2() throws RestClientException {
        // TODO: Complete this test.
        //   Reason: R013 No inputs found that don't throw a trivial exception.
        //   Diffblue Cover tried to run the arrange/act section, but the method under
        //   test threw
        //   java.lang.NullPointerException: https://api.example.org/2.0/user/emails
        //       at com.jdriven.adapter.out.RestClient.getEmailAddresses(RestClient.java:123)
        //   See https://diff.blue/R013 to resolve this issue.

        when(restTemplate.getForObject(Mockito.<String>any(), Mockito.<Class<EmailsResult>>any(), (Object[]) any()))
            .thenThrow(new NullPointerException("https://api.example.org/2.0/user/emails"));
        restClient.getEmailAddresses();
    }
}
1 Diffblue Cover is only able to write a test with an empty list response but not a list containing emails. It is also unable to write a test for a null response which results in 80% line coverage.
2 It creates a @Ignore test with a TODO comment asking us to complete the test ourselves. It seems to struggle creating some of the mock data.

Streams Test

For our final test let’s have a look at how Diffblue Cover handles some streaming.

    public List<Member> getUsersWithListing() {
        return restClient.getAllGroupsForUser() (1)
            .stream()
            .filter(this::isAuthorized) (2)
            .map(restClient::getAllListings) (3)
            .flatMap(Collection::stream)
            .filter(this::isActive) (4)
            .collect(Collectors.toList());
    }

    private boolean isAuthorized(UserGroup userGroup) {
        return userConfigurationProperties.getGroups().contains(userGroup.getUsername());
    }

    private boolean isActive(Member member) {
        String listingName = userConfigurationProperties
            .getNamePatternExpanded(member.getGroupName(), NameUtil.generateUniqueName(member.getDisplayName()));
        try {
            return listingService.getAllListings().stream()
                .anyMatch(listing -> Objects.equals(listing.getFullName(), listingName));
        } catch (Exception ex) {
            return false;
        }
    }
1 Fetching the groups that the member belongs to from the RestClient.
2 A filter for if the returned groups are in a configured list of authorized groups.
3 Fetching the listings for this member using the RestClient using the group.
4 Checks if the member is active or not by checking if there is an active listing under their name.

Here are the tests that it generated:

@ContextConfiguration(classes = {UserAdapter.class, UserConfigurationProperties.class})
@RunWith(SpringJUnit4ClassRunner.class)
public class UserAdapterTest {
    @MockBean
    private ListingService listingService;

    @MockBean
    private RestClient restClient;

    @Autowired
    private UserAdapter userAdapter;

    @Autowired
    private UserConfigurationProperties userConfigurationProperties;

    /**
     * Method under test: {@link UserAdapter#getUsersWithListing()}
     */
    @Test
    public void testGetUsersWithListing() { (1)
        when(restClient.getAllGroupsForUser()).thenReturn(new ArrayList<>());
        assertTrue(userAdapter.getUsersWithListing().isEmpty());
        verify(restClient).getAllGroupsForUser();
    }

    /**
     * Method under test: {@link UserAdapter#getUsersWithListing()}
     */
    @Test
    @Ignore("TODO: Complete this test") (2)
    public void testGetUsersWithListing2() {
        // TODO: Complete this test.
        //   Reason: R013 No inputs found that don't throw a trivial exception.
        //   Diffblue Cover tried to run the arrange/act section, but the method under
        //   test threw
        //   java.lang.NullPointerException: Cannot invoke "java.util.Collection.contains(Object)" because "that" is null
        //       at com.jdriven.adapter.out.UserAdapter.isAuthorized(UserAdapter.java:35)
        //       at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:178)
        //       at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
        //       at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
        //       at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
        //       at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
        //       at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        //       at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
        //       at com.jdriven.adapter.out.UserAdapter.getUsersWithListing(UserAdapter.java:31)
        //   See https://diff.blue/R013 to resolve this issue.

        UserGroup userGroup = mock(UserGroup.class);
        when(userGroup.getUsername()).thenReturn("janedoe"); (3)

        ArrayList<UserGroup> userGroupList = new ArrayList<>();
        userGroupList.add(userGroup);
        when(restClient.getAllGroupsForUser()).thenReturn(userGroupList);
        userAdapter.getUsersWithListing();
    }
}
1 Diffblue Cover is again able to write a basic test which we can then use as the bases for writing more complicated tests.
2 It is unable to generate mock data which means that the isAuthorized and isActive methods never get tested. This results in 53% line coverage.
3 It is unable to write the tests that require the creation of complicated mock data.

Conclusion

After testing Diffblue Cover on the above examples, the following observations can be made:

  • Diffblue Cover demonstrates an understanding of the code being tested and can create appropriate mocks. It recognizes that testing controllers may require a different approach compared to other classes.

  • Diffblue Cover struggles to generate complex mock data.

  • The generated code is verbose and could benefit from some clean-up.

  • It cannot generate mock data for classes without constructors, such as builders. Adding an all-argument constructor is necessary in such cases.

  • Diffblue Cover supports specific versions of Spring Core, Java, and JUnit. If you are using a newer version, like Spring Boot 3+, Diffblue Cover may be unable to generate tests.

  • It appears to have difficulties when the methods under test have no parameters.

  • Diffblue Cover has a somewhat opinionated approach to test generation and may not consider the style of similar tests in the same project.

  • Running Diffblue Cover on an entire project or even on packages is not possible as it has a limit on the number of tests it can generate at once.

I hope to see improvements in Diffblue Cover in the future. Currently, it can be useful for generating boilerplate code and in some cases can set up initial tests.

shadow-left