Various Methods for List Deduplication in Java Using Streams

1. Stream's distinct() Method

The distinct() method, introduced in Java 8, returns a stream consisting of unique elements; it relies on the elements' hashCode() and equals() implementations, so the class to be deduplicated must override these methods.

Stream<T> distinct();

1.1 Deduplication for a List of Strings

Since String already overrides equals() and hashCode() , deduplication works out of the box.

@Test
public void listDistinctByStreamDistinct() {
  List<String> stringList = new ArrayList<String>() {{
    add("A");
    add("A");
    add("B");
    add("B");
    add("C");
  }};
  out.print("去重前：");
  for (String s : stringList) { out.print(s); }
  out.println();
  stringList = stringList.stream().distinct().collect(Collectors.toList());
  out.print("去重后：");
  for (String s : stringList) { out.print(s); }
  out.println();
}

Output:

去重前：AABBC
去重后：ABC

1.2 Deduplication for a List of Entity Objects

Using Lombok's @Data annotation automatically generates equals() and hashCode() for the entity.

/**
 * Define an entity class
 */
@Data
public class Student {
  private String stuNo;
  private String name;
}

@Test
public void listDistinctByStreamDistinct() throws JsonProcessingException {
  ObjectMapper objectMapper = new ObjectMapper();
  List<Student> studentList = getStudentList();
  out.print("去重前：");
  out.println(objectMapper.writeValueAsString(studentList));
  studentList = studentList.stream().distinct().collect(Collectors.toList());
  out.print("去重后：");
  out.println(objectMapper.writeValueAsString(studentList));
}

Output:

去重前：[{"stuNo":"001","name":"Tom"},{"stuNo":"002","name":"Mike"},{"stuNo":"001","name":"Tom"}]
去重后：[{"stuNo":"001","name":"Tom"},{"stuNo":"002","name":"Mike"}]

2. Deduplication Based on a Property of Objects in a List<Object>

2.1 Create a New List Containing Unique Elements

@Test
public void distinctByProperty1() throws JsonProcessingException {
  ObjectMapper objectMapper = new ObjectMapper();
  List<Student> studentList = getStudentList();
  out.print("去重前    :");
  out.println(objectMapper.writeValueAsString(studentList));
  studentList = studentList.stream().distinct().collect(Collectors.toList());
  out.print("distinct去重后:");
  out.println(objectMapper.writeValueAsString(studentList));
  // Using TreeSet with Comparator to keep only one element per name
  studentList = studentList.stream().collect(
    collectingAndThen(
      toCollection(() -> new TreeSet<>(Comparator.comparing(Student::getName))),
      ArrayList::new)
  );
  out.print("根据名字去重后 :");
  out.println(objectMapper.writeValueAsString(studentList));
}

Output:

去重前    :[{"stuNo":"001","name":"Tom"},{"stuNo":"001","name":"Tom"},{"stuNo":"003","name":"Tom"}]
distinct去重后:[{"stuNo":"001","name":"Tom"},{"stuNo":"003","name":"Tom"}]
根据名字去重后 :[{"stuNo":"001","name":"Tom"}]

2.2 Using filter() with a Custom Predicate

A helper method creates a predicate that keeps only the first occurrence of a key extracted from each element.

private static
Predicate
distinctByKey(Function
keyExtractor) {
  Set
seen = ConcurrentHashMap.newKeySet();
  return t -> seen.add(keyExtractor.apply(t));
}

@Test
public void distinctByProperty2() throws JsonProcessingException {
  ObjectMapper objectMapper = new ObjectMapper();
  List<Student> studentList = getStudentList();
  out.print("去重前    :");
  out.println(objectMapper.writeValueAsString(studentList));
  studentList = studentList.stream().distinct().collect(Collectors.toList());
  out.print("distinct去重后:");
  out.println(objectMapper.writeValueAsString(studentList));
  // Filter by name using the custom predicate
  studentList = studentList.stream()
    .filter(distinctByKey(Student::getName))
    .collect(Collectors.toList());
  out.print("根据名字去重后 :");
  out.println(objectMapper.writeValueAsString(studentList));
}

Output:

去重前    :[{"stuNo":"001","name":"Tom"},{"stuNo":"001","name":"Tom"},{"stuNo":"003","name":"Tom"}]
distinct去重后:[{"stuNo":"001","name":"Tom"},{"stuNo":"003","name":"Tom"}]
根据名字去重后 :[{"stuNo":"001","name":"Tom"}]

3. Summary

The article demonstrates several ways to remove duplicates from Java lists, including the built‑in Stream.distinct() , property‑based deduplication using TreeSet with a comparator, and a reusable filter() predicate; it provides complete code snippets for both simple strings and custom entity objects.