Below are two WorldCat records for the same book.

The first shows the pre-publication record, with the title as provided by the publisher, a projected date, and little else (Figure 1). The second is the record created after the book’s publication (Figure 2). Now, we see a fuller description, with the ISBN as a key match point that the two records share.

Two records. One book. With little in the metadata saying so.

 

This happens when a record is created before the resource is published. At that point, all a cataloger has to work with is what the publisher has provided: a title, a publisher, a projected publication date, and an ISBN. Enough to order from, not much more. When the book is finally published, a cataloger working from the resource itself creates a fuller, more descriptive record.

Most catalogers have run into duplicates like these. Sparse records like the pre-publication record shown here are among the data quality issues we hear about most often from libraries, and I understand why. They add noise to WorldCat and take real time to untangle. When an existing record is incomplete or hard to find, creating a new one is often quicker than improving the old one, which adds yet another duplicate.

These are some of the hardest duplicates to identify at scale.

Where library science meets data science: a hybrid model

DDR (Duplicate Detection and Resolution), our traditional de-duplication tool, looks for evidence that two records belong together. It works best when records share enough consistent metadata to make that connection clear. In the case of pre-publication records, that overlap may be limited or missing by the time a fuller record is created. To DDR, the records can look unrelated.

That’s where machine learning comes in.

Machine learning gives us more flexibility in evaluating the relationship between records. Rather than requiring a certain amount of metadata to line up, the model can consider different signals together to determine whether two records are likely to describe the same resource. The ISBN, when present, is one particularly useful signal because it can remain consistent even as other parts of a record change. It can help the model connect a sparse pre-publication record with the fuller record created after publication.

Our machine learning model is also a more computationally efficient way to identify these duplicates than traditional de-duplication methods. With machine learning, we can conduct the analysis up front, identify the duplicates, and then process them for merging. What previously could take months can now be processed in a matter of days.

But here’s the thing: we’re not just creating a machine learning model and letting it loose on WorldCat. A vital step in our process has been keeping human expertise in the loop to continually analyze, train, refine, and adjust the model that’s running. When a cataloger tells us we’ve merged something incorrectly, we can roll back the merge, investigate what happened, and use what we learn to improve the model. This approach has helped us evolve the model with community input over time.

The result is a hybrid: machine learning, but also rules, safeguards, data quality expertise, and the collective judgment of OCLC and the library community. And that’s important because we’re trying to solve a problem that’s impractical to address manually. We can certainly merge these records one at a time. But when you have WorldCat’s volume of data, that’s not a sustainable solution to the scale of the problem.

The exciting thing is that we can see that this hybrid approach is working. Catalogers can report duplicate records to us when they find them, and the number of those requests has been declining over time (Figure 3).

 

At the same time, the number of duplicates we can merge through these processes has continued to climb. In fact, the number of duplicates merged since we launched our machine learning model in 2023 is over 19 million, with last year alone reaching an all-time high of 8.8 million. For me, that’s a good indication that we’re making a difference. We’re not just finding a way to merge more records. We’re addressing one of the persistent data quality challenges our users encounter as we continue to improve the quality of the data we all share.

WorldCat stewardship never stands still

WorldCat stewardship is an ongoing process of listening to libraries, identifying persistent problems, and evolving both human expertise and technology to solve them. The work we’re doing to identify and merge pre-publication duplicates in book format is one example of how we’re continually looking for ways to improve WorldCat data quality, but it is certainly not the only one.

Expanding de-duplication across languages, scripts, and formats. We partnered with libraries to expand our de-duplication work to non-Latin scripts, including Japanese, Chinese, and Korean, as well as other formats, such as music (scores and sound recordings), visual materials, maps, and continuing resources. To build confidence that we were handling these records safely and accurately, we worked with experts from the library community to review and validate the model.

Linked data and the evolution of WorldCat. Our data quality team has also been focused on enriching WorldCat data as linked data, including adding URIs for WorldCat Entities and for the RDA registry, and populating additional codes and fields that support current and future linked data applications.

Continuing to evolve the model. And we’re still moving de-duplication efforts forward. We have our sights set on another group of pre-publication duplicates for continuing resources and serials that still escape detection because they don’t meet the model’s minimum threshold for identification, even though human catalogers and data experts recognize that they belong together. Addressing these more elusive duplicates will require another round of intensive review by our data quality and data science teams, together with catalogers in the community, to understand where the model can improve.

And that’s the heart of WorldCat stewardship. The work doesn’t stop when we’ve solved one problem. As the needs of libraries change, the data evolves, and new technologies become available, we have to keep asking where we can do better for libraries.

WorldCat is a shared resource, and we take seriously our responsibility to the libraries that contribute to and rely on it. That means continuing to listen, continuing to learn, and continuing to innovate, with the same care and human judgment that have always been at the heart of this work.