Library information provider OCLC, US, has completed the development work to convert the underlying structure for its WorldCat database to Apache HBase, a distributed platform in use by many global information providers, including Facebook, Adobe and Salesforce.com. This marks the conclusion of a significant technical update to the WorldCat database of more than 300 million library records and more than 2 billion library holdings that will offer new options for data analysis and faster service to libraries and their users.
The Apache Hadoop software collection is a framework that allows for the distributed processing of large data sets across clusters of computers. HBase is a top-level Apache Software Foundation project built on Hadoop that provides major data handling improvements for these very large datasets. OCLC WorldShare applications for library management, resource sharing, metadata and discovery rely on access to a variety of large and growing datasets, including the WorldCat database.
The sheer scope of OCLC members' cooperative data is one driver of this change, as HBase provides better handling of very large datasets. In addition, HBase and Hadoop allow OCLC to represent library information in new ways for use in e-content and linked data systems while providing more consistent, reliable and faster service to libraries and their users.
Ron Buckley, Senior OCLC Technology Manager and leader of the Hadoop migration team, will be discussing this effort with leaders in the database management field at the 2013 HBaseCon conference in San Francisco on June 13, 2013.
Hadoop provides these enhancements, in part, by scaling data services across hundreds or even thousands of computers, each with several processor cores. This efficiently distributes large amounts of work across a set of machines, allowing for greater flexibility, speed and dependability. OCLC is running Hadoop across more than 150 servers in three clusters.
This technology has already had an impact on OCLC functionality and services. The recent addition of linked data elements to WorldCat.org relies on the features available in Hadoop. Also, the new WorldShare Metadata Collection Manager service takes advantage of the data handling benefits of its distributed infrastructure.
