Alibaba has been tackling the complexity of graph processing at hyperscale via a project called GraphScope – a one-stop large-scale graph computing system. The project has demonstrated a nearly 3X boost in performance on a trillion-scale graph on production workloads.
Alibaba deploys many deep learning frameworks for its e-commerce platforms involving many graph processing types and elements, from sampling to traversal to AI/ML, all working with other non-graph/ML systems.
As a result, different graph programming and runtime issues collide, and low-level code issues arise. In addition, bouncing between different systems means a lot of extra data movement, which leads to confusion. To counter this, Alibaba developed GraphScope.
GraphScope is a unified distributed graph-computing platform that provides a one-stop environment for performing diverse graph operations on a cluster through a user-friendly Python interface. It makes multi-staged processing of large-scale graph data on compute clusters simple by combining several critical pieces of Alibaba technology for analytics, interactive, and graph neural networks (GNN) computation, and the Vineyard store that offers efficient in-memory data transfers.
A collaborative development between Alibaba and the University of Edinburgh and Shenzhen’s Institute of Computing Sciences, GraphScope, outperforms state-of-the-art systems designed for different types of graph queries. It runs 2.86 X faster than a manually assembled pipeline with complex, multi-staged processing on large graphs in a real-life application at Alibaba.
Click here to read the original article published by The Next Platform.
Please give your feedback on this article or share a similar story for publishing by clicking here.
