Zero-Cost, Arrow-Enabled Data Interface for Apache Spark

arXiv (Cornell University)

Abstract

Distributed data processing ecosystems are widespread and their components are highly specialized, such that efficient interoperability is urgent. Recently, Apache Arrow was chosen by the community to serve as a format mediator, providing efficient in-memory data representation. Arrow enables efficient data movement between data processing and storage engines, significantly improving interoperability and overall performance. In this work, we design a new zero-cost data interoperability layer between Apache Spark and Arrow-based data sources through the Arrow Dataset API. Our novel data interface helps separate the computation (Spark) and data (Arrow) layers. This enables practitioners to seamlessly use Spark to access data from all Arrow Dataset API-enabled data sources and frameworks. To benefit our community, we open-source our work and show that consuming data through Apache Arrow is zero-cost: our novel data interface is either on-par or more performant than native Spark.

Authors 7

1. Sebastiaan Alvarez Rodriguez Aachen

   Leiden University

   Affiliation as printed

   Leiden University

   LIACS, Leiden University

2. Jayjeet Chakraborty

   University of California, Santa Cruz

   Affiliation as printed

   UCSC Santa Cruz

3. Aaron Chu

   University of California, Santa Cruz

   Affiliation as printed

   UCSC Santa Cruz

4. Ivo Jimenez

   University of California, Santa Cruz

   Affiliation as printed

   UCSC Santa Cruz

5. Jeff LeFevre

   University of California, Santa Cruz

   Affiliation as printed

   UCSC Santa Cruz

6. Carlos Maltzahn

   University of California, Santa Cruz

   Affiliation as printed

   UCSC Santa Cruz

7. Alexandru Uta Aachen

   Leiden University

   Affiliation as printed

   Leiden University

   LIACS, Leiden University