Data Processing Engineer - I/O
Mountain View, CA / Hyderabad, IN / Remote
About DataPelago:
DataPelago is at the forefront of revolutionizing data processing for traditional analytics and cutting-edge GenAI preprocessing. We are building an innovative data processing engine that is transforming how Apache Spark, Apache Flink, Ray and others operate on diverse, large-scale data. Our team of engineers drive and adopt advances in hardware-accelerated computing, parallel processing of large-scale data, query optimization, distributed systems, compilers, machine learning, and cloud-native computing. We are looking for specialists to join our engineering team and shape the future of accelerated data processing.
The Opportunity:
As a Data Processing Engineer - I/O, you will be a key individual contributor in advancing data
read and write capabilities of DataPelago’s data processing engine. You will enhance functional
breadth, performance, scale, and reliability of the DataPelago engine in reading and writing large scale data of various data types from diverse data sources and data sinks. This is a unique opportunity to make a significant impact on a category-defining product and work with a talented team of engineers.
What You'll Do:
• Architect: Influence the architecture of how our data processing engine interfaces with data
sources and sinks, catalogs, data formats.
• Design: Lead design of functional and performance enhancements to adapters/connectors,
data representations, data filtering, caching and more in our data processing engine.
• Core Development: Individually design, implement, test, optimize, and maintain components of the data processing engine.
• Innovation and Differentiation: Analyze technology roadmap of existing and emerging data
formats and libraries, open table formats, catalog services, and more (e.g., Apache Arrow,
Apache Parquet, Apache Iceberg) and identify opportunities for our engine to enhance technology and product leadership.
• Collaboration: Partner effectively with engineering and product management in defining and
realizing the data I/O roadmap of our product..
• Continuous Improvement: Foster best practices in design and code reviews, testing, CI/CD,
and issue resolution to maintain highest product quality, security, efficiency, & productivity.
What You'll Bring:
• Bachelor's degree in Computer Science or a related field with 7+ years of relevant experience OR a Master's degree in Computer Science or a related field with 5+ years of relevant
experience.
• 3+ years of deep technical experience in developing and optimizing data read and write interfaces for large-scale data processing, particularly related to Apache Parquet, Apache
ORC, Apache Iceberg, Apache Spark, and similar technologies.
• Demonstrated experience in instrumenting, analyzing, and optimizing the performance of
data processing engine components on benchmark and customer workloads.
• Demonstrated experience in the design, development, and successful release of high-performance data processing engine features for large production deployments.
• Good knowledge of the architecture of one or more of Apache Spark, Apache Flink, Presto/
Trino.
• Exceptional programming skills in C, C++. Rust experience preferred.
• Extensive development experience in Linux environments.
• Strong analytical and problem-solving skills with a passion for performance optimization.
Location Considerations:
We value face-to-face collaboration, but recognize that talent can be found anywhere. Our engineering team works at our headquarters in Mountain View, CA, at our India office in Hyderabad, and at remote locations.
Why Join DataPelago?
• Technology Leadership: Shape the architecture and development of how our core engine
works with advanced data store platforms.
• Cutting-Edge Innovation: Work on challenging problems at the forefront of accelerated
computing and data processing.
• Significant Impact: Your contributions will directly impact the performance and scalability
of our mission-critical platform.
• Growth: Expand your technical expertise and scope of responsibilities working with other
talented engineers and with a growing product.
• Competitive compensation, stock options, comprehensive benefits package, leadership de-
velopment opportunities.
Skills Required
- Bachelor's or Master's degree in Computer Science or related field
- 7+ years of relevant experience with Bachelor's or 5+ years with Master's
- 3+ years in developing and optimizing data read/write interfaces for large-scale data
- Programming skills in C, C++, and Rust (preferred)
- Extensive development experience in Linux environments
What We Do
DataPelago is redefining how enterprises process data for AI and analytics at scale. As organizations race to operationalize artificial intelligence, they are discovering that the greatest barrier to progress isn’t a lack of models or talent – it’s the infrastructure beneath them. Data pipelines remain fragmented across specialized systems for analytics, AI, and data engineering, each optimized for specific workloads but incapable of operating as a cohesive whole. The result is inefficiency: duplicated data, stranded compute resources, and escalating costs that slow innovation. DataPelago was founded to solve this challenge. Its flagship product, Nucleus, is the world’s first Universal Data Processing Engine (UDPE) – a new layer that sits between data lakes and query engines to unify data processing within a single, hardware-aware stack. Built from first principles for accelerated computing, Nucleus allows companies to process, move, and activate their data orders of magnitude more efficiently than existing systems. At its core, Nucleus dynamically orchestrates workloads across heterogeneous compute environments – CPUs, GPUs, TPUs, and FPGAs – ensuring every job runs on the optimal hardware for maximum performance and efficiency. This unified approach eliminates the need to maintain separate infrastructure for different data workloads, dramatically reducing complexity and total cost of ownership by up to 40%. Nucleus supports structured, unstructured, and semi-structured data in a single environment, enabling AI and analytics workloads to coexist seamlessly. It integrates easily with existing data ecosystems and open-source frameworks, providing enterprises with flexibility and performance without requiring code changes or proprietary lock-in. With Nucleus, data teams can accelerate queries, streamline pipelines, and scale AI initiatives faster, all while controlling infrastructure spend. Early adopters across industries are leveraging the platform to speed up data preparation, model training, and real-time analytics by up to 10x, turning data from a bottleneck into a competitive advantage. DataPelago’s mission is to make high-performance, cost-efficient data processing achievable for every enterprise. By bridging the gap between data infrastructure and AI innovation, the company is helping organizations unlock the full potential of their data, laying the foundation for a new era of intelligence at scale.
Why Work With Us
DataPelago is pioneering the world’s first Universal Data Processing Engine, unifying AI and analytics in a single, hardware-aware platform. We’re solving one of the biggest challenges in enterprise AI – making data infrastructure faster, simpler, and more efficient. Join us to build the foundation for the next era of intelligent computing.
Gallery
