Build and optimize distributed compute infrastructure for large-scale analytical and AI workloads. Responsibilities include improving query execution, scheduling, resource allocation, reliability, workload routing, and compute efficiency across Spark and related systems. The role involves debugging performance bottlenecks, optimizing joins, scans, shuffles, caching, partitioning, and memory usage, and working with lakehouse formats and cloud object storage. Candidates will implement workload optimization algorithms and may contribute to open source or research.

Build foundational lakehouse infrastructure for exabyte-scale AI data environments. Responsibilities include metadata and transaction systems, table maintenance, schema and partition evolution, snapshot isolation, compaction, clustering, file-layout optimization, object-store performance, columnar-format optimization, and query performance across major lakehouse engines. The role also involves debugging distributed systems, implementing compression and data-efficiency algorithms, and contributing to open-source or research efforts.