Building high-scale batch and
near-real-time data pipelines deployed on infrastructure we run ourselves
(on-prem), not managed cloud services. You will design and operate high-volume
analytical data systems end to end, with Apache Spark as the core processing
engine for both batch and streaming workloads.
Requirements
• 7+ years
of experience in data engineering and software development
• Ability to
write high-quality code in Java/Scala, Python, or equivalent languages
• Deep,
hands-on production experience with Apache Spark — batch and Spark Structured
Streaming (core requirement)
• Demonstrated
Spark performance tuning: partitioning, caching and persistence, broadcast
joins, shuffle reduction, data-skew handling, and Adaptive Query Execution
• Experience
operating Spark on self-managed clusters (YARN, Kubernetes, or standalone) —
executor sizing, resource allocation, and multi-tenant workloads
• Practical
experience with Kafka (or equivalent messaging systems) as a Spark source and
sink for high-volume workloads, including offset and checkpoint management
• Practical
experience with distributed query engines (e.g., Trino/Presto or similar)
• Practical
experience with ETL / data integration tools, commercial or open-source (e.g.,
Datastage, Informatica, Apache NiFi, or similar)
• Practical
experience with SQL-based transformation frameworks (e.g., dbt or others)
• Strong SQL
skills and understanding of data modeling and data warehousing for analytical
workloads
• Hands-on
experience with real-time / low-latency analytical stores (columnar or OLAP
engines, e.g., Apache Pinot/ClickHouse or similar)
• Practical
experience with big-data platforms and distributions (e.g., Cloudera, Hadoop
ecosystem, Databricks, or similar)
• Practical
experience containerizing and operating data workloads (Docker; Kubernetes a
plus)
• Experience
with workflow orchestration tools (e.g., Airflow or similar)
• Familiarity
with data lake table formats (e.g., Apache Iceberg, Delta Lake, or similar),
including schema evolution and compaction
• Familiarity
with data governance / cataloging tools (e.g., DataHub or similar)
• Familiarity
with lakehouse management systems (e.g., Apache Amoro or similar)
• Familiarity
using AI tools for development and debugging (Claude, Cursor, Codex)
Skills Required
- 7+ years of experience in data engineering and software development
- Ability to write high-quality code in Java, Scala, Python, or equivalent languages
- Deep production experience with Apache Spark for batch and Spark Structured Streaming workloads
- Experience tuning Spark performance, including partitioning, caching, broadcast joins, shuffle reduction, data-skew handling, and Adaptive Query Execution
- Experience operating Spark on self-managed YARN, Kubernetes, or standalone clusters
- Experience with Kafka or equivalent messaging systems as Spark sources and sinks, including offset and checkpoint management
- Experience with distributed query engines such as Trino or Presto
- Experience with ETL or data integration tools such as DataStage, Informatica, or Apache NiFi
- Experience with SQL-based transformation frameworks such as dbt
- Strong SQL skills and understanding of data modeling and data warehousing for analytical workloads
- Hands-on experience with real-time or low-latency analytical stores such as Apache Pinot or ClickHouse
- Experience with big-data platforms and distributions such as Cloudera, Hadoop, or Databricks
- Experience containerizing and operating data workloads with Docker; Kubernetes experience is a plus
- Experience with workflow orchestration tools such as Airflow
- Familiarity with data lake table formats such as Apache Iceberg or Delta Lake, including schema evolution and compaction
- Familiarity with data governance or cataloging tools such as DataHub
- Familiarity with lakehouse management systems such as Apache Amoro
- Familiarity with AI development and debugging tools such as Claude, Cursor, or Codex
What We Do
Techsa is a Cairo-based software services and development company that builds technology solutions tailored to client needs. Its work combines custom and mobile-app development, IT consulting, and software innovation with IoT/data-management platforms and analytics, including AI/ML-driven analysis. It serves sectors such as water and wastewater, logistics, roads, and property maintenance, and also describes itself as a partner for recruiting, onboarding, and managing software teams.
