Senior Data Engineer

What is the Data Engineer role?

CopilotIQ is seeking a Data Engineer to join our team as an individual contributor in a remote capacity. In this hands-on role, you will be responsible for designing, developing, and maintaining robust data pipelines, as well as optimizing our data warehouse and data lake infrastructure. You will also support the organization’s business intelligence needs by ensuring the availability, reliability, and quality of data across the ecosystem. We are looking for a highly collaborative and detail-oriented professional with strong problem-solving skills and a passion for building efficient, scalable data solutions that drive impactful insights.

What you’ll be doing:

• Operate and own production data pipelines, with a focus on data quality—reliability, accuracy, timeliness—and rapid incident resolution.
• Design, build, and optimize batch and streaming pipelines in Airflow, landing data in our Redshift-based warehouse.
• Lead hands-on development. Build, test, and deploy high-throughput Airflow workflows and Python/SQL transformations; introduce DataOps practices (CI/CD, versioned schemas, automated data quality tests) to raise the engineering bar.
• Ensure rigorous governance, lineage, and privacy controls across data platforms.
• Write clean, efficient, and high-performance code in Python and SQL for data transformations and automation workflows.
• Partner closely with business stakeholders to support reporting and analytics through BI tools such as Sigma Computing and Superset.
• Optimize relentlessly. Monitor performance, scalability, and infrastructure cost; tune queries, caching, and compression. Introduce new value-creating technologies (e.g., Redshift Serverless, Kinesis, Iceberg).
• Champion best practices in data engineering, including automation, data security, and operational excellence.
• Collaborate cross-functionally with analysts and software engineers to design and implement scalable, production-grade data solutions.

Requirements

• 5+ years of dedicated hands-on experience in designing, building, and operating production data pipelines and warehouses.
• Bachelor’s or Master’s in Computer Science.
• Programming & query languages: expert-level Python for data processing/automation and advanced SQL for analytical workloads.
• Core data-engineering stack: orchestration with Airflow (or Prefect); distributed processing with Apache Spark; AWS services (Redshift, Glue, Athena, S3, DMS, Lambda, and SQS); and at least one streaming technology such as Kinesis or Kafka.
• Deep understanding of dimensional / event-driven data modeling, partitioning, and performance tuning at terabyte-to-petabyte scale; comfortable balancing performance, cost, and governance in petabyte-scale environments.
• Foundational knowledge of NoSQL technologies such as MongoDB and DynamoDB.
• Excellent problem-solving skills, strong attention to detail, and the ability to thrive in a fast-paced, dynamic environment.
• A mission-driven, collaborative mindset with strong product thinking and a desire to learn, grow, and make meaningful technical contributions.

• AWS Certified Data Analytics – Specialty or Solutions Architect – Professional.
• Administering access control, datasets, and embedded dashboards in BI tools such as Sigma Computing and Superset.
• Experience rolling out DataOps practices (CI/CD for data, automated quality tests, lineage/observability tooling).
• Leading technical initiatives in a growing team.