We are seeking a Lead Data Engineer to drive the end-to-end modernization of a legacy Oracle platform into a scalable data warehouse built on Amazon Redshift. This role involves leading, accessing oracle procedural workloads, extracting business logic/rules, object dependencies and redesign/convert into Amazon Redshift-native, set-based, parallelized data processing with production-grade orchestration, validation, and operational controls integrated with workflow orchestration using Apache Airflow (MWAA).
Performance enhancement is one of the primary objectives by leveraging Redshift’s parallel processing capabilities. The Lead Data Engineer will be responsible for end-to-end delivery, including solution architecture, code conversion/refactoring to deliver enterprise-grade solution adhering to best practices/standards in performance optimization, scalability, and orchestration.
Experience10+ years of experience in Data Engineering, Data Warehousing, and Database Development.
Primary SkillsAWS Data Engineering, Data Architecture, Data Warehousing, Procedural SQL Development, Amazon Redshift, Airflow Orchestration, Python
- Lead end-to-end modernization of database programming objects and business logic from Oracle into Amazon Redshift and MWAA-based orchestration
- Strong Amazon Redshift expertise, including schema design, MPP concepts, sort / distribution strategy, datashares, serverless or provisioned deployment models, and query tuning
- Converting procedural database logic into set-based and orchestration-driven workflows
Must Have: Redshift, MPP, Oracle PL/SQL, PL/pgSQL, DAG, Airflow (MWAA)
Nice To Have: AWS (S3, Lambda), SQL, ETL/ELT, Data Modeling, Query Optimization, Performance Tuning, Linux/Unix, Git, CI/CD
Key Responsibilities- Perform deep code analysis of Oracle PL/SQL objects to determine what can be translated, what must be refactored, and what must be re-architected
- Identify patterns that are incompatible or inefficient in Redshift, including nested loops, row-by-row processing, procedural orchestration, excessive temp-table chaining, exception-driven logic, and transaction-dependent flows
- Understand, transform, convert and optimize complex Oracle PL/SQL procedures, functions, and scripts & storage objects to AWS Redshift to deliver the most optimal performance for Redshift MPP architecture
- Create design artifacts such as object inventory, dependency maps, pseudo-code, migration strategy, and Redshift rewrite patterns
- Implement production-ready SQL and Airflow DAGs with clear operational behavior, logging, alerting, and restartability
- Tune execution to meet SLA targets through parallelism, query optimization, workload isolation, and efficient data movement
- Review code quality, enforce engineering standards, and mentor junior or mid-level engineers contributing to the migration
- Architect and enhance analytics platforms built on Amazon Redshift
- Define and enforce best practices for data governance, security, and data quality
- Collaborate with cross-functional stakeholders to design and deliver robust data solutions
- Provide technical leadership by mentoring engineers, conducting code reviews, and guiding best practices
- Strong hands-on experience with Amazon Redshift, Oracle PLSQL, PL/pgSQL, Amazon MWAA (Apache Airflow), Aurora PostgreSQL, AWS DMS, S3, Lambda, Python, CloudWatch, GitHub
- Proven experience in enterprise-scale data warehouse architecture
- Expertise in performance tuning and large-scale data processing
- Prior experience in technical leadership or mentoring roles
- Deep hands-on experience with Oracle and/or PostgreSQL database programming, especially stored procedures, functions, cursors, transaction logic, and performance troubleshooting
Skills Required
- 10+ years of experience in Data Engineering, Data Warehousing, and Database Development.
- Strong hands-on experience with Amazon Redshift, Oracle PLSQL, PL/pgSQL, Amazon MWAA, Aurora PostgreSQL, AWS DMS, S3, Lambda, Python, CloudWatch, GitHub.
- Proven experience in enterprise-scale data warehouse architecture.
- Expertise in performance tuning and large-scale data processing.
- Prior experience in technical leadership or mentoring roles.
- Deep hands-on experience with Oracle and/or PostgreSQL database programming.
What We Do
NorthBay is an AWS Premier Partner focused on Database & Application migrations, data & analytics, DevOps & DataOps, application modernization and ML/Ai. Our practice areas include big data and analytics, machine learning, artificial intelligence and database migrations.
