Responsibilities
- Build and maintain automated data validation tests using Databricks notebooks and tools like Pytest
- Test data ingestion, transformation, and loading processes within the Databricks Lakehouse, specifically focusing on the Bronze, Silver, and Gold layers of the Medallion architecture
- Implement tests for data accuracy, completeness, consistency, timeliness, and uniqueness at different points in the pipeline to catch data issues early
- Reconcile data by comparing record counts, schemas, and values between source systems and target tables in Databricks
- Implement automated data quality checks within data pipelines to ensure no data regressions occur with new code deployments
- Implement automated monitoring and alerting for data quality metrics, identifying anomalies in data freshness, schema evolution, and volume
- Work closely with data engineers and product owners to understand data requirements and ensure data quality meets business needs
- Ensure compliance with data governance policies by building quality checks that validate data sensitivity, masking, and lineage, leveraging tools like Unity Catalog
- Communicate project status and new discoveries in a clear and timely manner during daily stand-ups
Requirements
- Bachelor’s or Master’s in Computer Science, Electrical Engineering, or related field and 5+ years of relevant experience
- Experience with data pipeline and data quality testing strategy and execution, with significant hands-on experience in the Databricks environment
- Strong proficiency in Python for developing and executing data validation scripts
- In-depth knowledge of Databricks, Delta Lake, and the Lakehouse architecture
- Proficiency in writing complex SQL queries for data validation, reconciliation, and troubleshooting issues
- Solid understanding of data warehousing concepts, including dimensional modeling (star/snowflake schemas)
- Hands-on experience with Azure, including Azure storage and data services that integrate with Databricks
- Ability to process data, interpret testing results and provide feedback to the team
- Desire to be part of a rapidly evolving organization, with compelling technology, and taking products and processes to the next level
- Self-awareness, integrity, authenticity, and a growth/entrepreneurial mindset
Similar Jobs
What We Do
Cellares is revolutionizing cell therapy manufacturing. We are developing a one-of-a-kind solution, The Cell Shuttle, to overcome the challenges associated with manufacturing so these life-saving therapies are affordable and widely available to patients who can benefit. The clinical impact of cell therapy in treating cancer has been proven, but this therapeutic approach has several limitations, especially in manufacturing, leaving extremely sick patients waiting for treatment and desperate for hope. Since cell therapy is currently produced for a single patient at a time, it is expensive to manufacture, requiring significant time and resources, and is difficult to scale. Preclinical and clinical scientists, as well as commercial cell therapy manufacturers also lack the options to fully automate their manufacturing process quickly, safely, cost-effectively and at the scale they need. The Cell Shuttle is an automated and closed end-to-end manufacturing solution that is flexible and scalable, enabling customers to run exact processes specified for their cell therapy. Compared with the current manual manufacturing processes for cell therapy, the Cell Shuttle’s next-generation automated manufacturing solution has 10 times the scalability (meaning 10 times more patient doses can be produced simultaneously), enables a three-fold reduction in process failure rates and will reduce the per-patient manufacturing cost by up to 70 percent for most processes.
