-
Core Specialty’s Data Environment is the organizational data ecosystem that supports enterprise requirements for data management, consolidation, insights, and reporting. The Data & Analytics organization ingests and manages data from over 200 distinct sources and growing, which serves business intelligence, AI/ML models, and enterprise decision-making. Maintaining the integrity, accuracy, and completeness of this extensive data environment requires a scalable and automated quality assurance strategy. The Director of Engineering – QA Automation will be responsible for designing and implementing this strategy, integrating quality processes throughout the data platform—including ingestion pipelines, semantic models, and machine learning workflows. This role will require 40% hands on involvement to start in order to build out the architecture, blueprints, and coding standards required for QA Engineers to adhere to. It may also include development efforts as needed.
Key Accountabilities/Deliverables:
Define and lead the QA automation strategy for data engineering, analytics, and machine learning initiatives.
Architect and implement scalable, reusable automation frameworks for data pipelines, ingestion models, semantic layers, and ML workflows.
Collaborate with Data Engineering, Data Science, and DevOps teams to integrate automated testing into CI/CD pipelines.
Establish coding standards, architectural patterns, and best practices for test automation across the organization.
Develop automated validation for data quality, schema enforcement, lineage tracking, and model performance.
Mentor and grow a high-performing QA team with deep domain knowledge in data and analytics.
Drive continuous improvement in testing tools, processes, and observability.
In addition to the above key responsibilities, you may be required to undertake other duties from time to time as the Company may reasonably require.
Technical Knowledge and Understanding:
Programming Languages: Python (preferred), SQL, Java/Scala (for Spark-based pipelines)
Automation Frameworks: PyTest, Great Expectations, dbt tests, Selenium, Airflow test suites
Data Technologies: Spark, Kafka, Airflow, dbt, Delta Lake, Parquet, Avro
Cloud Platforms: AWS (Glue, S3, Redshift), Azure, GCP (BigQuery, Dataflow), Databricks
Data Warehouses: Snowflake, BigQuery, Redshift
CI/CD & DevOps: GitHub Actions, Jenkins, Docker, Kubernetes
ML Testing: MLflow, TensorFlow Model Analysis, custom validation pipelines
Observability & Monitoring: Monte Carlo, Databand, OpenLineage
SDLC: Agile/SCRUM Frameworks, Kanban, JIRA
Data Quality Management: MS Purview
Preferred Skills
Experience with synthetic data generation and data fuzzing tools
Experience with Data Contracts and the Producer vs. Consumer model
Familiarity with data governance and compliance frameworks (e.g., GDPR, HIPAA)
Exposure to AI-assisted testing platforms
Experience:
Bachelor’s degree in computer science, data engineering, or related field required
Master’s degree in computer science, data engineering, or related field preferred
Minimum of 15 years of experience in Data Engineering (hands-on roles)
Minimum of 5 years of experience in a leadership position
Prior QA and automation experience within data engineering space required
Proven experience in testing large-scale data systems and ML workflows
Strong architectural thinking and ability to define scalable testing patterns
Excellent leadership, communication, and cross-functional collaboration skills
Applicants must be authorized to work for any employer in the U.S. We are unable to sponsor or take over sponsorship of an employment Visa for this position.
#LI-Hybrid
-
At Core Specialty, you will receive a competitive salary and opportunities for professional development and advancement. We offer medical, dental, vision, and life insurances; short and long-term disability; a Company-match of 100% of a 6% contribution 401(k) plan; an Employee Assistance Plan; Health Savings Account, Flexible Spending Account, Health Reimbursement Account, and a wellness program
Top Skills
What We Do
Core Specialty, through its subsidiary insurers, offers a diversified range of property, casualty, and marine insurance products for small to mid-sized businesses.
We have the capital to take on risk, the underwriting talent in place, decisive leadership team, infrastructure, and a proven track record of making things happen – fast!
When you’re ready to solve your toughest insurance needs, we’re ready to get it done for you.
We free customers up to focus on their business by taking the load of complicated specialty insurance off their hands.
We break down the walls of bureaucracy to provide optimal underwriting solutions for brokers.
We’re ready, equipped, and motivated to get the job done, efficiently and professionally, by empowering experts with what they need to move quickly on behalf of customers and their brokers.
Our specialty focus is the essential part of our identity.
It is at our core
.png){kind=logo}
