The Data Engineer builds and operates the Databricks-based data pipelines that power a new customer-facing benchmarking capability in Onit's AI-driven legal operations platform. Integrating data from multiple source systems across enterprise legal management, matter management, and spend management, this role turns inconsistent, duplicated, and incomplete source data into a clean, reconciled, and trustworthy foundation for analytics. Working closely with the Data Architect, Data Scientist, and Product, the Data Engineer delivers reliable ingestion, entity resolution, and data quality measurement at scale, with multi-tenant data isolation and privacy built in.
Key Responsibilities
• Pipeline Development: Design, build, and maintain scalable batch and incremental pipelines in Databricks (PySpark, Spark SQL, Delta Lake) that ingest data from multiple source systems into a medallion (bronze/silver/gold) architecture.
• Data Integration & Modelling: Implement the canonical data model defined with the Data Architect, so figures reconcile consistently across every product surface.
• Entity Resolution: Build production pipelines that resolve duplicate and conflicting records across systems that were never designed to reconcile with one another.
• Data Quality Engineering: Profile source data, implement automated data quality checks and expectations, and build monitoring and reporting so data quality stays continuously visible against defined targets.
• Orchestration & Reliability: Orchestrate and monitor jobs with Databricks Workflows, optimise Spark performance and cost, and troubleshoot pipeline failures.
• Governance & Privacy: Apply Unity Catalog governance, multi-tenant data isolation, and anonymisation and aggregation rules for pooled cross-customer analytics.
• Collaboration: Partner with the Data Scientist to productionise comparability models and data quality metrics, and with Engineering to expose curated data to the product.
Required Skills
• 5+ years in data engineering or analytics engineering, building production data pipelines in a cloud environment.
• Hands-on Databricks experience: PySpark and Spark SQL, Delta Lake, Databricks Workflows/Jobs, and Unity Catalog.
• Strong SQL and Python, with solid data modelling skills (dimensional and/or canonical models) and experience with the lakehouse/medallion architecture.
• Practical experience with deduplication, record linkage, or entity resolution, and with diagnosing and remediating data quality problems at scale.
• Experience implementing automated data quality checks, pipeline monitoring, and Spark performance and cost optimisation.
• Experience with cloud platforms (AWS preferred), Git-based version control, and CI/CD for data pipelines.
• Clear communication with data scientists, architects, and product stakeholders, including flagging data limitations early.
• Working proficiency with AI tools (e.g., Claude, Databricks Assistant, GitHub Copilot) to accelerate pipeline development, debugging, and documentation, with the judgment to validate AI-generated code before production use.
Preferred Qualifications
• Databricks certification (e.g., Data Engineer Associate or Professional).
• Experience with streaming or incremental ingestion (e.g., Auto Loader, Structured Streaming, Delta Live Tables).
• Experience building data for benchmarks, scoring models, or analytics intended for external consumption.
• Experience with anonymisation, aggregation thresholds, or privacy-preserving analytics.
• Background in enterprise B2B SaaS or the legal technology domain.
Skills Required
- 5+ years of experience in data engineering or analytics engineering, building production data pipelines in a cloud environment
- Hands-on experience with Databricks, PySpark, Spark SQL, Delta Lake, Databricks Workflows or Jobs, and Unity Catalog
- Strong SQL and Python skills
- Solid data modeling skills, including dimensional or canonical models
- Experience with lakehouse and medallion architecture
- Practical experience with deduplication, record linkage, or entity resolution
- Experience diagnosing and remediating data quality problems at scale
- Experience implementing automated data quality checks, pipeline monitoring, and Spark performance and cost optimization
- Experience with cloud platforms, preferably AWS
- Experience with Git-based version control and CI/CD for data pipelines
- Clear communication with data scientists, architects, and product stakeholders
- Working proficiency with AI tools such as Claude, Databricks Assistant, or GitHub Copilot, with the ability to validate generated code
- Databricks certification, such as Data Engineer Associate or Professional
- Experience with streaming or incremental ingestion, such as Auto Loader, Structured Streaming, or Delta Live Tables
- Experience building data for benchmarks, scoring models, or externally consumed analytics
- Experience with anonymization, aggregation thresholds, or privacy-preserving analytics
- Background in enterprise B2B SaaS or legal technology
What We Do
Onit is a global leader of enterprise software and artificial intelligence platforms and products for legal, compliance, sales, IT, HR and finance departments. Our software transforms best practices into smarter workflows, better processes and operational efficiencies. With a focus on enterprise legal management, matter management, legal spend management, contract lifecycle management and legal holds, we operate worldwide and help global companies and billion-dollar legal departments bridge the gap between systems of record and systems of engagement. Onit is the only company in our space with two platforms: Our leading no-code business process automation platform, Apptitude, and our business intelligence platform, Precedent. Apptitude allows customers to create, modify and deploy new software products and custom workflows. Onit’s legal AI platform, Precedent, enables our software products to read, write, and reason like a lawyer. Combined, the two platforms enable customers to digitally transform legal operations by automating processes, reducing costs and maximizing productivity with industry-leading cloud-based software.
