Lead Data Engineer
The Lead Data Engineer will lead the design, development, and administration of critical data pipelines and platform infrastructure within an Azure Databricks environment. This role is focused on building robust, scalable, and secure data solutions that enable efficient data consumption and digital transformation across the organization. The Lead Data Engineer will be responsible for catalog architecture, data quality, security compliance, and stakeholder engagement, while independently managing complex projects from concept to delivery while keeping the organization’s unique needs in consideration. Employee may perform other related duties as required to meet the ongoing needs of the organization.
Essential Responsibilities:
- Build, improve, manage and administrate a complex set of mission-critical data pipelines and data sets in a Databricks environment, with a heavy emphasis on stability, best practice and long-term continual improvement. Plan out and execute major catalog redesigns and rebuilds with a focus on stability, data integrity and zero data loss.
- Troubleshoot and resolve data pipeline, quality and resource issues efficiently with a high level of urgency.
- Lead adoption of the Databricks environment and modern methods of data consumption, digital transformation and automation solutions.
- Meet with stakeholders at every level of the organization to understand needs, objectives, requirements and set expectations. Train and assist users, teams and departments with furthering their adoption and utilization of Databricks. Assess and determine the success of current platform utilization while also identifying and defining new process improvement opportunities.
- Plan the adoption of new features and technologies within Databricks, Azure and the Power Platform.
- Independently manage multiple projects (solo and team-based).
- Support data security and governance within the Unity Catalog by maintaining data access controls, including PII-management, tokenization, row level security, masking and data usage logging and monitoring. Perform security audits and establish alerts.
- Perform data analysis, quality improvements, dashboarding, reporting and visualization of data. Assist data domain owners, report builders and analysts with report development (Power BI), improvement, migration and best practices.
- Serve as final approval of releases to production with thorough code and impact reviews, dependencies and change management.
- Serve as administrator of Power BI production workspaces, evaluation of best practices and compliance within reports, and certification of Power BI reports.
- Administrate all areas of Databricks including user access, resources, cost and security.
Essential Skills:
- Expert-level proficiency in developing advanced ETL pipelines using Databricks.
- Extensive experience with Unity Catalog, Delta Live Tables, data ingestion, transformation, cleansing, and governance methodologies.
- Advanced expertise in Python and SQL within Databricks Notebooks.
- Hands-on experience working with large, complex ERP data sets such as SAP, Infor, Dynamics 365, or Oracle.
- Strong understanding of Machine Learning (ML), Large Language Models (LLMs), and related concepts.
- Experience designing and implementing IoT and real-time streaming data pipelines.
- Proficiency with Azure DevOps, including repositories, version control, production deployments, and CI/CD best practices.
- Demonstrated experience in Azure and Databricks cost management, including resource administration, cost reporting, budgeting, and alerting.
- Bachelor’s degree in Computer Science, STEM, or a related field (or equivalent experience).
- Minimum 3 years of experience working in a technology-focused role.
- At least 1 year of direct, hands-on experience with Databricks in an Azure environment.
- Familiarity with Agile/Scrum methodologies and collaborative development environments.
- Willingness to travel to Brentwood site locations and attend industry conferences (approximately 10% travel).
- Preferred Certifications & Skills:
- Databricks Certified Data Engineer Professional
- Microsoft Certified: Azure Data Engineer Associate
- Experience with R, Scala, or other statistical programming languages
Brentwood offers professional growth potential, pleasant work environment, and an excellent wage and benefits package including 401K w/employer match. Brentwood Industries, Inc., provides equal employment opportunities to all employees and applicants for employment without regard to race, color, religion, sex, national origin, age, disability, or genetics.
At Brentwood, we have a passion for both our products and our people. Our goal as an employer is to help you excel as an individual and as part of a team by providing you with a satisfying, motivating and stimulating work experience. The varied nature of the environment at Brentwood allows you to work alongside industry professionals on a wide range of projects, contributing your knowledge and strengths to develop innovative, market-driven solutions.
8:00am - 5:00pm Monday thru Friday
Skills Required
- Expert-level proficiency in developing advanced ETL pipelines using Databricks
- Experience with Unity Catalog and Delta Live Tables and data governance methodologies
- Advanced expertise in Python and SQL within Databricks Notebooks
- Hands-on experience working with large ERP data sets (SAP, Infor, Dynamics 365, or Oracle)
- Experience designing and implementing IoT and real-time streaming data pipelines
- Proficiency with Azure DevOps, version control, production deployments, and CI/CD best practices
- Demonstrated experience in Azure and Databricks cost management and resource administration
- Support data security and governance within Unity Catalog including access controls, masking, and auditing
- Bachelor's degree in Computer Science, STEM, or related field (or equivalent experience)
- Minimum 3 years of experience in a technology-focused role and at least 1 year of direct Databricks on Azure experience
- Familiarity with Agile/Scrum methodologies and collaborative development environments
- Willingness to travel approximately 10% to Brentwood site locations and conferences
- Databricks Certified Data Engineer Professional
- Microsoft Certified: Azure Data Engineer Associate
- Experience with R, Scala, or other statistical programming languages
- Experience administering Power BI production workspaces and certifying reports
What We Do
Brentwood Medical provides custom packaging and injection molded components for the medical device, pharmaceutical, and electronic industries. Brentwood Medical is proud to be a full-solution provider, offering thermoforming and injection molding, a team of in-house design and process engineers, and dedicated account managers, working hand-in-hand with you through each project stage from concept to validated product. Brentwood Industries, Brentwood Medical's parent company, has more than 50 years of experience in thermoplastic molding and processes. Established in 1965 as a small thermoforming facility, Brentwood has transformed into a vertically integrated corporation and experienced rapid domestic and international growth. This continuous progress has allowed Brentwood to engage developing markets and further establish its worldwide footprint. Our 30,000 square-foot ISO class 7 cleanroom in Reading, Pennsylvania, is equipped with state-of-the-art machinery, offering the latest advancements in process control and automation to provide industry-leading quality at a competitive cost.
