Position Summary:
Join the Vallen team as a Senior Data Engineer. We are seeking to fill this role with an individual with 5–7 years of experience in data engineering and a strong background in Databricks. This role is ideal for someone passionate about building scalable data solutions, enabling analytics, and driving innovation in a modern data platform.
Essential Duties and Responsibilities:
- Design and develop scalable data pipelines using PySpark, Spark SQL, and Python within the Databricks environment.
- Build modular, version-controlled transformation workflows using DBT (Data Build Tool).
- Package and deploy Databricks workflows and notebooks using Databricks Asset Bundles (DABS) for CI/CD and environment management.
- Integrate Databricks workflows with Azure DevOps for automated testing, deployment, and version control.
- Develop and maintain robust CI/CD pipelines for data engineering workflows using Git, Azure DevOps, and DABS.
- Implement and optimize dimensional data models, ELT/ETL processes, and performance tuning in cloud data platforms.
- Collaborate with data scientists, analysts, and business stakeholders to deliver high-impact data solutions.
- Implement logging, alerting, and monitoring for data pipelines using tools like Databricks Jobs, MLflow, or Azure Monitor.
Job Qualifications:
- 5–7 years of experience as a Data Engineer, with at least 3 years working in Databricks.
- Strong proficiency in Python, PySpark, and Spark SQL.
- Hands-on experience with DBT in a cloud data platform.
- Experience with DABS for workflow packaging and deployment.
- Proven expertise in Azure DevOps, Git, and CI/CD pipeline development.
- Solid understanding of data modeling, ETL/ELT, and performance optimization.
- Experience implementing monitoring and observability for data pipelines.
- Excellent communication and collaboration skills.
Preferred Qualifications
- Experience in Azure cloud ecosystem.
- Familiarity with MLflow and other observability tools.
- Exposure to data governance and metadata management practices.
Work Environment & Physical Demands:
Reasonable accommodations will be evaluated and may be implemented to enable individuals with disabilities to perform essential functions of this position.
- This job operates in a professional office environment and routinely uses standard office equipment such as computers, phones, photocopiers, filing cabinets, etc.
- The work environment is generally favorable. Lighting and temperature are adequate and there are no hazardous or unpleasant conditions caused by noise, dust, etc.
- Long periods of time working on a computer and performing repetitive key-boarding activities.
- Long periods of time performing work over the phone and participating in group conference calls.
- Occasional evening and weekend work may be required as job duties demand and may include little to no advanced notice.
- Specific vision abilities required by this job include close vision, distance vision, color vision, peripheral vision, depth perception and ability to adjust focus.
- Frequently required to sit and/or stand.
Top Skills
What We Do
Vallen is a leading provider of indirect industrial supplies—all those materials that are not part of the products our customers manufacture. From earplugs to specialty tools, to abrasives and cutting tools, we cover the vast majority of our customers’ maintenance, repair, operations and production-related needs. Vallen is also a leader in safety services with offerings that include equipment inspection, maintenance, certification and solution design. Many of our customers look to us for integrated supply solutions and vendor-managed inventory, with Vallen associates on-site, bringing local knowledge and expertise to the specific needs of individual facilities.
