The Role
Designs and maintains scalable batch and streaming data pipelines in Databricks using Spark, PySpark, SQL, Delta Lake, and medallion architecture. Builds data models, automates workflows, develops Power BI reports, and delivers operational, financial, cost, quality, and schedule analytics. Partners with engineering, product operations, project management, and business leaders to support planning and performance decisions. The role also mentors junior colleagues, oversees work, performs statistical analysis, and drives process automation and efficiency improvements.
Summary Generated by Built In
Data Engineer II
Your role will include overseeing, supervising and reviewing tasks performed by team members to ensure effective execution of work; managing end-to-end processes and projects for both internal and external clients with responsibility for timely and accurate delivery; issuing clear instructions and directions to team members on tasks to be performed; and mentoring and guiding junior colleagues to Support their skill development, professional growth, and overall success.
ResponsibilitiesJob Responsibilities include:
- Generate and prepare reports to track KPI metrics along with data analysis and insights for Engineering Program execution and monthly revenue reporting
- Interpret data, analyze results using statistical techniques and provide ongoing reports.
- Design, build, and maintain scalable batch and streaming data pipelines on Databricks using Apache Spark, PySpark, Spark SQL, Delta Lake, Lakeflow Declarative Pipelines, Auto Loader, and medallion architecture patterns.
- Implement data models to support analytics, BI and downstream consumption
- Automate data workflows using Databricks Job and Workflows
- Represent data using PowerBI and advanced visualization tools.
- Prepare ad hoc reports pulling data from multiple data sources and applications
- Identify and automate operational tasks through process/technical automation
- Data analysis, gap identification and drive efficiency on day-to-day reporting
- Prepare Monthly operations reporting along with Portfolio analysis coordinating with engineering, Product Operations and Project Management Leaders.
- Supporting the GBE to drive efficiency in business operations and to achieve their Annual operating plan targets.
- Coordinate the Monthly Operations Report for the GBE and provide reporting of financial, cost, quality, and schedule metrics.
- Enable leaders to make informed decisions using metrics and reports to improve the program performance and customer satisfaction.
- Assist with Impact assessments with respect to changes in program scope, schedule, and priority.
- Coordinates affordability analysis used for census and program planning.
- Effective communication and Business Presentation skills.
- Ability to build data models using statistical methods
You must have:
- BTech or equivalent
- Minimum of 6 to 8 years’ experience in Data Engineering and data operations
- Advanced proficiency in Python, PySpark, and SQL, including complex joins, window functions, query optimization, reusable packages, API integration, automated testing, and performance tuning for large-scale datasets.
- Strong knowledge and relevant experience in creating and deploying pipelines in Databricks (Azure/AWS)
- Direct experience in creating AI Agents and Automation.
- Knowledge of statistics and experience using statistical packages for analyzing datasets
- Strong analytical skills with the ability to collect, organize, analyze, and disseminate significant amounts of information with diligence and accuracy.
- Experience working directly with business users to build reports, dashboards and solving business questions with data.
- Proficient Microsoft Office package skills e.g. Excel, Project, Outlook, PowerPoint etc.
- Strong Working knowledge of Power BI for data representation is preferred
- Experience working in an engineering, finance and/or program management organization is also preferred.
Good to have:
- Strong Working knowledge of Power BI for data representation is preferred
- Experience working in an engineering, finance and/or program management organization is also preferred.
Skills Required
- BTech or equivalent degree
- 6 to 8 years of experience in data engineering and data operations
- Advanced proficiency in Python, PySpark, and SQL
- Experience with complex joins, window functions, query optimization, reusable packages, API integration, automated testing, and performance tuning for large-scale datasets
- Experience creating and deploying pipelines in Databricks on Azure or AWS
- Direct experience creating AI agents and automation
- Knowledge of statistics and experience using statistical packages for analyzing datasets
- Strong analytical skills for collecting, organizing, analyzing, and disseminating large amounts of information
- Experience working directly with business users to build reports and dashboards and solve business questions with data
- Proficiency with Microsoft Office, including Excel, Project, Outlook, and PowerPoint
- Working knowledge of Power BI for data representation
- Experience in an engineering, finance, or program management organization
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company








