This job involves creating data pipelines using a configurable framework (developed using Databricks / pyspark), which
are used to load data from multiple sources to Google Cloud. It also involves
extensive collaboration with other team members (India & USA) to follow a
plan, execute workflows, ensure data load accuracy and debug in case of issues.
Essential Responsibilities
• Understand the overall requirement and
create & execute resilient and high performant data pipeline jobs,
developed in Databricks / PySpark
• Understand data and its relation /
significance from a business perspective
• Create custom scripts in python / pyspark
• Minor development and enhancement
activities as needed to handle new / custom requirements
• Validate data between source and target
& highlight issues and rerun jobs as needed
• Lead other developers, review their code
and ensure quality & optimal code is being delivered
• Provide regular updates on the tasks,
status and risks to project manager
Requirements
• Bachelor’s degree from a reputed
university
• 7 to 12 years overall experience
• Excellent SQL / data knowledge (6+ years)
in creating complex SQL queries, Stored Procedures etc.
• Experience in developing enterprise grade
ETL applications (any tool)
• Experience in Google BigQuery or any
cloud database
• Working knowledge in Pyspark,
Dataproc/Apache Spark, Airflow, Iceberg
• Good communication and positive attitude
• Attitude to take ownership and drive
tasks without follow-ups
Good to have
• Knowledge in Starburst, DBT
• Knowledge in US Healthcare domain
• Knowledge in Agile
Skills Required
- Bachelor's degree from a reputed university
- 7 to 12 years of overall experience
- 6 or more years of SQL and data experience, including complex queries and stored procedures
- Experience developing enterprise-grade ETL applications using any tool
- Experience with Google BigQuery or another cloud database
- Working knowledge of PySpark, Dataproc or Apache Spark, Airflow, and Iceberg
- Good communication skills and a positive attitude
- Ability to take ownership and drive tasks independently
- Knowledge of Starburst and dbt
- Knowledge of the US healthcare domain
- Knowledge of Agile
What We Do
Blackbuck Insights is a global data and AI consultancy providing data engineering services, cloud modernization, AI activation, and platform operations. The company helps enterprises modernize legacy environments, migrate and scale data platforms, build AI-enabled business applications, and establish trusted data foundations for analytics and intelligent decision-making. Its services support organizations seeking reliable, performant, and scalable technology solutions.
