Senior Data Engineer

Unleash your talents and venture into a new career that innovates and transforms products &
services globally!
Senior Data Engineer
Location: Indore
Experience: 5+ Years
[Key Responsibilities]
1. Gather and assemble large, complex sets of data that meet non-functional and
functional business requirements.
Skills: SQL, Python, R, Data Modeling, Data Warehousing, AWS (S3, Athena).
2. Create new data pipelines or enhance existing pipelines to accommodate
non-standard data formats from customers.
Skills: ETL Tools (e.g., Apache NiFi, Talend), Python (Pandas, PySpark), AWS Glue, JSON, XML,
YAML.
3. Identify, design, and implement internal process improvements, including
re-designing infrastructure for greater scalability, optimizing data delivery, and automating manual
processes.
Skills: Apache Airflow, Terraform, Kubernetes, AWS Lambda, CI/CD pipelines, Docker.
4. Build and maintain required infrastructure for optimal extraction,
transformation, and loading (ETL) of data from various data sources using AWS and SQL technologies.
Skills: SQL, AWS Redshift, AWS RDS, EMR (Elastic MapReduce), Snowflake.
5. Use existing methods or develop new tools/methods to analyze the data and perform required data sanity
validations to ensure completeness and accuracy as per technical and functional requirements.
Skills: Python (NumPy, Pandas), Data Validation Tools, Tableau, Power BI.
6. Work with stakeholders including Customer Onboarding, Delivery, Product,
and other functional teams, assisting them with any data-related technical or infrastructure-related issues.
Skills: Stakeholder Communication, JIRA, Agile Methodologies.
7. Provide actionable insights into key data metrics (volumes, trends, outliers,
etc.), highlight any challenges/improvements, and provide recommendations and solutions to relevant
stakeholders.
Skills: Data Analysis, Data Visualization Tools (Tableau, Looker), Advanced Excel.
8. Coordinate with the Technical Program Manager (TPM) to prioritize discovered
issues in the Data Sanity Report and own utility communications.
Skills: Project Management Tools, Reporting Tools, Clear Documentation Practices.
Expectations from the Role:
1. Be the primary owner of the data sanity process internally.
Skills: Data Quality Management, Python (Data Validation Libraries), SQL Auditing
Run defined sanity checks/validations on utility data to ensure data accuracy and completeness.
Skills: Python, SQL, QA Tools, AWS QuickSight.
Create the data sanity report using the established process and template.
Skills: Report Automation Tools, Excel Macros, Python.
Provide actionable insights into key data metrics, highlighting any challenges/improvements in the data.
Skills: Business Intelligence Tools (Tableau, Power BI), Statistical Analysis Tools.
Assess issues in the data as per the technical and functional requirements of the implementation and flag
critical issues to the TPM.
Skills: Root Cause Analysis, Data Profiling, Monitoring Tools.
Implement internal solutions to handle all issues in the data, such as creating custom transformers or
pre-ingestion logic.
Skills: Python (ETL Scripts), PySpark, AWS Lambda, JSON Processing.
Maintain and use data pipelines and processes to ingest required data in Bidgely.
Skills: AWS Data Pipeline, Apache Kafka, SQL.
In case of custom utility data formats, create custom data pipelines in Bidgely to support the ingestion
process.
Skills: Python, Apache Spark, AWS Glue.
On rare occasions, support utility data integration work by working with the utility's Datalake/IT team to
understand their data structure, data sources, and formats, and assist in extracting data from their source
systems.
Skills: Data Lake Architecture (Azure Data Lake, AWS Lake Formation), API Integration, Data
Formats (Parquet, ORC).
💼 We offer:
• Competitive salary
• Flexible work arrangements
• Professional growth opportunities
• Collaborative work culture