The Role
Design, build, and maintain scalable Spark/PySpark ETL pipelines and data models. Optimize SQL and Spark jobs, debug production workflows, collaborate with analysts, and document architectures to support large-scale analytics.
Summary Generated by Built In
Senior Software Development Engineer
Primary Skills
- Spark - Pyspark, SQL, SQL (Basic + Advanced), Python, Hive, Data Modelling Fundamentals
Specialization
- Data Analysis: Data Engineer
Job requirements
- Experience Range: 2 - 4 years of experience in software development, with hands-on expertise in data analysis and engineering
- 1. Design and develop scalable data processing solutions using Spark and PySpark to support advanced analytics initiatives
- 2. Implement robust data models and optimize SQL queries for efficient data retrieval and transformation
- 3. Collaborate with data analysts and business stakeholders to translate requirements into technical solutions
- 4. Build and maintain ETL pipelines leveraging Python, Hive, and SQL for large-scale data integration
- 5. Conduct thorough code reviews, performance tuning, and debugging to ensure high-quality deliverables
- 6. Monitor, troubleshoot, and resolve issues in production data workflows, ensuring data accuracy and reliability
- 7. Document technical processes, data models, and workflow architectures to facilitate knowledge sharing
- 8. Stay updated with industry trends in big data technologies and proactively recommend improvements
- 1. Advanced proficiency in Spark and PySpark
- 2. Strong knowledge of SQL (basic and advanced)
- 3. Expertise in Python programming for data processing
- 4. Experience with Hive for data warehousing solutions
- 5. Solid understanding of data modelling fundamentals
- 6. Ability to design and optimize ETL pipelines
- 7. Hands-on experience with large-scale data processing
- 8. Proficient in performance tuning of SQL and Spark jobs
- 9. Familiarity with distributed computing concepts
- 10. Competence in debugging and troubleshooting data workflows
- 1. Experience with cloud-based data platforms such as AWS or Azure
- 2. Knowledge of Airflow or similar workflow orchestration tools
- 3. Familiarity with Scala for Spark development
- 4. Understanding of data governance and security best practices
- 5. Exposure to machine learning frameworks in Python
- 6. Experience with continuous integration and deployment for data applications
- 1. Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related field 2. Relevant certifications in big data technologies or data engineering (such as Spark, Python, or SQL) are advantageous
Key Responsibilities:
Required Skills:
Preferred Skills:
Desired Qualifications:
Top Skills
Spark,Pyspark,Sql,Python,Hive
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Brillio is the leader in global digital business transformation, applying technology with a human touch. We help businesses define internal and external transformation objectives, and translate those objectives into actionable market strategies using proprietary technologies. With 2600+ experts and 13 offices worldwide, Brillio is the ideal partner for enterprises that want to quickly increase their core business productivity, and achieve a competitive edge, with the latest digital solutions.
