Databricks Data Engineer

Title: Databricks Data Engineer

Location: Pune / Hyderabad / Mohali

Job Summary:

We are seeking a Databricks Data Engineer skilled in Databricks, Python, Scala to join our team of data engineers within Client Information Technology. This team serves Client by ingesting data from multiple sources, making it available to internal stakeholders, and by interfacing with and exchanging data between a variety of internal and external systems. You will develop, deploy and troubleshoot complex data ingestion pipelines and processes. Your curious mind and attention to detail will be an asset, as will your extensive knowledge and experience in the data engineering space.

Responsibilities:

• Design, develop, optimize, and maintain data architecture and pipelines that adhere to ETL principles and business goals
• Collaborate with data engineers, data consumers, and other team members to come up with simple, functional, and elegant solutions that balance the data needs across the organization
• Solve complex data problems to deliver insights that helps the organization achieve its goals
• Create data products that will be used throughout the organization
• Advise, consult, mentor and coach other data and analytic professionals on data standards and practices
• Foster a culture of sharing, re-use, design for scale stability, and operational efficiency of data and analytic solutions
• Develop and deliver documentation on data engineering capabilities, standards, and processes; participate in coaching, mentoring, design reviews and code reviews
• Partner with business analysts and solutions architects to develop technical architectures for strategic enterprise projects and initiatives.
• Deliver awesome code

Technical Qualifications

• 5 to 8 years relevant and progressive data engineering experience
• Deep Technical knowledge and experience in Databricks, Python, Scala, Microsoft Azure architecture and platform including Synapse, ADF (Azure Data Factory) pipelines and Synapse stored procedures
• Hands-on experience working with data pipelines using a variety of source and target locations (e.g., Databricks, Synapse, SQL Server, Data Lake, file-based, SQL and No-SQL database)
• Experience in engineering practices such as development, code refactoring, and leveraging design patterns, CI/CD, and building highly scalable data applications and processes
• Experience developing batch ETL pipelines; real-time pipelines are a plus
• Knowledge of advanced data engineering concepts such as dimensional modeling, ETL, data governance, data warehousing involving structured and unstructured data
• Thorough knowledge of Synapse and SQL Server including T-SQL and stored procedures
• Experience working with and supporting cross-functional teams in a dynamic environment
• A successful history of manipulating, processing and extracting value from large disconnected datasets.
• Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
• Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases.
• Knowledge and understanding of Boomi is a plus

Qualifications:

• Excellent problem-solving skills and experience
• Effective communication skills
• Strong collaboration skills
• "Self-starter" attitude and the ability to make decisions with minimal guidance from others
• Proficiency in designing, developing, and deploying data pipelines using Delta Live Tables (DLT) on Databricks
• Innovative and passionate about your work and the work of your teammates
• Ability to comprehend and analyze operational systems and ask appropriate questions to determine how to improve, migrate or modify the solution to meet business needs
• Experience with data ingestion and engineering, specifically involving large data volumes
• Knowledge of CI/CD release pipelines is a plus
• Understanding of Python and knowledge of parallel processing frameworks like MapReduce, Spark, Scala
• Knowledge of the Agile development process
• Bachelors degree in computer science, information technology, business management information systems, or equivalent experience.

Company Description: Bits in Glass - India

• Industry Leader:
  • Bits in Glass(BIG) has been in business for more than 20 years. In 2021 Bits in Glass joined hands with Crochet Technologies, forming a larger organization under the Bits In Glass brand to better serve customers across the globe.
  • Offices across three locations in India: Pune, Hyderabad & Chandigarh.
  • Specialized Pega partner since 2017, delivering Pega solutions with deep industry expertise and experience.
  • Proudly ranked among the top 30 Pega partners, Bits In Glass has been one of the very few sponsors of the annual PegaWorld event.
  • Elite Appian partnersince 2008, delivering Appian solutions with deep industry expertise and experience.
  • Operating in the United States, Canada, United Kingdom, and India.
  • Dedicated global Pega CoE to support our customers and internal dev teams.
  • Dedicated AI and Data Engineering teams handling clients in various domains.
• Employee Benefits:
  • Career Growth:Opportunities for career advancement and professional development.
  • Challenging Projects:Work on innovative, cutting-edge projects that make a global impact.
  • Global Exposure:Collaborate with international teams and clients to broaden your professional network.
  • Flexible Work Arrangements:Support for work-life balance through flexible working conditions.
  • Comprehensive Benefits:Competitive compensation packages and comprehensive benefits including health insurance, and paid time off.
  • Learning Opportunities- Great opportunity to upskill yourself and work on new technologies like AI-enabled Pega solutions, Data engineering, Integration, cloud migration etc.
• Company Culture:
  • Collaborative Environment:Emphasizes teamwork, innovation, and knowledge sharing.
  • Inclusive Workplace:Values diversity and fosters an inclusive environment where all ideas are respected.
  • Continuous Learning:Encourages professional development through ongoing learning opportunities and certifications.
• Core Values:
  • Integrity:Commitment to ethical practices and transparency in all business dealings.
  • Excellence:Strive for the highest standards in everything we do.
  • Client-Centric Approach:Focus on delivering the best solutions tailored to client needs.