This position will primarily be responsible for designing, developing, and maintaining robust ETL/ELT pipelines for data ingestion, transformation, and storage. The role also involves designing and developing scalable data solutions.
The person will work with a team responsible for ensuring the availability, reliability, and performance of data systems. The candidate will design, develop, and maintain scalable data pipelines and infrastructure on cloud platforms (Azure, AWS, or GCP).
The person will lso work with a collaborative team responsible for driving client performance by combining data-driven insights and strategic thinking to solve business challenges. The candidate should have strong organizational, critical thinking, and communication skills to interact effectively with stakeholders.
Responsibilities:
- Design, build, and optimize data ingestion pipelines using Azure Data Factory, AWS Glue, or Google Cloud Data Fusion
- Ensure reliable extraction, transformation, and loading (ETL/ELT) of large datasets from various sources
- Implement and manage data storage solutions like Databricks, ensuring high performance and scalability
- Develop and orchestrate data pipelines using Apache Airflow for workflow automation and job scheduling
- Collaborate with database administrators and data architects to optimize database schemas
- Develop and maintain complex BI reporting systems to provide actionable insights
- Work closely with business stakeholders to understand requirements and translate them into technical specifications
- Monitor data quality and integrity across pipelines
- Maintain comprehensive documentation for data processes, pipelines, and reporting systems
- Collaborate with cross-functional teams to streamline data operations
- Stay updated on emerging data engineering technologies and best practices to enhance system performance
Technical Qualifications:
- Experience in data ETL tools and cloud platforms like Azure Data Factory, Azure Databricks, AWS Glue, Amazon EMR, Databricks on AWS, Google Cloud Data Fusion, or Databricks on Google Cloud
- Experience in Apache Airflow for workflow orchestration and automation
- Experience in SQL for data manipulation and querying
- Experience with at least one BI tool like Power BI, Tableau, or Google Looker
- Experience with relational databases (e.g., SQL Server, MySQL, PostgreSQL) and NoSQL databases (e.g., MongoDB)
- Proficiency in Python programming
- Proficiency in developing and maintaining data visualization and reporting dashboards
- Familiarity with data warehousing solutions and big data frameworks (Apache Spark)
Personal Skills:
- Strong analytical skills: ability to read business requirements, analyze problems, and propose solutions
- Ability to identify alternatives and find optimal solutions
- Ability to follow through and ensure logical implementation
- Quick learner with the ability to adapt to new concepts and software
- Ability to work effectively in a team environment
- Strong time management skills, capable of handling multiple tasks and competing deadlines
- Effective written and verbal communication skills
Education and Work Experience:
- Background in Computer Science, Information Technology, Data Science, or a related field preferred
- Minimum 3 years of total experience, with at least 2 years of relevant experience in data engineering and data pipeline development
- Certification in Databricks, Azure Data Engineering, or any related data technology is an added advantage
Similar Jobs
What We Do
Creative Capsule was founded in 2003 in Kansas City and since then has grown into a team of more than 200 motivated software professionals in Overland Park Kansas, Zürich Switzerland, and Goa India. Our founders continue to be involved in day-to-day operations, bringing an overriding customer focus and a dedication to technical excellence. Many of our best clients are rapidly evolving SaaS and software startup companies and life sciences companies who share our passion for innovation.
Creative Capsule builds lasting development relationships. We create highly skilled and effective software teams that integrate seamlessly with our clients’ organizations. We do not deliver one-off projects. Instead, we work to understand each client’s unique business needs and culture, and then construct cohesive teams that empower our partners and strengthen their delivery capabilities.
Creative Capsule helps clients to transform their software delivery capacity. For each client, we build a dedicated development group that works hand-in-hand with their organization, creating a single virtual team. We ensure a close fit with each client by selecting team members that have technical and personal attributes that resonate with the client’s needs and culture. Each of our teams spans continents, providing an optimized balance of responsive local presence and cost-effective offshore resources, while operating as a single innovative and agile force that shares risks and rewards.
