You will be responsible for the development, support, maintenance and implementation of an enterprise application that is scalable, resilient, and distributed.
What You'll Do
You will be responsible for creating innovative interoperability platforms, tools and solutions to enable seamless and secure data integration.
In this role, your solutions will be used to connect legacy, newly developed, and vendor applications across the datacenter and cloud environments, and you will be responsible for the full lifecycle of the solutions.
You will be developing specifications, designing infrastructure and interfaces, developing code. You will be responsible to design and build scalable, secured ETL pipelines in PySpark. You will develop complex PySpark code using SparkSQL, Dataframes, joins, transposes etc. to load data to a MPP Datawarehouse: Snowflake. You will utilize your good understanding of Python and Spark coding concepts e.g. SparkSQL, Dataframes, joins, transposes etc. You will create ETL data pipelines using PySpark to read Kafka topics, RDBMS, APIs and other sources and load into object storage (e.g. S3 etc.).
Who You'll Work With
You will join our Waltham or Atlanta office and be a part of our One Firm Tech-Cloud Data & Analytics team.
You will work with product managers, software engineers, architects and various platform teams. You'll be part of a team responsible for delivering technology enabled solutions of the future. You will be involved in all business value chain activities from understanding product needs to product development to on-going maintenance and enhancement.
Our vision & mission of the Cloud Data and Analytics (CDA) group is to give our firm colleagues the ability to generate valuable insights through a scalable, highly responsive and value driven data and analytics framework. We will achieve that by moving all our data and processes into Cloud.
The CDA group has several complementary teams working on various facets of delivering Cloud services.
Qualifications

• Bachelors/Master's degree in technology related field
• 5+ years of IT experience with about 3+ years in data engineering and ETL/ELT
• Experience in Java/J2EE tech stack.
• Experience in designing and developing data pipelines using PySpark in any Public Cloud e.g. AWS, GCP, Azure etc or hybrid environments.
• Proficient in SQL, data modelling and data warehouse concepts.
• Proficient in developing Microservices using Java, SpringBoot and REST API Creation and Consumption.
• Conceptual understanding of modern software engineering patterns, including those used in highly scalable, distributed, and resilient systems.
• Solid understanding of NoSQL databases like MongoDB and ElasticSearch, experience in Kubernetes, docker and CI/CD pipeline configuration.
• Experience in developing and delivering systems on AWS cloud platform or equivalent.
• Experience working on AWS SDK and Lambdas, any MPP data warehouses e.g. Snowflake, Big Query, Redshift, GraphQL API development, AWS Glue, Glue Studio, Blueprints
• Experience in implementing effective and successful Cloud based Data Migration and Data Integration strategies