The Role
Design, develop, and maintain scalable batch and real-time data pipelines using Python and PySpark. Build cloud-based data processing solutions, CI/CD pipelines, and streaming systems while ensuring data quality, performance, and scalability. Collaborate with cross-functional teams to translate business requirements into reliable data architectures. The role involves technologies including Kafka, Flink, Kubernetes, Hadoop, Spark, MongoDB, and cloud platforms such as Azure, AWS, or GCP.
Summary Generated by Built In
Job Summary:
Key Responsibilities:
We are looking for a skilled Big Data Engineer with strong experience in PySpark and data engineering. The ideal candidate will have hands-on experience in building scalable data pipelines, working with streaming data, and cloud platforms. Experience with Palantir Foundry is highly preferred
Experience Required: 5 – 10 years
- Design, develop, and maintain ETL/data pipelines using Python and PySpark
- Work with real-time/streaming data systems
- Build and optimize data processing solutions on cloud platforms
- Collaborate with cross-functional teams to understand business requirements
- Develop and maintain CI/CD pipelines
- Ensure data quality, performance, and scalability
- 5–10+ years of experience in Data Engineering
- Strong expertise in Python and PySpark
- Experience with Kafka, Flink, and Kubernetes
- Hands-on with Big Data technologies (Hadoop, Spark, MongoDB)
- Experience with cloud platforms (Azure, AWS, or GCP)
- Good understanding of data architecture and pipeline design
- Experience with Palantir Foundry
- Knowledge of Java
- Experience in consulting or Utilities domain
Skills Required
- 5-10+ years of experience in data engineering
- Strong expertise in Python and PySpark
- Experience with Kafka, Flink, and Kubernetes
- Hands-on experience with big data technologies including Hadoop, Spark, and MongoDB
- Experience with cloud platforms such as Azure, AWS, or GCP
- Understanding of data architecture and pipeline design
- Experience with Palantir Foundry
- Knowledge of Java
- Experience in consulting or the utilities domain
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Cognine Technologies is an IT services and technology solutions company with offices across the United States, India, and Europe. It provides application and product development, API and systems integration, cloud and DevOps services, quality assurance, and AI, machine learning, and automation solutions. The company’s work helps organizations build, modernize, integrate, and improve technology platforms and digital products for businesses worldwide.
.jpeg)
