Data Engineer II

Posted 6 Days Ago
Be an Early Applicant
Bengaluru, Karnataka
Mid level
Artificial Intelligence • Machine Learning
The Role
The Data Engineer will develop and manage data collection pipelines and ensure data quality while working with a distributed system. Responsibilities include building and scaling a data platform, implementing data quality checks, and collaborating with Data Science teams for AI model production.
Summary Generated by Built In

ABOUT US

We are Ai Palette. We think the world would be a nicer place to be if Food Companies could create products that the consumers really want. So we are making it happen. We want to be the most preferred Food AI company in the world. We’re making it possible by building an AI-powered SaaS platform based on our founders’ experience in the Food Industry & AI.

We are a Series A round funded company with experienced investors like pi Ventures, Exfinity ventures and Anthill ventures. We are building a global company and are already working with customers across the globe. Our customers include Fortune 500 companies and FMCG brands.

We have won several accolades & awards in our short period of existence such as:

  • Top 10 Global Food & Retail Tech startup at Kickstart Innovation'20 at Switzerland
  • Top 15 Global Food tech Startup at Slingshot'19 at Singapore
  • Top 200 Global startups at HKSTP Epic'20 at HongKong
  • Top 500 Global Food Tech startups by Forward Fooding


WHAT’S IT LIKE TO WORK AT AI PALETTE? 


We're a growing technology startup headquartered in Singapore with our Engineering base in Bangalore and a team in US, We PaletteeRs are a highly passionate and motivated bunch of people that help each other do remarkable things and achieve extraordinary results every single day. We are active learners, have a positive impact on consumers’ lives and settle for nothing short of excellence. We face challenges together and we win together. In our vision to build the World's First AI Platform for New Product Innovation, there isn’t a day that goes by where we don’t have “Aha” moments.  We strive together to deliver world-class solutions that transform the way consumer products of today and tomorrow will be created. Join us!

About the job

We are looking for a passionate Data Engineer who will be working in the Data Engineering division and core development of our AI Platform that will ideate consumer products of the future. As the Data Engineer, you will be working hand in hand with the Data Science and Full Stack team on the toughest and most challenging problems in Data Engineering and Cloud Computing handling millions of data points that include social media. You will get the opportunity to work and scale a growing Data Platform.

You don’t want a job that just pays the bills: you want a job to get out of bed for. You take more enjoyment from solving problems than your friends think is normal. If you are looking to find your “ikigai”, then your search stops here right with us!

RESPONSIBILITIES

  1. Build the data collection pipelines that can acquire and handle millions of public data points from various sources using APIs and web extraction techniques at scale
  2. Build the data cleaning, quality and integrity pipeline in the platform leveraging the Apache Spark Python and AWS services
  3. Architect and develop distributed systems that can handle large-scale data processing
  4. Programming Language: Python/Java, Apache Spark, Apache Flink (Good to have)
  5. NoSQL Database: Elasticsearch, Dynamo DB/Mongo DB
  6. Work closely with the Data Science Team for the preprocessing steps required for the AI models in Production Environment
  7. Scale and automate Data Platform Collection layer to handle the consistently incoming data
  8. Implement data quality checks and validation processes
  9. Troubleshoot and resolve performance issues
  10. Collaborate with cross-functional teams to support data-related initiatives
  11. Document data pipelines, data models, and other technical processes

REQUIREMENTS

  1. 4-6 years of experience in building data engineering pipeline
  2. Have hand ons experience on Apache Spark, PySpark, SQL, Python programming
  3. Worked with NoSQL database like Elasticsearch, Dynamo DB, Cassandra - Anyone
  4. Strong experience in AWS Cloud Platform - S3, EC2, Lambda, Elasticsearch etc
  5. Quick learner, excellent communication and team player

IDEAL

  • Deep technical understanding of AWS PaaS and IaaS services
  • Previous experience with Airflow, Spark, PySpark, Python, ElasticSearch, Web Crawling and Docker
  • Prior experience with social media data (twitter, reddit, blogs, etc.)

BENEFITS

  • Great progression opportunities - we want you to grow with us.
  • Look after yourself with health insurance including Hospital/Surgical.
  • Learn new skills with sponsored training on MOOCs such as Coursera, Udemy.  

EQUAL OPPORTUNITY: 

We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, gender, sexual orientation, age, marital status, veteran status, or disability status.

Top Skills

Java
Python
The Company
93 Employees
On-site Workplace
Year Founded: 2018

What We Do

Ai Palette enables CPG companies to create consumer-winning products using Artificial Intelligence and Machine Learning. Its patented technology identifies emerging trends across 61B data points collated from 150+ data sources in real time, uncovering consumer drivers and motivations, and helping in creating product concepts that meet unmet consumer needs. Its Natural Language Processing algorithm can also understand 18 different languages including Asian languages, making it truly one of a kind. With Ai Palette, you can: - Quickly access consumer trends and insights without spending time and money on traditional market research. FoodGPT leverages a vast pool of over 61 billion data points, ensuring you stay in sync with the dynamic consumer landscape. - Combine 1st party, 2nd party and 3rd party data and data sources in a single place to get faster research without investing a lot of time on digging into data sources. - Generate compelling product concepts, fine tuned to resonate with your target audience, with elaborate product claims and messaging crafted in consumer language. - Validate winning product hypothesis by marrying sales, consumer panel data, and secondary research data. - Leverage powerful predictions for sales forecasting with each new product launch. - Make new markets penetration or establish novel categories successfully backed by consumer data. Headquartered in Singapore, it is currently working closely with some of the world's largest food companies, including Fortune 500 giants like Kelloggs', Nestle, Olam, Diageo etc.

Similar Jobs

Hybrid
Bengaluru, Karnataka, IND
289097 Employees

Enverus Logo Enverus

Senior Data Engineer - 24447

Big Data • Information Technology • Software • Analytics • Energy
Basavanagudi, Krishnarajpet, Mandya, Karnataka, IND
1700 Employees

EchoStar Logo EchoStar

Lead Data Engineer

Aerospace • Cloud • Digital Media • Information Technology • Mobile • News + Entertainment • Retail
Bengaluru, Karnataka, IND
14500 Employees

Atlassian Logo Atlassian

Principal Data Platform Engineer

Cloud • Information Technology • Productivity • Security • Software • App development • Automation
Remote
Bengaluru, Karnataka, IND
11000 Employees

Similar Companies Hiring

Halter Thumbnail
Software • Machine Learning • Internet of Things • Hardware • Greentech • Business Intelligence • Agriculture
Auckland City, NZ
150 Employees
InCommodities Thumbnail
Renewable Energy • Machine Learning • Information Technology • Energy • Automation • Analytics
Austin, TX
234 Employees
RunPod Thumbnail
Software • Infrastructure as a Service (IaaS) • Cloud • Artificial Intelligence
Charlotte, North Carolina
53 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account