Lead AI Automation Engineer

Description

Digitain is seeking a highly skilled and strategic Lead AI Automation Engineer to drive the next generation of intelligent process automation across our enterprise ecosystem. In this role, you will lead the architecture, development, and scaling of end-to-end AI workflows—combining advanced web scraping, entity matching, Natural Language Processing (NLP), and Retrieval-Augmented Generation (RAG) systems to transform unstructured data into actionable business intelligence. 

As a technical lead, you will bridge domain-specific operational needs with cutting-edge AI architectures. You will set engineering standards for data extraction and automated decision-making pipelines, mentor team members, and ensure high reliability for mission-critical automations. 

Responsibilities

• Architect Intelligent Automations: Design and deploy resilient, end-to-end automated pipelines that integrate Large Language Models (LLMs), RAG architecture, and custom AI tools with core enterprise systems
• Data Scraping & Ingestion at Scale: Lead the strategy for large-scale web scraping, API extraction, and document processing, ensuring robust handling of anti-bot protections, changing schemas, and dynamic Web environments
• Intelligent Data Matching & Entity Resolution: Build sophisticated algorithms and semantic search solutions (vector databases, embedding models) to perform high-accuracy data matching, deduplication, and record linkage across disparate datasets
• RAG & Knowledge Retrieval: Design, evaluate, and optimize Retrieval-Augmented Generation (RAG) frameworks to ground LLM responses in proprietary databases, knowledge bases, and live web data
• Unstructured Text & Document Processing: Implement custom NLP pipelines (classification, named entity recognition, sentiment analysis, document parsing) to automate complex decision-making and report generation
• System Reliability & Monitoring: Establish best practices for exception handling, error recovery, LLM output validation (guardrails), and latency optimization across all automated workflows
• Mentorship & Team Standards: Guide and mentor AI automation specialists, conduct code reviews, and establish standard operating procedures for workflow documentation, API management, and prompt engineering
• Stakeholder Alignment: Collaborate with cross-functional leadership to identify high-value automation opportunities, map technical feasibility, and demonstrate ROI

Requirements

• 5+ years of total experience in software engineering, data engineering, or process automation, with at least 2–3 years hands-on experience building AI/LLM-powered automation systems
• Proven track record of leading complex automation projects involving unstructured data, web scraping, and NLP
• Advanced proficiency in Python (Pandas, Asyncio, Playwright, Selenium, BeautifulSoup, Scrapy, FastHTML/FastAPI)
• Deep expertise with LangChain, LlamaIndex, OpenAI API, Anthropic, Hugging Face, vector stores (e.g., Qdrant, Chroma, Pinecone, FAISS), and fine-tuning/prompting techniques
• Solid understanding of fuzzy matching, BM25, semantic search, vector embeddings, and probabilistic record linkage strategies
• Hands-on experience scaling headless browsers, managing proxy networks, solving CAPTCHAs, and working with complex web scraping frameworks
• Mastery of workflow automation tools and orchestrators (e.g., Apache Airflow, Prefect, n8n, Make, or temporal workflows)
• Strong architectural mindset with an obsession for error handling, data accuracy, and pipeline resilience
• Excellent communication skills to explain AI limitations, confidence scores, and strategic value to non-technical stakeholders
• Resourceful problem-solver capable of building workarounds for unstable APIs or challenging scraping targets
• Experience in high-volume, dynamic industries such as iGaming, FinTech, or E-commerce
• Familiarity with database technologies (SQL, ClickHouse, Redis, document databases) and message brokers (RabbitMQ, Kafka)
• Exposure to MLOps, LLM monitoring tools (e.g., LangSmith, Phoenix), and CI/CD for automation workflows