Jobs at DATAECONOMY

Build scalable ETL/ELT pipelines in Databricks using PySpark, Spark SQL, Delta Lake, Delta Live Tables, Workflows, and Unity Catalog. Ingest batch and streaming data across medallion architecture layers, develop transformations and data quality rules, and implement orchestration, monitoring, alerting, and automation. The role requires cloud data engineering, data modeling, performance tuning, CI/CD, and strong knowledge of lakehouse architecture, governance, and distributed systems.

Designs and builds secure, production-grade Generative AI and agentic AI applications using Python, LLMs, cloud services, and AI frameworks. Responsibilities include developing scalable AI pipelines, integrating Databricks data with AWS services, applying RAG and vector databases, evaluating models, implementing guardrails, researching emerging tools, improving application infrastructure, and guiding team members.

Build scalable production-grade ETL/ELT pipelines on Databricks using PySpark, Spark SQL, Delta Lake, Delta Live Tables, and Workflows. Develop data ingestion, transformations, quality rules, orchestration, monitoring, and automation across Bronze, Silver, and Gold layers. Apply dimensional data modeling, streaming, cloud platforms, CI/CD, governance, security, performance tuning, and lakehouse architecture. Explain technical concepts and architecture decisions using real project examples.

Designs and leads enterprise Snowflake data architectures, cloud data platforms, ELT/ETL pipelines, and AI-powered applications. Builds RAG, semantic search, Cortex Agents, vector embedding, and Cortex AI Function solutions using Snowpark, Python, SQL, and dbt. Establishes secure, governed, and cost-optimized data environments, mentors engineering teams, leads architecture reviews, and partners with stakeholders on scalable data and generative AI solutions.

Design and operate AWS production-serving infrastructure for fine-tuned LLMs. Deploy models using vLLM, TensorRT-LLM, or Triton; optimize inference and GPU utilization; implement shadow, canary, A/B, staged rollout, and automated rollback workflows; and build monitoring for latency, throughput, quality, and cost. The role also covers auto-scaling, high availability, Kubernetes-based ML workloads, continuous batching, quantization, and KV-cache management.

Build and maintain AWS-based pipelines that transform raw production traces into high-quality LLM training datasets. Responsibilities include ingestion, deduplication, format conversion, quality filtering, identity-aware train/test splitting, schema validation, sampling, and continuous retraining-data processing. The role requires strong Python data engineering skills, large-scale JSONL processing, Hugging Face Datasets, Arrow-based storage, AWS services, and familiarity with conversational and tokenizer-specific data formats.

Fine-tune open-source LLMs from experimentation through production on AWS. Run SFT and alignment methods including GRPO and DPO, debug distributed multi-GPU training, resolve loss and memory issues, and collaborate with evaluation and data engineering teams to improve accuracy. Scale models from 8B to 70B parameters based on offline metrics.

Develop and maintain enterprise-grade Java applications using Spring Boot, Hibernate/JPA, REST APIs, and microservices architecture. Build responsive interfaces with React or Angular, integrate databases and third-party services, optimize performance, implement security, and automate CI/CD pipelines. Troubleshoot production issues, participate in code reviews, mentor junior developers, and collaborate with cross-functional teams to deliver scalable, reliable solutions.

Design, develop, and optimize scalable Databricks data engineering solutions. Build ETL/ELT pipelines, manage clusters, implement Lakehouse, Delta Lake, Delta Live Tables, Medallion Architecture, and Unity Catalog. Optimize Spark jobs for performance and cost, monitor pipelines, resolve data quality issues, and apply DevOps and CI/CD practices while collaborating with architects, analysts, and stakeholders.

Build and manage the full lifecycle for self-hosted LLMs, including training data pipelines, SFT and DPO fine-tuning, evaluation, benchmarking, quantization, deployment, monitoring, and continuous improvement. Deploy models on AWS GPU infrastructure and SageMaker, automate ML CI/CD, and manage Docker and Kubernetes workloads on EKS. Implement A/B, canary, and shadow deployments with automated promotion and rollback based on performance and operational metrics. Collaborate with data science, ML engineering, and DevOps teams.

Analyze insurance policy, premium, claims, billing, and financial data; build KPI dashboards and reports; write and optimize SQL; validate and model datasets; identify trends, anomalies, fraud indicators, and leakage; and translate business questions into clear recommendations for underwriting, actuarial, claims, finance, and other stakeholders. The role supports pricing, reserving, portfolio management, retention, and loss-ratio analyses.

Lead a team building production-grade Generative AI and Agentic AI solutions for banking and financial services. Design RAG pipelines, multi-agent workflows, cloud-native AWS architectures, evaluation and observability frameworks, and reusable GenAI accelerators. Contribute hands-on Python code, guide architecture and code reviews, troubleshoot production issues, document solutions, and ensure security, scalability, compliance, and alignment with BFSI requirements.

Lead end-to-end delivery of data-centric technical projects, coordinating engineering teams and business stakeholders. Manage planning, timelines, risks, resources, dependencies, and communications using Agile methodologies. Oversee ETL pipelines, data models, integrations, code reviews, and data engineering practices while translating business requirements into technical plans. Collaborate with Product, Data Science, DevOps, and QA teams, provide Python development or debugging support, and track project metrics and stakeholder reports.

Execute end-to-end SharePoint migrations using ShareGate, including site discovery, readiness assessments, migration waves, cutovers, validation, permissions mapping, troubleshooting, and post-migration support. Manage multiple concurrent migrations, track timelines, risks, and issues, maintain documentation, and communicate with site owners, information stewards, business users, and project leads. Support content cleanup, stakeholder concerns, end-user validation, and escalation of complex migration issues.

Analyze investment and accounting data supporting Asia data modernization and IBOR/ABOR initiatives. Define data requirements, business rules, governance artifacts, and quality checks; perform reconciliations and root-cause analysis; validate Power BI and enterprise reporting datasets; and translate business needs for engineering, BI, product, investment, finance, and risk stakeholders.

Design and engineer Snowflake solutions using Cortex AI capabilities such as RAG, semantic search, and multi-agent systems. Apply capital markets expertise while communicating with and managing clients. Nice-to-have experience includes PostgreSQL, Talend, Azure or AWS, Denodo, Iceberg tables, and migration from Oracle or DB2 to Snowflake.

Build and optimize Starburst-based data pipelines and federated queries, integrate databases, data lakes, and warehouses, and ensure performance, reliability, and governance. Collaborate with analytics, AI/ML, and business teams to deliver scalable data platforms and solutions.

Develop and maintain scalable PySpark data pipelines and ETL processes, write optimized SQL for extraction and transformation, manage large structured and unstructured datasets, ensure data quality, resolve performance bottlenecks, and collaborate with cross-functional teams. The role also involves code reviews and applying data engineering best practices. AWS, Airflow, Hadoop, and modern data lake platform experience are preferred.

Design, build, and maintain scalable ETL/ELT pipelines, data platforms, warehouses, lakes, and models. Integrate structured and unstructured data, optimize distributed processing and SQL performance, and ensure data quality, governance, security, lineage, and observability. Lead cloud migrations, modernization, resilience, and incident response initiatives while collaborating with engineering, analytics, data science, risk, compliance, and product teams. Translate business requirements into reliable data solutions and mentor engineers.

Manage enterprise application releases across development, QA, and production environments, including weekly releases and emergency patches. Support Kubernetes and Red Hat OpenShift platforms, monitor cluster health, troubleshoot production issues, and maintain CI/CD pipelines. Configure environments using YAML and Bash/Shell automation, address security vulnerabilities, coordinate with development, QA, and infrastructure teams, and participate in incident management, root cause analysis, and continuous improvement.