Principal Backend Engineer

Job Position: Principal Backend Engineer 
Experience: 12+ Years 
Work Mode: Remote (Full-Time) - Minimum 4-hour overlap with US time zones

ROLE OVERVIEW 
We are seeking a Principal Backend Engineer to own the scalability, reliability, and architectural evolution of an AI platform as it transitions from a prototype to a production-grade platform. This is a high-impact, high-ownership role at the intersection of distributed systems engineering and applied AI. 

KEY RESPONSIBILITIES 
● Platform Architecture & Scalability: Own end-to-end backend architecture for the AI platform; design for multi-tenant, high-throughput extraction workloads   
● Extraction Pipeline Engineering: Architect and evolve AI pipelines, define pipeline abstractions, registry patterns, and execution strategies that balance accuracy, latency, and LLM API cost. 
● Multi-Provider LLM Abstraction: Enhance and harden the unified LLM provider layer ; implement multi-key round-robin pooling, structured output schemas, tool-calling protocols, and provider failover logic.
 ● Async Worker & Queue Architecture: Design the distributed worker system for heavy pipeline execution observable queue with dead-letter handling, retry policies, and backpressure management. 
● AWS Production Deployment: Lead the AWS deployment architecture (ECS/EC2, RDS PostgreSQL, S3, Bedrock, ALB, Secrets Manager, CloudWatch); define IaC, blue-green deployment strategies, and ensure the platform meets security, compliance, and data residency requirements. 
● API Design & Governance: Define and enforce API design standards (OpenAPI 3.1, spec-first, versioning, deprecation); own the 25+ FastAPI endpoints, request/response schema evolution, and backward compatibility guarantees.
 ● Data Architecture: Own the PostgreSQL schema for extraction metadata (holes, dimensions, GD&T, title blocks, notes, extraction traces, batches); design indexing strategies, multi-tenant isolation, and efficient querying patterns for the document review and dataset export workflows. 
● Security & Compliance: Implement secure service-to-service communication, secrets management via AWS Secrets Manager, IAM role-based access for Bedrock and S3, and CORS/auth policies   
● Observability & Reliability: Instrument the platform with structured logging, distributed tracing (AWS X-Ray / OpenTelemetry), and CloudWatch alarms; define SLOs for pipeline throughput, LLM call latency, and extraction accuracy.\
 ● Engineering Leadership: Mentor senior engineers, conduct architecture and design reviews, set coding standards, and drive the technical roadmap for the enterprise. 

REQUIRED SKILLS & EXPERIENCE
● 12+ years of backend engineering experience with a strong focus on distributed systems and production-grade platform design. ● Expert-level Python proficiency; deep hands-on experience with FastAPI, SQLAlchemy, Pydantic v2, and async/concurrent programming (asyncio, ThreadPoolExecutor). 
● Proven experience designing and operating microservices at scale — including service decomposition, inter-service communication, and failure isolation strategies. 
● Strong PostgreSQL expertise: schema design, indexing, query optimization, multi-tenant isolation (RLS or schema-per-tenant), and migration management.
● Hands-on AWS production experience: EC2/ECS, RDS, S3, ALB, IAM, Secrets Manager, CloudWatch, and ideally Amazon Bedrock. 
● Deep understanding of event-driven and async architectures: message queues, polling workers, idempotency guarantees, retry strategies, and backpressure handling. 
● Experience integrating LLM APIs (any of: Anthropic Claude, OpenAI, Google Gemini, Mistral, AWS Bedrock) in production — including rate limiting, structured output enforcement, and multi-provider failover.
● Experience with container-based deployments using Docker and Docker Compose; familiarity with ECS task definitions and service orchestration. 
● Strong API lifecycle management skills: OpenAPI 3.1, versioning, backward compatibility, deprecation policies, and governance frameworks. \
● Practical knowledge of distributed system patterns: sagas, circuit breakers, idempotency keys, at-least-once delivery, and graceful degradation. 
● Security engineering fundamentals: OAuth2, JWT, RBAC, CORS, secrets management, and cloud IAM role design. 
● Prior experience building developer platforms, internal tooling, or AI/ML serving platforms at large-scale organizations.

 PREFERRED / GOOD TO HAVE 
● Direct experience building AI/LLM extraction pipelines — structured output, tool-calling, multi-agent orchestration (LangGraph, CrewAI, or custom frameworks). 
● Familiarity with agentic pipeline patterns: planner-executor swarms, multi-round tool-use loops, shared notepad patterns, and confidence-based validation. 
● Experience with Amazon Bedrock cross-region inference and IAM-based model access (Amazon Nova, Claude via Bedrock). 
● Knowledge of ML model serving infrastructure: HuggingFace Transformers, PyTorch model loading, YOLO/object detection integration, ChromaDB vector stores.
 ● Experience with PDF processing pipelines: rasterization (PyMuPDF), coordinate normalization, multi-page extraction, and bounding box annotation workflows. 
● Familiarity with service mesh (Istio/Linkerd), Kubernetes, or EKS for cloud-native deployment evolution. 
● Experience with workflow orchestration frameworks (Temporal, Airflow, or equivalent) for long-running, stateful extraction jobs. 
● Experience in high-scale SaaS platforms with strict SLAs, multi-region deployments, and enterprise security requirements.

 SUMMARY This role is for a Principal-level engineer who can own backend architecture at the intersection of large-scale distributed systems and applied AI. The ideal candidate has built production platforms that serve real enterprise workloads, has deep LLM integration experience, and can lead a team while staying hands-on in the most critical technical decisions.