DISTINGUISHED, SOFTWARE ENGINEER

Role : Distinguished Engineer, Software Engineering  

About The Team  

Run Time Platforms (RTP), a division of Global Technology Platform (GTP), is dedicated to developing products and platforms that empower application teams to efficiently deploy and manage runtime environments. 

The Runtime Platforms pillar within Global Tech Platforms (GTP) owns the foundational infrastructure and developer-facing platform services—Expo, Pulse, CCM, and Secrets— that every engineering team at Walmart depends on to build, deploy, and run their software safely and at hyper-scale. This pillar is also responsible for an Enterprise Feature Store platform that will be used by US OMNI customers spanning search, personalization, Ads, Marketplace, Fraud, Payments and more. We are talking about a platform that needs to provide and adhere to an SLA p95 of 10ms for online feature retrieval. 

What you'll do: 

Technical Leadership & Strategy 

Define and own the short term and long-term technical vision for the ML data and feature store platform, spanning offline pipelines, online serving, and the infrastructure that connects them. 

Identify and systematically retire architectural debt across the platform — prioritising by production impact, latency, reliability, and engineering velocity. 

Serve as the primary technical authority in cross-functional design reviews involving ML engineering, data engineering, platform, and product teams. 

Influence the broader engineering organisation through published architectural standards, internal RFCs, and mentorship of senior and staff engineers. 

Platform Architecture & Design 

Architect end-to-end feature store data systems — from data ingestion and feature computation through materialization into online stores and delivery to model inference. 

Define and Design high performant, scalable, reliable Data and Control Mgmt planes for both online and offline data processing pipelines. 

Define and govern data quality standards, SLOs, and observability frameworks across the platform (feature freshness, materialization lag, training-serving skew detection, drift alerting). 

Drive platform decisions around schema design, storage backend selection, consistency trade-offs, and multi-cloud data replication strategies. 

Offline Data Systems (Big Data Ecosystem) 

Design and optimise large-scale batch data pipelines using Spark, Hive, or equivalent — covering feature computation, aggregation, transformation, and materialization into both offline and online stores. 

Define offline store schema and partitioning strategies (e.g., Parquet/ORC on cloud object storage) optimised for point-in-time correct training dataset generation. 

Establish pipeline reliability and observability standards: data freshness SLOs, quality checks at ingestion, deduplication, late-arrival handling, and lineage tracking. 

Provide guidance on orchestration and scheduling of batch workloads (Airflow or equivalent), including dependency management, SLA monitoring, and failure recovery. 

Online Data Systems (High-Throughput Serving) 

Architect and optimise online feature serving infrastructure targeting high QPM at consistently low latency (p99 in single-digit milliseconds), across multi-region, multi-cloud deployments. 

Design data access layers over low-latency stores (Cassandra, Redis, Spanner, or equivalent) — including read path optimization, caching strategy, connection pool management, and circuit breaking. 

Define online serving reliability patterns: multi-DC failover, graceful degradation, bulkhead isolation, and zero-downtime schema migrations. 

Establish capacity planning and load testing practices for online stores; translate traffic forecasts into storage and compute provisioning recommendations. 

SDK Design & Developer Experience 

Lead the design and evolution of the platform's Java and Python SDKs — from the current thick SDK (heavy direct data store coupling) to a thin SDK that handles only routing table fetching, feature name parsing, and request dispatch. 

Own the Java SDK's API contract, versioning strategy, backward compatibility guarantees, and deprecation policy (including serializer migration across major versions). 

Define ergonomic, well-documented Python SDK interfaces for data science and ML engineering consumers, ensuring consistency with the Java SDK's service contract. 

Establish SDK release governance: semantic versioning, changelog discipline, dependency management, and multi-version co-existence strategies. 

Key responsibilities for this role include: 

What you'll bring: 

Software Engineering Fundamentals  

10+ years of production software development experience, with demonstrable ownership of complex, distributed systems. 

Java (Expert): Concurrency primitives (AtomicReference, CAS, CompletableFuture, ForkJoin), SDK/library design, API versioning, backward compatibility guarantees, and dependency management. 

Python (Strong): Production-grade library and SDK design for data science or ML engineering consumers; idiomatic async client patterns; packaging and dependency management. 

Strong data modelling instincts across relational, wide-column, key-value, and columnar storage paradigms. 

Experience with observability tooling: distributed tracing (OpenTelemetry), structured logging, metrics (Prometheus/Grafana or equivalent), and SLO-based alerting. 

Cloud-Native & Infrastructure — Required 

Experience deploying and operating production services on Kubernetes at scale (GKE, AKS, or equivalent). 

Familiarity with GCP and/or Azure infrastructure: managed storage services, multi-cloud networking, IAM/RBAC, and cloud-native observability tooling. 

Understanding of CI/CD pipeline design for data and ML workloads — including pipeline testing strategies, staged rollouts, and data migration patterns. 

High-Throughput Online Systems — Required for candidates from an online/serving background 

Proven experience designing and operating services at high QPM with p99 latency targets in the low-millisecond range, across distributed, multi-region infrastructure. 

Deep expertise in low-latency data stores: 

• Cassandra: Data modelling, partition key design, consistency level trade-offs, compaction strategies, multi-datacenter replication, driver internals (DataStax Java Driver). 

• Redis: Eviction and expiry policies, cluster mode, pipelining, client-side connection management (Jedis, Lettuce). 

Expert-level understanding of distributed systems fundamentals: CAP theorem, consistency models, replication topologies, partitioning strategies, and eventual consistency trade-offs. 

Deep expertise in reliability engineering for online systems: circuit breakers, exponential backoff with jitter, bulkhead patterns, rate limiting, local vs. remote DC failover, and graceful degradation under store-side pressure. 

Experience with gRPC service design at scale: service contract definition, proto schema evolution, deadline propagation, streaming patterns, and observability (per-RPC latency histograms, error rate tracking).