Senior Data Engineer – Capacity Planning & Forecasting

Posted Yesterday
Hiring Remotely in Austin, TX, USA
In-Office or Remote
Senior level
Information Technology • Consulting
The Role
Build and maintain scalable data pipelines, analytical models, dashboards, and reporting for infrastructure capacity, utilization, cloud costs, and financial data. Develop demand forecasts and scenario analyses for GPU, TPU, CPU, memory, and storage requirements. Identify bottlenecks, underutilization, and cost optimization opportunities while partnering with Infrastructure Engineering, Finance, Procurement, Data Science, and business teams.
Summary Generated by Built In
Senior Data Engineer – Capacity Planning & Forecasting

Type: Remote | Brazil, Mexico, Chile, or Colombia
Coverage: Pacific Hours (8:00 AM – 5:00 PM PST)
Experience: 10+ years
Organization: Data Platform / Infrastructure Engineering

Job Description:

We are looking for an experienced Data Engineer to join our Data Platform organization, focusing on infrastructure capacity planning, forecasting, utilization analytics, and cost optimization.

The ideal candidate will be responsible for designing, developing, and maintaining scalable data pipelines and analytical data models that integrate infrastructure telemetry, cloud consumption, workload demand, and financial data.

This role will play a critical part in enabling data-driven infrastructure investment decisions by providing visibility into compute, GPU, TPU, storage, and cloud resource utilization, forecasting future capacity requirements, and identifying cost optimization opportunities.

The candidate will collaborate closely with Infrastructure Engineering, Data Science, Finance, Procurement, and Business Operations teams to build reliable data products that support strategic capacity planning.

Key Responsibilities:Data Engineering & Pipeline Development:

Design, develop, and maintain scalable ETL/ELT data pipelines to ingest, process, and transform large volumes of infrastructure and cloud consumption data.

Integrate data from multiple sources, including cloud billing systems, infrastructure telemetry, workload management platforms, and financial systems.

Develop optimized data models and analytical datasets for infrastructure utilization, capacity tracking, and cost reporting.

Implement data validation, reconciliation, monitoring, and quality checks to ensure data accuracy and reliability.

Optimize data pipelines for performance, scalability, and operational efficiency.

Capacity Planning & Forecasting:

Develop data-driven solutions to monitor and forecast infrastructure capacity requirements.

Analyze historical and current utilization trends across GPU, TPU, CPU, memory, and storage resources.

Build forecasting models to predict future compute demand based on workload growth and business requirements.

Identify capacity constraints, resource bottlenecks, underutilization, and potential infrastructure risks.

Support capacity planning through demand forecasting, scenario analysis, and capacity optimization.

Develop dashboards and reporting solutions to track available capacity, committed capacity, utilization, and projected demand.

Cloud Cost & Infrastructure Analytics:

Analyze cloud infrastructure consumption and billing data across AWS, GCP, Azure, or other cloud platforms.

Develop cost attribution models to understand infrastructure spending across workloads, teams, and business units.

Calculate key infrastructure cost metrics, including:

Cost per GPU hour

Cost per CPU hour

Cost per workload/job

Cost per inference request

Cost per million tokens

Identify cost optimization opportunities, including idle resources, underutilized infrastructure, and inefficient capacity commitments.

Partner with Finance and Procurement teams to reconcile infrastructure costs and support financial planning.

Data Modeling & Analytics:

Design and maintain dimensional and analytical data models for infrastructure capacity and financial reporting.

Develop reusable datasets and metrics to support operational and strategic decision-making.

Perform root-cause analysis of utilization anomalies, cost fluctuations, and capacity deviations.

Build automated reports and dashboards to provide actionable insights to stakeholders.

Ensure consistency between operational telemetry, consumption records, and financial reporting.

Stakeholder Collaboration:

Work closely with Infrastructure Engineering teams to understand workload patterns, infrastructure architecture, and capacity requirements.

Collaborate with Finance and Procurement teams on infrastructure budgeting, cloud commitments, and cost forecasting.

Partner with Data Science and Machine Learning teams to understand AI/ML workload requirements and compute consumption patterns.

Translate complex technical and financial data into meaningful insights and recommendations for leadership.

Qualifications:Required Technical Skills:

Programming & Data Processing

Strong proficiency in SQL, including complex queries, joins, window functions, and query optimization.

Proficiency in Python for data processing, automation, and analytical workflows.

Experience with Pandas, PySpark, or similar data processing frameworks.

Data Engineering

Hands-on experience building scalable ETL/ELT pipelines.

Strong understanding of data warehousing, data modeling, and distributed data processing.

Experience with Apache Spark, Airflow, Kafka, Trino, or similar technologies.

Knowledge of data quality frameworks, pipeline monitoring, and data governance.

Cloud & Infrastructure

Experience working with AWS, Google Cloud Platform, or Microsoft Azure.

Understanding of cloud billing, resource utilization, and consumption datasets.

Knowledge of compute, storage, networking, and infrastructure resource metrics.

Familiarity with GPU/TPU infrastructure and AI/ML workload consumption is desirable.

Capacity Planning & Forecasting

Understanding of capacity management, resource utilization, and demand forecasting.

Experience analyzing historical trends and predicting future resource requirements.

Knowledge of time-series forecasting, statistical modeling, and scenario analysis.

Ability to identify infrastructure bottlenecks and resource optimization opportunities.

Visualization & Reporting

Experience with Tableau, Power BI, Looker, or similar visualization tools.

Ability to develop operational dashboards and executive-level reporting.

Preferred Qualifications:

Experience in cloud infrastructure analytics or FinOps.

Exposure to GPU/TPU capacity planning and AI/ML infrastructure.

Understanding of machine learning inference workloads and compute economics.

Experience working with large-scale infrastructure telemetry datasets.

Knowledge of infrastructure cost allocation and unit economics.

Experience supporting financial forecasting, budgeting, or procurement decisions.

Key Performance Indicators (KPIs):

Data pipeline reliability and successful execution rates.

Accuracy and completeness of infrastructure utilization data.

Accuracy of capacity demand forecasts.

Visibility into infrastructure capacity and resource consumption.

Identification of cloud cost optimization opportunities.

Reduction in manual reporting and reconciliation efforts.

Timeliness and accuracy of capacity and financial reporting.

Educational Qualifications:

Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, Statistics, Mathematics, or a related quantitative discipline.

Equivalent practical experience in Data Engineering, Infrastructure Analytics, or Capacity Planning will also be considered.

Ideal Candidate Profile:

The ideal candidate is a technically strong Data Engineer with an analytical mindset and an understanding of infrastructure economics. They should be comfortable working with large-scale datasets, developing reliable data pipelines, analyzing resource utilization, and translating complex data into actionable capacity and cost optimization insights.

The candidate should demonstrate strong problem-solving skills, ownership, attention to detail, and the ability to collaborate across engineering and business teams.

Core Competencies: Data Engineering | SQL | Python | Cloud Analytics | Capacity Planning | Demand Forecasting | Infrastructure Optimization | FinOps | Data Modeling | Business Analytics

Skills Required

  • 10+ years of experience in data engineering, infrastructure analytics, or capacity planning
  • Strong SQL proficiency, including complex queries, joins, window functions, and query optimization
  • Python proficiency for data processing, automation, and analytical workflows
  • Experience with Pandas, PySpark, or similar data processing frameworks
  • Hands-on experience building scalable ETL/ELT pipelines
  • Strong understanding of data warehousing, data modeling, and distributed data processing
  • Experience with Apache Spark, Airflow, Kafka, Trino, or similar technologies
  • Knowledge of data quality frameworks, pipeline monitoring, and data governance
  • Experience with AWS, Google Cloud Platform, or Microsoft Azure
  • Understanding of cloud billing, resource utilization, and consumption datasets
  • Knowledge of compute, storage, networking, and infrastructure resource metrics
  • Understanding of capacity management, resource utilization, and demand forecasting
  • Experience analyzing historical trends and predicting future resource requirements
  • Knowledge of time-series forecasting, statistical modeling, and scenario analysis
  • Ability to identify infrastructure bottlenecks and resource optimization opportunities
  • Experience with Tableau, Power BI, Looker, or similar visualization tools
  • Ability to develop operational dashboards and executive-level reporting
  • Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, Statistics, Mathematics, or a related quantitative discipline, or equivalent practical experience
  • Experience in cloud infrastructure analytics or FinOps
  • Exposure to GPU/TPU capacity planning and AI/ML infrastructure
  • Understanding of machine learning inference workloads and compute economics
  • Experience working with large-scale infrastructure telemetry datasets
  • Knowledge of infrastructure cost allocation and unit economics
  • Experience supporting financial forecasting, budgeting, or procurement decisions
Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
HQ: Austin, Texas
87 Employees
Year Founded: 2007

What We Do

Virtasant is a fully-remote and globally distributed technology and business services company that brings the breadth and capability of more than 4,000 technology professionals in 130 countries. We started working in the public cloud in 2007, and over the last 13 years, our team has built or re-platformed over 200 applications, with over 600 million lines of code currently under management. In this time, our team has developed proprietary software and methodologies that help us quickly and accurately identify opportunities to maximize the speed, scale and cost advantages that come with native cloud computing. Virtasant combines industry-leading cloud specialization with a deep bench of global experts across every aspect of the product lifecycle to deliver superior results to our clients. Virtasant offers services including cloud assessments and migrations, custom product development, information management, machine learning, remote team management, application modernization, full-lifecycle technology outsourcing, and more

Similar Jobs

Qualtrics Logo Qualtrics

Enterprise Account Executive

Artificial Intelligence • HR Tech • Information Technology • Software • Business Intelligence
Remote
United States
7500 Employees
110K-144K Annually

Qualtrics Logo Qualtrics

Senior Industry XM Advisor - Financial Services

Artificial Intelligence • HR Tech • Information Technology • Software • Business Intelligence
Remote
United States
7500 Employees
94K-130K Annually

Qualtrics Logo Qualtrics

Senior Consultant

Artificial Intelligence • HR Tech • Information Technology • Software • Business Intelligence
Remote
United States
7500 Employees
126K-153K Annually

Qualtrics Logo Qualtrics

Front Line Sales Leader - Life Sciences - Enterprise

Artificial Intelligence • HR Tech • Information Technology • Software • Business Intelligence
Remote
United States
7500 Employees
120K-145K Annually

Similar Companies Hiring

NODA AI Thumbnail
Artificial Intelligence • Information Technology • Software • Cybersecurity • Defense • Defense Technology
Austin, TX
54 Employees
Golden Pet Brands Thumbnail
Digital Media • eCommerce • Information Technology • Marketing Tech • Pet • Retail • Social Media
El Segundo, California
567 Employees
RapidScale Thumbnail
Cloud • Information Technology • Consulting
Raleigh, NC
600 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account