Sr. Manager - Data Engineer

Reposted 5 Hours Ago
Be an Early Applicant
Hyderabad, Telangana, IND
Hybrid
Mid level
Big Data • Fintech • Information Technology • Business Intelligence • Financial Services • Cybersecurity • Big Data Analytics
The Role
Manage a single software engineering team (5–10), drive sprint planning and delivery, provide technical oversight and architecture guidance, collaborate with product and business stakeholders, and perform people management including reviews, mentoring, hiring, and enforcing Agile/SDLC and quality practices. Hybrid role requiring in-office presence at least two days weekly.
Summary Generated by Built In

TransUnion's Job Applicant Privacy Notice

Team Overview

An M02 is responsible for managing a development team, ensuring delivery of software solutions, and aligning execution with business and product goals.
The role balances:
* People management
* Technical oversight
* Delivery accountability
Typically owns a single team of engineers This is a hybrid position and involves regular performance of job responsibilities virtually as well as in-person at an assigned TU office location for a minimum of two days a week.

Role Overview And Core Responsibilities

Team Leadership & Delivery
  • Lead a team of software and data engineers (typically 5–10 members).
  • Ensure timely and high-quality delivery of platform capabilities, data processing solutions, and feature enhancements.
  • Drive execution of large-scale distributed processing initiatives and data platform modernization efforts.
  • Foster a culture of engineering excellence, innovation, and accountability.
Technical Oversight
  • Provide technical leadership for large-scale distributed data processing systems built on Apache Spark.
  • Guide architecture, design, and implementation decisions for scalable, fault-tolerant, and high-performance data processing workloads.
  • Establish best practices for Spark application development, optimization, observability, and operational excellence.
  • Ensure adherence to coding standards, performance requirements, scalability guidelines, and security controls.
  • Lead troubleshooting and resolution of complex production issues involving Spark jobs, data pipelines, cluster performance, and resource management.
  • Drive optimization of Spark workloads through partitioning strategies, query tuning, caching, memory management, and efficient resource utilization.
Data Platform & Spark Engineering
  • Design and oversee development of batch and real-time data processing pipelines using Apache Spark.
  • Collaborate on architecture involving Spark,  Iceberg, Hive, Hadoop,  AWS EMR, AWS Glue, GCP Dataproc, BigQuery, and other modern cloud-native data platform technologies.
  • Ensure reliability, scalability, and operational efficiency of enterprise-scale data workflows.
  • Drive adoption of data engineering best practices, CI/CD automation, testing frameworks, and monitoring for Spark workloads.
  • Champion performance benchmarking, capacity planning, and cost optimization initiatives across data processing platforms.
Stakeholder Collaboration
  • Work closely with product managers, architects, data scientists, platform teams, and business stakeholders.
  • Translate business and data processing requirements into executable engineering plans.
  • Communicate delivery status, technical risks, architectural decisions, and mitigation plans to leadership.
  • Partner with cross-functional teams to deliver data-driven products and platform capabilities.
People Management
  • Conduct performance reviews, coaching, and regular feedback sessions.
  • Mentor engineers in distributed systems, Spark development, performance optimization, and software engineering best practices.
  • Build technical depth within the team through knowledge sharing and career development.
  • Support hiring, onboarding, and growth of engineering talent with expertise in big data and distributed computing.
Process & Quality Management
  • Ensure Agile, SDLC and engineering governance practices are followed.
  • Track delivery, quality, reliability, and operational metrics.
  • Improve engineering productivity through automation, standardization, and platform improvements.
  • Drive continuous improvement initiatives focused on system reliability, performance, and customer satisfaction.

Required Knowledge And Experiences

Required Knowledge and Experience
Technical Expertise
  • Strong hands-on experience with Apache Spark (Spark Core, Spark SQL, Structured Streaming, DataFrames, Dataset APIs).
  • Proven experience building and operating large-scale distributed data processing systems.
  • Strong understanding of Spark performance tuning, partitioning strategies, joins, shuffles, caching, memory management, and resource optimization.
  • Experience with one or more programming languages such as Scala, Java, or Python.
  • Experience with modern data ecosystems including Hadoop, Hive, Iceberg, AWS EMR, AWS Glue, GCP Dataproc, BigQuery, or equivalent technologies.
  • Experience designing and maintaining batch and streaming data pipelines.
  • Strong understanding of distributed systems, cloud-native architectures, and scalability principles.
  • Hands-on experience deploying and managing Spark workloads on AWS and/or GCP.
Leadership Experience
  • Proven experience leading engineering teams and delivering complex technical initiatives.
  • Ability to balance hands-on technical leadership with people management responsibilities.
  • Experience driving cross-functional collaboration and influencing technical direction.
  • Demonstrated success managing technical roadmaps, execution planning, and delivery commitments.
Scope & Positioning
  • Mid-level management role positioned between Senior Engineer/Technical Lead and Senior Engineering Manager.
  • Responsible for engineering execution excellence and technical leadership of Spark-based data processing platforms.
  • Acts as a key decision-maker for delivery, architecture, operational reliability, and team growth.
Preferred Qualifications
  • Experience with AWS services such as EMR, Glue, S3, Lambda, EKS, ECS, Step Functions, and CloudWatch.
  • Experience with GCP services such as Dataproc, BigQuery, Cloud Storage, Dataflow, Pub/Sub, and Composer.
  • Experience managing terabyte-to-petabyte scale data processing environments.
  • Experience with Spark Structured Streaming, real-time analytics, and event-driven data processing.
  • Contributions to Spark optimization, platform engineering, or open-source data ecosystem projects are a plus.

TransUnion Overview:

At TransUnion, we encourage and are committed to creating a real, positive impact and shared sense of purpose within our Workforce for Good, which empowers our people to grow, innovate and contribute to a better future for our communities and customers. We strive to build an environment where our associates are in the driver’s seat of their professional development— while having access to help along the way. We recognize that success comes when our associates thrive both professionally and personally; that’s why we prioritize work/life flexibility and offer resources for our teams across the globe to collaborate and drive excellence.


Be a part of our Workforce for Good – you’ll work with great people, pioneering products and cutting-edge technology.


TransUnion Job Title


Manager II, Applications Development

Skills Required

  • Lead a team of software engineers (typically 5-10 members)
  • Own sprint planning, backlog execution, and delivery commitments
  • Ensure Agile/SDLC practices are followed and track delivery metrics
  • Provide technical oversight, guide architecture and design decisions
  • Conduct performance reviews, mentor and develop engineering talent
  • Support hiring and onboarding activities
  • Communicate progress, risks, and delivery status to stakeholders
  • Work on-site at assigned TransUnion office a minimum of two days per week (hybrid)

What the Team is Saying

Patrick
Tiana
Jason
Lauren
TC
Jay
Aayushi
Paul
Alex Barnett
Sheetal Wathare
TransUnion
TransUnion

TransUnion Compensation & Benefits Highlights

  • Healthcare Strength Healthcare is described as comprehensive with day-one medical, dental and vision coverage, plus HSA/FSA options and expanded mental-health support via Spring Health. Feedback suggests coverage depth and immediate eligibility are notable strengths.
  • Parental & Family Support Family support includes paid parental leave with a gradual return, adoption and caregiver assistance, and backup care through a Care@Work membership. External recognition highlights inclusive offerings such as fertility support and bereavement provisions.
  • Leave & Time Off Breadth Time-off programs feature flexible time off, paid holidays, two global wellness days, and paid volunteer time. Feedback suggests these leave elements contribute meaningfully to work-life balance.

TransUnion Insights

Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
HQ: Chicago, IL
13,000 Employees
Year Founded: 1968

What We Do

TransUnion is a global information and insights company that makes trust possible by ensuring that each consumer is reliably and safely represented in the marketplace. We do this by having an accurate and comprehensive picture of each person. This picture is grounded in our legacy as a credit reporting agency which enables us to tap into both credit and public record data; our data fusion methodology that helps us link, match and tap into the awesome combined power of that data; and our knowledgeable and passionate team, who stewards the information with expertise, and in accordance with local legislation around the world. Because of our work, organizations can better understand consumers in order to make more informed decisions, and earn their trust through great, personalized experiences, and the proactive extension of the right opportunities, tools and offers. In turn, consumers can be confident that their data identities will result in the opportunities they deserve. We make trust possible, so businesses and consumers can transact with confidence and achieve great things. We call this Information for Good®—it’s our purpose, and what drives us every day.

Why Work With Us

Our culture is welcoming, energetic and innovative. There’s an overall synergy that flows throughout TransUnion, creating a sense of unity in knowing that we’re all working to achieve the same overall goal. We’re dedicated to providing opportunities for our people to get involved and stay connected with their colleagues across the globe.

Gallery

Gallery
Gallery
Gallery
Gallery
Gallery
Gallery
Gallery
Gallery
Gallery
Gallery

TransUnion Teams

Team
Invested in Tech Teams
About our Teams

TransUnion Offices

Hybrid Workspace

Employees engage in a combination of remote and on-site work.

Typical time on-site: Flexible
HQChicago, IL
MX
Amsterdam, NL
Bengaluru, IN
Boca Raton, FL
Bogotá, Colombia
Burlington, ON
South Africa
Cerqueira César, Sao Paulo
Chennai, IN
Cherry Hill, NJ
Cork, County Cork
Crum Lynne, PA
Denver, CO
Greenwood Village, CO
Guaynabo, PR
Gurugram, IN
Hamburg, DE
Hyderabad, IN
Johannesburg, ZA
TransUnion UK Head Office
London, GB
Louisville, KY
Madrid, ES
Makati, PH
Mumbai, IN
New York, NY
Pune, IN
Reston, VA
San Luis Obispo, CA
Santiago, CL
Sydney, NSW
Toronto, ON
Ulloa, La Aurora
Washington, US
White Plains, NY
Learn more

Similar Jobs

TransUnion Logo TransUnion

Architect

Big Data • Fintech • Information Technology • Business Intelligence • Financial Services • Cybersecurity • Big Data Analytics
Hybrid
Hyderabad, Telangana, IND
13000 Employees

TransUnion Logo TransUnion

Senior Consultant

Big Data • Fintech • Information Technology • Business Intelligence • Financial Services • Cybersecurity • Big Data Analytics
Hybrid
2 Locations
13000 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account