Software Engineer II - Tester (Big Data & Cloudera Ecosystem)

Our Purpose
Mastercard powers economies and empowers people in 200+ countries and territories worldwide. Together with our customers, we're helping build a sustainable economy where everyone can prosper. We support a wide range of digital payments choices, making transactions secure, simple, smart and accessible. Our technology and innovation, partnerships and networks combine to deliver a unique set of products and services that help people, businesses and governments realize their greatest potential.
Title and Summary
Software Engineer II - Tester (Big Data & Cloudera Ecosystem)
Overview
Are you excited to work at the intersection of distributed systems, big-data platforms, and high-quality engineering? Do you enjoy validating complex, large-scale data pipelines and ensuring that mission-critical systems perform flawlessly?
We are seeking a Software Engineer II - Tester to join our Big Data Engineering Quality team. You will be responsible for testing and validating data engineering applications built across the Cloudera Manager ecosystem, including Kafka, Spark, HBase, Hive, Solr, NiFi, Hue, Livy, Unravel, and internal REST APIs.
You will design and maintain automation frameworks using any of the following languages: Python, Java, Scala, or Bash, and ensure full end-to-end coverage for ingestion, transformation, orchestration, and storage workflows.
Primary Responsibilities
1. Big Data Platform & Application Testing• Test and validate distributed data engineering applications across Cloudera Manager components: Kafka, Spark, HBase, Hive, Solr, NiFi, Hue, Livy, Unravel.• Test Apache NiFi pipelines including processors, flow configurations, data lineage, provenance, NiFi registry flows, and integration with downstream systems.• Validate ingestion pipelines, streaming jobs, batch ETL processes, orchestration flows, and REST APIs.• Perform system, integration, functional, regression, and performance testing across large-scale data systems.
2. Test Automation & Framework Development• Build robust automation frameworks using any of the following languages: Python, Java, Scala, or Bash, based on the codebase under test.• Develop automation for Spark job validation, Kafka event checks, HBase/Hive schema validation, NiFi flow output validation, and end-to-end data pipeline coverage.• Create automated tests for REST APIs, backend services, schema checks, and data correctness.• Build and maintain UI automation using Selenium for dashboards such as Hue, Livy UI and other internal tools.
3. CI/CD & DevOps Integration• Maintain and enhance Jenkins CI/CD pipelines to support automated test execution and continuous integration.• Leverage Docker or similar runtime environments to execute tests at scale.• Integrate automation suites into the build/release pipeline for seamless promotion and deployment.
4. Data Validation & Quality Assurance• Validate SQL queries, HBase tables, Hive datasets, Kafka event streams, NiFi flow files, and NoSQL outputs.• Ensure data accuracy, schema integrity, SLA compliance, and end-to-end pipeline reliability.• Conduct deep validation of NiFi processors (PutFile, GetKafka, MergeContent, RouteOnAttribute, ExecuteScript, etc.) and downstream integrations.
5. Collaboration & Agile Participation• Collaborate with Data Engineers, Big Data Platform Engineers, and QA peers to ensure complete test coverage.• Participate in Agile ceremonies-story pointing, defining acceptance criteria, backlog refinement, sprint reviews, and retrospectives.• Identify testing risks, propose mitigation strategies, and champion quality-first practices across the team.
Required Skills & Qualifications• 3-5 years of hands-on experience in software testing or QA engineering, ideally within Big Data environments.• Strong proficiency in any one (or more) of the following languages:
Python, Java, Scala, Bash.• Experience testing big-data applications on Kafka, Spark, Cloudera/Hadoop, HBase, Hive, Solr, NiFi, Hue, Livy, or similar components.• Practical experience validating NiFi pipelines: processors, flow logic, provenance, connections, and integrations.• Strong experience with test automation frameworks (API, backend, ETL/data validation).• UI automation experience using Selenium.• Proficiency in SQL and NoSQL data validation tasks.• Hands-on experience with Jenkins or similar CI/CD tools.• Provide production support for the applications as a developer/tester for two weeks in rotated shift.• Solid understanding of Agile/Scrum methodologies.• Strong analytical, debugging, communication, and problem-solving skills.
Preferred (Nice to Have)• Experience with Unravel, Cloudera Navigator, or monitoring tools.• Exposure to Docker, Kubernetes, or cloud platforms (AWS, Azure, GCP).• Familiarity with performance testing of distributed systems.
Mastercard is a merit-based, inclusive, equal opportunity employer that considers applicants without regard to gender, gender identity, sexual orientation, race, ethnicity, disabled or veteran status, or any other characteristic protected by law. We hire the most qualified candidate for the role. In the US or Canada, if you require accommodations or assistance to complete the online application process or during the recruitment process, please contact [email protected] and identify the type of accommodation or assistance you are requesting. Do not include any medical or health information in this email. The Reasonable Accommodations team will respond to your email promptly.
Corporate Security Responsibility
All activities involving access to Mastercard assets, information, and networks comes with an inherent risk to the organization and, therefore, it is expected that every person working for, or on behalf of, Mastercard is responsible for information security and must:

• Abide by Mastercard's security policies and practices;
• Ensure the confidentiality and integrity of the information being accessed;
• Report any suspected information security violation or breach, and
• Complete all periodic mandatory security trainings in accordance with Mastercard's guidelines.

In line with Mastercard's total compensation philosophy and assuming that the job will be performed in the US, the successful candidate will be offered a competitive base salary and may be eligible for an annual bonus or commissions depending on the role. The base salary offered may vary depending on multiple factors, including but not limited to location, job-related knowledge, skills, and experience. Mastercard benefits for full time (and certain part time) employees generally include: insurance (including medical, prescription drug, dental, vision, disability, life insurance); flexible spending account and health savings account; paid leaves (including 16 weeks of new parent leave and up to 20 days of bereavement leave); 80 hours of Paid Sick and Safe Time, 25 days of vacation time and 5 personal days, pro-rated based on date of hire; 10 annual paid U.S. observed holidays; 401k with a best-in-class company match; deferred compensation for eligible roles; fitness reimbursement or on-site fitness facilities; eligibility for tuition reimbursement; and many more. Mastercard benefits for interns generally include: 56 hours of Paid Sick and Safe Time; jury duty leave; and on-site fitness facilities in some locations.
Pay Ranges
O'Fallon, Missouri: $92,000 - $147,000 USD