Come work at a place where innovation and teamwork come together to support the most exciting missions in the world!
Staff Big Data Engineer
Job Description
We are seeking a Staff Big Data Engineer to define, design, and drive the evolution of the data platform and pipeline architecture powering Qualys' Enterprise TruRisk Platform. This role will be responsible for building and scaling distributed data systems that process billions of events and transactions daily while providing technical leadership across multiple engineering teams.
Responsibilities
● Define and drive the technical vision, architecture, and long-term strategy for the data platform and data pipeline ecosystem.
● Design, build, and maintain highly scalable, high-availability data processing systems using Apache Spark, Apache Kafka, and distributed data technologies.
● Lead architecture and design decisions across multiple engineering teams, ensuring solutions meet requirements for scalability, reliability, performance, security, and cost efficiency.
● Design and implement event-driven, streaming, and large-scale data processing architectures to support real-time and batch workloads.
● Troubleshoot, optimize, and resolve complex performance bottlenecks across data pipelines, distributed systems, and platform infrastructure.
● Partner with Product Management, Professional Services, and Sales Engineering teams to evaluate technical solutions, architecture trade-offs, and platform capabilities.
● Establish engineering standards, architectural guidelines, documentation practices, and operational best practices for the data platform.
● Research, evaluate, and recommend emerging technologies that improve platform scalability, performance, and operational efficiency.
● Mentor and guide engineers on distributed systems design, performance optimization, and big data technologies.
● Support architecture reviews, technical governance initiatives, and engineering excellence programs.
Required Qualifications
● Minimum 12 years of experience in software engineering, data engineering, distributed systems engineering, or big data platform development.
● Minimum 6 years of hands-on experience designing, developing, and optimizing Apache Spark-based data processing solutions.
● Minimum 6 years of experience building and operating large-scale data pipelines processing billions of events or transactions per day.
● Minimum 4 years of experience administering and operating Apache Kafka in production environments.
● Minimum 4 years of experience designing and implementing event-driven or streaming data architectures.
● Strong expertise in distributed data processing systems, data lake architectures, and large-scale platform design.
● Experience delivering highly available, scalable, and fault-tolerant distributed systems in production environments.
● Demonstrated experience leading architecture initiatives and influencing technical direction across multiple engineering teams.
● Bachelor's degree in Computer Science, Computer Engineering, Information Technology, or a related technical discipline.
Preferred Qualifications
- Experience working with high-volume, multi-tenant enterprise platform environments.
- Proficiency in Scala, Java, or Python for data processing applications.
- Proven experience mentoring junior or mid-level engineering team members.
Skills Required
- Minimum 12 years experience in software engineering, data engineering, or distributed systems engineering
- Minimum 6 years hands-on experience designing, developing, and troubleshooting Apache Spark-based data processing solutions
- Minimum 6 years building and supporting large-scale data pipelines processing billions of events or transactions per day
- Minimum 4 years administering and operating Apache Kafka in production environments
- Minimum 4 years designing event-driven or streaming data architectures
- Experience delivering highly available and scalable distributed systems in production environments
- Experience leading architecture and technical initiatives across multiple engineering teams
- Experience mentoring engineers and providing technical guidance on large-scale platform development
- Bachelor's degree in Computer Science, Software Engineering, Information Technology, or related technical discipline
- Apache Spark
- Apache Kafka
- Hadoop ecosystem technologies
- Data lake architectures
- Event-driven architectures
- Distributed data processing systems
- Oracle Database
- Cassandra
- Redis
- Linux/Unix environments
- Performance tuning and benchmarking of large-scale systems
- Experience with Elasticsearch or Apache Solr
- Experience with Trino
- Experience with Apache Airflow
- Experience with distributed caching technologies
- Experience implementing Lambda, Kappa, or Kappa++ architectures
- Experience with Apache Flink and real-time stream processing
- Experience with rule-engine platforms
- Experience deploying and supporting machine learning models in production
- Experience administering enterprise Big Data platforms and services
Qualys Compensation & Benefits Highlights
The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about Qualys and has not been reviewed or approved by Qualys.
-
Affordable Benefits — Benefits costs are widely viewed as low for employees and dependents, with healthcare often described as almost fully paid for. Feedback suggests this affordability helps offset perceptions of lower base pay in some roles.
-
Healthcare Strength — Healthcare offerings are broad, including multiple medical plan options, dental and vision coverage, mental health support, and disability insurance. Benefits are described as “pretty amazing” or “great,” reinforcing perceived quality and coverage depth.
-
Equity Value & Accessibility — Equity participation is accessible through company stock plans and an employee stock purchase plan. Compensation packages commonly include equity alongside salary and bonus, which some consider a meaningful part of total rewards.
Qualys Insights
What We Do
Qualys, Inc. (NASDAQ: QLYS) is a pioneer and leading provider of disruptive cloud-based security, compliance and IT solutions with more than 10,000 subscription customers worldwide, including a majority of the Forbes Global 100 and Fortune 100. Qualys helps organizations streamline and automate their security and compliance solutions onto a single platform for greater agility, better business outcomes, and substantial cost savings. The Qualys Cloud Platform leverages a single agent to continuously deliver critical security intelligence while enabling enterprises to automate the full spectrum of vulnerability detection, compliance, and protection for IT systems, workloads and web applications across on premises, endpoints, servers, public and private clouds, containers, and mobile devices. Founded in 1999 as one of the first SaaS security companies, Qualys has strategic partnerships and seamlessly integrates its vulnerability management capabilities into security offerings from cloud service providers, including Amazon Web Services, the Google Cloud Platform and Microsoft Azure, along with a number of leading managed service providers and global consulting organizations. For more information, please visit http://www.qualys.com
