Senior Systems Engineer, Qolo

Posted 2 Hours Ago
Be an Early Applicant
Hiring Remotely in USA
Remote
Senior level
Information Technology
The Role
Leads Qolo’s Production Support and SRE team, overseeing incident command, on-call operations, monitoring, alerting, observability, payment-platform reliability, job and file recovery, automation, release support, security controls, documentation, and technical escalations. The role mentors engineers, drives root-cause remediation, improves operational processes, and coordinates with Engineering, DevOps, Security, Product, clients, banks, and vendors.
Summary Generated by Built In

Job Description:


The Senior Systems Engineer is the technical and team lead for Qolo's Production Support / Site Reliability Engineering (SRE) team. The role is responsible for the stability, integrity and availability of the Qolo payments platform, including issuer authorization, settlement and funds movement, scheduled jobs, reports and file transfers, and client and internal portals.

This is a senior individual contributor who also leads the team day to day. The Systems Engineer III acts as Incident Commander for major incidents, serves as the L2 escalation point for team members and for machine-generated alerts, sets technical direction for monitoring, automation and runbooks, and mentors the team. The role works closely with Engineering, DevOps, Security, Product, Client Success, Onboarding, and external banks and vendors, and acts as the liaison between those teams and Infrastructure & Support leadership.

Production Support / SRE keeps detection, monitoring, incident triage and incident command. Day-to-day settlement and ACH subject-matter ownership sits with Payments Operations, and code-level remediation sits with Sustaining and Feature Engineering. This role works across those boundaries and drives the requirements that reduce manual work for the team.


Essential functionsIncident command and on-call

·         Serve as Incident Commander for Sev 1 and major Sev 2 incidents: open and run bridges, set severity and priority, manage SLA clocks, coordinate stakeholders, and send incident communications on a defined cadence until resolution.

·         Act as the L2 and high-level escalation point for triaging and resolving technical issues, including SQL and transaction diagnostics, manual interventions that require bank approval, and coordination with Engineering and external parties.

·         Lead root cause analysis: complete RCAs within the 48-hour target using blameless methods such as 5 Whys, and drive corrective actions into runbooks, alerts, automation or engineering work items.

·         Participate in the Production Support / SRE 24x7 on-call rotation. Help design and maintain the rotation, PagerDuty routing and escalation policies aligned to reliability domains (issuer, jobs and reports, portals, network and infrastructure), and the Engineering Areas of Responsibility (AOR) escalation model.

·         Own the quality of on-call handoffs, escalation criteria and incident documentation for the team.

Monitoring, alerting and observability

·         Design, tune and maintain alerts for the issuer path, scheduled jobs, reports, file transfers and portals using Azure Monitor, Log Analytics, Application Insights and Kusto Query Language (KQL).

·         Improve alert quality by reducing noise, tracking actionable versus non-actionable alerts, and retiring or refining alerts that do not drive action.

·         Define and track service level indicators and objectives (SLIs, SLOs) and error budgets for critical paths, including availability, latency and integrity of authorization, settlement and funds movement, reports and file delivery.

·         Run recurring audits, including the quarterly Azure alert action group review, to confirm recipients, routing, severity and on-call coverage.

·         Monitor and report on system performance, and recommend capacity and configuration changes.

Jobs, files, reports and funds movement operations

·         Oversee operation and recovery of scheduled jobs and file flows, including VisualCron jobs and Azure runbook jobs: diagnose failures, determine partial loads, back out and reprocess, and rerun safely.

·         Confirm that files are created, transmitted, received and posted, and follow up on file exceptions. Rerun, deduplicate, correct and re-deliver failed reports and malformed client files.

·         Maintain the jobs and file inventory (Qolo File Management Tracker) as the source of truth for job names, schedules, owners, dependencies and SLAs.

·         Support daily settlement and funds-movement operations for sponsor banks while ownership transitions to Payments Operations, and continue to own detection, triage and escalation for settlement failures.

·         Investigate payment incidents by building and running targeted SQL queries and by researching card network and payment portals to resolve issues.

Automation and engineering

·         Build and maintain SQL, PowerShell and Azure-based automation (Azure Automation, GitHub workflows and runners) for jobs, audits, reruns, diagnostics and other repeatable tasks.

·         Convert recurring manual work into documented, restartable and monitored automation, and write clear operational requirements (alerting, restartability, rollback) for work that Engineering or DevOps must implement.

·         Help the team protect engineering time by tagging and reporting work by category (incident, monitoring, automation, client support, configuration, audits) and by escalating when reactive work exceeds the target cap.

·         Identify and prioritize the team's top recurring pain points and turn them into an automation or tooling backlog, with measurable reduction in manual effort.

·         Install, configure, upgrade and enhance monitoring, operating system, database and utility software that supports production operations.

·         Prepare, test, evaluate and implement changes that support infrastructure and operations, following Qolo change controls.

Release and deployment support

·         Support production releases: send client and vendor deployment notices, monitor Application Insights and Azure Front Door during rollout, validate post-deployment health, and participate in rollback and hotfix steps per playbooks.

·         Define minimum observability, post-deployment validation and rollback requirements for services and releases, and confirm that a tested rollback or hotfix path is linked from each release record and service runbook.

·         Move production traffic and adjust scaling during releases as directed by the release plan.

Escalated client and internal support

·         Provide technical support for escalations from Client Services, Client Success and Onboarding when they cannot resolve an issue: reproduce the problem, pull logs, run targeted SQL and data checks, document the outcome, and escalate to Product or Engineering when needed.

·         Execute program, BIN and other non-financial configuration requests in line with Qolo change controls, and help define the requirements for self-service tools that remove Production Support as middleware.

·         Administer and review access for production platforms and tools, and support security investigations (for example phishing and suspicious activity) in partnership with Security.

Certificates, security and compliance

·         Maintain the inventory, renewal calendar, alerts and runbooks for certificates that affect production (TLS, mTLS, card network and bank certificates), in partnership with Security and Engineering.

·         Apply Azure security best practices to the resources the team administers, and support audit requests with evidence of controls, access reviews and incident records.

·         Operate within access-control and least-privilege policies for on-call and production changes. Policy and governance remain with the CISO.

Documentation and knowledge management

·         Develop and maintain technical documentation: runbooks, incident playbooks, SOPs, knowledge base articles (KBIs), deployment support procedures, client notification templates and diagnostic query libraries.

·         Make sure every frequent alert type has a linked runbook that defines first checks, mitigation and escalation paths, and that RCA findings feed back into runbooks and SOPs.

·         Document completed projects clearly for the next engineer and for stakeholders.

Team leadership and mentorship

·         Act as technical and team lead for the Production Support / SRE team: set day-to-day priorities, review work, coordinate coverage and handoffs, and keep intake disciplined.

·         Mentor and train team members and L1 resources, including alert triage, SQL and KQL diagnostics, runbook use and incident communication. Model "play your position" discipline and cross-training so team members are interchangeable.

·         Serve as liaison between the Production Support / SRE team, stakeholder teams and Infrastructure & Support leadership. Advise on current internal and external work, and report team health, workload and risks.

·         Proactively research and introduce new technology and best practices, and recommend the technical direction for operations and reliability projects.

·         Lead large or complex projects as assigned: define the plan, coordinate participants and make sure work is delivered accurately and on time.

·         Expand your own education, technical stack and relevant certifications.

·         Perform other duties as assigned.

QUALIFICATIONS
Experience

·         Seven or more years of successful, extensive experience in systems engineering, production support, site reliability, infrastructure operations or a related technical field.

·         Demonstrated experience as an incident commander or technical escalation lead for production systems.

·         Experience in payments, financial services or other regulated, high-availability environments is strongly preferred.


Education and Certifications

·         Bachelor's degree or training from an accredited university, college or trade school in a related technology field, or equivalent professional and technical experience. Advanced degrees are a plus.

·         Advanced certifications recommended, for example Microsoft Azure Administrator or Azure Security Engineer, ITIL, or equivalent.


Technical skills

·         Expert-level Azure operations and monitoring: Azure Monitor, Log Analytics, Application Insights and KQL.

·         Advanced SQL Server skills, including reading and writing diagnostic queries and working with stored procedures.

·         PowerShell scripting and automation. Experience with Azure Automation and GitHub workflows is preferred.

·         Windows and Linux server administration, and Azure networking and security fundamentals.

·         Experience with job scheduling and file-transfer tools (for example VisualCron and SFTP), and with incident and alerting platforms (for example PagerDuty).

·         Strong working knowledge of Qolo's solutions and service offerings, or the ability to build it quickly, including card issuing and authorization flows and money movement (ACH, settlement).

·         Familiarity with SRE practices: SLIs and SLOs, error budgets, blameless postmortems and progressive delivery.

Other requirements

·         Strong time management, organizational and communication skills, with the ability to multitask in a fast-paced environment.

·         Ability to provide 24x7 monitoring and support through participation in an on-call rotation.


As a forward-thinking software provider, Computer Services, Inc. (CSI) helps community and regional financial institutions solve their customers’ needs through open and flexible technologies. In addition to its award-winning core banking platform, these include the latest in lending, digital banking, payments, financial crime prevention and cybersecurity. Building on its 60-year track record of personalized service, CSI is shaping the future of banking and empowering its customers to rival their competition. For more information about CSI, visit www.csiweb.com 

CSI provides rewarding and challenging career opportunities for our employees. When determining your pay, we consider various factors such as your skills, qualifications, experience and location. Along with a competitive salary, this position includes eligibility for incentive awards based on both individual and business performance. We also offer a comprehensive range of benefits. To learn more about our benefits, visit: Benefits Summary

CSI is an Equal Opportunity Employer.  All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, pregnancy, sexual orientation, gender identity, national origin, age, physical and mental disability, marital status, veteran status, or any other characteristic protected by applicable law. If you need an accommodation during the recruitment process, please email us at [email protected] and we will work with you to meet your accessibility needs.

For applicants residing in California, please read Privacy Notice for California Residents | CSI (csiweb.com)

Visa Sponsorship: We are unable to offer visa sponsorship for this position. Applicants must be authorized to work in the United States without the need for sponsorship now or in the future.

Skills Required

  • Seven or more years of experience in systems engineering, production support, site reliability, infrastructure operations, or a related technical field
  • Demonstrated experience as an incident commander or technical escalation lead for production systems
  • Bachelor's degree or equivalent professional and technical experience in a related technology field
  • Expert-level experience with Azure operations and monitoring, including Azure Monitor, Log Analytics, Application Insights, and KQL
  • Advanced SQL Server skills, including writing diagnostic queries and working with stored procedures
  • PowerShell scripting and automation experience
  • Windows and Linux server administration experience
  • Azure networking and security fundamentals
  • Experience with job scheduling and file-transfer tools such as VisualCron and SFTP
  • Experience with incident and alerting platforms such as PagerDuty
  • Familiarity with SRE practices, including SLIs, SLOs, error budgets, blameless postmortems, and progressive delivery
  • Ability to participate in a 24x7 on-call rotation
  • Experience in payments, financial services, or other regulated high-availability environments
  • Experience with Azure Automation and GitHub workflows
  • Strong working knowledge of card issuing, authorization flows, ACH, settlement, and money movement
  • Advanced certification such as Microsoft Azure Administrator, Azure Security Engineer, or ITIL
Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
HQ: Singapore
2,688 Employees
Year Founded: 1965

What We Do

As a forward-thinking technology partner, we accelerate our delivery of cutting-edge solutions to empower our customers’ growth. CSI heavily invests in research and development, product partnerships and strategic acquisitions to provide the most innovative technologies—all while maintaining our service-first mentality. We drive financial institutions and other businesses forward with a variety of services, including: Core Banking With more than 57 years as a leader in core bank processing, CSI’s enterprise banking solutions empower institutions to compete in an open banking environment while growing assets, developing additional revenue streams and reaching new customers by integrating the latest digital technologies. Regulatory Compliance As an established leader in compliance, top global brands trust CSI as their regtech partner and rely on us for our advanced compliance software—including industry-leading fraud and anti-money laundering solutions. Managed Cybersecurity CSI offers cloud-based cybersecurity services, with 24/7 monitoring, real-time incident remediation, seamless compliance integration and industry-leading support for enhanced threat mitigation. IT Management As one of the largest, most trusted managed services partners in the country, we take a consultative approach to IT management to provide a scalable, efficient IT enterprise that supports institutions’ growth and regulatory needs. Document Distribution From digital document delivery to cost-efficient print and mail services, CSI provides a full range of document distribution solutions to help financial institutions stay connected with their customers. Advisory Services CSI’s Advisory Services—which include IT governance, compliance and risk management services—deliver cost-effective solutions that reduce the time and expense of managing security and compliance while ensuring technology is aligned to business objectives.

Similar Jobs

Remote
United States
152 Employees
90K-150K Annually

John Deere Logo John Deere

Data Science Tech Lead

Artificial Intelligence • Cloud • Internet of Things • Machine Learning • Analytics • Industrial
Remote or Hybrid
2 Locations
69000 Employees
138K-207K Annually

John Deere Logo John Deere

2027 Intern - Product Engineering

Artificial Intelligence • Cloud • Internet of Things • Machine Learning • Analytics • Industrial
Remote or Hybrid
IL, USA
69000 Employees
1-1000K Annually

Hilton Logo Hilton

Manager, Creative Operations & Activation

Events • Food • Travel • Hospitality • Hotels & Resorts
In-Office or Remote
3 Locations
500000 Employees
40K-110K Annually

Similar Companies Hiring

NODA AI Thumbnail
Artificial Intelligence • Information Technology • Software • Cybersecurity • Defense • Defense Technology
Austin, TX
54 Employees
Golden Pet Brands Thumbnail
Digital Media • eCommerce • Information Technology • Marketing Tech • Pet • Retail • Social Media
El Segundo, California
567 Employees
RapidScale Thumbnail
Cloud • Information Technology • Consulting
Raleigh, NC
600 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account