Platform Reliability Analyst - BOT

The Platform Reliability Analyst is responsible for ensuring the continuous monitoring and overall health of our cloud infrastructure hosted on platforms such as AWS, Rackspace, Expedient, and Heroku. This role involves proactive monitoring of system performance, coordinating incident response efforts, and collaborating with development, cloud, and operations teams to address issues before they impact the business. The ideal candidate will have a process-oriented mindset, strong communication skills, and a foundational understanding of cloud technologies to facilitate rapid resolution of incidents and optimize system performance.

Key Responsibilities:

• Proactive System Monitoring: Oversee system performance and availability through continuous monitoring of alerts from various APM tools (New Relic, Cloudwatch, etc.). Provide feedback on alert tuning, identify patterns in incidents, and pinpoint optimization opportunities (e.g., identifying idle systems that could be shut down).
• Production Support: Build and maintain a comprehensive understanding of all software systems and their variations. Ensure readiness to support production systems by identifying potential issues before they affect customers.
• Outage Management: Lead the incident command center during outages with a focus on rapid resolution. Coordinate incident response by:
• Recording incident start/end times and affected systems.
• Notifying internal stakeholders and support teams of the incident status.
• Coordinating the involvement of the correct teams and ensuring all relevant details are shared.
• Providing and executing runbooks, or coordinating with cloud teams for execution.
• Running incident bridges, ensuring systems, logs, and traffic are monitored and relevant experts are involved.
• Documenting facts versus theories in real-time during incident resolution.
• Incident Communication: Notify the company about incidents and coordinate with support to inform customers. Eventually, manage status updates on a future status page for system transparency.
• Incident Prevention and Follow-Up: Be the first line of defense—proactively identify system issues before customers are impacted. Conduct root cause analysis (RCA) after incidents to determine underlying issues and implement preventative measures. Update and create runbooks as needed.
• Collaboration and Coordination: Regularly set up meetings with cloud and development teams to address and resolve recurring issues. Communicate proactively with leadership about any potential cost increases or system inefficiencies.
• System Health Metrics: Monitor traffic, system health, security perimeter, and overall performance. Track key metrics such as the percentage of issues identified proactively versus reactively.

Key Skills and Qualifications:

• Strong Communication Skills: Clear, concise English to convey the status of incidents and performance issues to both technical and non-technical stakeholders.
• Process-Oriented Mindset: Ability to follow, document, and improve processes to ensure smooth incident management and resolution.
• Attention to Detail: Capability to record key details about system health, performance, and incident facts versus theories in real-time.
• Familiarity with Monitoring Tools: Experience using monitoring and alerting tools such as New Relic, Cloudwatch, or Datadog, and familiarity with logs, traffic monitoring, and system health metrics.
• Coordination and Leadership Skills: Ability to lead incident response teams, coordinate with various technical experts, and manage communication effectively during outages.
• Basic Technical Understanding: While not an engineering role, some technical familiarity with cloud environments, system alerts, and security practices is important. Entry-level engineers with an interest in coordination roles are encouraged to apply.
• Collaboration: Ability to work cross-functionally with development, cloud, and support teams to ensure smooth operations and proactive issue resolution.