Issues with Events mode

Incident Report for Penji

Postmortem

This incident was caused by an internal data system using more resources than expected, which then caused processing of Events mode changes to become backlogged. Once additional server capacity was added, the backlog of changes was processed and the issue was resolved. We have adjusted our monitoring to more quickly identify this issue and are working on improvements to better allocate server resources to avoid this in the future.

Posted Apr 24, 2024 - 04:24 UTC

Resolved

This incident has been resolved.
Posted Apr 20, 2024 - 16:51 UTC

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Apr 20, 2024 - 16:35 UTC

Identified

The issue has been identified and a fix is being implemented.
Posted Apr 20, 2024 - 16:16 UTC

Investigating

We are currently investigating this issue.
Posted Apr 20, 2024 - 15:56 UTC
This incident affected: API, Admin Portal, Mobile App, and Web App.