Issues accessing Penji

Incident Report for Penji

Postmortem

Between 12:55am and 01:50am UTC, users experienced elevated loading times and errors due to server issues. This incident was caused by performance issues in the events mode recurrence calculations when under high server load. Our monitoring systems alerted and identified the performance issue, and affected servers began load shedding to handle as many users as possible during the incident. To resolve the issue, we applied performance fixes to the affected area and increased server capacity. Remediation items include additional load testing, capacity planning, and performance improvements to critical areas.

Posted Aug 30, 2023 - 09:07 UTC

Resolved

This incident has been resolved.
Posted Aug 29, 2023 - 02:12 UTC

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Aug 29, 2023 - 01:59 UTC

Identified

The issue has been identified and a fix is being implemented.
Posted Aug 29, 2023 - 01:35 UTC

Update

We are continuing to investigate this issue.
Posted Aug 29, 2023 - 01:22 UTC

Investigating

We are currently investigating this issue.
Posted Aug 29, 2023 - 01:07 UTC
This incident affected: API, Admin Portal, Mobile App, and Web App.