Completed -
Scheduled maintenance has been successfully completed. All systems are operating normally, and the IncomeXpert API is fully available.
Feb 19, 22:11 EST
Verifying -
Verification is currently underway for the maintenance items.
Feb 19, 21:50 EST
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Feb 19, 20:30 EST
Scheduled -
This evening the IncomeXpert API will undergo maintenance. The plan for this window will be to enhance the underlying infrastructure to provide more compute capacity to production and deploy some more memory efficient application software. Further analysis on recent API issues indicated more RAM capacity would help avoid RAM utilization spikes due to server scaling events and staging deployments.
Feb 19, 11:46 EST
Resolved -
Blueprint has completed its monitoring of this issue. We have current corrective actions to address the original issue which were effective. These were restarting stuck/unresponsive server instances. Going forward we will adjust the planned scaling schedule to earlier in the day and avoid scheduled scale up events during the workday.
Feb 18, 16:24 EST
Monitoring -
The IncomeXpert API is now operating nominally. The Blueprint team will continue to monitor systems throughout the day to ensure the system remains nominal.
The preliminary root cause of the issue indicates a subset of server instances are experiencing high RAM utilization after scale out and becoming slow to respond. This impacted a fraction of the API server instances, hence some clients saw nominal performance and other experience degraded performance.
Feb 18, 12:37 EST
Update -
IncomeXpert UI users have nominal performance restored. The degradation to the API is still being investigated.
Feb 18, 10:48 EST
Investigating -
We have identified a slow response situation impacting some users. Users may experience slow or variable response when using the UI or API. Our team is investigating and will update status periodically until resolved.
Feb 18, 09:29 EST