Back to overview
Downtime

Database maintenance

Dec 07 at 04:15am HST
Affected services
blahaj.social

Resolved
Dec 07 at 11:56am HST

blahaj.social is back after a very intense half day in PostgeSQL...
Long story short there were 3 accounts in the database that somehow causes out of memory issues to be thrown when fetched, this happened for a very long time in for example backups...

It took manually extracting the data with the accounts filtered out, followed by re-inserting it. Then finally completing a backup. Upgrading the database, re-inserting all data and by hand fixing all foreign key issues resulted by those accounts

painful as it was i can sleep now i know there is no longer a time bomb in the database i could only fix with full harddisk rollbacks...
Sorry for all downtime caused and bash script crimes commited

Updated
Dec 07 at 05:27am HST

This is taking longer than expected due to a failure of the default tooling, we wrote a data export system specifically made to rule out the broken rows in the table. This requires data export of 433 995 rows in batches of 500, this can take an hour.

Thanks for your patience

Created
Dec 07 at 04:15am HST

We are currently working on fixing a longer term database corruption of the accounts table. To run the repair we needed to take the instance online while running the scripts. We'll be back soon!