EDIT, Update: We've been able to partially recover about 25% of the media uploaded to Pawb.Social.
Upon further investigation, data from pawb.social was priorly moved to the new server and was inadvertently deleted as part of the earlier incident affecting furry.engineer and pawb.fun's cache while we were preparing the new server.
furry.engineer and pawb.fun media is unaffected, but all media directly uploaded to pawb.social is unrecoverable and has been lost. A miscommunication on my part OK'd the removal without fully checking the scope of data that was being removed; This was intended to only remove test data we were using to stress test the new hardware, but affected live data as well.
A large majority of posts are externally hosted and referenced (e.g. images from media.kbin.earth, d.furaffinity.net, etc.) are NOT affected. Only posts directly uploaded via the Create Post > Image field on pawb.social are affected.
We had previously transferred pawb.social to the new hardware as a preliminary trial and I had forgot this occurred thus failing to bring it up prior to the deletion going through.
I'm looking into options to be able to in-place reupload media on behalf of posters, but as the UUIDs are randomly generated by Pict-rs (the media service used by Lemmy), I'm unsure how to facilitate this. Any advice or information would be appreciated, and can be sent to network@pawb.social.
I understand this is a serious incident and reflects poorly on us, and I can only apologize for this loss and work to prevent this happening again in the future.
EDIT: I believe this will also impact profile pictures and community images (logos and banners), so please re-upload them at your earliest convenience.___
15 Comments
KoboldCoterie@pawb.social · 16 pts · 342d
No worries, mate. Thanks for the heads up that it happened; you're providing a free service to us, we aren't going to complain about the occasional hiccups, especially considering how infrequent there've been any.
Kolanaki@pawb.social · 16 pts · 342d
My face was so ugly in the pic I uploaded, that the server deleted every single picture just to be sure. 😞
Kovukono@pawb.social · 14 pts · 342d
You guys are running a free server with 99.9% uptime and barely any notable issues in the time I've been here. Losing some pictures in a migration over a mistake is not as serious as you think it is. You guys do amazing work, and we love you for everything you've done.
crashdoom@pawb.social MOD · 11 pts · 341d
ℹ️ Thanks to the efforts of one of our moderators, @southernwolf@pawb.social, we've been able to recover some of the lost data from pawb.social that was still active in Cloudflare's regional caches.
Currently, this is about 25% of all of the known uploaded media files stored on pawb.social.
We'll be reviewing the final tally of recovered data in the morning and trying to work on getting what we can reimported into Pict-rs.
southernwolf@pawb.social · 6 pts · 341d
For those who see this before we release the final tally: As it stands right now, I've recovered ~1/3rd of media posts via Cloudflare caching. I'll have a final total when I'm truly done, but honestly that's pretty damn good to have been able to rescue all of that via cache recovery alone.
southernwolf@pawb.social · 6 pts · 341d
This is subject to change, but here is the preliminary tally:
DarrenTheFoxcoon@pawb.social · 2 pts · 341d
Would the Pawb staff be open to using volunteers as secondary backups? A few GB seems like absolutely no problem for my own rig
natebluehooves@pawb.social · 4 pts · 339d
Amusingly we do have backup infrastructure on site. 30tb pawb nas, 80tb nas for backups, and lto tape backups periodically. my problem is getting it all set up. i’ve been battling depression and trying to catch up on housework.
the issue here was doing a partial migration and allowing a couple months to pass before trying to finish, by which point I had forgotten pawb was moved.
we plan on pawb to take a lot more storage as time goes on. we are at about 1.5tb plus ~450gb database size right now.
edit: to make some sense of this: the 1.5tb includes cached media from other instances. our users don’t use a lot of storage yet.
southernwolf@pawb.social · 1 pts · 338d
Please see an update for this situation here: https://pawb.social/post/33046379
southernwolf@pawb.social · 1 pts · 341d
alex@pawb.fun · 10 pts · 342d
@crashdoom 🫂 (no hug react on Telegram so you're getting one here)
Ethanol@pawb.social · 8 pts · 342d
This is fine, you're doing a great job. Mistakes happen and re-uploading my profile picture or banner is not a hassle at all. No need to beat yourself up. Thank you all for the work you put into this instance :3
LordAmplifier@pawb.social · 7 pts · 341d
It's a bit of a bummer to see some of the "history" of pawb disappear, but shit happens. You're providing one hell of a good service completely for free, and that's cool as heck. Thank you for the transparency and all the good work you do behind the scenes <3
Wahots@pawb.social · 7 pts · 342d
Happens to the best of us! My work just nuked a ton of data in a similar incident, so you're in good company, haha.
Dojan@pawb.social · 6 pts · 341d
Sometimes, shit happens. We've all been there, one way or another. Thanks for all your hard work.
Foxfire@pawb.social · 5 pts · 342d
No worries, good to know the cause of the image issues. I'll simply edit my posts to reupload the missing data since it was all my own art anyway.