I just threw some additional logging into the piefed code to try to figure out what's going on, this has been happening too long and it's being elusive =(
Hmmm, I applied the latest updates a couple days ago but not seeing any errors in the logs. I just pulled in the latest changes yet again in case there was a fix, but I'll dig more later today.
For the curious: We didn't have the send_queue cronjob enabled, so scheduled messages weren't sent out when they were supposed to. Piefed seems to have a bug where the cron is enabled way in the future, it spams those old scheduled messages regularly.
As for why the struggle fixing it... this is embarrassing but I was looking at the wrong database. Last week I copied our "pyfedi" db to "pyfedi_utf" as part of correcting a problem with the db character encoding. I've been applying changes to the "pyfedi" db to fix this, instead of the "pyfedi_utf" database.
Lesson learned - clean up the old database once you know the new one is good.
It's off now
Sorry missed this reply, I've turned it off now.
I also did a bunch of performance tuning on the infra + db, and patched a few slow queries in piefed. Should definitely be zippier!
b
5!
1
I just threw some additional logging into the piefed code to try to figure out what's going on, this has been happening too long and it's being elusive =(
If you see this happen with a post / thread, can you follow up here with it for me to check the logs against please?
There seems to be some unknown bug in piefed (or our infra possibly) that I've had no luck in figuring out with Rimu. It's an ongoing battle =(
Argh. I think I've found the problem and applied a fix, let me know if it persists after this please!
No still seems to be missing data, digging more...
Hmmm, I applied the latest updates a couple days ago but not seeing any errors in the logs. I just pulled in the latest changes yet again in case there was a fix, but I'll dig more later today.
It happens even if you don't specify a video url and it happens on piefed.social too, seems to be a new bug.
I've got vengaboys stuck in my head now, thanks
Sigh, this is fixed properly now.
For the curious: We didn't have the send_queue cronjob enabled, so scheduled messages weren't sent out when they were supposed to. Piefed seems to have a bug where the cron is enabled way in the future, it spams those old scheduled messages regularly.
As for why the struggle fixing it... this is embarrassing but I was looking at the wrong database. Last week I copied our "pyfedi" db to "pyfedi_utf" as part of correcting a problem with the db character encoding. I've been applying changes to the "pyfedi" db to fix this, instead of the "pyfedi_utf" database.
Lesson learned - clean up the old database once you know the new one is good.
test3
Test!
6
Sweet