We identified bot abuse on this endpoint and temporarily took it down until I can implement some bot restriction on it. I'll bring it back online shortly.
Our traffic today jumped 4x our previous daily high. I had to enable some extra protection to give the server some breathing room to recover. One of the side effects is the mobile apps are temporarily unavailable until I get everything tweaked. It will be back in a few days. The good news is that I already got my hands on a newer more powerful server that will be going into production in the next few weeks.
For the next few days if you want to scroll the site you will need to use your mobile browser. I understand its totally not the same experience as voyager but I promise you i’ll get everything back up as it was as soon as I can.
Thanks for your trust and patience!
Edit: Access via the mobile app Voyager has been reinstated.
alright, turns out it had nothing to do with cloudflare and more related to my nginx settings. I made some changes and it appears to have fixed the issue. Let me know if this has fixed it on your side as well.
I believe this might be related to the fact that we are behind cloudflare. I notice other instances that are behind cloudflare appear to have the same issues while those that are not behind cloudflare appear to be ok. Lemmy.world appears to strip the embedded video function completely. See https://lemmy.world/post/49302693 as an example
Hey all, most of today's outage was caused by me. I implemented Anubis in attempt to differ some bot/scrapping traffic. We are presently serving over 350 million requests per month and its starting to show in the stability in the instance. I will be switching to the horizontal scaling approach in hopes to stabilize things more.
I'll also be providing an long overdue update once I get everything back under control.
Sorry for the stability issues as of late, you all deserve better.
@dbtng@eviltoast.org I appointed you as mod on the community. Like others have mentioned its recommended that you have an account on the instance but it's not a requirement.
Thank you for volunteering your time to the fediverse!
Thanks for asking, I think transparency around infrastructure is important.
This instance is hosted on equipment that we own, located in a local data center in Montreal, Canada. Backups are stored within the same facility; however, the backup hardware itself is not currently owned by us. Our S3 storage is hosted in a separate Montreal data center also on equipment that we do not own or control.
Our long-term objective is to operate our own dedicated backup infrastructure and S3 storage environment. However, the significant increase in RAM costs has temporarily delayed that rollout. Someone recently donated a few new servers however they were stripped of ram beforehand.
It’s also worth noting that we are currently using Cloudflare, but my plan to align with the broader Fedecan infrastructure strategy and transition to Deflect.ca in the future.
I am very well aware of the slowness and timeouts that people are experiencing. I'm working on getting a few upgrade put into place to bring things back to where they need to be. Thanks for the patience!
Hey all, I'm travelling at the moment but have been in discussions with the other admins about this issue. I return Friday oct 31st and will schedule to get us upgraded to 0.19.13 shortly after. I won't be giving the typical 5 day official notice to not create any extra delays. The upgrade to 0.19.13 is minor and shouldn't take us too long to run. Thank you for being patient.
We are not blocking any IP ranges. The server has had an increase in traffic over the last few months. I'm in the process of identifying new hardware to buy to make things speedy again.
Our Terms of Service will be likely be reviewed in the future, but our core philosophy of “don’t get us in trouble” will stay the same. We’ve always believed that the actions of a few shouldn’t negatively impact the experience for everyone else—and that belief still stands.
When it comes to defederation, we handle it on a case-by-case basis. We only consider it under very specific circumstances, or if there’s a clear mandate from the community. For example, we may choose to defederate from an instance if it's causing harm or violating our rules, especially if their admin team is unresponsive or unreachable.
Whenever we do take such action, we usually start a discussion with the community to keep everyone informed. In fact, this has happened recently—you can read more about it here.
Hey all,
We identified bot abuse on this endpoint and temporarily took it down until I can implement some bot restriction on it. I'll bring it back online shortly.
Hey @westingham
Thanks for starting a post about this.
Our traffic today jumped 4x our previous daily high. I had to enable some extra protection to give the server some breathing room to recover. One of the side effects is the mobile apps are temporarily unavailable until I get everything tweaked. It will be back in a few days. The good news is that I already got my hands on a newer more powerful server that will be going into production in the next few weeks.
For the next few days if you want to scroll the site you will need to use your mobile browser. I understand its totally not the same experience as voyager but I promise you i’ll get everything back up as it was as soon as I can.
Thanks for your trust and patience!
Edit: Access via the mobile app Voyager has been reinstated.
Hey! these screenshots are taken from the cloudflare panel
The Tesseract frontend has been pulled.
For those interested you can see the announcement here: https://sh.itjust.works/post/64232628
alright, turns out it had nothing to do with cloudflare and more related to my nginx settings. I made some changes and it appears to have fixed the issue. Let me know if this has fixed it on your side as well.
Thank you
I believe this might be related to the fact that we are behind cloudflare. I notice other instances that are behind cloudflare appear to have the same issues while those that are not behind cloudflare appear to be ok. Lemmy.world appears to strip the embedded video function completely. See https://lemmy.world/post/49302693 as an example
I also noticed someone did open a github issue on the lemmy-ui repo https://github.com/LemmyNet/lemmy-ui/issues/4001
I'll take a look and see if there's something in the nginx settings that could fix this
Hey all, most of today's outage was caused by me. I implemented Anubis in attempt to differ some bot/scrapping traffic. We are presently serving over 350 million requests per month and its starting to show in the stability in the instance. I will be switching to the horizontal scaling approach in hopes to stabilize things more.
I'll also be providing an long overdue update once I get everything back under control.
Sorry for the stability issues as of late, you all deserve better.
@dbtng@eviltoast.org I appointed you as mod on the community. Like others have mentioned its recommended that you have an account on the instance but it's not a requirement.
Thank you for volunteering your time to the fediverse!
Hey, we did an upgrade earlier today. All tests were passed but its possible something was missed.
Could you tell us which app you are using?
Done
Hey!
Thanks for asking, I think transparency around infrastructure is important.
This instance is hosted on equipment that we own, located in a local data center in Montreal, Canada. Backups are stored within the same facility; however, the backup hardware itself is not currently owned by us. Our S3 storage is hosted in a separate Montreal data center also on equipment that we do not own or control.
Our long-term objective is to operate our own dedicated backup infrastructure and S3 storage environment. However, the significant increase in RAM costs has temporarily delayed that rollout. Someone recently donated a few new servers however they were stripped of ram beforehand.
It’s also worth noting that we are currently using Cloudflare, but my plan to align with the broader Fedecan infrastructure strategy and transition to Deflect.ca in the future.
We have not upgraded yet but will be pushing the update this week.
Hey all,
I am very well aware of the slowness and timeouts that people are experiencing. I'm working on getting a few upgrade put into place to bring things back to where they need to be. Thanks for the patience!
Hi @kertain@sh.itjust.works
I plan on making a post to share all this information and more in the following weeks.
Thank you for your thoughtfulness and consideration!
I've been considering running it in parallel to a lemmy instance but it comes down to a few things:
This issue appears to have been fixed. Can anyone confirm whether they are still having any problems?
Hey all, I'm travelling at the moment but have been in discussions with the other admins about this issue. I return Friday oct 31st and will schedule to get us upgraded to 0.19.13 shortly after. I won't be giving the typical 5 day official notice to not create any extra delays. The upgrade to 0.19.13 is minor and shouldn't take us too long to run. Thank you for being patient.
Hi there,
We are not blocking any IP ranges. The server has had an increase in traffic over the last few months. I'm in the process of identifying new hardware to buy to make things speedy again.
I have cycled the services to see if that helps.
Let me know if things get better for you.
Yes, this will not change. The /c/Agora will continue to be a platform for members to use to shape this instance.
Our Terms of Service will be likely be reviewed in the future, but our core philosophy of “don’t get us in trouble” will stay the same. We’ve always believed that the actions of a few shouldn’t negatively impact the experience for everyone else—and that belief still stands.
When it comes to defederation, we handle it on a case-by-case basis. We only consider it under very specific circumstances, or if there’s a clear mandate from the community. For example, we may choose to defederate from an instance if it's causing harm or violating our rules, especially if their admin team is unresponsive or unreachable.
Whenever we do take such action, we usually start a discussion with the community to keep everyone informed. In fact, this has happened recently—you can read more about it here.