my-place.social had a 3 hour outage. Here's what happened

At around 3:50 AM EST (8:50 UTC) my-place.social stopped responding.
It ran out of memory, and the out-of-memory manager killed mariadb. Without the database, Friendica was dead. I rebooted around 7 AM EST.

For days, the server has always had about 4GB free, plus 4GB of unused swap space. Just before the crash, there were numerous Apache processes running, and some were consuming considerable memory, causing the server to run out of memory. These are likely Friendica workers.

I don't know what changed that required a sudden need for additional memory.

I've reduced the database cache from 8GB to 6GB, and added 11GB additional swap space. This should prevent further crashes, although performance may suffer when swap space starts being used.

I will do further analysis over the next few days to see if I should reduce the maximum number of workers, or further reduce the database cache. Scaling up is also an option, but less preferred due to cost.

I am so sorry for any inconvenience this caused.

Jerry