We had this 503 error after each login on our older 3.x instance. After a refresh of the page, the session then works fine.
Now we have installed a plain vanilla 4.6.3 and migrated the mongodb.
Surprisingly the 503 issue is happening as well on the new instance.
Setup is with nginx and 3 NodeBB processes on same VM, and Redis cluster
I assume for some weird reason, the nodebb rate limiting kicks in, but strange that this only occurs after each new login, and reproducible.
Any suggestions how to diagnose or how to modify the rate limiting check to be less aggressive (or to exclude logins)?
Thanks Stefan
16 Comments
baris@community.nodebb.org · 2 pts · 251d
You can lower it and old passwords will still work, new passwords will use the new rounds value.
big_steve@community.nodebb.org · 2 pts · 251d
Checked for 12 --> 10 and works. Thanks!
baris@community.nodebb.org · 1 pts · 252d
Did you modify bcrypt_rounds in config.json? Check cpu usage during login. You can tweak the traffic management values in acp to make it less sensitive as well.
baris@community.nodebb.org · 1 pts · 252d
With the default settings it shouldn't what about the 2 values set at /admin/settings/advanced#traffic-management?
baris@community.nodebb.org · 1 pts · 252d
Maybe try a higher value for first one, on this forum it is set to 300ms. You can read more about what those values are here https://github.com/STRML/node-toobusy?tab=readme-ov-file#tunable-parameters
big_steve@community.nodebb.org · 1 pts · 251d
Can there be another OS config issue on the machine which can cause bcrypt to be that slow? I am thinking of the /dev/random vs. /dev/urandom scenario...
big_steve@community.nodebb.org · 1 pts · 252d
Ay, after setting the first parameter to 500, the 503 does not occur anymore. Will dig into testing some more combinations tomorrow. Thanks 🙏
big_steve@community.nodebb.org · 1 pts · 252d
I deactivated the traffic management and don't get a 503 anymore after login. But I don't want to completely leave that disabled, should I?
big_steve@community.nodebb.org · 1 pts · 252d
We do not have any traffic right now, nodeBB is runnning with 3 processes each < 1% CPU% and with traffic management enabled and the standard settings we get the 503 once after login. With traffic management disabled the 503 does not happen.
Can I enable some more debugging/tracing to see why traffic management shuts down the session after login?
big_steve@community.nodebb.org · 1 pts · 252d
As said these are 100 500
big_steve@community.nodebb.org · 1 pts · 252d
I did a quick test with setting the first parameter to 300, did a restart. Logout, login, and the 503 is still there.
big_steve@community.nodebb.org · 1 pts · 252d
Which way is less sensitive - tuning values up or down? Currently they are set to 100 and 500 - and unfortunately the descriptions do not explain their meaning.
big_steve@community.nodebb.org · 1 pts · 251d
baris Is it possible to change bcrypt_rounds on an existing system which has already users and (hashed) passwords saved? (I am thinking of reducing the default of 12 to 10 as a test)
big_steve@community.nodebb.org · 1 pts · 252d
We did not modify the bcrypt_rounds parameter. You mean the password hashing in bcrypt could drive a cpu% spike which the traffic management erroneously interprets as too high load?
Does that really sound realistic that the CPU is that busy with a single login and the following hardware - one VM for nodeBB and the same for mongoDB Debian GNU/Linux 13 (trixie), Kernel 6.12.57 4 Cores x86_64 Architektur 8 GB RAM
baris@community.nodebb.org · 1 pts · 252d
Try disabling traffic management at /admin/settings/advanced#traffic-management and see if it's caused by that.
big_steve@community.nodebb.org · 1 pts · 252d
Thanks, much appreciated. If bcrypt is causing that, then lowering the rounds value should also resolve that, at least for a test. Will post the results.