Alright I got somewhere. It's the DNS. Specifically my Adguard Home setup. By default it uses rate limiting of 20 requests per client. I removed it. And now Firefox is loading as fast as Brave was loading it previously.

Although websites on Brave are loading much faster, Firefox is now pretty usable. A few hundred ms difference is fine for me. The slowdown started happening since Firefox 150ish so probably something changed then that affected Firefox but Chromium based browsers worked fine. IDK what but for now removing rate limiting works for me. Hopefully my router won't crash because of this. Thanks for the help. Also if I use Firefox DoH with custom Controld endpoint, everything loads at lightening speed. But I can't use it as there are DNS rewrites in my Adguard Home which allows me to access my services w/o typing the IP all the time.
This is firefox
This is Brave origin.