Since the dawn of LLMs it's virtually impossible to scrape web content. Headless browsers have become basically useless. I actually have to automate keyboard inputs to simulate the navigation. I could maybe try to write the javascript cache to file but honestly it's just faster that way.
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
replies: