There was actually one published a few days ago that concluded that it can be effective:

Participants with depression experienced a 51% reduction in symptoms, the best result in the study. Those with anxiety experienced a 31% reduction, and those at risk for eating disorders saw a 19% reduction in concerns about body image and weight.

However the person who did the study shares your concerns:

I asked Heinz if he thinks the results validate the burgeoning industry of AI therapy sites. “Quite the opposite,” he says, cautioning that most don’t appear to train their models on evidence-based practices like cognitive behavioral therapy, and they likely don’t employ a team of trained researchers to monitor interactions. “I have a lot of concerns about the industry and how fast we’re moving without really kind of evaluating this,” he adds.

Also they did another article about difficulties and pitfalls of making these things