The problem is that you only roll once they leave the feywild. Up to that point time between the two planes works in sync. You effectively just time travel when leaving depending on the result of your roll.

There's a 5% chance that days become years. Based on just that alone, for every 20 days spent in the feywild you're missing a year in the rest of the world. I got a factor of 22.7 on average for a 7-day week, and 23.3 if it's ten.