It's not about infinite growth.

Given that not everyone lives to old age, we would expect children to outnumber old people even in a zero (or slightly negative) percent growth scenario.