I'm not a neurologist, I'm not going to make any grand claims with absolute certainty.

But, from what I've heard from scientists, the brain does do a sort of passive pruning, memories (or at least the neural pathways and patterns that represent those memories) are left to disconnect if they go unused/unaccessed for long enough. And conversely, when neural pathways are used more often, those synaptic connections are reinforced. And when those synaptic connections no longer exist, that's data loss (forgetting).

To be clear, I think there's a lot about memory that is still not well understood, and my description is simplified and I certainly may have remembered some parts wrong, but from what I've read and been told about how the brain works, that's the gist of it.