This thought used to haunt my brain as well and it has the potential to kill all your joy. Regardless of difficulties, you have to be able to enjoy things in your life or eventually life itself becomes a burden which could eventually become unbearable. Doing things you enjoy is not trivial or unimportant. Enjoying things is the difference between wanting to live on this planet and wanting an express ticket into the unknown. There's a balance, but ignoring the things that make life worth living tends to make life not worth living. Please pursue your joy.