I had a similar day today to yours yesterday. I did still manage to get a few extraneous things done. But, as a distraction came up, I asked it are you more urgent than the other tasks on my list. Mostly, the answer was no. But a few times, the answer was yes.
Now, sometimes a short, unimportant task can be more urgent than a long, important task, because clearing yourself of it unburdens you, so is sometimes good to do immediately. But you will balance all of these things against the urgency of your top priority task.
This means, on a busy day, aka a day with many urgent important tasks, your "filler," do it right now tasks have to be equally urgent and important. On a less busy day, not only are your main tasks less important, but the filter for which "filler" tasks you let in lowers as well.
Is this unavoidable? I think yes and no. Obviously, if there is something you are repeatedly doing that are a waste of your time, you should stop. But other tasks, with a generally very low urgency, tend to build up over busy stretches. This is why I have a sink full of dishes at the moment: for better or worse, I consider this a low urgency task. (It's for the worse.)
But I need to clean my dishes eventually. If you have a slow day, and all you get done is laundry, dishes, exercise, answering emails, resting, drinking tea and reading the news, well, just look at it as clearing your stack for your next busy day.
That said, if you find yourself slipping into bad habits, yes, it might be time to find some high urgency tasks to kick your butt back into gear.