What worldbuilding do I do that isn't necessarily about the story itself but is a personal indulgence? It depends on the subgenre I'm writing.
There is so much worldbuilding necessary in second-world fantasy--the kind of story that doesn't take place in a recognizable period on Earth--that everything I include has to be relevant to the story; otherwise, the worldbuilding dominates the plot and character development...or I end up with a 300k tome. The first round of cuts I do in my second-world fantasies are the TMI details of the world. Does the reader really need to know where the water is sourced? Purified? Only if there's a plot-relevant problem with the water--scarcity, poisoned, monsters--or if an important character is employed in that industry. Infrastructure usually takes the first hit in the cuts, because I absolutely write that info in the draft. The administrative necessities of running a nation/tribe/horde? Again, it gets cut if it's not plot-relevant. A little bit of mundane is necessary, but too much can bog down the pacing and distract from the story rather enhance it. Weeds. I know them too well. Alas, getting lost in them is part of my process of immersing myself in the world I'm creating. Gods bless editing.
In contrast, for my upcoming Urban Fantasy series (releasing Jan 30th!), the worldbuilding there is all about pointing out the uncommon amid the common. That is where I allow myself the luxuries of sneaking details that--if they were missing--wouldn't impact the plot. Silly things like how dust is applied to shelves or why guys are wearing flip-flops in the dead of winter. I indulge my love of creating fantastical explanations for the one-offs in our everyday lives. I try not to be heavyhanded with them, but they do make me snicker.
In the end, worldbuilding indulgences are a lot like pepper on pasta. Flavor enhancements.