I don't want to alarm anyone but I think I found that secret. One of them anyway. It's this: There are no right answers. Not to any question about writing.
There are answers that might be better than others but even that word 'better' is up for debate. Better for whom? And why? Who gets to decide that? Maybe now you have some insight into how and why I overthink everything ever. It's a gift and a real pain in the ass. It turns the whole question of what to write next into a whirlwind of second guessing and analysis paralysis. There's very little fun involved. Given all of that, I had to come up with a system. Two systems actually.
1. Orders of precedence - write first anything constrained by contract or owed to another entity you do not wish to disappoint. If no contract, write first anything that is already underway and write it to completion. If nothing is currently in the works or contracted, write whatever will feed the fire of whichever audience burns brightest. If there's no obvious audience salivating for a book from you, write what you please.
2. Bright, shiny new ideas that popcorn up while I'm in the middle of a WIP must take a number and stand in line.
In practice, this means I have a way to direct myself toward completing a project. I have a few checks to weigh story ideas against. Most of the time, there's a clear answer to what I should write. The second system - the one about new ideas is a defense mechanism. There's nothing like hitting the sagging middle of a WIP to generate bright, shiny, compelling, BETTER story ideas. The big secret about that, though, is that those stories, too, have saggy, boring middles to be muddled through. Ask my pile of half finished projects how I know this. I had to come up with a redirect that worked for me. When a new idea comes courting, I stop what I'm doing and jot down notes about the idea - bare bones. I want just enough to scratch the itch of capturing this flash of seeming brilliance </sarcasm> so I can follow the thought later when I come back to the idea to flesh it out. If I can. By dignifying the idea with a page of summary and a named file folder in Dropbox, my brain lets me get back to what I had been writing without the tugs and pulls of needing to chase the new thing.
However. I'm finding I need a new system to handle what I should write next and that's because there comes a time in most every life where writing fails. Or we fail. Or imagination fails. Or physical or mental illness claims all the space and the spoons we had to write. Sometimes the 'should' part of 'what should I write next' is the worst and most self-sabotaging question we can pose to ourselves. Some days you can't answer that 'should' question. Then it might help to default to asking 'what can I write?'
Just because we sit or stand at computers all day and make stuff up doesn't mean we aren't working our hearts out. We're spinning some of these stories out of the essence of ourselves and yes, it's likely that we get energy back from what we do, or else why do it? But the rest of our lives might not be quite so generous about returning dividends on spent energy and we overspend ourselves. Writing can be a refuge, it's just a good practice to treat a refuge gently, with respect. It can be freeing to throw off expectation for a little while and plunge into a story that doesn't have a logical place in your catalogue and nothing to recommend it other than it might be fun to try to write.