Readers demand a sense of progress. Nearly every time I've felt bored with a story, it's because there was some "lull" where it didn't feel like any of the characters were failing to move toward their intended goal. (This is part of why it's so difficult to write a story about a character dealing with depression.) The audience wants to feel like we are marching toward some conclusion. The "video game" way to do it is the way that Zelda does: "Collect the six medallions!" or "Rescue the seven maidens!" It allows you to have a continuous through-line toward a singular goal even as you explore a variety of different scenarios and settings. It feels like every scene is moving us toward the conclusion of the story.
Constantly moving the goalposts violates our sense of progress. "Your princess is in another castle" is perhaps the most egregious of this, because it effectively says, "Haha, you thought you were getting closer to your goal, but it was actually all for naught! Your efforts in this castle were a complete waste! You're no closer to your goal than when you entered this castle!" Note, however, that "moving the goalposts" isn't the same thing as "changing goals." For example, in the film Star Wars, Luke Skywalker's goals escalate as the story goes on. When Luke and friends get captured and trapped on board the Death Star, Luke's goal is "escape." Then, when he discovers that Leia is trapped on board, the goal becomes "rescue the princess." Finally, after they escape, the goal becomes "blow up the Death Star to save the rebels who are under attack." With each escalation, the stakes increase. When Luke and friends succeed in rescuing the princess, their story isn't over yet; there's another mission that they have to complete. But we still feel like we've made forward progress.
The latest episode of the Writing Excuses podcast delves into this subject. I highly recommend giving it a listen.
As an aside, I actually don't have a problem with being told "Your princess is in another castle" in the context of a game like Mario, because when I'm playing a Mario game, I'm not playing because I want to rescue the princess. I'm playing because I enjoy pressing the A button to jump and bounce on goomba's heads. So the "fake ending" actually is a small delight, because moving the goalposts back actually means I get more of what I came for. Mario isn't really a plot-driven game, and as such general principles of plotting don't really apply to it. And Mario provides us with a sense of progress on a different level, because when we clear world 1, we get to advance to world 2, and clearly going from 1 to 2 is a sign of foward progress, even if we didn't succeed in rescuing the princess.
There's a point to be made here about the mystery genre, where you often have "red herrings" and other places where characters pursue the wrong conclusion, only to discover, "Sorry, this suspect is actually innocent; the real murderer is hiding in another castle." So all of the time that we spent chasing this suspect is effectively "wasted" because they weren't the real killer. Except it really isn't. The mystery is set up so that we
anticipate and
expect uncertainty, so we're really not all that suprised when the first suspect turns out to be innocent. In fact, finding suspects innocent can actually feed into our sense of progress. If we begin the story with a list of eight suspects, one way to find the real killer is by process of elimination. Every suspect that we cross off the list gets us one step closer to discovering the real killer. There's also the fact that the pursuit of a red herring is not all for naught. Often, we discover that the suspect is innocent, but in the process of chasing him (or possibly interrogating him after the fact), the detective gathers one more clue or critical piece of evidence that allows them to re-orient themselves and begin marching in the right direction. It never feels like we are "marching backwards" or "running in place," because if the story is done well, even the time "wasted" chasing red herrings gets us closer to solving the mystery.
Mysteries get away with red herrings and false information to mislead the reader, because
that is the entire point of a whodunnit story. Reader expectations are already prepared for surprise. They
want there to be a surprise, otherwise they wouldn't have picked up a mystery novel. The reader reaction is never, "Oh wow, I wasn't expecting there to be a surprise," it's always, "Oh wow, I wasn't expecting you to surprise me
in that way." You can surprise the reader by fulfilling their expectations in unexpected ways. By setting up the story as a murder mystery, the author has already primed audience's expectations, so when their expectations are subverted, it feels like you are fulfilling a promise, not breaking it.
If a "plot twist" frustrates readers, it's probably because it wasn't foreshadowed properly. If you want to set up a "main villain" and then later reveal that he's not really the "big bad" and is actually just a puppet in a larger scheme, you have to plant those seeds of doubt early. When your reader gets to the big reveal, the reaction you want is, "OF COURSE! The writing was on the wall the whole time! How did I not see that coming?" If your readers say, "What the heck? That came out of nowhere," then it means you're doing something wrong.
