This is a rambly rant, in which I write about points as they occur to me, and not all on surprise endings.

1) The surprise ending will ideally match the tone of the rest of the story, or be a chord of it.

I have a serious, serious bias against stories that end with puns. But if the rest of the story has been humorous or punning fantasy (why, yes, I do consider them different subgenres—the one is Terry Pratchett, the other is Piers Anthony), then it can fit, though I still think it’s not always good to do so. When puns are the greatest humor you can think of, I believe you need to put the jokebook down for a while.
But what about a horror story that ends with a pun? Or a humorous, light fantasy story with cartoonish violence that leaves no one dead, where the protagonist at the end expires in a bloody and utterly serious battle? These don’t work, for me. They feel as if the author has confused “surprising” with “totally and completely blasting away all sense of the prior story.” (For a different expression of this, see point 6).
Please, don’t get focused on the surprise ending to the exclusion of all else. Pick one that will work for your story. It should certainly alter things, or it wouldn’t be a surprise. But why do you want to throw out the previous tonal work? Often the story, or the ending, or maybe even both, isn’t working if that’s the case. The author may have gotten bored, or thought of the pun first and then slapped any old story above it in his eagerness to share, or decided that, damn it, this story and this ending are going together, no matter how poorly they fit. But do reconsider before using something like this. It could very well improve your work.

2) Building on clues in-story is damn useful. Be quiet, be discreet, be slow.

I prefer this tactic for in-story revelations. Unless they really are meant to change and alter everything that’s gone before—see point 5—they have to grow from clues that the author’s already provided. The trick is to keep the audience from recognizing them as clues at the time.
What are some ways to do this? Well:

  • Include bits of character introspection that will, when the audience reflects on them, contradict or jar against something another character is thinking, but aren’t the exact opposite of those thoughts (which is often too obvious). For example, perhaps your heroine believes the villain is challenging her to be his equal and offering her clues about his past. Meanwhile, in the villain’s scene, he thinks about how enemies would never make it easy for one another. When the villain springs a horrible trap, then we can see whose perception was flawed, his or the heroine’s. Did the villain subconsciously betray himself? Or did the heroine, who thought she was gathering material to defeat the villain, follow the path of his will like a good little puppet?
  • As an alternative, don’t have the characters introspect about things happening around them. Many fantasy characters spend way too much time in self-analysis anyway. Build up the events slowly, naturally, making the characters’ acceptance of them equally slow and natural, and then provide a sudden alternative explanation. It could have been there all along, but the reader, lulled by the character’s unconcern, is not looking for it.
  • Have phrases repeat out of context, but give them multiple meanings. With only one meaning, they become very unsubtle and unsuitable foreshadowings, like the character who hears a line from a song about a hero dying and thinks of it over and over in relation to his best friend. Is it a surprise when his best friend dies? Of course not. If the character thinks of the line in relation to his best friend, himself, a random person he hears about who dies trying to save a bunch of children, a dog who dies defending her pups, and his king, then when it finally shows up in the context it was destined for all along, the audience will recognize it on (at least two) levels at once.
  • Have the revelations develop as a rational, connected explanation between several “intuitive,” seemingly unconnected occurrences. Here’s a good use for that fuzzy language like “somehow” that I’m always ranting at. When the character thinks that “somehow, she made him uneasy” and “somehow, he saw her master’s power shining through her” and “though he could not say why, he felt his stomach tighten at the emotion in her voice,” and so on, it won’t seem, done right, like a bait-and-switch when the woman he was traveling with turns out to be not the main villain’s lieutenant but the main villain herself.

3) Don’t rely solely on language to get your epiphany across.

This is a short one, because it’s simple. You can’t make something surprising or shocking by tacking on exclamation points, capital letters, italics, or puns. Tricks of language pump the moment full of artificial drama, and no more. If it was completely obvious and prosaic that the “boy” traveling with the swordmaster is really the princess in disguise, then having everyone exclaim “He’s a SHE?!?” when the reveal comes is not going to make it marvelous.
(This is also another reason I dislike pun endings. The author’s done nothing on the level of plot, characterization, theme, setting, or anything else, just dumped in a trick of language and expected me to accept it. Not this time, buster).

4) Either linger on the protagonist’s emotional scenery, or build up a complete person who will react in a knowable way to the surprise ending.

Sometimes the author does an excellent job with a revelation or surprise ending on the level of plot or theme, but falls down on character. This isn’t a great problem if the story isn’t about character. But quite a few of them are. And if we can’t tell how the protagonist is going to react to that surprise ending, or if she has a reaction that makes no sense, the surprise is, once again, not a surprise. It’s just something the author thought would be cool and tossed in there, and if he really wanted it to work, he should have chosen a different viewpoint character. (Something like this will be part of the rant on choosing a viewpoint character).
For in-story revelations, spend some time on the protagonist’s reaction. She’s just found out that her sister has betrayed her, that the Dark Lord is her father, that she’s really the heir of some secret and ancient bloodline prone to genetic insanity—there, that’s better. It might be a revelation you’ve been building towards all story, or one that happens fairly early on. Whichever it is, it makes no sense to toss this bombshell at the character and then dash past it, or have her total reaction consist of flat statements like, “She was shocked.” Don’t tell us, show us. If showing is ever appropriate, it is here. It might have to wait until the characters reach a safe place if they’re on the run, or for the full implications to sink in, but they had better damn well be there. The revelation is a revelation only if you show it rearranging your character’s emotional scenery. Otherwise, it’s no different than any other random piece of worldbuilding; the character might have known this all along and just never bothered to mention it. That’s the feeling I get from some of these fantasy “secrets.”
When you end, you don’t get to spend more time with the character. The audience must know what the character is going to do with the surprise ending. Obviously, if it’s one that oversets her whole worldview, we might have a pretty good idea. But if it’s not, if she just found out that she’s prone to genetic insanity, and the story ends, then what? Are we supposed to consider that all her actions in the story are now insane? Are we supposed to worry about her future? Are we supposed to decide that other members of her family in the story are insane, but she’s not yet? If the author has not done a good job of connecting the protagonist to madness and/or family history somehow, we might know what she ate for breakfast and what she would think of her sister working for the Dark Lord, but we won’t know this.
Work on it. Once again, don’t sacrifice the story to the ending. Work with both so that the ending really is a surprise, not random.

5) Plot “twists” are for the center of the story. Keep them out of the ending as much as possible.

I can say this because of the visual I have for “plot twist,” which is strands braiding and, yes, twisting in a new direction. I assume that after the twist, we’re going to get something more. The author will have to show how that changes the plot, the assumptions the characters have been working with so far, the other revelations the plot strands were building towards, and so on.
In an ending, you don’t get any more strands. A surprise ending has to be able to feel like not only a surprise, but also an ending.
Yes, serial fiction is an exception to this. Yes, stories that are meant to be open-ended are an exception to this. But I don’t think every fantasy that throws a plot twist into the very ending is serial or meant to be open-ended; in fact, sometimes the author openly states that this is the end of the story/series/whatever. It’s the author, again, wanting to do something cool and unusual, and forgetting that that’s not enough.
So what changes in plot do work for an ending? I present to you Limyaael’s Personal List of Neat Ways to Twist And Yet End:

  • Once again, building on clues. The clues can function on the level of plot as well as character; in fact, plot’s probably easier to do that with than characterization is, since you can manipulate objects and minor characters and dialogue and narrative as well as protagonist viewpoint.
  • An ending that pushes the protagonist at a dilemma and then lets him invent a third way out. This has to be done well not to come off as gimmicky, and it can’t be utterly random (see point 6, again), but I love it when it succeeds.
  • Similarly, letting a particular way out seem to exist and then canceling it by virtue of something else that’s been lurking in the story all along, unnoticed. I like brutal fantasy, and I admire tragedy, so that’s why this works for me.
  • Acknowledging an open ending by letting the characters think about it and be at peace with it. If I suddenly realize that the protagonists don’t know if they’ll ever win free of their debts of loyalty to the evil emperor, but the protagonists pick up the optimism and start devising a new way to get out of it, then I feel more at peace than if the author simply pulls a “Ha-HA! Will they or won’t they get out of it?” trick (As I’ve said before, that kind of ending only works once, and since a ton of your readers will have read “The Lady or the Tiger?” you’re kind of beaten to the punch already).
  • Having the protagonist meet the “twist” with a “twist” of his or her own. This is where protagonist personality gets to be more than a helpless tool of the plot. A surprise that may look gimmicky can have a whole new light thrown on it by the protagonist’s response, something new and interesting and which only your character, not Generic Hero #37363, could think of. This, obviously, also depends on good characterization prior to the ending.

6) Surprise endings that throw out all the story’s rules are suspect.

The biggest examples of this I know of are stories that play quietly by all their genre’s conventions until the very last paragraph, and often combine genre-switching with deus ex machina. The detective is trapped in a room he can’t get out of. How does he get out of it? Aliens descend to save him. The protagonist has used all her three wishes and doesn’t have a fourth as the villain backs her up to a cliff. How does she escape? The villain whips off his mask and reveals that the whole thing was just a play, and he and the protagonist take their bows. A scientist who’s been investigating a strange group of life-threatening diseases finds the final secret. What is it? God has been infecting evil people with the diseases. (So that’s all right, obviously).
There are smaller examples than this. I’m sure you can think of some. The point is: if you set up rules and play by them faithfully until the surprise ending, you’re only cheating yourself. It’s random, rather than a surprise. The reader can feel you wrote the whole story solely for the “Gotcha!” moment, and an author looking cool and smart at reader expense is few people’s idea of a good time. Most of all, it can look as though you couldn’t think of an ending and slapped whatever you could think of on the poor story in lieu of actually finishing it off, which is not actually, y’know, being a brilliant writer so much as a sloppy one.
There are, possibly, some brilliant examples of this, though I can’t think of any. I don’t consider stories that pull a reveal like this when there have been clues hiding in the text to be stupid or random. If aliens are a plausible explanation for the detective’s bad guys, if certain things pointing at a play appear in the story earlier than the ending, if the scientist is intensely religious and there have been clues to indicate God was probably behind the diseases earlier, okay.
But a lot of them don’t do this. Waaaaay too many authors think they’re being cool and smart, and forget that “Gotcha!” is not actually a lot of fun for anyone other than the person playing the game.
And, yes! The brutal fantasy rant is next.