A lot of people seem rather Emphatic about having the madness rant next.
1) Have fun with the causes.
Most of the madness I’ve seen portrayed in fantasy stems from either torture—or other causes of intense pain, but I refuse to type that out each time, so “torture” it shall be—or jealousy. Character A spends a long time in the villain’s dungeon. Ergo, he goes mad. Character B is really, really in love with Character A, but is convinced that his true love is Character C. Ergo, she goes mad. Meanwhile, Character C is in the corner being mad from something else, but because it’s not pain or jealousy, the author doesn’t pay much attention to her. Poor Character C.
Why not syphilis?
I realize there may seem to be no circumstances whatsoever under which the question “Why not syphilis?” is appropriate, but bear with me a moment. It has several advantages over the other two. For one thing, it’s physical, attacking the sufferer’s brain, so the author doesn’t need to perform the very delicate characterization work to create someone who would go mad under torture or from jealousy. (Frankly, I find it hard to believe in 90% of the madness cases I see, because it seems as if the stubborn, level-headed, practical character snaps too easily, but that’s me). For another, it’s something that would provide a convincing secret for the character to keep, without being so over-the-top that it makes the reader snort when she finds it out, or so minor that the reader spends the rest of the book thinking, “That was it?” Finally, in a love triangle, it provides a really, really convincing reason for Character A, once he recovers, not to marry Character C. If you must introduce a love triangle, this would at least be a novel way of solving it.
If you don’t want to touch the sexual diseases (you sure?), there are other means. A curse could be fun, and certainly more novel than a curse that dooms the seventh son to change shape or become a vampire, which is so last bad “paranormal romance” novel. Madness-causing mages could find work among nobles who want to do something to incapacitate their enemies but, for various reasons, need them alive, and they’d probably be less traceable than assassins. “Yes, poor Aunt Emeraldrilla. It’s sad, really. Put her in her rooms, there’s a good fellow. Now, where was Prince Coresth again?”
You could also have sheer, straightforward mental disorders, which don’t see a lot of play in fantasy—with the exception of multiple personalities, which often turn out to have a magical explanation—and a family inheritance of them. Perhaps they think it’s a curse, but it’s really not, and the reason all those magical healers are baffled is that they’re looking in the wrong place.
And if you go for one of the other causes, you stand a chance of creating…
2) True bastard villains who use madness as a weapon.
Torture is one of the marks of the Big, Bad Dark Lord. So it’s not a real surprise when the hero snaps (though it may be unconvincing, and is usually temporary) under his whip, or his acid, or his knives, or his poker, or his imaginative machine with the really scary name that the author just made up, but kind of looks like an Iron Maiden with a side of rack. After all, the Dark Lord is Evil. He’s the kind of person who would cause intense pain just to see what happens to someone.
But that’s not the same as using madness against his enemies. Madness may be a treat, but the pain is what he wants, whether for revenge or sexual pleasure or because the heroine kind of looks like the girl who spit on him in grade school.
Take someone, instead, who decides that what he would like is to be the power behind the throne. He takes pleasure in seeing people do what he wants, but he’s fundamentally lazy and doesn’t want the work of actually fighting his way to the top and ordering people around. Besides, he’s seen that usurpers tend to have short lifespans, because inexplicably there are peasants who will rally to the cause of any dumb kid with a crown-shaped birthmark. The king has conveniently died, leaving the queen regental and the young prince. But trying to kill the queen is risky, and he can’t seduce her because she’s suspicious of him, and giving her some kind of poison to induce a lingering illness is, well, chancy, because who knows when he might accidentally overdose her? Or she might be one of those pesky monarchs who takes small doses of poison to make themselves immune to them. They’re everywhere these days, you just can’t be sure.
What does he do? He either calls in a mage to strike her with madness, or he starts a rumor that she is mad, all on her own, and must be removed from the regent position for the good of the country. So sad, of course. It’s horrible, when a disease disfigures the mind and not the body. And to take them so young? Horrible, of course, horrible, thank you for your sympathies, my lord, yes, I was the one who found her raving that the young prince was a demon and she must slay him, I’m going to keep her in a room for her own safety, not a burden, not when it’s for the good of the country, but it’s so kind of you to say so…
Madness, believed in by other people, doesn’t have to be real. It’s a more effective prison than a dozen iron bars. Who’s going to believe someone who rants and raves about anything anyway? Particularly if the madness-inducing villain really does perform bizarre but meaningless rituals at certain points, just to confuse her, and thus make her sound even wilder?
By the time that the bastard gets what’s coming to him, your audience will be rooting for him not just to fall over a cliff, as seems to happen to so many Dark Lords, but to meet a lot of knives on the way down. Flechettes, for preference.
3) Tone down the gibbering.
Mad people don’t have to gibber and spit and roll around on the floor and tear at their clothes. Another example will do nicely, I think.
Imagine that you spend most of your day, alone, in a few small rooms of a house. You don’t need to do any housework, because the servants do it for you. You don’t need to take care of your children; there are servants to raise them for you, too. There is no television, no computer, no telephone, no recorded music, and you’re not supposed to read novels, because reading novels is not ladylike, as they might, gasp, mention sex. Your husband is out most of the day. Your friends might visit, except that most of them live a bumpy, uncomfortable, and perhaps dangerous carriage ride away. It’s not the social season, so there are no places to go in town. You’re not supposed to go to the theater without your husband, as that is also unladylike, and anyway, it is a vulgar, lower-class entertainment. You can sew, play the piano, or read conduct books and religious ones—possibly the newspaper, but you’ve already read it for the day. And that’s about it.
Quiet. Lots of it. Sitting in rooms while the clock pendulums sway very slowly…
This was the situation of a lot of Victorian middle-class wives. They could spend some time with others, but an awful lot of it was supposed to be spent alone, in quiet pursuits that must have really worn after a while. It’s no wonder at all that so many of them were diagnosed with “melancholy,” or “hysteria,” or another version of “women’s complaints” that we would now probably call clinical depression or a dozen other names.
Someone can go mad quietly, genteelly, and, above all, subtly. It doesn’t have to be a woman, though so many fantasy novels have women set up in exactly this kind of situation that I wonder the only noblewomen we see are the quiet, cow-brained “jealous” ones and the “feisty” ones who want to practice swordplay. And if you’re telling the story from the point of view of someone who doesn’t think much about this person’s situation, like an oblivious husband, then the madness may not be noticeable for quite a long time.
4) Show us the institution.
A large part of the common “madness” image was formed by London’s Bedlam (Bethlehem) Hospital, and the horrible conditions there. Yet, for all that, madhouses aren’t actually very common in fantasy. I can only remember one off the top of my head, in Glen Cook’s Deadly Quicksilver Lies, and it’s only a brief stop on the hero’s quest to try to find the bad guys; they sling him in there because he’s too close to finding out their dirty secrets. It was impressively nasty.
But, come on. Surely not all fantasy societies deal with their madmen and women by locking them up in the dungeons, or the attic for that matter. There may be middle-class families who don’t have the money to take care of them, noble families who don’t want the embarrassment associated with them, and times when the mad person is dangerous to themselves or others (please try to restrain the gibbering, though? Please?) and needs care that the family can’t provide. Madhouses may be the most practical solution, though not the kindest one.
And hey, madhouse attendant would be an unusual and impressive career for a fantasy protagonist. Imagine her encountering the one person that everyone thinks is crazy, but he, of course, has a different story to tell…
Oh, yes. That ought to be attended to.
5) Madness doesn’t have to be a 100% or 0% deal.
I’ve sometimes been mildly annoyed. Here is this interesting crazy person, who may also see visions that happen to be true or can help the hero on his quest, but still. Crazy with a capital C. His or her version of reality is going to be entertaining to read about. I hope the author will manage, if he or she writes from this person’s perspective, to make it entertainingly different, without resorting to stream-of-consciousness, because I have an allergy to stream-of-consciousness.
And, instead, it turns out that the person is sane, and everyone just thinks she’s crazy. And this isn’t a case where some bastard, or even a person with good intentions, has an investment in making everyone think so. People are just too blind to deal with “the shining light of truth,” or similar.
Oh, come on. Stir a little madness into the mix! Have her convinced that evil fairies and moon goddesses are chasing her, and only have the evil fairies turn out to be real! That would keep the suspense up. That would balance the reader on the edge. Too often, the moment the mad character looks up and says, “I’m not really mad, you know,” and the protagonist starts believing her, I know there’s no suspense there. All her enemies are real, she’s the chosen sought secret one who has the key to open the gate of the Blahblahblah on the full moon of Nahnahnah-Neener Night, and it becomes a fairly straightforward quest fantasy. The fact that other people think she’s crazy doesn’t really hinder them, either, because they run away from everyone who thinks that. In the worst cases, I wonder why the author bothered to include the madness at all; it would have worked with the protagonist just finding the hunted person hiding in his room one day.
Have the mad characters hover between sanity and in-, and have some people who are never quite sure about them. It makes things interesting.
6) Have madness creep up on people.
There’s something uniquely fascinating about having an enemy living inside your own head, which, I think, is why so many fantasies do include madness in some capacity. But authors strip that creepiness away too often by being unsubtle (see point 3), or having the character go mad quite suddenly (see point 1). Characters often recover from madness, too, even if they’ve spent a good part of the book spiraling towards it.
What about characters who spiral towards it, except the reader doesn’t notice, until she suddenly looks back and finds that their minds have gone “ping!”? That would scare the fuck out of people. That would set up a whole new level of horror and darkness. The buckets of gore don’t need to be dumped all over the place, and the werewolves can have an early night.
Now, granted, this takes a lot of subtle work, at least as subtle as setting up a character who will believably go mad from jealousy, and probably more. It has to be handled like a mystery, planting clues that only make sense later, but give the reader a growing sense of disquiet, and it probably also needs to be counterbalanced by pyrotechnics of another kind, such as battle, that will give the audience something to watch instead of focusing all their efforts on rooting out the main character’s secrets. But, done right, this can be a rip-roaring creepfest.
…I talked about mad protagonists all the way through, I think, or at least mad neutral characters, and not insane villains.
Oh, well. The insane villains aren’t exactly uncommon, anyway.