This is less of a rant and much more of a listing of pros and cons. In this case, the pros and cons refer to whether or not to actually put stories and poems from your world’s mythology/history/culture into the narrative itself. Basically, it comes down to what kind of writer you are, but some things might also depend on the individual story, the poem or tale in question, how important it is to the plot, and so on.

Advancing the plot.

This is probably the most common use for stories and poems. The hero hears a prophecy chanted by a half-awake mage. A bard sings a song that will reveal a mysterious character’s dark past. The protagonist reads a fairy story that will later reveal the key to a ancient riddle. And so on.
Pros: Well, I just listed them. It can be done subtly- with the possible exception of a prophecy, which is usually supposed to be dramatic- and is a better way of advancing the plot or revealing an answer than the Gandalf ripoff sitting the hero down and telling him everything flat out. The author can deepen the world at the same time as he’s moving the story along, which is always a plus. If the song or poem or tale at first doesn’t seem to be very important, it can be a way of showing us what these characters do when they have time to pass in an inn or riding down those long and dusty trails to defeat the Dark Lord.
Cons: I said this was great when done subtly. The problem is that a lot of authors can’t do it that way. The tale all but jumps off the page proclaiming, “Look at me! I am THE ANSWER!” The reader can often then use this to figure out the story’s central mystery before the characters do, which is always a disappointment. (A lot of first-time fantasy authors tip their hands this way). Also, if the tale or poem is one that has a forced introduction into the story, like a bard just walking up and reciting it out of nowhere, it feels a lot more like a plot device.

Paraphrasing the story or poem.

This gets used sometimes when the writer doesn’t really trust his or her ability to get the story across in suitably mythic language. “He then chanted the old song of Arthur the Great, who came from the north and conquered the Hundred Kingdoms…” instead of “This is the song of Arthur the Great/ Who wore the Mantle of Heroes and the Crown of Fate.”
Pros: The writer using this sometimes really doesn’t have the ability or taste for mythic language, so the paraphrase is a good idea. For poems, it might be an even better one than for tales, since few fantasy authors are also talented poets, and a lot of fantasy poetry comes out sounding ridiculously laughable. It also leaves what was actually said up to the reader’s imagination, which can create a vision that satisfies that reader better than anything the writer might have said. For a superb example in paraphrase, read the scene in Guy Gavriel Kay’s The Darkest Road where he has characters competing for the crown of the Dwarves making speeches. He describes them, rather than speaking them, and since he’s a very good descriptive writer, it works wonders.
Cons: Paraphrasing creates a problem for the writer who then wants to use the exact wording of a story, poem, or prophecy later in the story. Referring back to a line like “the Mantle of Heroes and the Crown of Fate” does no good if that line was not actually spoken in the book. The reader will rightfully feel that this is a cheat. Also, the expository passages like “He then chanted the old song of Arthur the Great…” have the unfortunate tendency to turn into infodumps in and of themselves, only this time, instead of dialogue, we’re getting a secondhand recital we can’t really “hear.” This robs the passage of a lot of the vestiges of interest that remain. Expository writing is a tool that writers instinctively pick up, and one of the hardest ones to use well.

Showing off culture.

You want to give your readers a sample of your elven language, so you include a few lines of a song (which you may or may not translate). Or you want them to appreciate what a rustic backwater they’re passing through, so you mention that these villagers know the complete story of an event which has disintegrated into less than myth in the culture your heroes come from.
Pros: Done well, this works dazzlingly in fantasy. A lot of fantasy worlds don’t have the printing press, so oral storytelling is the great storehouse of their culture. It just makes sense that your rustic backwater would preserve the story, handing it down and remembering it long after the nobles, who might be able to read and who are more likely to be in contact with other cultures, will have given it up. It can also give your readers a taste of the alien. Tolkien has seven lines of an Elvish song in FOTR which is never translated in the text. Yet it looks beautiful, and obviously means something. This is part of what gives his Middle-earth the richness and depth that so many imitations lack.
Cons: This can get really, really obtrusive if you do it too often, and start smacking too much of coincidence. “Oh, those villagers JUST happen to know the story they’ve been looking all across the world for? How amazing!” It goes straight back to the story or poem being a plot device. A fantasy writer who doesn’t know her limitations can take the opportunity to insert her awful poetry in there, and when she means readers to be swooning or gaping in awe, they’ll be laughing instead. And too much untranslated language will make your readers annoyed as hell. In small amounts, it shows off your worldbuilding skill. Entire passages in it, and you’ll probably have people skimming to get back to the story.

Referring to a back history.

So this character is singing this song not because it’s going to be important in the future or even because it’s typical of his culture, but just because you’d like people to know about the first time a hero defeated a dragon. Or you want to share the legend of the princess who danced in the forest of glass a hundred years ago, so that gets raised. Or your character had a nurse who told him the fairy tale about the maiden and the foxskin, which was always his favorite, and which he keeps remembering even though his nurse died twenty years ago.
Pros: This creates another level of meaning in the story, reminding- or persuading- your readers that your fantasy world doesn’t begin with this story and doesn’t end when it ends. It has a richness and depth of history that goes back, and back, and back. This is similar to showing off the culture, but that often includes the present as well. Strictly historical stories, songs, and poems can show what scholars get up to. It can create “depth” by reminding people that a battle was won once before, without necessarily guaranteeing, as a prophecy would, that such a battle could be won again. Or it can subtly foreshadow something that some readers might understand and some might not. Tolkien does this with the song of Beren and Lúthien that Aragorn sings to the hobbits in FOTR, since Aragorn and Arwen are basically reliving the story of Beren and Lúthien. Yet other than a few comments on Arwen’s beauty being like Lúthien’s and the “eagerness” in Aragorn’s face as he sings, we don’t get many direct references to it. The immense mythic structure surrounding the relationships of Elves and Men in The Silmarillion is being hinted at here, without destroying everything for readers who want to go exploring.
Cons: This demands that the writer have a coherent backstory before she start writing, or else be good at making such things up on the fly. Having a scattered and disjointed history that you aren’t good at filling in will make most attempts at historical art a disaster. Also, the stories should be genuinely inspiring and interesting. This might not sound like much of a requirement, but just as with the idea that any poems should actually be the products of a skilled poet, it’s one that slips by some writers. I’ve found myself skimming past fairy tales and prophecies before, especially prophecies. Good prophecies are darn hard to write. (That might be part of my next rant).

Using bits of the story as repeating motifs.

This is a pretty common one. The character hears a proverb, and keeps repeating the proverb to himself for the rest of the story. A tale gets recited around the campfire one night, and after that phrases from the story are touchstones between the heroes. A song is important to the heroine as a child, and she thinks of that song when she sees the place it describes for the first time with her own eyes.
Pros: This can bind the story together without requiring the kind of effort that historical references do. A proverb doesn’t require as much effort to think up as a poem. Also, it can lead to better characterization. What kind of person is this hero, that that strange proverb is so important to him? It might even play into the structure of the story. Guy Gavriel Kay’s Sarantine Mosaic duology is structured like, surprise, a mosaic, and bits of important scenes and lines from certain characters are always popping up later in the story. Even extremely minor characters who could safely show up only once might have a cameo later. Given that his hero is a mosaicist and thinks like a visual artist, this works on the level of characterization as well.
Cons: It bears the peril of all repetitions: It can get sickening. By the time I finally quit reading Jordan, I was ready to spit every time someone said, “The Wheel weaves as the Wheel wills,” or “Death is lighter than a feather, duty heavier than a mountain.” They’d been repeated too many times, over too many books, and any freshness or resonance they had left was long gone. And as far as characterization goes, they can start acting as crutches instead of roads leading the reader (and the author) deeper into the protagonist’s mind. Rather than, “What kind of person is he?” it becomes “He’s the kind of person who says this phrase.” That’s no more characterization than decreeing that your heroine is blonde and blue-eyed and can read minds and is kind to animals is. It’s a flat and simple trait. Show us the “why” and the “how,” not just the “what.”
Like I said, very personal. I dropped songs and poems into my stories all the time when I started writing fantasy, because that was how Tolkien had done it, and I thought that was part of how you wrote fantasy. Later I mostly cut those. I also have to say that other than Tolkien and Susan Cooper, not a whole heck of a lot of fantasy authors have impressed me with admiration for their poetic skills, especially in prophecies.