Villains are sometimes a dime a dozen, so what separates the good bad guys from the ones that are merely mediocre?
Up to this point I've only covered the characters that are directly under the player's control. But just as important to the story in a lot of cases is the force that you're up against: the antagonist. Whether it be one man/demon/god or a shadowy organization that you see a few select members of, the antagonists drive the story forward just as much as the player does. Compelling antagonists give the player a reason to continue moving forward: to defeat them, stop their plans or run the damage control if the unthinkable happens and they succeed (well, it's not unthinkable because it does often happen, but you get my drift).
Much like protagonists, villains can be mostly silent or have personalities. Unlike with the protagonist, however, since the character is never in control of the player there is a level of detachment that means that either choice is seemingly a little easier to pull off. Let me give you some classic examples with some analysis, and where things have gone right and wrong.
It might not come as a shock (in fact it better not) but I really like the character Grahf, from Xenogears. He's not the main antagonist of the game, but he is one of the major villains present therein. When he first appears all you really know about him is that he has some remarkable power, since he has the ability to endow other creatures and people with terrible powers. Likewise Grahf has some connection to the main protagonist: Fei Fong Wong: why does Grahf pursue Fei? More distressingly, why does he claim that it is the destiny of both himself and Fei to destroy God?
Grahf never shows up for extended periods of time, but whenever he does he leaves quite an impression. He's certainly not a good guy, but his goals are also not in-line with some of the other antagonists. Through leaning his backstory you come to see that he could be considered a tragic character, but that the tragedy he suffered does not condone or justify his terrible actions. He's a complex character and I believe he comes across as a rather well designed one, although that could just be my obvious bias talking.
Let me contrast with a rather surprising choice for what I'd consider a silent villain: a somewhat legendary and divisive figure, Sephiroth. The main antagonist of Final Fantasy VII, Sephiroth also makes spare few appearances within the game, and those he does make often have him speaking much, if at all. Despite this when talk of antagonists, particularly ones in RPGs, comes up his name is always among the first bandied about. It certainly doesn't hurt that he has a striking visual design, but consider that beyond this and his defining moment of killing Aeris (or Aerith, or whatever the hell) he isn't exactly the most fleshed out villain. His motivation remains rather vague, and he mostly comes off as a lunatic with somewhat deep complex regarding a being from space which may or may not be his mother -- on a side note I do not consider Sephiroth to be suffering from an Oedipus Complex in the strictest sense of the term, but that's neither here nor there -- certainly he is evil, but he's not really an overly remarkable character when put on paper.
So what accounts for his popularity? Well, as I mentioned his design has certainly endeared him to fans of both genders (which for the sake of my own sanity I'll leave at that), and there is an aspect of tragedy in his backstory; a lone warrior, driven insane by the weight of what he finds when he seeks his origins. Still, aside from that and a large dose of being stoic, there's nothing that special. Perhaps that is why he is so popular though, because he displayed little characterization and is best known for his actions the aforementioned cases of people filling in the blanks in terms of background allow them to give him whatever sort of set of characteristics they'd like him to have. In the end as well I guess you can't discount the whole "badass with the big sword" effect either. I'll also spare you the phenomenon that many have seen as a problem, where villains are given far more sympathy than they should rightfully have (if you're that curious tv tropes has a large section under "Draco in Leather Pants")
I would argue that in the end it's a lot less of a fine line to walk to make a villain than a hero. You can screw up on some points from a villain and the audience will probably discount it as stuff that they don't need to know about and aren't bothered by not knowing. Why does a bad guy do bad things? Because he's the bad guy of course. In the end you seem to need precious little more than that at times. As long as they do despicable acts and stand in the hero (and thus the players) way then there's a lot of leeway to have. Of course the a good or great villain will do more than this, but at the same time be able to be more complex, and perhaps even sympathetic to the audience, or at the very least have an understandable -- if still unforgivable -- justification.