"Mary Sue" is a one of the newest literary terms, but in reality, it's just giving a name to an older problem in writing.
A Mary Sue character is when the author inserts him or herself into the narrative in a fashion of wish-fulfillment, and the Mary Sue character becomes so overly powerful, popular, or amazing that it undermines the reader's willing suspension of disbelief. In other words, the writer is writing themselves into the story as the hero of the piece, and alienating their readers by doing so.
The name has its origins in fanfiction, where-in a fan of the particular TV show, book, or movie would insert themselves (often under a pseudonym or avatar) into the story. The character would the save the day, romance the characters the writer was in love with, and completely ruin the original narrative by not replacing it with something that makes any sense.
(Fanfiction was a large portion of the nerd internet culture in the 1990's, but it's popularity diminished when the internet's capacity for transmitting data allowed for both pictures and videos to be sent more quickly and the rise of websites like YouTube.)
The most noted Mary Sue is the character of Wesley Crusher, on Star Trek: The Next Generation. Wesley is believed to be a stand-in for Star Trek creator Gene Roddenberry, as he is far too young to accomplish most of what he does. Mary Sues, however, also appear in literature.
Two contemporary literary characters noted to be Mary Sues are Stieg Larsson's Mikael Blomkvist from The Girl with the Dragon Tattoo and Anita Blake from Laurell K. Hamilton's urban fantasy series. Essentially, both of these characters have lost their original narrative purposes at some point in their stories - Anita, especially, has grown so overly powerful and overly sexual that it's hard to find a real person in her character at all anymore.
The problem of a Mary Sue, though, is one that has plagued writers of all generations, not just this one. Long before there was fanfiction, there were novels written as thinly veiled memoirs, where-in the author's choices would elevate him or her past where they actually succeeded in life. These were the original Mary Sues.