Leicester trailed 2-0 with 18 minutes remaining before Richie De Laet began Leicester's comeback in the 72nd minute. Jamie Vardy dragged Leicester level with a close-range finish and Nathan Dyer won it in the 89th, heading in while being flattened by goalkeeper Brad Guzan.
Goals from Jack Grealish and Carles Gil had put Villa in command, but Leicester's win puts it second in the table behind Manchester City after five rounds.
"Fantastic, fantastic, spirit and good character," Leicester manager Claudio Ranieri said. "At 2-0 I watched my players and they believed everything was possible. You can lose, but you have to believe that anything is possible."
Dyer collided with the onrushing Guzan just as he got his head on the ball. He remained on the field for a couple of minutes after his goal but eventually got to his feet.
"Until I got up and the ref said 'You've scored' I didn't know," Dyer said. "I was watching the ball all the way, but it was a nice feeling to score. I was just dazed."
Earlier, Ryan Mason's late goal gave Tottenham a 1-0 win over last-place Sunderland.
Mason scored in the 82nd minute in northeast England, taking Erik Lamela's through-ball before clipping over goalkeeper Costel Pantilimon.
Sunderland is last on goal difference after just two points from five games, while Tottenham climbed to 12th on six points. Man City leads on 15, with Leicester second on 11.
Related story
Tottenham remains unbeaten in EPL but held by Leicester
