The 44-year-old took charge in July 2008, but recent defeats have seen Bayern crash out of the Champions League after a 4-0 hammering at Barcelona while Saturday´s 1-0 defeat by Schalke leaves them third in the German league.
"Of course, I am very disappointed at the moment," said Klinsmann in a statement on the Bayern website.
"Nevertheless, I would like to extend my heartfelt thanks to Bayern Munich, the fans, the trainers, the players and my colleagues for an exciting time.
"We have laid the foundations for the future. I still believe that the team can be German champions this season."
Former Bayern trainer Jupp Heynckes will take over as head coach for the remaining five league games with Hermann Gerland as his assistant and the pair will take training for the first time on Tuesday.
In a press conference in Munich, Bayern chairman Karl-Heinz Rummenigge scotched rumours the club had already approached Arsenal boss Arsene Wenger, but said the club aimed to have Klinsmann´s successor in place by May 31.
"I can tell you all: forget everything you have heard until now," he said.
"We have deliberately had no contact with any coaches."
And team manager Uli Hoeness said the team could now go "with renewed force to do the unthinkable and still win the Championship".
Both stressed that Heynckes was only a temporary solution and that the search for a new coach would begin as soon as possible.
"It is now very clear that this appointment is only valid for four weeks, for the next five games," added Rummenigge, who said Klinsmann was "surprised" at the news.
Klinsmann, who along with assistant coaches Martin Vasquez and Nick Theslof, will leave with immediate effect and was told of the decision in a meeting with Hoeness and Rummenigge earlier on Monday.
"We did not come to this decision lightly," said Rummenigge.
"But the results of the past weeks, the way in which they came about and above all, the situation we find ourselves in five games before the end of the season forced us to act out of a sense of responsibility to the club.
"In Jupp Heynckes and Hermann Gerland, we have two experienced coaches who will take over the running of the team until the end of the season."
Having guided Germany to third place at the 2006 World Cup, Klinsmann´s Bayern contract was until June 30, 2010, but he only lasted 10 months in his first role in charge of a domestic team and lost seven of his 29 league games.
The first signs of trouble came last September in the Bundesliga when Werder Bremen embarrassed Bayern 5-2 in front of their own fans.
And having been knocked out of the German Cup 4-2 by Leverkusen in March, dreams of repeating last season´s domestic League, Cup and League Cup treble quickly faded.
The alarm bells then rang when Bayern were hammered 5-1 at Wolfsburg on April 4 in the Bundesliga before being routed 4-0 at Barcelona at four days later in the first leg of the Champions League quarter-final.
And Saturday´s 1-0 home defeat to Schalke was the final straw as Rummenigge refused to speak to reporters when leaving the stadium.
Klinsmann´s replacement Heynckes already has a pedigree at Bayern having won two successive Bundesliga crowns in 1989 and 1990 in his first coaching stint in Munich between 1987 and 1991.
He then coached Spanish outfit Athletic Bilbao and German side Eintracht Frankfurt before moving to Real Madrid where he delivered the Champions League to the Spanish giants in 1998 after a 32-year wait.
Related story
Bayern Munich routs Arsenal to put foot in quarterfinals
