Nicolas Hofler cancelled out Dayot Upamecano's opening goal for Bayern before Holer converted from the spot late on after Jamal Musiala handled in the box.
Freiburg lost last year's final on penalties to RB Leipzig but Holer was nerveless, smashing home his kick to send his side through to the semis for the second year in a row.
Bayern, the record 20-time German Cup winners, have failed to reach the last four for the third consecutive year, while Freiburg hope to lift the trophy for the first time.
An "angry" Bayern captain Thomas Mueller said "that certain something was missing... it's very bitter."
In his second game in charge, Bayern manager Thomas Tuchel said the team needed to take responsibility for the loss.
"I'm mega disappointed, but why should I be mad? It's our own fault, we know that."
A typically calm Freiburg coach Christian Streich barely cracked a smile after his team won away to Bayern for the first time in club history.
"We defended passionately and we defended well," he said.
"It's a nice evening, but you don't need to overstate it either."
Bayern took the lead midway through the first half when France defender Upamecano rose highest to head in a Joshua Kimmich corner.
The hosts continued to press but Freiburg would be next to score.
A scuffed clearance from Bayern winger Kingsley Coman fell to Hoefler, who unleashed a screamer from outside the box to level the scores.
"Such a shot, in Munich, in such a game - it's the goal of my career," Hofler told German TV.
As the clock wound down, Freiburg continued to sit deep under pressure, managing only the occasional foray into Bayern's half.
However with extra time looming, Musiala's attempt to block a shot saw the ball bounce off his arm and the referee point immediately to the spot.
Freiburg forward Holer stepped up and whacked the ball into the top of the net, giving Yann Sommer no chance.
Bayern will at least have the opportunity for immediate revenge when they head to Freiburg in the league on Saturday.