McCullum, who was appointed in 2022 just after Ben Stokes was made Test captain, helped transform England's red-ball fortunes with a swashbuckling brand of play that was quickly nicknamed 'Bazball'.
While he has a winning overall record coaching the side in Tests, England have struggled more recently, losing seven of their last nine long-form games, including a 4-1 defeat in Australia in the Ashes this year.
He will continue as head coach of the men's white-ball teams, which the 44-year-old took charge of in January 2025 under a contract running through 2027.
McCullum's departure comes in the wake of England's recent 2-1 home Test series defeat against New Zealand, near the end of which Stokes announced his retirement from international cricket.
"Of course I'm gutted not to be continuing, but I respect the decision," McCullum said in an ECB statement.
"I've absolutely loved coaching the Test team, and I'm incredibly proud of what we've achieved together," he said.
England's resounding Ashes defeat this year led to an ECB review of their performance, broader criticism of which included their seemingly casual preparations for the marquee series.
There have also been questions about the side's dressing-room culture, with an incident involving Harry Brook and a nightclub bouncer in New Zealand last year remaining a distraction, while Stokes and fast bowler Gus Atkinson were suspended for England's second Test against New Zealand last month after breaching a team curfew during a night out in London.
ECB Chief Executive Richard Gould paid tribute to former New Zealand captain McCullum's work.
"Brendon breathed new life into England's test side during an exciting period that delivered some memorable victories, and we're grateful for everything he has given to the role," Gould said.
The ECB's search for a new head coach of the Test team will begin immediately.