The 44-year-old spent 20 seasons with the New England Patriots, winning six Super Bowls, before relocating to Tampa Bay and leading the Buccaneers to a Super Bowl championship last season.
The ESPN report came as something of a surprise after Brady led the league in touchdown passes and passing yards this season and could still potentially win his fourth league MVP trophy next month.
Brady's agent Don Yee told Reuters that people should wait to hear directly from Brady about his plans.
"I understand the advance speculation about Tom's future," Yee said in an email.
"Without getting into the accuracy or inaccuracy of what's being reported, Tom will be the only person to express his plans with complete accuracy.
"He knows the realities of the football business and planning calendar as well as anybody, so that should be soon."
The Buccaneers said they could not confirm the news.
"Unfortunately, we are not in a position to know anything relating to the various reports," the team's communications manager Chris King told Reuters.
"I don't have anything to offer at this point in time."
The Buccaneers tried to convince Brady to come back for another season but he chose to prioritize his family and his health, ESPN reported.
But Brady's father disputed the ESPN report, telling KRON4 News that his son was not retiring.
Brady called the Bucs to inform them that he was "not even close" to making a decision on retirement, the Athletic reported late Saturday, citing a source.
Earlier in the day, Brady's company TB12 appeared to confirm the ESPN report when it celebrated Brady in a since-deleted tweet listing his accomplishments.
"Thank you for it all," the tweet said.
ESPN said it stood by its reporting, adding that Brady had wanted to make the announcement on his own terms.
'SATISFIED' WITH CAREER
In a podcast recorded after Tampa Bay were eliminated from the playoffs by the Los Angeles Rams last weekend, Brady said he was "satisfied" with his career, ramping up speculation that he was ready to walk away from the game.
"I said this a few years ago, it's what relationships are all about," Brady said.
"It's not always what I want. It's what we want as a family. And I'm going to spend a lot of time with them and figure out in the future what's next."
Brady is the father of three and he and wife Gisele Bundchen have been building a house together near Miami.
"I'll know when I know," Brady said during the podcast.
"I think for all of us, you know, we can all decompress a bit. It's been six straight months of football. Every day consumed by day in and day out football. And I think now it's just some time to spend some time with my family and spend some time with my kids."
The Northern California native, who was selected with the 199th pick of the 2000 NFL draft, is the league's all-time leader in passing yards, passing touchdowns and quarterback wins.
He appeared in a record 10 Super Bowls, winning the game's MVP award five times, and his 15 Pro Bowl selections is the most in league history.Source: Reuters