Mr. Broad said he reached the decision in recent weeks after long discussions with his wife, Edythe, who he said has long urged him to retire. Mr. Broad has prostate cancer, diagnosed a decade ago, which is in remission, and he undergoes daily physical therapy for intense back pain, though he said his health was not a factor in his decision.
In an interview Thursday, Mr. Broad noted wryly that he was disclosing his decision to retire in the same week that Senator Dianne Feinstein, the California Democrat who is also 84, announced that she was seeking a sixth term. “She’s got more energy than I have,” said Mr. Broad, who attended a fund-raiser for Ms. Feinstein here earlier this week. “Oh, yeah.”
The practical ramifications of Mr. Broad’s decision may be limited; the announcement in many ways marks the end of what has been a slow-motion fade. In August 2016, he named Gerun Riley, 41, a longtime aide, as executive director of the Broad Foundations, which oversee his investments in art, science and educational causes. Last month, he added four new members to the board of directors of the Broad, another sign of his yielding control.
And he has his longtime deputy, Joanne Heyler, as founding director of the Broad Museum and director and chief curator of the Broad Art Foundation. The Broad has been open since September 2015 and is going full throttle, having drawn 1.5 million visitors since the doors opened. Tickets for the “Yayoi Kusama: Infinity Mirrors” exhibition, which opens later this month, quickly disappeared when 150,000 people swarmed onto the website the moment they were posted a few weeks ago.
Mr. Broad also spearheaded the effort to build the Walt Disney Concert Hall, the Frank Gehry-designed building that has become an anchor of downtown. His decision to locate the Broad museum just up the street from the concert hall — bypassing Santa Monica and Beverly Hills — has also been seen as crucial to downtown’s emergence.
He came to symbolize a Medici style of philanthropy that some of this city’s biggest boosters say has been in distressingly short supply. Mr. Broad, in the interview, said he thought that was changing, noting, for example, that David Geffen, the entertainment mogul who lives in Beverly Hills, among other places, recently donated $150 million to the construction of a new building for the Los Angeles County Museum of Art.
He read from prepared notes as he listed other people who might fill his shoes, including Steve Ballmer, the former chief executive of Microsoft who is now the owner of the Los Angeles Clippers; Bob Iger, the chief executive of the Walt Disney Company, Jeffrey Katzenberg, a former chairman at Disney, and Nicolas Berggruen, a philanthropist and investor who is building a think tank in the Santa Monica Mountains.
The decision to build a museum to house the Broads’ sweeping personal collection of contemporary and postwar art — 2,000 pieces, including works by Jasper Johns, Andy Warhol, Jean-Michel Basquiat, Damien Hirst and others — came in a city where, until recently, many fine works of art have been hidden away in private mansions.
In the interview, Mr. Broad reminisced about meeting many of these artists as he and his wife traveled the world, amassing their collection at a time when many pieces were far more affordable than today. He noted that his years as a philanthropist marked the fourth career of his life: He started as an accountant and moved into insurance and construction before settling into philanthropy.
“Every time I’ve stepped aside in the past, I haven’t missed it,” he said.
Mr. Broad alluded to his own reputation as being sometimes prickly, saying he found being a business executive much less challenging than working in philanthropy or dealing with political figures. In 2012, he published a memoir called “The Art of Being Unreasonable.”
“In business you can make decisions,” he said. “You don’t have to be collegial with all sorts of outsider organizations. It’s a different role in philanthropy.”
Since he moved to California, Mr. Broad has lived within two miles of his current home in Brentwood. “We were never excited about the glitz of Beverly Hills,” he said. “We liked Brentwood.” He said that Los Angeles had changed dramatically since he arrived, in no small part because of what has happened with the contemporary art scene.
“Los Angeles is now the contemporary art capital of the world,” he said. “We have emerged as one of the four major cultural capitals of the world, together with London, New York and Paris.”