Longtime homeowners are getting socked with double-digit rate increases or notices that their policies won’t be renewed. Out-of-state home buyers who have flocked to Florida during the pandemic are experiencing sticker shock. Insurers that are swimming in red ink are cutting back coverage in certain geographic areas to shore up their finances.

Various factors are at play, insurance executives and analysts say. Two hurricanes that slammed the state—Irma in 2017 and Michael in 2018—generated claims with an estimated cost of about $30 billion. The cost of reinsurance, which insurers take out to cover some of the risk in the policies they sell, is swelling. Of particular concern, executives say, are excessive litigation over insurance claims and a proliferation of what insurers see as sham roof-related claims.

A large group of Florida homeowner insurers tracked by Marsh McLennan’s Guy Carpenter business had $1.58 billion in underwriting losses in 2020, more than double the $664 million loss in 2019.

“The industry is in a panic because it is losing so much,” said Barry Gilway, chief executive of Citizens Property Insurance Corp., a state-backed insurer of last resort that is growing rapidly as private-sector insurers retrench. Barring changes, he said, “rates will continue to skyrocket and it absolutely will have an impact on the real-estate market.”

Eric Firestone, a 37-year-old teacher who lives with his wife in the Miami area, received a letter from their insurance company in February saying their policy wouldn’t be renewed because the carrier was no longer servicing their area. When their insurance agent shopped for an alternative, the cheapest one she could find had a $9,644 annual premium—an 85% increase over their most recent premium of $5,205.

“Where am I going to get the extra $4,000?” Mr. Firestone said. “Worst-case scenario, I will have to go into credit-card debt.”

Florida is the most expensive state in the U.S. for home insurance. Residents are projected to pay on average $2,380 in premiums this year, a 21% increase over the $1,960 paid in 2018, according to estimates by trade group Insurance Information Institute. By contrast, the average American homeowner is expected to pay $1,297 this year, up 4% from $1,249 in 2018.

Out of 105 rate increases for multi-peril homeowners policies approved by the Florida Office of Insurance Regulation last year, more than half exceeded 10%, compared with six of 64 in 2016, state Insurance Commissioner David Altmaier informed lawmakers.

Ana Regina Myrrha, an insurance agent in the Orlando area, said she hired three more employees because clients’ policies that used to automatically renew now require “re-quoting,” or finding cheaper alternatives. Her office used to tell clients to expect a single-digit rate increase upon renewal, she said, but “we now tell them that a 20% increase is a gift.”

Florida lawmakers are considering bills that supporters say would help curb premium increases. Among the provisions of a Senate measure backed by the insurance industry and passed by the Republican-led chamber are limits on attorney fees and roof coverage that proponents say would reduce incentives for frivolous lawsuits and claims.

A measure in the GOP-led House that industry representatives consider weaker is expected to come up for a vote in the next few days. If it passes, lawmakers will have until the end of the session on April 30 to negotiate a compromise.

Many Democrats and plaintiff lawyers oppose the bills, saying they would hurt consumers by limiting legal recourse against insurance companies. Opponents say the insurance industry is exaggerating the role of fraud in its deteriorating finances and playing down the significance of losses from natural disasters and rising reinsurance rates.

While Florida made up 8% of all homeowner insurance claims in the U.S. in 2019, it accounted for 76% of homeowners’ insurance lawsuits, according to an analysis by Mr. Altmaier’s office.

A survey by his office of Florida insurers representing about 60% of the market showed they received 90,950 roof claims in 2020, up from 56,657 in 2019. Questionable claims often involve alleged roof damage from hail or wind that the carriers’ adjusters and engineers conclude was due instead to normal wear and tear, which isn’t insured, according to industry executives.

In Florida, as in other states, policyholders with disputed claims can sue their insurers. Critics say that Florida law currently allows too-easy access to insurer-paid legal fees, encouraging contractors to drum up business and lawyers to file frivolous lawsuits over claims, seeking the legal fees.

“Let me help you get a new roof through insurance!” reads one door hanger that Mr. Altmaier shared with lawmakers.

Mandy Wells, 40, a real-estate agent in Cape Coral, met with a representative of roofing company Marlin Construction in 2019 regarding storm damage. The representative guided her on filing a claim and said she would get a new roof, according to Ms. Wells. She signed an agreement authorizing Marlin to receive payment directly from the insurer.

But Ms. Wells said Marlin, which has generated numerous complaints to the Better Business Bureau, never performed any work, and broke off contact when she refused to sign additional documents. Scott Hertz, a Marlin attorney, said Ms. Wells cut off Marlin after receiving an insurance payment, and the parties are now locked in litigation, partly over that money. He said the complaints against Marlin are mostly tied to the employee who met with Ms. Wells, who is no longer at the company.

Though Florida’s real-estate market remains strong, spiraling insurance premiums could hobble it, real-estate agents say. Some out-of-state buyers streaming into Florida say they are stunned by their insurance bills.

Ricardo Calina, 45, a former Wisconsin resident who closed on a house in Winter Garden, near Orlando, in January, said he was upset to learn he and his wife would pay roughly 25% more in homeowners insurance, even though the 4,500-square-foot Florida property is 500 square feet smaller than their Wisconsin residence.

“It is frustrating that we are paying more for less,” Mr. Calina said.

Meanwhile, in Davie, near Fort Lauderdale, Alexander Barr is worried about getting priced out of the house where he grew up and is now raising his two children with his wife. Earlier this month, they received a renewal notice from their insurance company advising that their premium was increasing to $4,006, a 63% jump from the $2,451 they paid last year.

“I can only imagine what’s going to happen next year,” Mr. Barr said.