Some 6,700 West Bromwich Building Society customers will see their tracker rates rise by 2% on 1 December.
That is despite the fact that the Bank of England has kept interest rates on hold for four-and-a-half years.
The building society said it was forced to increase rates, as the funding costs for such mortgages had gone up, and its terms and conditions allowed a change.
Michelle Elewaar, a full-time landlord with four properties, is one of those affected.
She has a tracker buy-to-let mortgage with the West Bromwich, which she thought had a rate of 1.99% over the Bank of England base rate for the full term of her mortgage.
"I was really shocked. Like any responsible landlord, I made provisions if there were interest rate rises," she says.
"Never in a million years did I think would have to make provisions for, in my opinion, West Bromwich breaching their contract."
Ms Elewaar remains convinced that the contract she signed did not make it clear that a rate rise by the lender could be implemented.
"One woman against the bank isn't going to get very far and I realise this. But I'm a member of an online forum called Property 118, together with around a thousand others. We're joining forces."
More than 150 of these cases have been sent to Justin Selig of The Law Department, a firm of solicitors.
Mr Selig is also representing customers of the Bank of Ireland who were affected by unexpected rate hikes earlier this year.
"I am a borrower with a tracker mortgage as are millions of other people in the country," he told the BBC.
"There is a risk that if we don't do anything about these two situations that other lenders will follow suit."
He believes that the lenders have not been clear with their customers.
"You shouldn't have to look through the small print. Things like this should be on the key facts document and they usually are. In fact, in relation to the documentation we've seen so far it is not clear in the initial document that they have the right to do this."
Mr Selig also claims that up until a few weeks ago the West Bromwich Building Society had this definition on their website:
"Tracker mortgages give you the certainty of knowing that the rate you pay will move in line with Bank Base Rates."
Back in May, 13,500 Bank of Ireland buy-to-let mortgage holders also saw their tracker rates rise.
Then, in October, the bank raised tracker rates for 12,000 of its customers for the second time this year.
Ray Boulger, from the mortgage broker John Charcol, believes that other lenders may follow suit.
Out of the 10 million or so people with residential mortgages, he estimates at least two million have fixed-term tracker mortgages.
"Because the buy-to-let market is not regulated by the Financial Conduct Authority (FCA), some lenders, including West Bromwich, think they can get away with doing things they might not want to try on with the residential mortgage market," he says.
"However, if they are allowed to get away with this, some lenders may well be tempted to try to increase the tracker margin on residential rates as well."
Mr Boulger also believes that even though the buy-to-let sector is not regulated, the individual lenders are.
He is urging customers to write to the FCA directly. If the regulator could be persuaded that these tracker rate rises are unfair, it could force lenders to backtrack.
The regulator is going to conduct a consultation on the buy-to-let sector later this year.
A spokesman from the FCA said: "There is a wider issue to be debated with the industry about fairness in the context of changes to mortgage contracts. We intend to publish a discussion paper on this subject later this year."
The West Bromwich Building Society argues that the wholesale cost of funding buy-to-let mortgages has risen significantly.
Long-term interest rates have risen, with the market now expecting the Bank of England to raise base rates in 2016.
"Market conditions have changed significantly since these buy-to-let mortgages were taken out, resulting in an increased cost of funding them," a spokesman for the building society told the BBC.
It also argues that landlords have enjoyed a dramatic fall in interest payments, as base rates have gone down.
Assuming they did not cut rents when their borrowing costs fell, it also claims they would have seen an increase in their income.
"We have held off on making any changes to rates for as long as we feel reasonable, but have acted now to balance the interests of the Society's wider membership, particularly our savers, whose income has fallen," it said.
"An additional percentage of 2% will be added to these buy-to-let mortgages from 1 December 2013 to make up an overall rate of interest."
It also insists that the change is permitted under the terms and conditions of the accounts.
However, it said it expects that additional 2% margin to reduce over time, as wholesale funding costs become cheaper.
While landlords now decide whether to press ahead with legal action, the question for tenants is whether their rent will rise as a result of the higher mortgage rates.
Michelle Elewaar says she will not have to put the rent up in the short term, but she is worried that she may be forced to do so if other mortgage lenders follow the lead set by the West Bromwich and the Bank of Ireland.