The watchdog issued the fine after its review found that only two of 164 mortgages sales met acceptable standards.
The bank, which is still supported by the taxpayer, was guilty of “serious failings” in its procedures, the FCA said.
RBS received a 30% discount on the original £20.7 million fine by agreeing to pay early.
However, it still puts another significant black mark on the taxpayer-owned bank, which reported a loss of over £8 billion for last year.
RBS and NatWest, among the country’s largest mortgage lenders, were criticised for poor standards of advice between June 2011 and March 2013.
This ranged from weak affordability checks to a failure to advise customers about the best mortgage term for their needs. The banks also neglected to offer proper debt consolidation advice, the regulator said.
The FCA’s director of enforcement, Tracey McDermott, said the regulator had made its concerns clear to the RBS Group in November 2011, but that it had taken over a year for the banks to act.
“Where we raise concerns with firms we expect them to take effective action to resolve them without delay. This simply failed to happen in this case,” she said.
NatWest's mortgage information was not up to scratch, the FCA ruled.
RBS chief executive, Ross McEwan, who joined the bank in 2012, said the failings were “unacceptable and should never have happened”.
“Our customers have every right to expect the very best service when making this decision.
“It is clear that in the past the bank just didn't get this right.”
Mr McEwan said he had worked hard to change procedures at the bank following his arrival to ensure there was no repeat of this latest verdict.
"When I joined the bank we completely overhauled our processes, and took all our mortgage advisers off the front line for an extensive period of time to get the training required," he added.
The failings at RBS highlight why the latest set of mortgage rules, known as the Mortgage Market Review, were deemed to be necessary.
The new rules, introduced earlier this year, require lenders to undertake more thorough affordability checks on new borrowers, and to consider the full extent of a customer’s budget before making recommendations.
This is to prevent borrowers from entering a situation where they are no longer able to repay their loans.
"Poor advice could cost someone their home, so it’s vital that the advice process is fit for purpose,” said Ms McDermott.
“Both firms [RBS / NatWest] failed to ensure that their customers were getting the best advice for them.”