If you begin a letter with Dear Sir/Madam, or To whom it may concern, then it should be signed off with Yours faithfully.

Everything else i.e Dear Alan, or Dear Mr Jones, should be signed off with Yours sincerely.