Can I avoid mortgage referral fees and does it pay the effort to avoid them?

Referral fees are payments made by service providers to other parties as quid pro quo for referring customers. For example, a real estate agent is rewarded for sending a borrower to a loan provider. The reward, whatever form it takes, is a referral fee.

Better, you consult any other lender & negotiate with him. Also compare total fees that they are charging.