$30000 Home loan

I own two acres of land worth around 15,000, I have $3000 dollars to use as a down payment. I am going to need a conventional loan. I am building a pole barn style building with an office and garage for my business and a living area. I was told I could use my land which I owe nothing on and have the deed in hand as my down payment. The probelm I am worried about is I have a 643 Transunion credit score. The bank I plan to go through I have done business with quite a few times on auto loans and have never missed a payment with them. Some misunderstandings with medical bills dropped my credit score. I just want to know if I can possibly get my conventional loan for around 30000-35000 using the land as collateral plus $3000 dollars with my 643 transunion credit score.