Do i need to just give up on him or what?

try to ask more specific questions maybe that will help with the conversations. or you can try counseling. Other than that idk. you have to ask yourself "Is he worth the effort?"