A man i'm seeing only asks me out during major fights?

This man & I are on our late 30s. We are doing well in our careers & we have our own baggages to carry. We've been seeing now for more than 5 months. In those 5 months, we never, ever go out on dates like dinner or movies, we just stay glued infront of his TV and then sleep together. Most of the time I cook dinner, but that's it. I would occasionally complain but he'd say he barely even have time for himself so we just make the most of available time we have-we'd just spend the night at his or my place and have . We would constantly fight after seeing each other(over text msg, he seldom calls). I've been paranoid & jealous but he'd say he's not seeing anyone else, he'd say he couldn't even spend more time with me, how could he go around look somewhere else. What bothers me is that everytime we get into big fights, that's the time he'd ask me out to dinner and that pisses me off even more! I mean, why would he want to go out with an upset date, right? Of course, that is also the time he'd repeatedly call & I wouldn't answer coz I'm pissed & I'd turn his dinner invite down telling him I could've appreciated it if he'd ask me out not when we're having major fights(on the brink of us/me quitting). I know he's just playing nice & I hate that and the same time I want to tell him he needs to work harder and realize his inconsistencies and selfish attitude and he knows this. Did I do the right thing turning down his dinner invite? What if he wants to talk & finally ask me to be his official girlfriend? But why does he have to do it when we're awfully fighting?