I also struggle with this a lot. Its almost kind of dibilitating. I think we need emotions and pleasure to function in the world properly. Otherwise it feels pointless and your just doing things you need to do because its expected of you.
I have quit coffee many times. I always come back to drinking it. I limit myself to one cup, that gives me the pleasure of the ritual every day, without getting too much jitter.
I do think for me though that being caffeine free is optimal… but one cup is much better than 3.