I started quitting caffeine 14 days ago and during the first week, it was difficult for me to even do a water fast. Day 4, 5 and 6 were the worst for me, basically it was hell, but the efforts definitely paid off.
So for me, the withdrawal symptoms alone were too stressful.