Watch his “coffee makes you fat” video but as others have said calorie wise you are fine. Gist is it raises cortisol and hurts REM sleep. Both lead to weight gain.
Caffeine on an empty stomach can cause discomfort, jitters too from my experience. Tea might be a slightly better alternative to experiment with