While you do not need to start over per se, eating a very small meal and fasting again will slow your metabolism. If I did that, I would have a giant refeed and then start again to keep my metabolism running high. I also consider fasting more about refeeding than abstaining.