Weight loss requires a caloric deficit. Part of the idea with IF is that by restricting your eating window, you will eat fewer calories than you would if you were grazing all day long.
You can also create a caloric deficit by burning more calories than you consume. Some people do that with a higher activity level.
If counting calories helps you to make sure you are in a caloric deficit, then it works for you.