You really don’t need to worry about them 98% of the time. It’s basically only relevant if you have a deficiency or pathology. Like if you have an iron deficiency and you’re eating fortified breakfast cereal to try to address it you should avoid having tea or coffee with that meal since the phytates and polyphenols inhibit iron absorption. Or if you have kidney disease oxalates will bind to calcium and can cause kidney stones, but this is very unlikely to happen in a person with healthy kidneys.
Outcome data shows that fruits, vegetables, and legumes are positively associated with better health outcomes and lower mortality. If antinutrients were a big problem then you’d expect people eating lots of foods that contain them would be less healthy, but the reverse is true.