The best time for you to hold your tongue is the time you feel you must say something or bust.
Art is for anyone. It just isn't for everyone. Still over the past decade its audience has hugely grown and that's irked those outside the art world who get irritated at things like incomprehensibility or money.