You can, but I don't much like the second sentence with either. Firstly, I expect a verb after "have to" and have to readjust when that does not happen. Secondly, it is ambiguous whether you speak only about your son, or whether you speak on other matters but speak about your son only when you have to. Assuming the former, it would be better, in my opinion, to say something like: