I prefer to punish myself by not allowing me to speak to you.<br/><br/>I prefer to punish myself by not speaking to you.<br/><br/>Are the above sentences grammatically right?