It is ungrammatical. Say: I used to do it that way. To is part of the infinitive and there's no such infinitive as to would because would is a modal/defective auxiliary and has no infinitive form.
New words, one handy idiom, and a 2-minute quiz — delivered to your inbox to keep your streak alive.