Only expand integers for integer math once