There is really no difference in meaning, although "little brother" is more informal. However, if your younger brother is six feet tall and weighs 250 pounds, you will certainly get some smiles and maybe some joking comments if you introduce him as your "little brother."