Neural templates for phonemes in one's native language are formed early in life; these can be modified but are difficult to form de novo. These can be examined with mismatch negativity (MMN). Three phonemic contrasts were presented to adult native English compared with Japanese speakers who acquired English later in life: vowels native to both languages (/i//iy/), consonant-vowel contrasts (/da//wa/) phonemic in both languages, and consonant-vowel contrasts phonemic in English but not in Japanese (/ra//la/). For vowels, no MMN differences were found. For /da//wa/, MMN amplitude was significantly reduced in Japanese speakers. For /ra//la/, only 50% of the Japanese group showed an identifiable MMN. This suggests that phonemic templates are formed early in life, and non-native consonant contrasts are difficult to learn later.