There are contradictory results on lateralisation and localisation of rhythm processing. Our aim was to test whether there is a hemispheric dissociation of metric and non-metric rhythm processing. We created a non-metric rhythm stimulus without a sense of metre and we measured brain activities during passive rhythm perception. A total of 11 healthy, right-handed, native female Hungarian speakers aged 21.3 ± 1.1 were investigated by functional magnetic resonance imaging (fMRI) using a 3T MR scanner. The experimental acoustic stimulus consisted of comprehensive sentences transformed to Morse code, which represent a non-metric rhythm with irregular perceptual accent structure. Activations were found in the right hemisphere, in the posterior parts of the right-sided superior and middle temporal gyri and temporal pole as well as in the orbital part of the right inferior frontal gyrus. Additional activation appeared in the left-sided superior temporal region. Our study suggests that non-metric rhythm with irregular perceptual accents structure is confined to the right hemisphere. Furthermore, a right-lateralised fronto-temporal network extracts the continuously altering temporal structure of the non-metric rhythm.