Event-related functional magnetic resonance imaging (fMRI) was used to explore how the human brain models gaze-object relations. During scanning participants observed a human model gazing towards or away a target object presented either in isolation or flanked by a distractor object. In two further conditions the model's gaze was shifted and subsequently maintained away from the stimulus/i. These four conditions were implemented within a factorial design in which the main factors were "type of observed behavior" (gaze vs. gaze-away) and "context" (target alone vs. target flanked by a distractor). Results revealed that premotor, parietal and temporal areas, known to sub-serve the understanding of other people actions, were significantly more activated by the observation of the model gazing towards rather than away from the stimulus/i. In addition, a significant interaction indicated that, when the target was presented in isolation, neural activity within the inferior frontal gyrus, another key area for action understanding, was influenced by gaze-object relations. Our findings suggest that this area is important for the establishment of intentional gaze-object relations and indicate that the presence of a distractor interferes with the representation of such relations.