Brain lateralization is a phenomenon widely reported in the animal kingdom and sensory laterality has been shown to be an indicator of the appraisal of the stimulus valence by an individual. This can prove a useful tool to investigate how animals perceive intra- or hetero-specific signals. The human-animal relationship provides an interesting framework for testing the impact of the valence of interactions on emotional memories. In the present study, we tested whether horses could associate individual human voices with past positive or negative experiences. Both behavioural and electroencephalographic measures allowed examining laterality patterns in addition to the behavioural reactions. The results show that horses reacted to voices associated with past positive experiences with increased attention/arousal (gamma oscillations in the right hemisphere) and indicators of a positive emotional state (left hemisphere activation and ears held forward), and to those associated with past negative experiences with negative affective states (right hemisphere activation and ears held backwards). The responses were further influenced by the animals’ management conditions (e.g. box or pasture). Overall, these results, associating brain and behaviour analysis, clearly demonstrate that horses’ representation of human voices is modulated by the valence of prior horse-human interactions.