Visual social attention is an important part of the social life of many species, including humans, but its patterning may vary between species. Studies on human–pet relationships have revealed that visual attention is also part of such interspecific interactions and that pets are sensitive to the human visual attentional state. It has been argued that domestication and/or repeated experiences with humans have shaped and refined these decoding abilities. Little is known on how the species’ evolutionary history may play a role in determining visual attention patterns during interactions, nor how the human’s own social skills may influence the animal’s attention patterns in human–animal interactions. In the present study, we investigated the visual attention patterns directed to the partner in dog–child and cat–child interactions in their home environment. We also compared these patterns between a group of children with autism spectrum disorders (ASD) and children with typical development. We found that the attention patterns differed according to species, with dogs displaying more gazes and cats more glances toward their human interlocutor, while children showed gazes toward both species. Only slight differences were observed according to the developmental status of children: ASD children displayed much more visual attention with their pet cat than with their pet dog and the same amount of visual attention toward their pet, whatever the species, as typically developing (TD) children. Because humans rely a lot upon visual communication in their own social encounters, where direct gazes play a major role from early on, they may be especially sensitive to the gazing behavior of their dogs. People with ASD, with a less typical pattern of interaction, may be more comfortable with the less “invasive” short glances of cats. These results suggest not only that interspecific communication has to be associated with processing and storing the other species’ ways of communicating in order to be successful but also that visual attention patterns during interactions, even when interspecific, are, for a large part, the result of the species’ own evolutionary history.