Turn-taking is a universal pattern of human conversations characterized by a fast exchange of turns between speakers and an avoidance of overlaps. Language is embedded in this conversational skill acquired well before it during infancy, through everyday interactions with caregivers. The earliness of this skill and its link with language allows us to test whether social environment shapes early language development. We therefore study turn-taking perception of 6-month-old infants by measuring their gazes during video presentation of three different conversational situations where the turn is explicitly given, normally taken or taken with an overlap. We studied 51 infants to cover several family and infant characteristics: infants' sex, presence of siblings, and family socioeconomic status (SES). We found that infants looked more at the second speaker when she overlapped the first speaker than in the other situations, but not all infants were equally sensitive. Indeed, infants from high-SES families reacted differently to the three situations, while infants from the two lower SES categories did not. Also, only singletons reacted differently by looking more at the second speaker after the overlapping and turn-giving situations, and not after the turn-taking situation. Our results emphasize the importance of early social experiences on language development.