Listeria monocytogenes, a contaminant of raw milk, includes hypervirulent clonal complexes (CC) like CC1, CC4, and CC6, highly overrepresented in dairy products when compared to other food types. Whether their higher prevalence in dairy products is the consequence of a growth advantage in this food remains unknown. We examined growth kinetics of five L. monocytogenes isolates (CC1, CC4, CC6, CC9, and CC121) at 37 and 4 °C in ultra-high temperature (UHT) milk and raw milk. At 4 °C, hypovirulent CC9 and CC121 isolates exhibit better growth parameters in UHT milk compared to the hypervirulent CC1, CC4, and CC6 isolates. CC9 isolate in raw milk at 4 °C exhibited the fastest growth and the highest final concentrations. In contrast, hypervirulent isolates (CC1, CC4, and CC6) displayed better growth rates in UHT milk at 37 °C, the mammalian host temperature. Proteomic analysis of representative hyper- (CC1) and hypovirulent (CC9) isolates showed that they respond to milk cues differently with CC-specific traits. Proteins related to metabolism (such as LysA or different phosphotransferase systems), and stress response were upregulated in both isolates during growth in UHT milk. Our results show that there is a Listeria CC-specific and a Listeria CC-common response to the milk environment. These findings shed light on the overrepresentation of hypervirulent L. monocytogenes isolates in dairy products, suggesting that CC1 and CC4 overrepresentation in dairy products made of raw milk may arise from contamination during or after milking at the farm and discard an advantage of hypervirulent isolates in milk products when stored at refrigeration temperatures.