Breastfeeding protects against diseases, with potential mechanisms driving this being human milk oligosaccharides (HMOs) and the seeding of milk-associated bacteria in the infant gut. In a cohort of 34 mother–infant dyads we analyzed the microbiota and HMO profiles in breast milk samples and infant’s feces. The microbiota in foremilk and hindmilk samples of breast milk was compositionally similar, however hindmilk had higher bacterial load and absolute abundance of oral-associated bacteria, but a lower absolute abundance of skin-associated Staphylococcus spp. The microbial communities within both milk and infant’s feces changed significantly over the lactation period. On average 33% and 23% of the bacterial taxa detected in infant’s feces were shared with the corresponding mother’s milk at 5 and 9 months of age, respectively, with Streptococcus, Veillonella and Bifidobacterium spp. among the most frequently shared. The predominant HMOs in feces associated with the infant’s fecal microbiota, and the dominating infant species B. longum ssp. infantis and B. bifidum correlated inversely with HMOs. Our results show that breast milk microbiota changes over time and within a feeding session, likely due to transfer of infant oral bacteria during breastfeeding and suggest that milk-associated bacteria and HMOs direct the assembly of the infant gut microbiota.