Crowther was born with the name Ajayi in Osogun, in the Egba section of the Yoruba people, in what is now western Nigeria. When about 13, he was taken as a slave by Fulani and Yoruba Muslim raiders and sold several times before being purchased by Portuguese traders for the transatlantic market. Samuel Ajayi Crowder became the first slave boy in Africa to become a Bishop.