After my little trip to Swaffham this week, this is what The Concise Oxford Dictionary Of English Placenames have to say about the name origin of the town.
Swaffham, Norfolk. Swafham in Domesday Book, Swaffham in 1130. The first element is the tribal name Swaefas in its stem form. The name means the ‘ham’ of the Swaefas, or Swabians in early sources.
The Swaefas were a German tribe, or the Angles, who came to settle in Britain as visible in the below image (https://upload.wikimedia.org/wikipedia/commons/5/56/Anglo-Saxon_Homelands_and_Settlements.svg). So, Swaffham was a settlement where the Angles resided, one of their earlier places of residence in the country, and the town name has also hardly changed in the last millennium.