He came to Nantucket with his step-father Richard Swain. His father resided at Ipswich and Topsfield, Mass and was the sone of William Bunker, a French Huguenot who fled to England to avoid religous persecution. His mother married RIchard Swain after George Bunker was lost at sea, and they took the five young Bunker children with them to Nantucket.