Portman was the son of John Portman, who was buried in the Temple Church on 5 June 1521, by Alice, daughter of William Knoell of Dorset. His family was long established in Somerset, having given its name to the former manor and present village of Orchard Portman, and he served as Justice of the Peace for that county from time to time. He was a barrister who was successful enough to be personally known to King Henry VIII. In 1532, he acquired 270 acres adjacent to the NW of the City of London, which estate stretching from today's Oxford Street to the Regents Canal, known as the Portman Estate, is still held by his descendants the Viscounts Portman.[3] In 1533, Henry VIII gave him a wardship, and he was one of the administrators of the will of Catherine of Aragon.
He married Elizabeth, daughter and heiress of John Gilbert, who was connected by descent with the eminent legal family of Fitzjames. By her he had a son and a daughter:
Sir Henry Portman, knight (died 1590), married Jane Mitchell and produced issue: