Colfox was the son of Colonel Thomas Alfred Colfox of Coneygar, Bridport, and Constance, daughter of Edward John Nettlefold of Hallfield, Birmingham. Born in Bridport, Dorset, he was educated at Eton College and at Woolwich. Serving in 132 Royal Field Artillery in Fermoy 1908-9 and Ewshott 1909-12 he joined 14 RFA at Allahabad 1912-14 and Secunderabad 1914. He served as a Major in France and Belgium in 128, 130, 129, G/38, 6DAC and D/162 RFA during World War I, and was wounded twice in 1917, receiving a Military Cross. He then taught classics and mathematics at Eton for the rest of World War I. In later life, when fully retired, he taught mathematics to children in Bridport as a volunteer.
Six hundred years after Sir Nicholas Colfox was involved in the murder of the King's uncle, Thomas of Woodstock, Duke of Gloucester, Sir Philip had returned Colfoxs to the national scene. Sir Nicholas was pardoned by King Henry IV in 1404. Colfox's family had a long connection with Symondsbury, Bridport. The family is first mentioned in the town records in 1280. In 1357 Peter Colfox was commissioned by Edward III to rebuild Marshwood Castle, sold by Colfox's brother-in-law in the early 20th century. In 1394, Sir William Colfox, Vicar of Chilthorne Dommer near Yeovil, witnessed an Inquisition relating to part of the Manor of Symondsbury which Sir William Philip Colfox inherited in 1924.
Personal life
In 1920, he married Mary Frances Bullen, the daughter of John Bullen Symes Bullen of Catherston, Charmouth Dorset. They had 2 sons and 2 daughters: Thomas Andrew, John , Susan Frances, and Bridget Alice. He was a Unitarian worshipping in the Chapel in Bridport built by his ancestor Thomas Collins Colfox in 1797. In Bridport, he continued to teach children Mathematics and Classics and was Chairman of the Governors of the Colfox School, so-named in honour of his father who had given the land and refounded the previous Unitarian grammar school. Sir Philip and cars were a notorious combination - he parked his car in Bridport perpendicularly to the pavement and policemen were warned that everyone else but him were to be fined. A daughter was asked at a party in Kent - "Are you related to that bad driver from Dorset?" Colfoxes had been on the Bridport Bench as magistrates continually since the 15th century. On one occasion Sir Philip, Lady Colfox heard a misdemeanour offence against the couple's son charged by the police for the road traffic offence of leaving mud on a public road - which he freely admitted. John was defended by his brother-in-law. His father, mother and uncle found him not guilty and awarded costs against the police. This made local press but only in inside pages in 1950.