Astor represented Great Britain in rackets at the 1908 Summer Olympics, winning the gold medal in the men's doubles competition together with Vane Pennell, and winning bronze in the men's singles event. Astor had been the British Public Schools rackets champion in 1904–1905, and in the same year as his Olympic competition he played singles and doubles in the British Army rackets championships. Despite a later loss of leg, he was able to play and win against younger opponents at squash on a prosthetic limb.
Military service
He served in the 1st Life Guards, which he joined in 1906 after a year at Oxford, and was Aide-de-Camp to Baron Hardinge, Viceroy of India between 1911 and 1914. Within his regiment he was promoted Captain in 1913 and Major in 1920. In World War I, he was wounded serving with his regiment at Messines in October 1914. After recovering he returned to the Western Front, rising to the rank of Lieutenant-Colonel commanding 520 Household Siege Battery of the Royal Garrison Artillery and awarded the Légion d'Honneur as a Chevalier. In September 1918, near Cambrai, his right leg was shattered by a shell and later amputated. He was Honorary Colonel of the Kent and Sussex Heavy Regiment, Royal Artillery, between 1927 and 1946 and Honorary Colonel of the 23rd London Regiment, between 1928 and 1949. In World War II he was Lieutenant-Colonel of the 5th Battalion, City of LondonHome Guard, a unit drawn from newspaper employees, between 1940 and 1944.
He was a director of the Great Western Railway between 1929 and 1946. In 1926, Astor was Lieutenant of the City of London, then held the offices of Justice of the Peace from 1929 and Deputy Lieutenant of Kent from 1936 until 1962. He was a director of Hambros Bank between 1934 and 1960. He was Vice-Chairman of Phoenix Insurance between 1941 and 1952 and Chairman of between 1952 and 1958. He was a director of Barclays Bank between 1942 and 1952. on page 117 of "some recollections by A.W. Tuke AND R.J.H Gillman" Barclays Bank Limited 1926-1969 Barclays Bank Limited 1972 under appendix I for Dover from 1922 to 1945. On 21 January 1956 he was created Baron Astor of Hever, of Hever Castle in the County of Kent. In 1962, he moved from England to France.
Selected artworks from the family's vast collection were bequeathed to the National Gallery including the prized "Thames below Westminster" by Claude Monet. John Jacob V and Violet are buried together on the grounds of Hever Castle, which, since 1983, has been owned by Broadland Properties Limited and is a major tourist attraction. Eldest son Gavin succeeded him as Baron.