Peters was born at 38 Queen Street in Salford, in Lancashire, England. James Peters's Jamaican father, George, had been mauled to death in a training cage by lions. His mother, Hannah Gough originally from Shropshire, was unable to look after him so allowed him to join another circus troupe as a bareback horse rider. He was abandoned by the circus at age 11, having broken his arm, and found himself in Fegan's orphanage in Southwark, and then Little Wanderers' Home in Greenwich, where he captained many of their sports teams. Greenwich Admirals Rugby League Club now celebrate Peters' life with an annual challenge game. Peters trained in printing and carpentry; his trade brought him to Bristol, where he played as a fly-half for Dings Crusaders, Knowle and Bristol Rugby Club, and represented the Somerset County team between 1900 and 1903. His presence at Bristol was opposed by some on racist grounds: a committee member at Bristol resigned in protest at his selection for the team, whilst a local newspaper described him as a "palid blackamoor", and complained that he was "keeping a white man out of the side". Peters then moved on to Plymouth in 1902, representing Plymouth RUFC, and the Devon county side until 1909. He was the star player in the Devon side which won the County Championship in 1906, resulting in the press calling for his selection for the national team.
England international
On 17 March 1906, Peters won his début cap for England against Scotland. However, The Yorkshire Post pointed out, "his selection is by no means popular on racial grounds". In the match, he set up two tries for England: The Sportsman commented that the "dusky Plymouth man did many good things, especially in passing." He was to play a further game, against France, in which he scored a try. In October through to December 1906 Peters became embroiled in controversy regarding the touring South African Springboks, as some members objected to him playing against them for Devon due to his colour, and he was not picked for the International game against South Africa, with a number of newspapers citing racial grounds. He went on to be capped for England three more times in 1907 and 1908. In 1910, Peters lost three fingers in a dockyard accident, and although it was initially feared he could not play again he continued to play until 1912 when politics forced Peters out of rugby union.
Rugby league
Clubs in the south-west of England attempted to form a Western League and play competitive fixtures. He was suspended for accepting payment from Devon Rugby Club, which was illegal according to the codes of rugby union. Many players, including Peters, and also RFC Plymouth were suspended by the RFU. Plymouth's ground closed, signalling the end of Plymouth RFC. The rules of professionalism often owed more to politics than finance. Peters, by then 34 and disillusioned with rugby union, was accepted into rugby league. Returning to his native north-west of England, Peters played for Barrow in 1913, and then transferred to St. Helens in 1914 until his retirement from rugby.