A utility back, Bayly made his first-class debut as a 16-year-old, appearing for West Coast against the touring New South Wales team in 1882. He played his first match for the following year, and played for the province until the 1901 season. Bayly was a member of the New Zealand team that toured Australia in 1893. On that tour he played in all 10 matches, scoring three tries. In 1894, he captained New Zealand in their 6–8 loss to the touring New South Wales team at Lancaster Park, Christchurch. After missing selection for national side in 1896, Bayly was recalled in 1897 to captain the team on their tour of Australia, making nine appearances. In all, Bayly played 20 games for New Zealand, including 10 as captain, scoring six tries and one drop goal for a total of 20 points. He did not play any Test matches, as New Zealand did not play its first full international match until 1903. In 1899, Barney Armit was fatally injured during a provincial match between Otago and Taranaki. Armit attempted to hurdle Bayly, but Bayly rose up and caught him around the ankles; Armit was subsequently up-ended and seriously injured his spine in the fall. Armit died 11 weeks later and became the first New Zealand international to die from injuries sustained on the rugby field. According to his obituary, Bayly was extremely distraught by the incident and "wept like a child" on Armit's deathit may have even shortened his life. Bayly served as a Taranaki selector from 1891 to 1898, in 1901, and in 1906, and was a national selector in 1901 and 1905. He was president of the Taranaki Rugby Football Union between 1899 and 1906, and president of the New Zealand Rugby Football Union in 1907, dying in office.
Cricket
He played six first-class matches for Taranaki between the 1891/92 and 1897/98 seasons. Over his first-class career he scored 39 runs, at an average of 6.50 and a high score of 20 not out. He took nine wickets at an average of 22.77, with best bowling figures of 6 for 54.
A farmer at Toko, Bayly sold his property to the government in about 1905 and moved to Stratford. Owing to ill health, he moved to Whanganui in 1907, and died there on 14 December 1907. He was buried at Stratford Cemetery.
