Lyttelton played a match for the "Gentlemen of Worcestershire" in 1866, when he was 14, playing alongside two of his older brothers, Charles and George. The Lyttelton family was closely associated with cricket in Worcestershire, and most of the family appeared for the county at some time. He played for Eton in his final year at the school, and appeared in the annual fixture against Harrow that year, his performance being of little note. Eton won the match by 21 runs, in which Lyttelton scored two runs in the first innings and remained not out with five runs in the second. He batted at number ten in both innings, and did not bowl. He suffered a pair during a match between Worcestershire and Herefordshire in 1871. During his time at Cambridge University, Lyttelton frequently appeared for the "Quidnuncs", a cricket club generally populated by former university cricketers who had earnt a blue. He never played first-class cricket for the university, but did appear against them in one match in 1872; his only first-class appearance. Lyttelton was part of the Marylebone Cricket Club team which played the university at Lord's Cricket Ground in June that year. He batted at number seven, and after scoring a duck in the first innings, he scored four runs in the second of a low-scoring match in which only W. G. Grace passed 50 runs in an innings. He made a second appearance for the MCC that summer, as part of a side which beat a Worcestershire team containing two of his brothers by three wickets. He played little more notable cricket, turning out for the Quidnuncs and the Free Foresters infrequently. He was described in Scores and Biographies as being "Like the rest of the family he is a fine free hitter, and an excellent field at long-leg, or middle-wicket-off."
After graduation
Lyttelton was ordained as a deacon in Oxford in 1876, and as a priest the following year. He began his career with a Curacy at St Mary's, Reading, and in 1879 became a tutor at Keble College, Oxford, a post he retained until 1882. In 1880 he married the women's activist Mary Kathleen Clive, daughter of the Liberal politician George Clive; they had three children: Margaret Lucy, Archer Geoffrey, and Stephen Clive.
In 1882 Lyttelton moved with his wife to Cambridge to take up the post of first Master of Selwyn College. When they arrived, the college was still a building site with only the West front completed. Initially, they made do with a suite of adapted student rooms until the Master's Lodge was built. Lyttelton's moderate political views along with his political connections to the prime minister helped ensure acceptance of the new institution within the university. However, he did cause some consternation when he ruled that in general only members of the Church of England would be admitted to the college, despite government acts of 1856 and 1871 which allowed undergraduates and faculty members of any religion, or no religion, to be admitted. He was described in a history of the college as a "fine teacher and a reserved, aloof-seeming man of judgement, decision and piety."
