He read for the Bar and was called by the Inner Temple in 1946. In the same year he entered the Prize Fellowship examination at All Souls, applying as a candidate in Law, even though he had not read for a Law degree. He was elected in November 1946 and began a long connection with the college in various categories of fellowship. Between 1961 and 1966 he held the office of Estates Bursar, at a time when he was working as a silk and regularly appearing in court and giving advice. Fisher was a junior counsel from 1946 to 1960, in which year he took silk. After service on the Bar Council he was Chairman of the Bar from 1966 to 1968.
Bench
At the beginning of 1968 Fisher was appointed a High Court Judge in the Queen's Bench division. In July 1968 he was the junior member in a Court of Appeal decision that quashed the conviction for obscenity entered against the publishers of Last Exit to Brooklyn. This case sounded the death knell for prosecutions on the grounds of obscenity of books with literary merit. Much of his time on the bench was spent out of London trying criminal cases, and living in judges' lodgings.
Missing the intellectual challenge of arguing complex cases he concluded that he had made a mistake and resigned from his judicial appointment in 1970. He moved to the City of London and became a director of J. Henry Schroder Wagg & Co under the chairmanship of his friend Gordon Richardson. From 1981 to 1987 he was Chairman of the Appeal Committee of the Take-over Panel, and founder chairman of Imro.
Academia
In 1973 he became a member of the Governing Body of Imperial College, and from 1975 to 1988 was its chairman. In 1975, he was elected President of Wolfson College, Oxford, on the retirement of its founding president, Isaiah Berlin. It was this position which Fisher put first in his Who's Who entry. He presided over Wolfson's move from temporary premises to its newly completed buildings on the banks of the River Cherwell. During his time at Wolfson he took and completed an Open University degree in mathematics. He was chairman of the Howard League for Penal Reform from 1983 to 1991.
Enquiries
During this time he was frequently called upon to undertake the chairmanship of high-profile enquiries. In the Confait inquiry Fisher was asked by the Crown to examine a gravely flawed prosecution which had resulted in the conviction of three young men for arson, murder and manslaughter. On a reference by the Home Secretary, the Court of Appeal presided over by Lord Scarman had quashed all the convictions. In the ensuing investigation, Fisher concluded that there had been a blatant disregard of the Judges' Rules . His report recommended radical changes to the system and led directly to the appointment of the Philipps Commission and thence to the enactment of the Police and Criminal Evidence Act 1984 and the Prosecution of Offences Act 1985. The following year the Crown Prosecution Service was established. In 1979, Fisher was asked by Lloyd's of London to conduct an inquiry into self-regulation at the institution. The following year his report recommended the adoption of a new constitution which he had drafted, including the creation of a new governing council and effective disciplinary procedures. It brought in protection for council members from claims for negligence: liability was made to depend on bad faith.