After his education, Rahim joined the Pakistan Civil Services, picking up the first bureaucratic assignment in Foreign Service of Pakistan. Rahim served as the Foreign Secretary under the government of Prime minister Muhammad Ali Bogra. For some time, he remained associated with Communist party, but also built personal relations with Zulfikar Ali Bhutto in 1965. After attending the socialist convention at the residence of Dr. Mubashir Hassan, J. A. Rahim played a key role in writing the party's socialist manifesto: "Islam is our religion; democracy is our politics; socialism is our economy; power lies with the people", on 30 November 1967. This manifesto was officially first issued on 9 December 1967. J.A. Rahim was made Pakistan Peoples Party's first secretary general after writing the party's constitution. Rahim earned public notability after his name was announced as a Bengali member of delegation of Pakistan Peoples Party to launch a negotiation with Awami League party under Sheikh Mujibur Rahman. In 1970, Rahim along with Ghulam Mustafa Khar, returned to West Pakistan, telling Bhutto that the "meeting with Mujib was of no use". After the 1971 war, Rahim stayed in what remained of Pakistan, governing the Law ministry, Justice minister, and the Township planning and agrovilles. In 1972, Rahim was appointed as the first Minister of Defence Production which he governed until 1974.
Disillusionment with Bhutto
His relations with Bhutto deteriorated after Pakistan People's Party began purging the radical and ultra-left wings of the party and J. A. Rahim was also lined by Bhutto later. In July 1974, Rahim himself got disillusioned with Bhutto after seeing Bhutto's handling of internal affairs and publicly disagreed with Bhutto as he wanted Bhutto to deal with the matters efficiently, not by force. He was appointed Pakistan Ambassador to France by Bhutto just to get him out of the way and away from Pakistani politics. But he returned to Pakistan unscheduled. Rahim was then tortured by the members of the secret police, the Federal Security Force, and was thrown into jail in 1976. Shortly afterwards, he was released. Later after Bhutto had formally issued an apology to him, he again left for France to complete his tenure as ambassador. Some people say that Bhutto, in 1979 during his final days in Zia-ul-Haq's death cell, regretted most was his fall-out with his former mentor, J. A. Rahim.
