Foster, was born in Dublin, Ireland, was educated at Trinity College, Dublin, where he graduated B.A. in 1839 with honours, and became a student for the bar, but abandoned the legal profession in favour of a colonial career. The colony of Victoria, then only the Port Phillip District of New South Wales, was his choice, and he landed there on 28 March 1841. At first he devoted himself to pastoral and agricultural pursuits.
Politics
In June 1846 Foster was elected as representative of Port Phillip in the New South Wales Legislative Council, and in July 1848 again came forward for election as one of the six members allotted to Port Phillip. The feeling was, however, so strong that this so-called representation was a mere farce, that the majority were desirous not to elect any more members until separation was secured. In deference to the protests of this party Foster's nomination was withdrawn, but a few days later he was put up as a candidate for the Town of Melbourne, when the non-election party nominated Earl Grey, the Secretary of State for the Colonies, in opposition to him, in the hope that his election would call public attention in England to the grievances of which the colonists complained. In the result Earl Grey was returned by a large majority, and Foster on a subsequent occasion declined to stand. Ultimately, however, he was elected on 1 June 1849, to sit in the Sydney Parliament till he left for Ireland in 1850. In 1853 Foster returned, with the appointment of Colonial Secretary of the colony of Victoria, which had in the meantime been constituted. Foster held the post of Colonial secretary from 20 July 1853 to 4 December 1854, and was a nominated member of the Victorian Legislative Council from August 1853 to December 1854. He was thus virtually Premier during the difficulties caused by the discovery of gold and the rigid enforcement of the unpopular diggers' licences. The troubles culminated in the Ballarat riots, of which Foster was made the scapegoat. Aware of the hostile feeling against him, he tendered his resignation to Sir Charles Hotham, by whom it was accepted on the ground that the Queen's government of the colony was endangered, and with the implied pledge that compensation should be given Mr. Foster for the pecuniary loss which he would sustain by his retirement from the public service. Mr. Foster, however, failed in all attempts to obtain any recognition of his claims to compensation. During his tenure of office he introduced and passed the measure which embodied the new constitution of Victoria, and which for the first time included the principle of an elective Upper House. In 1854 he turned the first sod at Williamstown, Victoria of the great system of railways which has since been developed, and was also instrumental in introducing telegraphs into the colony. Considerable difference of opinion existed as to some of the measures proposed by him; but it is remarkable that every one of them has since been adopted by subsequent legislative action under the new constitution—as, for instance, the abolition of the gold diggers' licence, and the appropriation of the Land Tax to purposes of general utility instead of expending it on immigration. The contracting of loans for public works, which he proposed as the necessary complement of his policy, has since been largely developed. When his conciliatory policy with reference to the diggers' licences was reversed by the Governor, Sir Charles Hotham, and unfortunate results ensued at Ballarat, he retired from office. Subsequent legislative inquiry proved that for such results he was in no way responsible. After the concession of responsible government he sat in the first Victorian Legislative Assembly as member for Williamstown, and acted as Treasurer of Victoria in the first Administration of Sir John O'Shanassy from 11 March to 29 April 1857 soon after which he returned to England, where he since resided.
From 1857, Foster lived in England. In accordance with the will of his uncle, the last Lord Fitzgerald and Vesci, he assumed the name of Vesey and Fitzgerald in addition to his own name of Foster, by which latter he was known in Australia. Foster married in 1851 Emily, daughter of Rev. J. J. Fletcher, D.D., and administered the government of Victoria from May to June 1854, during the interval between the departure of Charles Latrobe and the arrival of Sir Charles Hotham. Several of his relatives achieved distinction in Australia, including his three first cousins—Sir William Foster Stawell, Justice Foster of New South Wales, and Charles Griffith of Victoria.