When Margrave Leopold III died in 1136, he was succeeded by his third-born son Leopold IV, probably because Henry already administered the Rhenish possessions of his mother's now-extinct Salian dynasty. In April 1140, the Hohenstaufen king Conrad III of Germany enfeoffed him with the County Palatine of the Rhine, which he ruled for only a short time as he was appointed Bavarian duke and margrave of Austria when his younger brother Leopold IV unexpectedly died in October 1141. Leopold had received the Duchy of Bavaria in 1139, after King Conrad had deposed Duke Henry the Proud in the course of the dispute between the Welf and Hohenstaufen dynasties. Henry took his residence in the Bavarian capital of Regensburg. In May 1142 he married Gertrude, daughter of Emperor Lothair and widow of Henry the Proud. She died after less than one year, giving birth to her only child with Henry.
The duke remained a loyal follower of the Hohenstaufens and in May 1147 accompanied King Conrad on the Second Crusade. When they suffered a disastrous defeat at the Battle of Dorylaeum against the Seljuk Turks in October, Henry narrowly escaped together with Conrad's nephew, young Frederick Barbarossa. On their way home, Henry stayed at the court of the Byzantine emperor Manuel I Komnenos, where he married his niece Theodora in late 1148. Elected king of Germany in 1152, Frederick Barbarossa tried to reach a compromise with the Welfs and endowed Henry the Lion, son of the late Henry the Proud, with the Duchy of Bavaria in 1156. A replacement duchy had to be found for the Babenberg family, which was accomplished by the Privilegium Minus, through which Frederick elevated Henry's Margraviate of Austria to a duchy with complete independence from Bavaria. Unlike his father, who had resided in Klosterneuburg most of the time, Henry moved his Austrian residence to Vienna in 1145. Only by this act could Vienna surpass other cities within the duchy, such as Krems, Melk, and Klosterneuburg. Since then, it has remained the capital of the country. In addition, in 1147 the first St. Stephen's Church was completed, becoming a visible landmark for the city and showing its prominence. In 1155, Henry founded the Schottenstift monastery in Vienna, in the courtyard of which a statue of him stands to this day. In November 1176, while his Austrian lands were campaigned by the forces of Duke Soběslav II of Bohemia, Henry II with his horse fell from a rotten bridge near Melk and suffered a femoral neck fracture. Henry II succumbed to his injuries on 13 January 1177 in Vienna. According to his last will, he was buried in the Schottenstift monastery.