A soldier tried to disarm the Austrian attacker with pepper spray before opening fire at least four times, police say.
The 26-year-old Austrian "died on the spot" at around midnight after a soldier, who was standing guard outside the building, opened fire.
The soldier tried to disarm the attacker with pepper spray before opening fire at least four times, police spokesman Harald Soeros said.
According to early indications, the soldier "acted within the rules", a spokesman from the ministry of defence was quoted as saying.
The official added that the soldier, who is from the Tyrol region, was treated in hospital for a wound to his arm.
Police have ordered reinforcements around all diplomatic missions in the city.
The incident comes just days after two knife attacks in Vienna, including one that left a couple and their 17-year-old daughter with severe injuries.
A 23-year-old Afghan national confessed to those assaults but denied any political motive.