About 6,400 troops will be sent home, with the rest moved to other Nato countries such as Italy and Belgium.
President Donald Trump said the move was a response to Germany failing to meet Nato targets on defence spending.
But it has attracted widespread opposition in Congress from those who believe it will embolden Russia.
Senior German officials have also expressed concern.
“We don’t want to be the suckers anymore,” Mr Trump told reporters at the White House on Wednesday shortly after the move was announced. “We’re reducing the force because they’re not paying their bills; it’s very simple.”
Mr Trump has long complained that European members of Nato should spend more on their own defence. He has said that Nato members should no longer be relying so heavily on the US to shoulder the costs of maintaining the alliance.
The argument between the allies focuses around the target agreed by all alliance members that defence spending should reach 2% of GDP (gross domestic product, the total value of goods produced and services provided in a country) by 2024. Germany, along with many other countries, has yet to meet this target.
Defence Secretary Mark Esper struck a different tone when announcing the decision, suggesting it was part of a broader plan to reposition US forces in the region.
He said it was “a major strategic and positive shift” that would “unquestionably achieve the core principles of enhancing US and Nato deterrence of Russia”.
The move is set to cost the US government several billion dollars and will reduce the country’s military presence in Germany by more than 25%.
A squadron of fighter jets would be moved to Italy while some troops could be relocated to Poland, Mr Esper said.
The decision was criticised by German officials, with the chairman of the country’s foreign affairs committee suggesting it would “weaken the Nato alliance”.
And the head of the German state of Bavaria, Markus Soeder, said he regretted the decision. “This puts a burden on the German-American relationship,” he told reporters.
There was also bipartisan criticism in Washington. “This is a self-inflicted wound… against American interests,” Democratic Senator Jack Reed said.
Republican Senator Mitt Romney described the decision to remove troops from Germany as a “grave error” and a “a slap in the face at a friend and ally”.
The plan was first announced by Mr Trump last month, when he accused Berlin of being “delinquent” in its payments to Nato. He also accused Germany of treating the US “very badly on trade”.
He also claimed that Germany was profiting from having the troops there because soldiers spend their money in the country.
The US military presence in Germany is a legacy of the post-World War Two Allied occupation of the country. Germany currently hosts by far the largest number of US forces in Europe, followed by Italy, the UK and Spain.
Some US personnel based in Europe support non-Nato operations and US military numbers fluctuate as forces are rotated in and out of Europe.