The web giant has scanned millions of books and made them available online via its eBooks platform.
Google had negotiated the deal to settle a six-year-old class action suit claiming infringement of copyright.
But the New York court said the deal would “simply go too far”, giving Google an unfair competitive advantage.
Under the agreement with the Authors Guild and the Association of American Publishers, Google would continue to digitise books and sell access online.
In return, the company would pay $125m (£76.9m) in royalties every year to the copyright owners of the books being scanned.
However, copyright concerns persisted, as the ownership of many of the works being scanned by Google could not be established, meaning many would be unable to claim the royalty payment.
“The [amended settlement agreement] would give Google a significant advantage over competitors, rewarding it for engaging in wholesale copying of copyrighted works without permission, while releasing claims well beyond those presented in the case,” said judge Denny Chin.
The US Department of Justice has approved the ruling and said it was the “right result.”
It has been critical of Google’s deal, saying it would give Google exclusive rights to profit from “orphan works”, where the rights holders are unknown or cannot be found.
Gina Talamon, a Justice Department spokeswoman, said the agreement “created concerns regarding antitrust, class certification and copyright issues.”
The agreement is also separately being investigated by the US Department of Justice on competition and copyright grounds.
Google responded to the ruling saying it was ‘disappointing’.
“We’ll review the Court’s decision and consider our options,” said Google’s managing counsel, Hilary Ware.
“Like many others, we believe this agreement has the potential to open up access to millions of books that are currently hard to find in the US today,” she added.
“Regardless of the outcome, we’ll continue to work to make more of the world’s books discoverable online through Google Books and Google eBooks.”
Google has already scanned some 15 million books.