WASHINGTON (AP) — Donald Trump says he was surprised that Newt Gingrich hammered him for noting the Mexican ancestry of the American judge overseeing lawsuits against Trump University.

Trump tells Fox News Channel on Monday that the former House speaker’s criticism was “inappropriate.” Trump says he’s only defending himself against relentless questions about Trump University by questioning the impartiality of U.S. District Judge Gonzalo Curiel. Trump says most customers were satisfied with the school.

Curiel was born in Indiana. Still, Trump says, the judge cannot be impartial in the cases because Curiel’s parents were born in Mexico and Trump wants to build a wall to keep people from sneaking over the border.

Gingrich is one of several prominent Republicans urging Trump to lay off Curiel and unify the GOP.