BOSTON (AP) — Federal prosecutors say there’s no proof the judge overseeing gangster James “Whitey” Bulger’s trial is biased.

Bulger’s attorneys say U.S. District Court Judge Richard Stearns has a conflict of interest because he was a federal prosecutor in the 1980s, when Bulger was an FBI informant. They say Stearns would have known about an agreement for immunity from his crimes that Bulger says he had with the FBI.

Stearns has rejected calls to step aside, but an appeals court scheduled a Jan. 8 hearing in the case.

The Boston Globe reports that prosecutors argued in filings Friday that Stearns was never privy to matters involving Bulger because Bulger’s case was handled by a special task force. They also argued Bulger has failed to show any immunity agreement existed.

Bulger is accused of participating in 19 murders.