Eric Swalwell's senior staffers are distancing themselves from him ... while acknowledging not everyone has the means to do so.

Top staffers for Swalwell's congressional office and his gubernatorial campaign released a joint statement ... which begins with indignation over his alleged abusive behavior.

The two sides stand with their former colleague who accused Swalwell of raping her in 2024 and other women who have come forward and accused the congressman of sexual misconduct.

That said, the staffers also want to make clear that people who choose to keep working for Swalwell are not necessarily doing so out of a sense of loyalty ... but because they have to be realistic about their finances.

The statement reads, "We recognize that not everyone -- in particular out junior staff -- can immediately forfeit their income and benefits without significant personal risk or consequence. Our responsibility now is to them."

The staffers end by saying that they -- more than Congressman Swalwell -- understand they have a responsibility to the people of California's 14th Congressional District ... and those who remain on staff will serve those constituents, not Swalwell.

As you know ... a staffer who spoke anonymously to CNN said she went to a bar with Swalwell and blacked out -- remembering just a snippet of the evening when she woke up and he was on top of her.

She says she tried to say no and push him off ... but, claims he did not stop.

Top CA Dems have called on Swalwell to drop out of the race for Governor despite the candidate protesting his innocence.

And, the Manhattan District Attorney has now opened an investigation into the alleged incident.