Kim Kardashian and Kris Jenner scored a legal victory in their battle against Ray J ... because his countersuit over their sex tape settlement will now play out in private arbitration, not a public courtroom.

A judge ruled Friday that Ray J's countersuit against the famous mother and daughter -- claiming they've violated a settlement agreement by continuing to bring up Kim's sex tape with Ray J on "The Kardashians" -- belongs in private arbitration.

The judge agreed with Kim and Kris that the parties signed a contract to take disputes over that agreement ... and so that's where they'll handle this disagreement.

To be clear, Ray J's countersuit is the only thing headed to arbitration now ... the defamation case Kris and Kim filed against Ray J is still currently in court.

Kim and Kris' lawyer, Alex Spiro, tells TMZ ... "We expected the court to rule in our favor and we're pleased with the decision."

As you know ... Ray J's was sued for defamation for claiming Kim and Kris were being investigated for racketeering charges.