I’ve had this also in the same circumstances you describe OP. My guess was that the realm was fine when we went into the dungeon (≤85 players including us). But when we exit the dungeon, the realm count has been inflated past 85 by cheaters. The game thus (mis)judges we are trying to connect illicitly to a full realm, and unfairly disallows it.
Also had it several days when attempting to exit a dungeon after the server has gone into “20 minute restart countdown”, which always used to allow rejoin regardless of the countdown, ss from 5 days ago:
Same outcome, kicked back to login screen.
They have said:
Hopefully this will encompass this problem too.