1. Create a publish site that is joinable.
2. Add a user to the site.
3. Deactivate the user from step 2.
4. Login as the user from step 2 and go to their Membership tool.
The site does not appear on their My Current Sites, which sounds correct to me.
The site does appear as a link on the Joinable Sites which it does not sound correct to me. Click on the site link on the Joinable Site page does nothing. It would be a 'nice to have' if this link was not displayed to the user.