Rinse immediately, every time
The single most important step in caring for luxury swimwear is rinsing in cold, clean water within an hour of removing the piece. Salt water accelerates the breakdown of elastane fibres. Chlorine bleaches dye and degrades fabric structure. Both continue working on the fabric after you leave the water if not rinsed out promptly.
Cold water is important. Hot water relaxes the elastane in the fabric and reduces its capacity to return to shape. Over repeated washes, this is what causes swimwear to lose its hold and begin to sag. Even warm water is enough to cause this damage over time.
If you are travelling and do not have access to clean water immediately, rinse as soon as you can. A salt-water rinse is better than no rinse at all. The priority is removing chlorine first: pool water causes more rapid colour degradation than sea water in most fabrics.