It's an inescapable fact that when we fall in love, we bring all our previous sexual experience and conquests into the new relationship. However much we want to make a fresh start, and in the words of Madonna approach our new partner “like a virgin”, it's hard to throw off the past. For many people, the legacy is a positive one, especially if the ex has helped them to feel safe and secure. However, if he or she was possessive, abusive or unfaithful, the past can cast a shadow over subsequent relationships.

Surprisingly, the ghosts of ex-lovers is less of a problem at the very start of a relationship. This confuses many of the couples in my marital therapy office. Rachel and Mike, both in their early thirties, sought help because their sex life had gone from being a source of great pleasure to one of conflict.

“It was really passionate for the first six months, but once we moved in together sex started to dwindle and now it happens only if I initiate, and then not always,” said Rachel.

So how can things change so quickly? When we first make love, we are very aware of past lovers and how we measure up. As intercourse is all about possession and surrender, casting out the ex is part of the excitement, drive and passion. However, once lust, the other ally at the beginning of a sexual relationship, has begun to wear off, ghosts can creep back into the bedroom.

If the sexual honeymoon has been a short one, I normally discover that the problems are as much rooted in a past relationship as the current one. This proved to be the case with Rachel and Mike. He had been springing out of bed early at the weekends, not to get on with chores, but in case he felt pressured to perform. He had fallen for a common myth about male sexuality: “A man is expected to be interested, always ready,” he said. Worse still, Mike's previous partner had also bought into the myth. “If I was not interested in having sex, she'd assume there was something wrong with her, that she'd put on weight or that I didn't fancy her. She couldn't accept that I might just be tired or that I simply had a lot on at work.”

Past performance casts a shadow

On a couple of occasions in his previous relationship Mike had not been able to sustain an erection and his previous girlfriend would sulk. So now, unless he was 100 per cent sure of delivering, he stayed on his side of the bed. Once Rachel understood the shadow cast by Mike's ex, she saw her own behaviour in a fresh light: “So if I come across the bed, even though it might be just for a cuddle, you interpret that as a demand for sex?” Mike nodded. With this understanding, Rachel and Mike were able to put his past behind them and lovemaking became a pleasure again.

For other people, the ex is not a turn-off but a turn-on. Alice, 48, had become preoccupied with her first teenage love: “When I'm having sex I imagine my old boyfriend. I've started having day dreams about tracking him down and meeting for a coffee in a hotel lobby. Except it doesn't stop there; we're soon upstairs naked in a four-poster bed.” She had started to question her 20-year marriage and whether she and her first love were destined to be reunited.

However, it soon became clear that Alice's fantasy boyfriend and the real one were only tangentially connected. “Of course, he would probably have lost that long Seventies hair and the tight waist, and he wasn't that good a lover: hurried and a bit rough,” she admitted. So I asked a couple of questions. How did Alice appear in her fantasy? “I'm young and beautiful, too.” What did that time in her life symbolise? “I was about to go to college. So, I guess, new possibilities, doors opening.” By contrast, her life today seemed stuck and dull. The fantasies were indeed an insight into what was lacking in Alice's life, but not in the literal way that she had first thought. So she stayed in her marriage but took a further-education course.

Cheating has its own rewards

The third kind of couple who are affected by exes in the bedroom are those where one half has recently had an affair. In my ongoing survey into the impact of adultery in the UK, many couples report a boost to their sex life in the first weeks and months after discovery. “What can I say, it's been amazing” says a woman, 42, whose partner had cheated on her for “a few weeks”. Another woman, 47, after her husband had an eight-month affair, reported “a massive improvement; we were in a rut. I have rediscovered my passion for him. I think I was trying to reclaim him, but it's not just physical. It's about reconnecting too.”

However, like when couples first get together, this initial passion can quickly burn away. At this point, if they cannot talk about their love life and address the underlying issues, the ex will become a wall between them. “I have started to obsess about what my wife and the other man had done. I want to know everything but I want to know nothing,” said Martin, 39. However, during counselling, he focused on how he and his wife's love-making could be different. “I need to make her feel more cherished. I could spend more time cuddling, pleasing her.” The result was that his wife seemed more turned on and his confidence grew. Love-making became about the two of them and the ex was banished.

So what is the key to unhooking the past and enjoying a satisfying love life? First, it is important to understand that there are two parts to our sexuality: our inherent sexuality (what touch excites us, personal preferences, the physical process of reaching orgasm) and relationship-specific sexuality (chemistry, the give and take between each partner's inherent sexuality, what's happening in the wider relationship). Particularly for people whose libido has been damaged by past experiences, it is easy to forget that they have a personal inherent sexuality independent of their ex.

Secondly, we must acknowledge our own and our partner's pasts. The fantasies and ghosts spiral out of control and gain in power only when they are ignored. Finally, the issue further underlines the value of good communication. If you can talk freely with your partner, and put everything on the table, a good sex life is almost guaranteed.

Andrew G.Marshall is a marital therapist and the author of I Love You But I'm Not in Love With You (Bloomsbury, £8.99). To take part in his survey into the impact of adultery in the UK, visit www.andrewgmarshall.com

Times recommends

• Live chat: Thrifty ways to update your winter wardrobe
• Is anybody out there really listening?
• Nutrition Q&A: Will eating late make me fat?