Sometimes I think I have no right to feel hard-done-by, but mostly I feel angry. My first wife, Susan, and I met at university and had a great connection: same sense of humour, same interests, even the same profession. We're accountants, which doesn't sound exciting, but I'm in corporate venture transactions, and the financial rewards are thrilling. Susan is a partner in a large firm. We had a lovely home and great social life and everything jogged along quite easily.

I'd never been faithful to Susan. Soon after we met I had a fling with a girl I met in a bar and realised that if I was discreet, no one need ever know what I was doing. That pattern continued, made easier by my job taking me away a lot. When we were in our late thirties Susan brought up the subject of children and we decided we were ready for parenthood. Four weeks later she was pregnant and we were delighted, though I barely had time to celebrate before heading to Spain, ostensibly for a work weekend but in reality to enjoy golf and serious luxury at a top hotel. A colleague introduced me to a high-flyer who was joining our division the next week and I was instantly attracted to her. Julie was 28 and stunning, but my professional reputation was important to me so I made a big effort not to flirt and we had a platonic start to our relationship. That weekend was spent discussing her role in the bank.

Two weeks later Julie and I began an affair. It started after working late one night. We were trying to grab taxis in the rain and were under her umbrella. I kissed her. She pulled me into an alleyway, and we had sex against the grimy bricks. In the taxi later, we frantically sorted our clothes and parted at the station, where I tried to smarten myself up before getting the train home.

Susan was in bed and I sat up for hours. I knew that this was more than an ordinary affair and if Susan hadn't been pregnant, I'd have left then. Instead, that was the beginning of three years of ecstasy and agony, to coin a cliché. Julie and I were totally professional in front of colleagues, but took risks in ways that I still can't believe. She would make an appointment with me via my secretary, come in with files, lock the door and take her skirt off to reveal stockings and suspenders. We often ate in a restaurant downstairs from our office that many of our colleagues also used, but I doubt they also had regular sex in the alleyway behind the restaurant. I lived in fear of getting caught as I would have been instantly sacked, but nothing could have stopped me.

One day at work there was much talk about what had caused the scrapes across the conference table. We joined in, well aware that Julie's stilettos had done the damage the previous night. The riskiest time was at a conference: I went into a stationery cupboard and Julie followed me and, despite knowing that if anyone opened the door we'd be seen by 200 people, we had sex.

But it wasn't just about sex. Julie was good at her job, fascinating to talk to and great fun, and always ready to stand her ground. I finally told her about Susan's pregnancy three months before the baby was born. She was furious, not just about the baby, but because I had lied to her. She even punched me, but I almost enjoyed it, so besotted was I.

When Richard was born I was in turmoil. Julie was hurt by my instant love for him and Susan was starting to question the distance between us. I felt torn. Life was a constant juggle between Richard, working and seeing Julie until, two years on, she said that since I couldn't make a choice, she would: she was applying for a transfer. I couldn't bear to lose her so I promised to leave Susan. The next few months were hell, as I told Susan, bought a flat and organised seeing Richard as much as possible. Susan simply couldn't believe it and though I said I'd only recently got together with Julie, she didn't believe me, finally asking if we'd begun our affair while she was pregnant. I denied it, but I don't think she believed or forgave me. The next five years were punctuated by broken arrangements and rows about access. Julie moved within the bank, so we were not in the same department before we went public with our relationship, which caused little comment.

I was surprised when Julie said she wanted to get married - she'd always said marriage and children weren't on her agenda. I wasn't keen but eventually agreed, though the next year I was taken aback when she said she wanted a baby. I was adamant that a baby wasn't part of our agreement but the arguments raged for months - alongside our sex life - until she said she would leave me and have a child with someone else. So I agreed, and Alice was born six years ago. I love her as much as Richard and the two of them adore each other, while Susan seems to feel less angry.

But Julie has time and energy for everything else apart from us; our sex life is a desultory once or twice a week - in bed with the lights out. We still enjoy the children, work for the same bank, meet for lunch and have lovely holidays. But I look at this beautiful, passionate woman who has caused me as much pain as pleasure and I don't know her. Life isn't all about sex, but it was a big thing for us for so long and now she just shrugs and says “Oh, I know it's me. I should give you more attention”, so I feel pathetic, begging for attention. I feel so short changed and full of fury that at times I hate Julie, but can't bring myself to admit it or do anything about it. I'm back where I started, in a mediocre marriage.

Times recommends

• Live chat: Thrifty ways to update your winter wardrobe
• Is anybody out there really listening?
• Nutrition Q&A: Will eating late make me fat?