What happens when you are born with the American dream fulfilled? I was born just off the beach in Malibu, California. My father, Peter Lawford, was a movie star and a member of the Rat Pack. My mother's brother Jack would be president of the United States. I was given wealth, power and fame when I drew my first breath. How do you follow that?

My mother gave birth to me in St John's hospital in Santa Monica, California, on March 29, 1955. That day, Judy Garland gave birth to her son, Joe, in the same hospital. Western Union delivered a boatload of telegrams.

"We're so happy for you both. he'll be quite a boy. Love — Jeanne and Dean Martin."

"Dear friends — I'm so happy for you both and may I say you picked my favorite hospital for this epic event... Hello to Sister Mary David — Bing Crosby."

"So thrilled for you both. Love Gary & Rocky Cooper."

At the moment of my birth, my father was having lunch at one of his hangouts and talking to Cary Grant. Cary was reassuring him. Not about becoming a father but about his career. "Don't worry, old man. As soon as you get a little grey in your hair, you'll work all the time. I didn't work for two years, my temples got grey, and it was a whole new ball game."

Then the call came, and half an hour later he opened the door to her room to find her sitting up in bed with a bottle of J&B scotch, ready to celebrate. "Come on in, we've got a big, beautiful boy. Let's have a drink." A few minutes later, the big, beautiful boy was delivered to his celebrating mom and dad. My father looked down at me and declared: "That's not my kid. He looks Chinese. Hey, wait a minute, Pat, wasn't the gardener Asian?" They laughed. And had another scotch.

I was the first boy born to a mother who was the product of a family with a long and lusty tradition of glorifying and supporting the male. You can't get more fawned over than a Kennedy male. My mother had struggled against the yoke of being a talented and wilful female in a family that didn't really care what the women were up to as long as they were having lots of babies. Her marriage to my father and her subsequent life in California were early attempts to find her own identity and be noticed outside of "the Family".

"I fell in love with your father the moment I laid eyes on him," she told me. "He was so handsome. Grandpa [Joseph Kennedy] sent me on a trip around the world to get him out of my system. It didn't work. I got to Japan and turned right around."

My grandmother Rose wrote a letter to my mom not long after I was born, advising her not to "wear falsies that are too prominent as they are not only cheap but tempt you know whom"! I assume Gramma was talking about my father. Well, it didn't work: after me, they had three more children. Three girls: Sydney, Victoria and Robin. A blonde, a brunette and a redhead. All the bases were covered. I was the only boy, the oldest, the king. The way it should be. Pure Kennedy.

When my mother married my father she made a monumental statement of independence from her own father, whom she adored. Joseph P Kennedy, my grandfather, was the man from whom everything flowed. He was the power, the money and the brains. My mother was my grandfather's favourite. He was the architect of our world. There was a sense that everything the Kennedy family was came about as a direct result of my grandfather's will. On September 7, 1957, my grandfather predicted one of his sons would be president, one would be attorney general, and another would be a United States senator, all this simultaneously.

It was not just about money and power with my grandfather. He was first and foremost about family. My mom told me: "Grandpa gave each of us a million dollars when we turned 25. All of his friends told him not to do it, saying his kids wouldn't give him the time of day if they got all that money. It wasn't true. We all still can't wait to come home." You couldn't keep my mom and her siblings away. Later I figured out that although Grandpa gave his children the money to realise their independence, he never taught them what to do with it.

My mother was more like my grandfather than any of her siblings. She had a mind for money, a strong independent streak, and she could cut you off at the knees with "the look". There was no mistaking it when Joe Kennedy was unhappy with you. His displeasure burned in his eyes and straight into whoever was unlucky enough to cross him. I don't recall ever getting "the look" from my grandfather, but my mom more than made up for it . She also had her father's instinct and luck when it came to making money. My grandfather often said: "The one with the best business head is Pat. If she put her mind to it, she could easily take over the business."

Times recommends

• Are eldest children really a cut above?
• Peaches Geldof: censored
• Top 10 style-setting films of all time