It is rodeo day at Louisiana's state penitentiary, home to 5,200 murderers, rapists and armed robbers, and thousands of spectators are having a grand old time. They have seen prisoners hurled from bucking broncos, and others competing to hold a wild cow long enough to extract a cup of milk. They revelled in “Convicts' Poker” - a perennial favourite that involves four prisoners playing cards at a table as they are charged by a bull: the last one sitting wins.

Three men have already been taken to hospital, but the convicts don't mind. They are volunteers, and the rodeo offers them a rare chance to win money, have fun and, for once in their wretched lives, stand tall.

Now it is time for the finale - “Guts and Glory”. About 50 inmates will vie to pluck a $600 (£305) poker chip from between the horns of a rampaging 2,000lb bull. The huge beast erupts from its pen. It charges at the nearest men, who scatter. It catches a man on one horn and tosses him into the air. Another inmate is trampled. Nobody dares to get near the raging animal.

Then one prisoner - tall, black and strongly built - finds himself alone in the centre of the ring, facing the bull. He stands his ground. The animal hesitates and lowers its head menacingly. The man darts forward and vaults over the flailing horns, snatching the chip as he does so.

Alan “Foots” Pharr, 43, has served 21 years for shooting a shopkeeper in a robbery. As the crowd stands and applauds, he approaches the VIP box. There he throws the chip to Burl Cain, the prison warden.

It was a gesture of gratitude, Pharr explains afterwards. Winning “Guts and Glory” was the greatest achievement of his troubled existence. “Warden Cain gave me the chance to make something of my life, and I wanted to show him that I'd done that.” That is a remarkable thing for a convict to say about his jailer, but it is a sentiment heard often in this extraordinary prison, commonly known as Angola.

“He's the best warden we've ever had,” says Jerry Williams, 51, one of hundreds of convicts selling food and handicrafts outside the stadium while an inmate band serenades the visiting public. Williams has served 31 years for murder.

“God always uses a vessel, and God has used Warden Cain,” says Carlwyn Turner, 47, a convicted rapist who is a disc jockey for the only federally licensed radio station in a US prison - the “Incarceration Station”. “Warden Cain has given a lot of guys purpose. That's what keeps them going,” says Lane Nelson, 53, another convicted killer and “Death Row” correspondent of The Angolite, the prison's award-winning newspaper.

It is hard to argue with such accolades. Cain, 65, a fervent Christian with a deep Southern drawl and the build of a refrigerator, believes he was sent to Angola to do God's work, and what he has achieved there in 13 years is little short of a miracle.

He has transformed the most violent maximum security prison in America into its safest. He has turned Angola into a place where families with young children happily consort with convicted killers at the spring and autumn rodeos. He has brought hope where there was only despair.

Angola occupies a remote loop of the Mississippi and, but for the six heavily guarded camps where the prisoners live, it could be a manicured private estate. Its roads are lined with flowerbeds and white fences. They dissect 18,000 acres of rich farmland on which the inmates tend cattle and grow ample crops for the prison's needs. With labour costing just four cents an hour, it is one of the last places in America where cotton is still picked by hand.

There are no watchtowers or fences around the 26-mile perimeter because the swirling Mississippi is virtually unswimmable, and the fourth side is flanked by impenetrable forested hills. The prison keeps a pack of bloodhounds just in case.

On a fine spring day Angola looks almost beautiful, but until recently its history was one of relentless suffering. Before the Civil War it was a slave plantation. Afterwards convicts worked it, and were so brutally abused that between 1870 and 1900 3,000 died. Many were buried in the levees they were building along the Mississippi.

In the 1930s guards inflicted thousands of floggings on prisoners. In 1951, 31 convicts cut their Achilles tendons in protest at their appalling treatment. As recently as the mid-1970s gangs ran wild, new inmates were routinely sodomised and prisoners slept with knives beneath their pillows. At least 40 were killed between 1972 and 1975, prompting Time magazine to label Angola “the bloodiest prison in America”.

Even today it represents, in one sense, the worst of the US penal system. It is a prison from which only a handful of inmates will leave in anything but a coffin, no matter how exemplary their behaviour.

Lifers account for 71 per cent of them, and in Louisiana “life” means life - even first offenders are ineligible for parole. The rest are serving sentences averaging 91 years. “A lot of people are in for crimes of passion,” says David Garrett, a former dentist serving life for murder. “It was a once-in-a-lifetime thing, but in Louisiana you're locked up for ever.” As if that is not enough to break prisoners' spirits, most are abandoned by their families. Their wives find someone else, their children stop visiting, their mothers die. Four fifths of Angola's overwhelmingly black and ill-educated inmates receive no visitors. A third are buried there because no one claims their bodies.

“It's downright evil,” says Douglas “Swede” Dennis, 72, who arrived in Angola half a century ago. The former Marine was hitching through Shreveport one night in 1957 when the police detained him for no apparent reason, and he accidentally killed a man during a brawl in the holding cell.

In Angola the next year he lost an eye when attacked by fellow inmates. In 1964 he killed a notoriously violent prisoner in self-defence and received a second sentence. He escaped in 1979 but was recaptured ten years later and is now recovering from open heart surgery. The only way he will ever leave now is in a body bag, he says.

“Do I still think I belong here? No. I think I should have been out decades ago.”

Cain agrees. “Prison should be for predators, not dying old men,” he says. “Some people need to die in prison. Some people we can't change. But just because you can't change some doesn't mean you can't change any.” At least half Angola's inmates are redeemable, some are genuinely good men and a few are probably innocent of any crime, he says.

Such hopelessness would seem to be a recipe for violence among prisoners with nothing left to lose. But Cain sees opportunity in their despair. He regards them not as evil men to be locked up and forgotten, but as souls to be saved. He preaches “moral rehabilitation”. He seeks to persuade his captive flock that they can still reach Heaven if they mend their ways. He recalls his mother's edict when he took the job: “If you don't see that those prisoners have a chance to know Him, He will hold you accountable for their souls.”

Cain, a self-styled “benign dictator”, has doubled the number of prison chaplains, built chapels in all six camps, and opened a Baptist Bible college that trains inmates to become ministers.

He calls Angola “the land of new beginnings” and has had a biblical verse inscribed in stone at the prison entrance: “I have learnt to forgive and forget about the things that are behind me. I am pressing forth and reaching for the things that are before me.” It is a message that falls on soil as fertile as Angola's silt-covered fields.

Cain claims that at least half his inmates have found God. More than 140 have graduated from the Bible college and serve as ministers in Angola or as “missionaries” to other Louisiana prisons. About 250 others have been baptised in the past three years.

Where violent gangs once ruled, the inmates have formed dozens of churches and religious groups with names such as “Men of Hope” and “Wonders of Joy”. Volunteers look after dying men in the prison hospice. Others repair wheelchairs for disabled Third World children, or make toys for deprived kids. “They are learning how good it feels to do the right thing,” says Cain.

In three years there have been no killings in Angola, and only two suicides. Last year there were 286 assaults - a record low, down from 1,061 when Cain arrived in 1995.

The atmosphere is remarkably relaxed. The guards - mostly unarmed - banter with inmates. I talked to prisoners in their blocks without guards present, and without feeling threatened. Every other inmate described himself as “born again”.

Pharr, the “Guts and Glory” winner, expressed deep remorse for his crime and strives to steer younger inmates towards God. “I've learnt that Heaven and Hell are real and I want to do things here, even if I never get out, to make sure my life on earth is not in vain,” he says.

Turner, the DJ, is in the Incarceration Station's cinderblock studio. He plays gospel music interspersed with preaching and Bible readings for 12 hours every Monday and Wednesday (gangsta rap, heavy metal and songs with violent lyrics are banned). “We need the Lord to direct our path,” he croons, as he puts on Order My Steps by the Mighty Clouds of Joy.

Turner, an erstwhile rapist who is now an ordained minister, broke down in tears when asked if he had children. His 27-year-old son was also in Angola for murder, he confessed. “I feel bad. I feel responsible,” he wept, adding that he cried out his son's name while on air.

Ron Hicks, 38, convicted of murder 19 years ago, is now pastor of a flourishing inmate church with 200 members. “I'd love to make peace with my victim's family,” he says. Meanwhile, he likes to think that God has chosen him to work in a prison. “God can use me inside or out,” he says. “I would prefer it was outside but if I have to do His will right here, I have to do His will.” Jerry Williams, another convicted killer, is selling food at the rodeo to raise funds to buy clothes and medicines for other inmates. “I changed my life when I came here,” he says. “We're not terrible people. I just made a mistake at a young age and I'm paying for it.”

Cain doesn't forget Angola's 83 Death Row inmates. He was racked by guilt for failing to console the first of the six men executed during his tenure. Since then he has ministered to each prisoner in his final hours, and held his hands as the lethal chemicals course through his veins. “God is waiting for you. You're going to ride away on a chariot. Your time has come,” he told one killer, Antonio James, as he died. After each execution he announces that the prisoner's soul has been “sent to God for final judgment”.

Not all inmates are gentle and compliant, of course, and Cain does not rely solely on religion. He also employs a more earthly system of sticks and carrots.

Some 1,400 inmates have earned trustee status through years of good behaviour. That entitles them to substantial privileges - good jobs (prison angler, for example), higher pay, less supervision, access to hobby shops and participation in the rodeo. A few are even allowed out of Angola to preach or to sing in inmate choirs.

But Cain swiftly removes those privileges from transgressors, or from whole groups if prisoners know that one of their number is breaking the rules and don't inform on him. Real troublemakers face banishment to the infamous Camp J, where they live in isolation, wear jumpsuits and are deprived of personal possessions.

Cain's methods have generated interest in a penal system not known for its compassion - several prisons across the South are starting Bible colleges, hospices and rodeos. But it is public opinion in Louisiana that needs softening if the state is ever to relax its “life means life” law. That is why the rodeos, which bring 10,000 people a time to Angola, are important.

Cain says: “It's the inmate's chance to show ‘I'm not a killer any more. I did a horrible thing one time and I'm really sorry and I won't ever do it again'. And then the people go back and say ‘I couldn't believe it. I could've left my children with some of them'.” Garrett, the convicted dentist, agrees. “When you're put in prison you're ashamed and embarrassed,” he says as he sells wooden boxes and bowls at the handicrafts fair. “But here people see that you're not an animal or a totally bad person. It makes you feel good about yourself.”

Cain's next mission is to encourage reconciliation between inmates and their victims' families, because no politician would dare to consider parole for convicted killers unless those families were on board.

At present the families must make the first move, and few are inclined to do so. But the warden believes that his prisoners are ready, that most are truly repentant, and quotes from Chronicles: “If my people ... shall humble themselves, and pray, and seek my face, and turn from their wicked ways, then I will hear from Heaven, and will forgive their sin, and will heal their land.”

He continues: “This land has seen more human suffering than anywhere in America, but look at it now. It's healed, and that's a God-thing. These inmates have turned their faces, and He has healed this land. It's probably safer now than New Orleans.”

Explore Women

• Fashion

• Beauty

• Diet & Fitness

• Relationships

• Families

• Celebrity

• Body & Soul

• The Way We Live

Times recommends

• The dirty weekend guide to Europe
• Sharing the McCanns’ hell
• Steve Coogan, weathering the storms