This is a snapshot of how teenage boys live in inner-city Britain. A 14-year-old stands above an underpass and throws stones at the cars below: “Just to get the chase, just to get the thrill.” A 16-year-old with a troubled face says this: “I didn’t like showing I was nice inside because it would make me out to be some soft person. It’s not about being hard, it’s about acting hard.” An 18-year-old has grown up with one overwhelming thought: “You just get into the idea, I’m never going to do nothing in my life, so why should I bother?”

No, these are not middle-class boys who expect to become doctors and lawyers and rich City people, they are boys who failed at school and they hail from deprived areas where the only achievers are drug dealers. They are also white, and while this may confound the middle-class assumption that low achievers are likely to be black, this makes them typical. Overwhelmingly, say Professor Robert Cassen and Dr Geeta Kingdon in their report Tackling Low Educational Achievement, it is white boys who fail in Britain. Remove the middle classes from the equation, look just at children from deprived areas, and more than three-quarters of low achievers are white, British and male.

They are visible at the age of 3, Cassen says. “The child from a professional middle-class home hears 1,500 different words a day. Aworking-class child hears 500. They don’t recover because we don’t have an equalising education system.”

That is one view. In recent weeks I’ve spoken to 20 white boys aged 13 to 25. All have underachieved, some to the extent that their lives have been at risk, though many are starting to work towards their potential because they are now in a system where support is available. I’ve also spoken to as many professionals who work with them. The questions are obvious: why do these boys fail, and what can be done to help them? How can we interrupt the cycle of deprivation, chaos and despair that is set to brand another generation as no-hopers with no future?

The answer to the first question is hard to pin down without resorting to Little Britain clichés about feckless single mothers, though it is clear that this debate is not about education but rooted in emotional welfare. Let’s start with the view from the street.

Dom is 16, bright, and he writes poetry and cooks a mean chicken roulade. He likes to talk and can see the big picture. As he is currently a resident of Aycliffe Secure Services, a children’s home near Darlington (thanks to a conviction for arson with intent to endanger life, his seventh offence), I make a convenient captive audience. This is his story.

Until he was 10 he was top of his class. Then his stepdad started to abuse his sister sexually and to hit Dom and his mam. Dom got into fights, was an alcoholic by 12, smoked weed and took pills. He was expelled from school, went to live with his dad on an estate in Leeds, got into trouble over drinking and fighting, and remembers finding a heroin addict sleeping in the place where you put the rubbish. Often. His dad beat him and threw him out. He was homeless for six months and, finding himself in a situation where no one was in charge of him, took charge himself by doing burglaries to get money. Back at his mam’s, he lasted five weeks in the next school. And so on.

“My mam is a star. I quite like being the centre of attention . . . My friends are one of the most important things to me . . . I think my mam thought I was going to fail. I know my dad did. He was shocked when I told him I was doing quite good here . . . I don’t think schools are making it exciting. Being excluded was great. I was only 13. Police couldn’t do nothing . . . I reckon money’s a big part of it. If your mam and dad are saying ‘I don’t want ’owt to do with you’, then it’s like, I’ve got no money. I’m too young to get a job. Then you get into the wrong group of mates, some of them with habits to feed. You never hear about kids who are doing bad stuff achieving. It’s not something kids are interested in. There are role models, and the type of things your friends are into influences you. I’ve heard people say, ‘I’ve shot this person’.”

This is said with no sense of drama because, on the estate Dom knows, shooting people is unremarkable. He is touching on many of the themes that crop up in conversations with these boys: loyalty to a single-parent mother who has offered no guidance or structure, low self-esteem, an absent father, no male role models other than gang leaders and drug dealers, peer pressure, low expectations and even an assumption of failure, truancy, a belief that schools are at fault for failing to offer exciting education, and exclusion, which, without careful management, tips boys on to the streets. There’s more: ambiguous discipline, or none at all, and from that a failure to recognise the connection between cause and effect, until, as with Dom, there is intervention that enables him to understand what has happened to him with a piercing and poignant maturity.

“There’s no respect for the community, just respect between friends . . . There’s no discipline from when you’re born. I reckon it’s about the no-smacking rule. How else are you supposed to discipline them? My mam couldn’t discipline me and there was no one else so I thought I was top dog. I don’t agree with people who say you can’t do anything.

You’ve just got to try. If you give up on kids they’re going to give up on themselves.”

Gill Palin is Aycliffe’s manager. “What we see is children with no structure being allowed to do what they want because mam’s too busy to control them,” she says. “It’s about parenting. The majority of the children in here – their parents have never said ‘no’ to them. There are mental health problems, domestic violence, alcohol problems, substance misuse, unemployment and this is happening to second and third generations. The parents have never asked for support, or if they’ve asked they haven’t got it.”

At 25, Adam has had most of the experiences a parent would want their child to avoid: heroin addiction, alcohol problems, prison, homelessness. He spent six months sleeping under a car park in Birmingham. He’s the oldest of five children who have different fathers, he’s had a lot of stepdads and he doesn’t know the man who he calls his sperm donor, though he’s seen him on the estate. He has two children by an ex-wife. He is tinged with sadness and finds it hard to believe in himself. He has worked, in a factory, because that’s where his careers teacher told him he was heading, as a brickie, a warehouse supervisor, a window fitter, but he knows that he has people skills and wants to get into youth work. He says, he could have got A*s at GSCE, but he didn’t go to school much because it was boring. When did he first realise he was good at something?

“A few months ago. I’m bright but I don’t seem to use it. I’ve adapted. I’ve had to fit in to get along. A few friends are still in prison. In and out. In and out. You wear it like a badge thinking, I can’t do this because of what I’ve done. People are telling you you’ll always be an alcoholic, a drug addict. It’s like when you were in school and they said you’ll be useless. That’s not encouraging. There was no guidance and if you’re not getting that at home . . . School should be more fun.”

It’s often said that there is an antieducation culture among boys, that it’s not cool to be seen to do well at school. None of the boys I spoke to recognises this, though their replies tend to bear it out. “It’s where you’re from more than anything,” Adam says. “Once you get in that in-crowd you’re like sheep, you just follow what everyone else does. I was always messing around. Where I lived it was all single parents in council houses. The dads aren’t paying any child support, they beat the mams. You can see there’s nothing for you.”

Not all the underachieving white boys I spoke to were from single-parent families, or had engaged with crime. Some were the quiet ones you can imagine slipping through the school system without attracting attention. Some, such as Lee McConville, 22, who was mentored by my colleague Phil Webster earlier this year and is now working on becoming a TV journalist, have two loving parents, which is probably why Lee finds himself the sole survivor of a group of friends who are either dead or in prison. “You’ve got parents telling you you need to go to school, but a lot of kids have got a problem with authority. So teachers telling them what to do makes them worse. There’s so many distractions. You just want to join in because you want a laugh as well. It wasn’t that you didn’t want to work, you just didn’t think about it. I never did the bad stuff, more like putting windows through. My mum used to cry, she thinks it was bad parenting. I tried to explain it’s not her, it’s me. In deprived areas there’s a big black cloud hanging over everyone who’s trying to make it. There’s lack of support, there’s nothing to do, that’s why you hang round on the streets. I was a cheeky kid, then my attitude did change at 14 – look at me, you can’t touch me.”

Dave and Jeanette McConville live in a Victorian terraced house that belongs to a housing association in Lozells, Birmingham. Most of the shootings and stabbings you hear about on the telly are within a mile radius, Jeanette says. She didn’t go to school because her dad hit her and the bruises embarrassed her. She was pregnant at 16 and, like her husband, she is unemployed. She is forthright but probably not as tough as she sounds. The messages are mixed. “Lee seemed to build an anger. We don’t know where it came from. It’s peer pressure. Kids want everything but they don’t want to work for it. Most of the kids round here are drug dealers . . . As far as I’m concerned once the kids are in uniform the school’s responsible.”

Dave says that Lee has never liked discipline. “I think he just wanted to be free.” Why do they think he has a problem with authority? “There’s no discipline at school.”

Later they admit that they are soft on their kids. Dave: “She’d send me upstairs to give them a slap and we’d all end up laughing. We love our kids.”

There is a pattern to these conversations. Parents who are themselves poorly educated and low achievers seem unable to provide structure or boundaries. Admittedly, this is not an easy thing to do, but the prevailing culture seems to be a moral vacuum in which responsibility lies elsewhere. Neither do these parents understand how to negotiate the school system, or have experience of making progress. Their own negative experiences have left them feeling unmotivated, their children pick up on that and, once they start to fail, they become accustomed to failure, says Steve Belcher, an outreach development worker at the Birmingham branch of Fairbridge, the charity that has helped Adam and Lee to understand that they have a future.

“If young people are labelled as failures, they’ll learn to fail and reach a point where they don’t know how to succeed. There’s often a catalyst that causes change. They lose a parent or a grandparent, that causes them to do things differently, they lose confidence and motivation. They look for blame and they say, ‘I come from a bad area, I go to a crap school, the teachers say I’m thick, my mum and dad say I’m useless’. Once they’ve got the label they keep it.”

The easiest way to counter that low self-esteem and lack of belonging is to latch on to street culture where boys readily find more credibility than from performing at school. This is why they tell you that it’s their friends who matter most to them. At no stage is there a sense that they have any choice; that concept is for the privileged classes.

The other thread that runs through this debate is the use of words such as excitement and fun. School is failing them because it isn’t exciting. Naughty stuff – many lads use these coy words to refer to crime – may be a matter of economic necessity, but sometimes it is done for a thrill. Where does that come from? Neil Ezard, assistant head in education at Aycliffe, is one of many professionals who note that these boys learn language from screens rather than from people, and that accustoms them to instant gratification.

“The role model in their own homes is usually grandma or mum or auntie. The male role model – the key element in forming a young boy’s impression of who they’re supposed to be – can come from a hero in a video game or an action adventure, which tends to be violent and which tends not to include good relationships with women. So the only thing that they value is fun and excitement associated with their role model. Some of the things they do are because of a lack of understanding that what they do to other people is real and has a devastating effect on their lives. You do despair.” Over the past decade Camila Batmanghelidjh has offered support to vulnerable children and young people through the charity she founded, Kids Company. “The biggest barrier to learning is the emotional state in which you come to the classroom, and that depends on whether you have had a robust attachment figure in your life,” she says. “If you don’t have that, a great deal of energy goes on just piecing yourself together every day, let alone navigating a learning task.”

Batmanghelidjh refers to all failing children. Yet we know that girls consistently do better at school than boys. Why? It is generally accepted that girls are more likely to have female role models within their families, and that they are more suited than edgy rumbustious boys to an education system that rewards diligence. But what about specific ethnic groups? For all the white boys’ lack of education they have acute antennae about perceptions of racism, which makes them uneasy discussing this, though Dom observes that “the majority of white people will be alcoholics and there won’t be many black or Asian alcoholics.”

Lee suspects that the respect for adults that has evaporated from many white families is alive in their black counterparts. “I think black people have more respect for their parents because they have to, their parents are probably stricter. Asian communities are strong because of religion.”

Some professionals are uncomfortable making ethnic distinctions, but those who are prepared to do so agree that black and Asian families offer more structure and discipline and that their communities offer the sense of belonging that poor white boys struggle to find.

At Fairbridge, Steve Belcher and his colleague Philip Rattigan believe that white boys lack a sense of cultural identity. “The sense of community around the black and Asian minority groups is stronger than for the white British lad,” says Rattigan, a development tutor. “Both the social community and the religious community. A young black lad who hasn’t got his father around will still have a male role model within the church or the youth centre or the community.”

Belcher talks of a sense of belonging in nonwhite communities. “Black kids are proud to be black, Asian kids are proud to be Indian or Pakistani. There’s not a lot of cultural identity for white Anglo-Saxon males.”

Several head teachers in inner-city schools tell me that, in their experience, white boys are more likely to play truant than boys from other ethnic groups. One believes that single mothers collude in truancy because they like to have a male around for company and to do shopping and odd jobs – even if he is at primary school.

In southeast London, Caralyn Betts, head of school links at Lewisham College, agrees. “There’s little sense of identity for white males. They’re often lacking a father figure, there’s a poor work record in the family, there’s no sense of people achieving and their learning needs are masked by poor attendance. There’s a sense of strong identity among ethnic minority groups. White boys are isolated.”

Breaking the cycle of failure and despair isn’t easy because, as one teacher said, these are hard-to-reach boys with hard-to-reach parents. The Government has tried through its Sure Start programme, but the consensus among those who work with failing boys is that it succeeds for families capable of finding it, but is failing to reach those who need it most.

“The centres need to be in the sink estates, we need joined-up agency work with health and social care in schools,” says Gill Palin, at Aycliffe.

This understanding – that professionals have to find these families, rather than expect families to find help themselves – is behind the success of School-Home Support, a charity that places specialist workers in schools with the aim of reaching the parents of failing and unhappy children and offering practical and emotional support, and parenting classes. SHS workers operate with care, subtlety and absolutely no coercion, they build trust and facilitate friendships and their tactics work, but as the schools have to find their salaries, funding remains a problem.

Fairbridge recruits vulnerable young people through agencies that work with them, including hostels, and uses confidence-building tasks and mentoring to reverse their expectations. Lewisham College offers vocational training peppered with accessible male role models, many of whom come from similar backgrounds to the boys they teach.

So there are models of good remedial practice that offer hope not just to failing white boys but to all children and young people at risk of becoming dysfunctional adults. The most imaginative work I saw was at Francis Bacon Maths and Computing College, a former grammar school that draws its intake – many disadvantaged – from estates near St Albans. To reverse falling rolls, the school has raised its GCSE A-C pass rate from 41 to 47 per cent over the past two years, and closed the gap between boys and girls from 25 to 8 per cent. This coincides with two initiatives designed to raise boys’ achievement.

The first is a series of “dads and lads” evenings for boys and an adult male role model. They are given an activity – such as building a barbecue – that they carry out as a team, they are fed and there is lots of bonding. David Graham, the deputy head teacher, remembers a conversation between an absent dad who said that his only involvement with his son was ripping him apart when he failed, and another dad who said he discussed his son’s progress with his wife and boy. “I’m getting it really wrong,” the first dad said. He now sees his son three times a week.

The other programme is called Bugs – Boys Underachievement Group. Boys identified as needing extra support at the end of Year 7 are offered a weekly class, and a star-chart system that applies to all their activities, and rewards good behaviour, attitude as well as learning and leads to prizes. How do you sell this to the boys, I ask Sonia Turner, who runs the scheme. “I tell them I want them to help me debug the school,” she says. “Boys learn more through hands-on learning, if it’s made exciting and if there’s competition.”

This year’s new Bugs swing in their chairs and have trouble concentrating, but with two teachers and in a group of eight they are engaged by the task of designing a school garden and it’s obvious that they enjoy the attention and the sense of belonging that comes with their Bug status.

Then I meet a group of older boys, two of them former Bugs, who can only be described as mature, positive and happy. I know that schools like to wheel out their successes, but this session is remarkable and moving because of the respect that they show each other and their teachers. They have a sense of responsibility, which is hard to nurture in children who may not always have had a stable, caring adults in their lives. These are their words.

Ashley: “My behaviour was going wrong because I wasn’t seeing my dad. I have anger and I lash out. Being part of Bugs I feel comfortable to talk to people about my anger. You don’t need to keep it inside.”

Jamie: “When you go to Lads and Dads it makes you feel a lot prouder of yourself, like you can do it. You knock yourself before. It’s proof.”

Jeron: “I used to walk out of classes, now I just let things go. With Bugs you get friends a lot easier and you’re not scared to ask like a proper smart person. You’re thinking, ‘it’s my life now’. It makes you proud.”

Jack: “We were all over the place a couple of years ago. We’re all together now, helping each other. There’s an impression that if you’re good at school, you’re not popular. Who needs popularity when you’re going to do something with your life? Since we’ve been doing this – yey! – we’ve been achieving more. It’s having somewhere where you know someone is going to support you.”

*Dom’s name has been changed

Times recommends

• Jamie Oliver's boiling over
• Family homes for the 21st century
• What makes men commit?