Twelve of the best new novelists By John Mullan

 To find the most promising new writers John Mullan and a panel of judges read piles of debut novels. What did they discover about the state of British literary fiction today?

                                                                                                                                Photograph: BBC

The books dozen (standing, from left): David Abbott, Deborah Kay Davies, Eleanor Thom, Adam Haslett, Evie Wyld, Rebecca Hunt, Jim Powell; (seated, from left): Samantha Harvey, Stephen Kelman, Ned Beauman, Jenn Ashworth, Anna Richards.

 The growth of British literary fiction has been one of the most extraordinary publishing phenomena of recent decades. Not everyone has been pleased. The label "literary fiction" is often used disparagingly, as if "literary" were synonymous with "pretentious" or "plot-free". "The two most depressing words in the English language are 'literary fiction'," declared David Hare recently in this newspaper. Some like to say that there is no such thing: there are only good novels and bad novels. Yet authors and publishers and readers recognise that literary fiction exists and offers its own particular pleasures. Its surprising commercial health has given would-be novelists the confidence to experiment, to trust they can find readers interested in the new shapes fiction can take.

 I was recently asked by BBC2's The Culture Show to chair a panel of five judges in an effort to identify promising debut novelists. Publishers submitted their outstanding first novels of the past couple of years, and we had to choose the 12 "best". What we got were examples of what we have come to call "literary fiction". We found our dozen, and in the course of reading 57 novels I got a picture of the state of British literary fiction. Reading all these first-time authors you could see the representative habits and ambitions of the would-be literary novelist, and see, sometimes all too clearly, the influences of established literary fiction.

 What is literary fiction? It is not genre fiction. Hilary Mantel's Wolf Hall is a historical novel. Kazuo Ishiguro's Never Let Me Go was shortlisted for the Arthur C Clarke Award, the leading British prize for science fiction. Yet you only have to think about these two examples to see how they escape their genres. Mantel's novel revisits the favourite stamping ground of historical fiction – Henry VIII and his wives – in order to rethink what it might be to see events filtered through the consciousness of a person from a distant age. Ishiguro takes a dystopian hypothesis – human clones being bred for their organs – and then declines to put in place any of the sci-fi framework that would allow us to understand how this could be. Indeed, the whole interest of his story is in the limits placed upon its narrator. These are both "literary" novels because they ask us to attend to the manner of their telling. And, despite their narrative demands, they have both found hundreds of thousands of readers willing to do so.

 People have been talking of "literary fiction" (a phrase still unrecognised by the Oxford English Dictionary) since the 1960s, but it was in the early 80s that it became established. There were earlier progenitors, such as John Fowles, whose novels combined dark or misdirected sexual passion with an obstinate bookishness – and showed that you could be self-consciously literary and make money. Clearly, The French Lieutenant's Woman lies behind some of the more playfully erudite English novels of later decades, notably AS Byatt's Possession. It is no accident that Fowles's publisher was Tom Maschler, who was instrumental in establishing the Booker prize in 1969.

 The key date was 1981, when Salman Rushdie's Midnight's Children won the Booker prize. The Booker had already begun to acquire notoriety – the year before had seen a much publicised run-off between William Golding and an entertainingly grumpy Anthony Burgess – but now it seemed to be revealing something new. Rushdie was an unknown, discovered by the judges, and Midnight's Children seemed like nothing in the English novel before. For the first time the prize ceremony was televised.

 Interviewed for our programme, Ishiguro recalled this moment being "absolutely pivotal" for young writers attempting to stretch the bounds of the English novel. "I think people are right to think of it as an iconic book that ushered in a new sort of attention for literary writing." It was unparochial not only because of its subject matter and its Indian narrator, but because of its formal ebullience. It owed more to Günter Grass and Gabriel García Márquez than to the modern masters of English fiction (although its literariness included a kinship with Laurence Sterne's 18th-century classic Tristram Shandy).

 Ishiguro is convinced that if his own first novel, A Pale View of Hills, had been published five years earlier it would have disappeared without trace. Formally divided between two narrative sections and two countries – England and Japan – it arrived just as readers and critics were embracing, even looking for, narrative experiment. The appetite for such fiction made it rapidly newsworthy, and younger novelists became celebrities. In 1982 the first branches of Waterstone's opened. Any book lover now middle-aged will recall how those first stores made novels, in particular, look like alluring commodities. Also interviewed for our programme, former Granta editor Ian Jack remembers how "the book became a fashionable object, the novel became a fashionable object in the 1980s". In 1983, Granta published its list of "The Best of Young British Novelists" in an issue that included excerpts from such writers as Martin Amis, Julian Barnes and Ian McEwan. The cover, featuring an exploding pen and a union flag, shouted that British fiction was the new thing. The first extract was the opening of Amis's as yet unpublished Money, bursting on the reader with a linguistic verve and lexical daring that seemed almost un-English. Now even sentences could be exciting.

 As a marketing exercise, it worked. Novelists began featuring in newspapers and magazines; new novels – and the advances that novelists had received – became news items. The Booker prize became a passport to commercial success. "Literary novelist" started to look like a rewarding career path, not an after-hours occupation. Prizes and lists were ways on to this path. Jack, himself responsible for the 2003 Granta list, is blunt: "Literary novels really depend on prizes, and they depend on lists." Partly it is just a matter of needing maps. The territory is thronged. Those Waterstone's tables may be already highly selective, but they are still crowded. We all need signs. But it is also a matter of perceived critical judgment. Literary fiction seems to invite discrimination.

 This is because such fiction calls attention to form. Once this was the property of avant-garde fiction; in recent years it has become a tendency in fiction that appeals to a mass readership. A leading case is David Mitchell's 2004 novel Cloud Atlas. This has a narrative structure unimaginable in a bestseller 20 years ago. Its author describes it as a "russian doll": six separate narratives, one inside another, set in six different times (ranging from the 19th century to some distant, post-apocalyptic future) and written in six different genres. Each narrative appears as some material remnant in the subsequent narrative.

 The formal playfulness is elaborate, yet it was not just shortlisted for the Man Booker prize, but was also selected as the Richard and Judy Book Club book of the year. Similarly, when Sarah Waters's 2006 novel The Night Watch employed a reverse chronology, it did not stop it being Radio 4's Book at Bedtime – or reaching a large reading public. Narrated in three sections, dated 1947, 1944 and 1941, it has multiple plots that require us to discover the causes of effects that we have already witnessed. It is the kind of narrative sophistication that ordinary readers now take in their stride.

 In the press and on radio and TV, discussion of new fiction is frequent but is almost all about its content. In fact it is formal restiveness, even tricksiness, that has distinguished much of the literary fiction of the past couple of decades. When I talked recently to a novelist of my interest in the "tricks" of contemporary fiction, he objected to that word. To him it implied formal gimmickry – a substitution of devices for narrative skill or literary depth. Yet this need not be the implication of detecting the tricks of contemporary fiction. Think of this moment in one of the best-selling and most widely admired literary novels of the 21st century:

 She knew what was required of her. Not simply a letter, but a new draft, an atonement, and she was ready to begin.

BT

London 1999

 This is the mock-ending of Ian McEwan's Atonement, which looks as if it is signing off (that tell-tale reversion to the novel's title) but in fact comes with a thickness of pages still remaining. It is a trick, certainly. The first time you read the novel you are almost certainly taken aback – until you realise that BT are the initials of the protagonist, Briony Tallis, and that what you have been reading is her story, not McEwan's. The trick is deeply satisfying, for it enacts what it reveals: that fiction might be a way of making up for reality. The atonement mentioned in the last sentence is not the subject of the story – it is the narrative itself. You can tell that the trick is not shallow, because when you read the novel again, it still works. You know that the story is being made up by one of the characters, yet it is impossible not to believe it.

 Narative tricks were certainly in evidence among the first-time novelists we read, sometimes announcing a literary ambition that was unsupported by gifts for characterisation or consistent plotting. Many novels were divided between different narrative voices or different tenses, or both. Several were made up of alternating narratives set in the present day and then in some distant historical period. First-person narration, which predominated, was often characterised by some expressive impediment. It was as if debut novelists had been reading The Remains of the Day or The Curious Incident of the Dog in the Night-time and thought of new ways in which they could fashion narrators with cruelly constrained powers of expression. There was also a fair amount of what Dr Johnson would have called "fine writing" – sentences designed to call attention to their style. Dimly some recalled that literary novelists did not use sentences just to get from A to B.

 Yet among our dozen, there were one or two who seemed wonderfully oblivious to formal trickery. I remember beginning Jim Powell's The Breaking of Eggs and experiencing an unusual narrative voice that was neither inadequate nor self-consciously stylish, and a story that proceeded in traditional Greene-ian fashion, from self-delusion to disillusion. The narrator is a man in his 60s, living in Paris and subsisting on the travel guide to eastern Europe which, because of his communist sympathies, he has been editing for most of his adult life. With the collapse of communism, and a visit to his long-lost brother in the hated United States, his ideological convictions begin to crumble. Powell was one of our two chosen novelists over the age of 60 – like his protagonist. The fashion for literary showiness seemed to have passed him by.

 The pressure for narrative playfulness does not just come from other books. There is also the influence of film and TV narrative. A novelist such as Andrea Levy uses complicated patterns of flashback and multiple viewpoints that derive from the screen rather than the page. The debt of fiction to film and TV narrative has not been fully measured by critics, though readers now take it for granted. You could often see this in our debut novels, particularly in the readiness to cut rapidly and without explanation from one carefully described scene to another, making narrative out of montage. Rose Tremain, the representative on our programme of the original 1983 Granta list, thinks that the influence of film distinguishes post-70s fiction. "I think that the other thing that perhaps affected my generation is that we had all seen an awful lot of cinema, we were all great cineastes. And I think cinema, and the swiftness of cinema, and the conciseness of imagery in cinema can inform fiction marvellously well."

 The new force is creative writing. Of the 57 novels submitted to our panel, almost a third were by graduates of creative writing courses. Invariably, writers who have been on such courses announce the fact in the information printed below the author's photograph on the dustjacket of their first novel. It is as if graduation from such a programme is a further recommendation to the potential reader. When Manchester University hooked Amis to teach creative writing, it confirmed the status of an academic subject now able to recruit the biggest beasts of contemporary fiction. Here was the scourge of cliché and flabby thinking announcing his desire to give seminar-room guidance on how to write. A still young subject seemed vindicated.

 Like many academic critics, I fancied myself resistant to the wiles of the creative writing graduates. Yet, disconcertingly, I noticed some of the debut novels that I most admired were by authors who were the products of such courses. Samantha Harvey's The Wilderness, for instance, sounds in outline like a creative writing exercise. It is the story of Jake, an architect who, it slowly becomes clear, is suffering from the first stages of Alzheimer's. Narrated in the third person, it follows the glitches and gaps in his own recollections. Like many literary novels now, it is narrated in the present tense, but with the irresistible purpose of inhabiting his experience. Sections in the past tense revisit islands of memory, allowing the reader to piece together the story of his adult life. Yet I found myself both seized and intrigued by its method, which allows the reader to inhabit the escalating errors of the protagonist.

 I wonder if the growth of creative writing is a symptom as much as a cause, profiting from an increased interest among readers – as well as writers – in formal experiment in narrative. It was not just what the novels were about, but the way they got written, that became more various and unpredictable. And what has driven this has surely been academic, too. During the 1980s and 90s, academic literary criticism sealed itself off from the general reader. What went on in seminar rooms and academic books became remote from the world outside. Yet many – probably the majority – of successful literary novelists had taken English literature degrees. Everyone knows that McEwan and Ishiguro did the MA in creative writing at the University of East Anglia, but both of them already had English degrees from other universities. Some novelists (Graham Swift, Alan Hollinghurst, William Boyd, Waters) had been Eng lit postgraduates.

 While pretending hieratical removal from the world of ordinary readers, literary academics were shaping the novel because they were teaching the novelists-to-be. Some successful novelists have been explicit about how the study of fiction inspired their own writing. I remember Louis de Bernières telling an audience of his readers that an MA course had helped him to discover the multi-narrator form that made Captain Corelli's Mandolin a literary bestseller. Mitchell has recalled in an article how Cloud Atlas had its origins in his study of Italo Calvino's If on a Winter's Night a Traveller during his MA in comparative literature at the University of Kent. Calvino's novel is composed of a sequence of interrupted narratives, inspiring Mitchell to pursue the conceit: "How many narratives deep could I go? Four? Six? Nine? Italo Calvino got to 12 in If on a Winter's Night a Traveller, but he never 'came back' to recontinue his interruptions. Could a novel boomerang back through the sequence, picking up where narratives C, B and A had left off, in reverse order? The problems would be knotty, but knottiness encourages original escapology."

 So another kind of a literary novel is one that has a close relationship with classic literary works – and that expects readers who can recognise the kinship. Thus the opening sentence of Zadie Smith's On Beauty:

One may as well begin with Jerome's e-mails to his father.

 You are supposed to recognise, of course, the mock-insouciant opening sentence of EM Forster's Howards End:

 One may as well begin with Helen's letters to her sister.

 But it is not just an opening sentence. The reader of On Beauty who has read Howards End will get things that other readers miss, for Smith imitates and wryly updates the main episodes of her original. It is all, we might say, a joke on her own characters, who think that they are making their own story but are comically doomed to imitate the story of others. Waters, another novelist who has studied literature as well as read it, offers delicious surprises to those who have done likewise. Fingersmith, her version of a Victorian sensation novel, has a plot twist that is all the more stunning if you think you have been cleverly recognising the plot of  Wilkie Collins's The Woman in White.

 Alan Hollinghurst's 2004 Man Booker winner The Line of Beauty does not just feature a protagonist, Nick Guest, who is researching a PhD on style in Henry James: it also holds out special pleasures for readers who are themselves familiar with James's fiction. Hollinghurst became renowned for introducing a middlebrow readership to explicit gay sex, but the commercial success of his deeply literary narrative inventions is more significant. The minutely discriminating, free, indirect style of The Line of Beauty, following the patterns of Nick's own thoughts, is a kind of homage to the style of "the Master", with its small qualifications and quizzical inferences. If, like author and character, you relish James, you will relish the novel the better.

 There are more graduates from literature, especially English literature, degrees than ever. And there are all those book groups, dedicated to the critical discussion of novels. Literary fiction is not the invention of marketers; it answers the interests of readers. We have become so used to the thought that any higher literacy is in retreat before the forces of electronic media and consumer idiotism, that perhaps to imagine the opposite has become impossible. Perhaps the simple truth is that literary fiction has flourished because there are more literary readers than ever.

The panelists explain their choices

Sam Leith, journalist and novelist

•Ned Beauman's Boxer Beetle (Sceptre) is a piece of staggeringly energetic intellectual slapstick featuring collectors of Nazi memorabilia, gun-toting occultists with Welsh accents, vicious six-legged hardnuts, a country house murder mystery and a Raiders-of-the-Lost-Ark ending. It's crammed with strange, funny and interesting things.

•Adam Haslett's Union Atlantic (Tuskar Rock) is a big muscular American state-of-the-nation novel in the grand tradition: alive to the cross-currents of class and money, it's a deft psychological study of big-business hubris. Accomplished and gripping.

•Set in backwoods Australia, After the Fire, A Still Small Voice by Evie Wyld (Vintage) is intricately patterned and fastidiously written novel looks at what it means to be isolated, how men find a place in the world and how we struggle to avoid repeating our fathers' mistakes. It is rich, ambitious, and touched with sinister magic.

Helen Oyeyemi, novelist

•David Abbott's The Upright Piano Player (MacLehose Press) is not unlike a nocturne in its tone and mood; it is a melancholy and evocative treatment of a man's post-retirement crisis. Henry Cage is sketched with just enough subtlety, and allowed just enough sympathy – no more, no less – to make his failings devastatingly real.

•Rebecca Hunt's Mr Chartwell (Fig Tree), the original Black Dog, is just delightful in his blithe audacity – and that's how I think of her as a writer – she's got swagger. She took Churchill's term for his clinical depression – "the black dog", hyper-realised it and placed the result right in the centre of a story about grief.

•Anna Richards and Little Gods (Picador) – I warn you that from the first chapter you'll be swept up into a sort of struggling ball of poignant hilarity and will emerge hundreds of pages later laughing and saying "ouch", with an extraordinary giantess on your mind and in your heart.

Janet Lee, editor of The Culture Show

•A Kind of Intimacy by Jenn Ashworth (Bliss) is laugh-out-loud funny and completely compelling. The protagonist Annie is obese, unloved and deluded. In fact she misreads every situation she's in and from this disjuncture comes the comedy. It's a dark, dark tale, but a tale for today.

•Deborah Kay Davies's True Things About Me (Canongate) is a brutal story in brutal prose. The unnamed narrator works in a benefit office, a criminal walks in and literally claims her. Desire is portrayed here as a kind of breakdown – everything is wrecked in its pursuit.

•In The Wilderness by Samantha Harvey (Vintage) a man with Alzheimer's questions the very nature of self-hood. Beautifully written with recurring motifs it represents the disorientation of Alzheimer's better than any factual piece I've ever read.

Alex Clark, critic and broadcaster

•I was completely engaged by Stephen Kelman's Pigeon English (Bloomsbury), the story of Harrison, an 11-year-old Ghanaian boy who has come to live in London. One of the hardest things in fiction is to write from a child's point of view – Kelman does it brilliantly.

•The Breaking of Eggs by Jim Powell (Weidenfeld & Nicolson) piques the interest from the first page. Its narrator is a 61-year-old Pole who has lived in Paris for most of his life, devoting himself to updating a guidebook to the communist countries of Europe. The novel moves cleverly between the comic, the serious and the terribly painful.

•In The Tin-Kin by Eleanor Thom (Gerald Duckworth) a present-day discovery unlocks a previously hidden history – to create a vivid portrait of a Traveller community in the 1950s. Both the contemporary and the historical stories are compelling, and there's a very skilfully handled tension as the links between the two slowly emerge. She is also extremely talented at drawing big, memorable characters.

New Novelists: 12 of the Best, a Culture Show special, is on BBC2 at 9pm on 5 March and is part of Books on the BBC 2011. www.bbc.co.uk/tv/seasons/books.

©www.guardian.co.uk