In literary circles, there's much talk of the "new Yiddishists" movement. This includes writers such as Michael Chabon, Ayelet Waldeman, Nicole Krauss, Jonathan Safran Foer, Lev Grossman, and Dara Horn. All of these writers bring somethings of their Jewish identity to the page: Jewish characters, the Yiddish and Hebrew languages, allusions to holidays, midrashim, shared history, and the like. These writers are quite gifted...I'd highly recommend Krauss's The History of Love and Horn's In the Image for those who want to see spiritual introspection balanced with imaginative storytelling. However, their books aren't "Orthodox".
Many if not most of the novels published by Orthodox publishers were written originally as serials in magazines or by those used to writing for magazines. This is not bad--in fact I enjoy many of these books--it's just a very different style of storytelling. Writing a serial for the first time myself right now, I can see that as fun as they are to read and to write, they are different. You have deadlines and word counts hanging over your head. You have publishers and readers who will be furious if you don't complete the project. Readers want to see a lot going on in each episode, yet be able to keep the plot in their head from week to week. There are mashgichim at both the religious magazines and religious publishing houses who must be satisfied. For them, the message is considered at least as important as the form, if not more.
In the secular world, a novelist has different requirements. You need space for character development. Time to ponder and reconsider and revise. You might want to capture a wider range of emotions and topics than necessarily acceptable to a mashgiach. You might want to venture into experimental structure. You might want an audience beyond the religious community. The artful novelist does not necessarily thrive under the conditions usually found in the frum publishing world.
There are exceptions. Here's one: currently, HaModia has an amazing serial called This is America. I've been following it for a year and am praying that it will end up in a novel form at its conclusion because I want to recommend it to all my friends. It's THAT good. Also, Sarah Shapiro's writing--though not fiction--shows a flair and precision of language that is rare in even the secular world. It often reads like a novel even when it is non-fiction.
In the past few years, the climate for Orthodox literature has changed. Interestingly, it seems to be occurring when secular publishers put out Orthodox books, partly I think because of the success of the likes of Chabon, Horn, et al.
For a couple of decades, Rochelle Krich has been a trailblazer in this department. Her mysteries are particularly well-written and substantive. I'd describe the pacing and plotting of her early books as the stuff of bestsellers, not the "literary novel." However, her more recent Molly Blume novels have become increasingly literary.
Krich's successors are finding their works in the bookstores and libraries across America now. Risa Miller has put out two prize-winning books: Welcome to Heavenly Heights and My Before and After Life. (I particularly enjoyed the later.) Ruchama King Feuerman's Seven Blessings has invited comparisons to Jane Austen. She autopsies the shidduch culture of BTs in Jerusalem, yet does so with humanity, not scorn. Though not a novelist, the award-winning poet Yehoshua November also demonstrates that it is possible to be sophisticated in form and substance and frum.
I was extremely hopeful when I saw the website for The Writer's Cafe, an Orthodox literary magazine. Perhaps this would be a format for Jewish writers to print more material "outside the box." However, it appears that the project is at least temporarily suspended. I was disappointed at the news and hope it comes back. The new Ami magazine (disclaimer, my new serial appears in their "tween" supplement) also aspires to a different style of writing.
I'm hoping that readers will buy into this new model, because I think the scarcity of reading material that is "kosher" in the market right now drives more avid readers to read secular material that contains inappropriate language and ideas. I also see that improving the style of Jewish literature and its accessibility brings with it the opportunity for Jews to be a light unto the nations. Books like Seven Blessings and My Before and After Life bring healthy hashkafa into the lives of non-Jews as well as Jews who might not pick up a more stereotypical "frum book".