There’s never been a better time in history to be the kind of person who enjoys not eating things. In my corner of New York City – as in most major cities, these days – your dining options aren’t limited to not eating meat or not eating dairy products. There are restaurants you can visit to not eat gluten or wheat; to not eat anything our prehistoric ancestors wouldn’t have eaten; or to not eat food that has been cooked. And you can wash it all down with a milky coffee containing neither milk nor much in the way of coffee (a decaf skinny soy latte, known sarcastically among baristas as the Why Bother?). It’s a strange kind of self-indulgence, this conspicuous non-consumption. Yet the non-diners of contemporary New York stand in a proud contemporary tradition. It includes Immanuel Kant, who barely ate anything for breakfast, and who once wrote that the urge to eat dinner, if you’d already had lunch, “can be considered a pathological feeling”. I harbour the fantasy of one day opening a chain of upscale Kant Diet restaurants, open for dinner only, serving nothing. Oh, and with a special, high-priced tasting menu, for those who can afford to deny themselves a truly extravagant range of dishes.

It’s easy, of course, to disdain the absurdities and exploitation of the diet industry: the ridiculous menu items, the miles of profit-motivated books on the paleo, Dukan, Atkins, and South Beach diets, or the 5:2 plan for intermittent fasting – and the desperation with which we fling ourselves at them, over and over, in the face of bitter experience. Yet perhaps no other corner of modern culture is the crucible for so many of the ways in which we struggle to transcend our human limitations. For one thing, the person embarking on a new diet longs to overcome the power of temptation once and for all: to live like Ulysses, strapped to the mast of his ship in an effort to resist the Sirens, but to do it entirely through force of will. Furthermore, the dieter is seduced by the dream of pure self-creation: the hope that we can become who we choose to be, unhindered by previous bad decisions or circumstances beyond our control. Nobody seeking fame as a diet guru is going to tell you your tendency to chubbiness might be fixed in your genes.

This, surely, is what explains the powerful allure of each new nutritional regime, even though the basic facts of healthy eating are generally obvious and undisputed. True, there are debates about the advantages and disadvantages of red meat, grains, and dairy, among others. But if you’re a normally unhealthy 21st-century Westerner, your dietary to-do list is simple: you need to eat many more vegetables, plenty of fruit, and far fewer processed foods, especially those with added sugar. But we’re not primarily consulting diet books and experts for new nutritional knowledge. We’re seeking something more profound: therapeutic wisdom that might enable us finally, somehow, to square the circle of no longer feeling guilty about what we eat – while not forgoing pleasure (“Eat the foods you love and still lose weight!”), improving our physical attractiveness, and boosting our mood.

Behind it all, not very well hidden, is the ubiquitous desire to transcend death – or at least to extend life sufficiently that we needn’t think too hard about death for a while longer. No wonder the more extreme varieties of diet begin to resemble the ascetic practices of ancient mystics, intent on achieving spiritual enlightenment through denial of the physical body. We may talk about merely getting ready for beach season, but we too are seeking transcendence. “All religions, nearly all philosophies, and even a part of science,” wrote the biochemist Jacques Monod, “testify to the unwearying, heroic effort of mankind desperately denying its contingency.” And the philosophies of clean eating, low-carb, or macrobiotics are no exception.

The trouble is that diets don’t simply express the human yearning to transcend limitation; they also confront us, time and again, with the impossibility of doing so. To start with, there are physiological reasons why people who successfully lose the poundage find themselves in a worse position to remain at a stable weight than those who hadn’t lost any. But the more frustrating traps are mental. Diets, as the neuroscientist Michael Graziano has pithily expressed it, “cause the psychological struggle that causes weight gain”. You stick to your diet all week – and then reward yourself with a weekend binge so unhealthy it reverses the gains. Or you don’t stick perfectly to your diet – and feel so bad about it that you soothe yourself with cheeseburgers and ice-cream, making things worse than if you’d set less demanding rules to begin with. In one dispiriting study, overweight people who were exposed to “weight-stigmatising messages” – about the importance of staying slim – were more likely to consume unhealthy confectionery afterwards, in an effort to allay their feelings of anxiety. The more general, and annoyingly ironic, point is that dieting entails a relentless focus on what you’re eating; yet that means food becomes more salient, hunger pangs more noticeable, and mental images of prohibited treats more alluring. The quest for a healthy diet begins to resemble the quest for happiness: the more vigorously you pursue the goal, the more it eludes your grasp.

All of which might seem to suggest a counterintuitive solution: focus on something else instead. When my son was born, late last year, I gave myself permission to temporarily abandon my endless, unsuccessful efforts to eat more virtuously. For a few months, I figured – with apologies for the pun – I’d have enough on my plate. If burgers and potato chips were what got me through the sleep deprivation and new-parent panic, so be it. And sure enough, for a few weeks, I ate appallingly. But then something shifted. I got fed up with eating rubbish. I turned to oatmeal for breakfast, and grilled salmon with steamed vegetables for dinner, because I wanted to, not because of a regimen I’d previously vowed to follow. My earlier rule – no more than one burger every two weeks – had left me in the same situation as the rats in research studies, provided with and then deprived of unhealthy food: it’s only when the food is withdrawn that it acquires addictive qualities. By contrast, when I could eat all the burgers I wanted, it turned out I didn’t want many.

I suspect that the real problem here – deep down, beneath all those layers of melted cheese and discarded potato-chip packets – is that we desperately seek a sense of autonomy and control in a world that doesn’t offer much of it. This is the urge that leads us to embark on diets in the first place, resolving to assert our willpower over our mysterious and disobedient bodies. But it’s also the urge that causes those diets to fail: the newly-imposed rules begin themselves to feel imprisoning, and so we rebel against them, if only to prove that we won’t be ordered about in such a fashion. We resist being dictated to – even when the dictator is us.

What to do? It won’t work to emulate my early parenthood strategy, by consciously abandoning self-discipline: if you’re doing that with the intent of eating more healthily, then by giving up dieting, you’re really only dieting in a sneakier way, and you’ll be subject to all the same pitfalls. No, the real, unglamorous answer – that staple of ethical philosophy, echoing all the way back to its founding – is surely to pursue the path of moderation. The most charming aspect of the legend of the Buddha is that he finally achieved enlightenment not through extreme self-denial, but when he finally gave up extreme self-denial, and accepted a bowl of rice pudding from a local girl. There’s a lesson here – about the futility of idealised extremes, and the wisdom of acknowledging reality as it is, including our bodily urges. The lesson is not, just to be clear, that you should eat rice pudding if you want to achieve spiritual awakening. Though I assume someone’s already planning to publish The Buddha’s Enlightenment Diet: all rice pudding, all the time. If you see a copy alluringly displayed in the window of a bookshop, I recommend not succumbing to temptation.

From New Philosopher 'Food' - available from newsstands, bookstores, airports, or order online here (delivery from Australia).

Artwork by Genís Carreras