The last decade saw the end of cheap oil, the magic growth ingredient for the global economy after the second world war. This summer's increase in maize, wheat and soya bean prices – the third spike in the past five years – suggests the era of cheap food is also over.

Price increases in both oil and food provide textbook examples of market forces. Rapid expansion in the big emerging markets, especially China, has led to an increase in demand at a time when there have been supply constraints. For crude, these have included the war in Iraq, the embargo imposed on Iran, and the fact that some of the older fields are starting to run dry before new sources of crude are opened up.

The same demand dynamics affect food. It is not just that the world's population is rising by 1% a year. Nor is it simply that China has been growing at 9% a year on average; it is that consumers in the big developing countries have developed an appetite for higher protein western diets. Meat consumption is rising in China, India and Brazil, and since it takes 7kg of grain to produce 1kg of beef (and 4kg to produce 1kg of pork), this is adding to global demand.

Farmers have been getting more efficient, increasing the yields of land under production, but this has been offset by two negative factors: policies in the US and the EU that divert large amounts of corn for biofuels and poor harvests caused by the weather.

If the World Bank's projections are anything like accurate, further massive productivity gains from agriculture are going to be needed over the next two decades. There will be an extra 70m mouths to feed every year, which will result in a 50% increase in demand for food by 2030. Meanwhile, the amount of arable land per person will continue its long-run downward trend.

The extent of this challenge has been highlighted by the extreme drought in the US this year. Failure of the maize harvest – down by more than 100m tonnes on what was expected – has had a knock-on impact on wheat, which has not been affected by the lack of rain. Prices of both crops have jumped by $100 a tonne this summer. The latest data from the World Bank showed that food prices rose 10% between June and July and have now exceeded the previous peak in early 2011.

It will take time for these increases to have their full impact on consumers. In the short run, the cost of meat will not be affected because there is glut caused by livestock owners slaughtering their herds in order to save money on expensive feed. But by the end of the year, food will be dearer.

Central banks are unlikely to tighten policy in response to higher inflation, since the increase is seen as an external shock that will have a depressing effect on the spending power of consumers. They should not, however, assume that the spike will be a one-off, since grain stocks are at such low levels that bad harvests in 2013 would see rocketing prices, probably accompanied by panic-buying, export bans and food riots.

A recent report from Oxfam said the US should expect further severe droughts in the coming decades. "The US experienced $14bn-dollar disasters in 2011 – an historical record – including a blizzard, tornadoes, floods, a hurricane, a tropical storm, drought and heatwaves, and wildfires." The current year has already seen wildfires, a windstorm, heatwaves in much of the country and the most severe drought in half a century.

This seems to be an apt moment for the west to reassess the wisdom of biofuels. The US ethanol distilleries used 120m tonnes of maize in 2011 and there have already been calls from the United Nations Food and Agriculture Organisation for the reduced maize crop to be used for human food. There is also growing political opposition in the US to the country's Renewable Fuel Standard, which mandates 15.2bn gallons of biofuels for 2012, of which 13.4bn gallons can come from corn-based ethanol. Unsurprisingly, livestock and poultry producers have been at the forefront of calls for the mandate to be suspended.

The whole point about biofuels was that they were supposed to be a cost-free and a pain-free way for developed nations to show that they were responding to climate change. Rising crop yields meant there would be enough grain left over each year to turn into ethanol, and this would mean western consumers could do their bit to save the planet without in any way compromising their living patterns. That now looks like a highly questionable assumption.

So what happens next? A lasting solution to the food question will require either action on the demand side, action on the supply, or more likely both. The two obvious ways of limiting demand are to check population growth or to change dietary habits so that meat consumption is reduced. Neither is going to be easy to achieve.

On the supply side, the short-term response should be to find alternatives to biofuels. Longer term, the hope is that the pressures of a rising population coupled with the incentives provided by rising food prices lead to a second green revolution that will dramatically increase yields in those parts of the world – such as sub-Saharan Africa – where they are currently low. One of the few beneficial impacts of high commodity prices is that farmers will be able to afford to buy fertilisers for their land.

A recent study by Fidelity looked at some of the other recent developments to boost food supply, including precision agriculture – the use of high technology to apply the optimum amounts of seed, water and fertiliser for maximum efficiency – and a wider use of biotechnology and GM crops. The report also highlighted what is known as "fast food" from animal cells, a process by which scientists "create artificial meat by delivering an electric charge to the animal muscle cells in a mixture of amino acids, which causes the cells to multiply".

Given the predicted growth in consumption in developing countries, Fidelity says this could become an "environmentally acceptable option" as traditional meat becomes more expensive. Whether this approach is "environmentally acceptable" remains to be seen. The Fidelity report does, however, clarify one point, namely that hard choices have to be made.

The current assumption seems to be that the world can have a rising population, ever-higher per capita meat consumption, devote less land to food production to help hit climate change targets and eschew the advances in science that might increase yields. This is the stuff of fantasy. It is possible to have more intensive farming using the full range of technological breakthroughs in order to feed a bigger, meat-hungry population. Or it is possible to have lower yields from a more organic approach to feed a smaller population eating less meat. But not both.