Retirement systems that manage money for firefighters, police officers, teachers and other public workers aren’t pulling back on costly bets at a time when markets are rising around the world.

Some public pension funds are adding to traditional allocations of stocks and bonds while both are expensive. Others are loading up on more private-equity or real-estate holdings that are less liquid and sometimes carry high fees.

How much risk to take is a question facing all investors as they enter 2018. « Everything is overvalued,” said Wilshire Consulting President Andrew Junkin, who advises public pension funds. “There’s no magic option out there.”

In the public pension world, the willingness to chase expensive assets is the product of the core challenge most funds face—how to fulfill their mounting obligations to workers and retirees.

Decades of low government contributions, overly optimistic assumptions, overpromises on benefits and two recessions have left them with deep funding holes at a time when retirees are accelerating cash outflows. Estimates of their current combined funding shortfall vary from $1.6 trillion to $4 trillion.

The goal of most pension funds is to pay for future benefits by earning 7% to 8% a year. After the 2008 financial crisis, many funds tried to hit those marks by lowering their holdings of bonds as interest rates dropped, and by turning to real estate, commodities, hedge funds and private-equity holdings.

These so-called alternative investments rose to 26% of holdings at about 150 of the biggest U.S. funds in 2016, according to the Public Plans database, compared with 7% more than a decade earlier.

That strategy has its own risks. Venturing deeper into alternative investments diversifies holdings but can also add fees, complexity and the challenge of selling illiquid assets quickly if the fund needs cash. Returns from these investments can be volatile. Only private equity outperformed stocks between 2010 and 2016, according to a study by the Center for Retirement Research at Boston College, while hedge-funds returns barely exceeded 1% and commodities lost money.

Despite the recent embrace of alternative investments, most public pensions are still heavily linked to the ebbs and flows of global markets. Public pension funds had a median 56.69% of their holdings in equities as of September 30, according to Wilshire Trust Universe Comparison Service, as compared with 54.37% a year earlier. They also had 23.3% in bonds, down from 25% a year earlier.

The stance paid off during 2017’s market rally, as public pensions had one of their best years of the past decade. They earned 12.4% in the 2017 fiscal year ended June 30, according to Wilshire Trust Universe Comparison Service. It was their best annual result since 2014.

But some in the pension world are predicting it won’t last. Over the next decade Wilshire Consulting is predicting a 6.25% compound return for U.S. equities and 3.5% return for core bonds. International equities have a projected compounded return of 6.45%. Most pensions’ return targets remain at 7% to 8%.

Some pensions are trying to confront the challenge by changing the make-up of their investments more frequently. The board of the New York City Employees’ Retirement System has asked investment staff to re-examine the fund’s asset allocation less than two years after the previous lineup was approved, said Mike Haddad, deputy chief investment officer for public markets.

The effects of high stock valuations, a U.S. tax overhaul, the Federal Reserve’s move away from quantitative easing and a rising rate environment are all new points of discussion. “It’s a unique set of market conditions that’s very challenging for a long-term pension,” Mr. Haddad said.

The California State Teachers Retirement System in 2017 shifted $10 billion into a mix of bonds and alternatives it said was designed to hedge risk. That money was divided into 39% long-dated Treasurys and 61% hedge fund-type investments. The second-largest public pension in the U.S. by assets, it rolled back its fixed-income target to 13% from 15% and shaved its equities target to 54% from 55% effective Jan. 1.

“The truth will be told in the next bear market on which strategy provides better diversification,” said Calstrs Chief Investment Officer Christopher Ailman.

One major Canadian pension fund is planning a bigger bet on illiquid assets. The $202 billion Canadian pension fund Caisse de dépôt et placement du Québec plans to move money into investments such as real estate, private equity, infrastructure and corporate credit, said President and Chief Executive Officer Michael Sabia, of CDPQ.

“Today, liquid assets—traditional government bonds and public equities—account for the majority of our investments,” Mr. Sabia said in a statement. “A few years down the road, this will no longer be the case.”

The biggest public pension in the U.S., known by its abbreviation Calpers, has been backing away from alternative investments as a way of reducing complexity and fees. In December, directors considered a 34% allocation to equities, which would have involved selling more than $50 billion in stocks based on its holdings at end of October. They also considered a higher allocation that would have meant buying about $30 billion. With their bond holdings, they thought about doubling them as well as holding steady.

In the end, the fund opted to raise its equities target to 50% from 46% as of July 1 and its fixed income target to 28% from 20%. It had 49.8% of its $341.5 billion portfolio in equities as of Oct. 31, according to the fund’s website.

No matter which move Calpers made, it faced challenges. Scaling back Calpers’ equity investment would have reduced the fund’s projected 7% return at a time when the fund has just 68% of the assets needed to pay for future benefits. That would have meant higher contribution costs for local governments across California.

But increasing its allocation to stocks is also risky. “This may not be the most opportune time to take on additional equity risk,” investment manager Dianne Sandoval said at a December board meeting.

—Tom McGinty contributed to this article.