Power Sampling in LLMs

GRPO (Shao et al., 2024) has become a dominant RL method for improving reasoning in LLMs. However, it requires expensive training, hyperparameter tuning, and building ad-hoc reward functions, which often reduces sample diversity and generalization ability. A natural question is: Can we unlock the same reasoning performance without costly RL training?

Sub-Optimal Sampling

Our goal is to sample high-likelihood trajectories \(\mathrm{p(x_{0:T})}\). Since we don’t have look-ahead oracles, the likelihood is often approximated by autoregressive property:

Despite the simplicity, decoding locally may be only sub-optimal. Consider a simple example of a length-2 sentence with the token vocabulary of \(\{a, b\}\). The sequence probabilies become

where we know that \(\mathrm{p(x_0=a)=0.45, p(x_0=b)=0.55}.\) In this example, decoding the token \(\mathrm{b}\) looks like the optimal solution, which seems unreasonable since \(\mathrm{p(ab)}\) has the highest probability.

Power Sampling

To sample \(\mathrm{p(ab)}\), (Karan & Du, 2026) proposed a simple recipe by sampling the power distribution \(\mathrm{p^{\alpha}}\):

Take the above example, if we choose \(\mathrm{\alpha=2.0}\), we have

The sub-optimal decoding issue is easily solved by sampling from a sharpening distribution \(\mathrm{p^2}\). By contrast, the well-known low-temperature strategy fails in this case

The above example shows that \(\mathrm{p^{\alpha}}\) motivates future high-likelihood trajectories.

Metropolis-Hastings

For a general length, we first set a block size \(\mathrm{B}\), e.g. 192, to save computations and denote \(\mathrm{\pi_k(x_{0:kB})\propto p(x_{0:kB})^\alpha}\). To sample from \(\mathrm{\pi_{k+1}}\), (Karan & Du, 2026) proposed to first initialize a base sample \(\mathrm{x^{(0)}\rightarrow x}\):

where \(\mathrm{p_{\text{proposal}}}\) is a temperature-\(\mathrm{1/\alpha}\) sampler from the base model.

Next for a random \(\mathrm{m\in \{1, 2, \cdots, (k+1)B\}}\), we accept the new sample \(\mathrm{x'}\)

with a probability \(\mathrm{1\wedge A(x', x)}\) via the Metropolis-Hastings algorithm and repeat \(\mathrm{N_{\text{MCMC}}}\) times

Reasoning Performance on Math500

Empirically, power sampling significantly improves over the base model and achieves single-shot reasoning performance comparable to GRPO, without any post-training. Moreover, because it preserves sampling diversity, it delivers stronger Pass@k performance across multiple samples.

The following shows the impact of $\mathrm{\alpha, N_{\text{MCMC}}}$ where $\mathrm{\alpha, N_{\text{MCMC}>1}}$ significantly improves the performance.

Citation

@misc{deng2026_power,
  title   ={{Power Sampling in LLMs}},
  author  ={Wei Deng},
  journal ={waynedw.github.io},
  year    ={2026},
  howpublished = {\url{https://www.weideng.org/posts/power_samplers/}}
}