A new nonlinear near-wall turbulence model is developed on the basis of realizability constraints to predict turbulent flow and heat transfer in strongly nonequilibrium flows. The linear k-epsilon-f(mu) model of Park and Sung (Fluid Dyn. Res., 20 (1997) 97) is extended to a nonlinear formulation. The stress-strain relationship is derived from the Cayley-Hamilton theorem in a homogeneous flow. The ratio of production to dissipation (P-k/epsilon) is employed to solve an algebraic equation of the strain dependent coefficients. A near-wall treatment is dealt with by reproducing the model coefficients from a modified strain variable. An improved explicit heat flux model is proposed with the aid of Cayley-Hamilton theorem, which includes the quadratic effects of flow deformations. The near-wall asymptotic behavior is incorporated by modifying the f(lambda) function. Emphasis is placed on the model performance on the, truncated strain terms. The model performance is shown to be generally satisfactory. (C) 2002 Elsevier Science Inc. All rights reserved.