Traditional musical-instrument classification methods mainly use regions in the time or/and frequency characteristics, cepstrum characteristics, and MPEG-7 characteristics, and they often lead to erroneous classification. Therefore, there is need to develop a more suitable method that is more applicable to the nonlinear characteristics of musical-instrument signals and can avoid the abovementioned problems. In this paper, a musical-instrument classification method that couples the optimized phase-space reconstruction (OPSR) with a flexible neural tree (FNT) is proposed. As per nonlinear dynamic theory, a principal component analysis and correlation coefficient are used to optimize the phase-space reconstruction (PSR) method. Multidimensional PSR results for different musical-instrument signals are extracted as the main components, and the dimensionality is reduced by the OPSR method. A probability density function (PDF) is introduced in the feature extraction step to differentiate musical instruments according to the phase-space-reconstructible characteristics. A FNT is adopted as a classifier to tackle the variability in musical-instrument signals and to improve the adaptive ability of various target classification problems. Experimental testing has been conducted to show that the proposed OPSR–PDF–FNT algorithm gives superior performance over other comparable algorithms and can classify 12 musical instruments with an accuracy of 98.2 %.