PRBS orders required to train ANN equalizer for PAM signal without overfitting

Abstract

Artificial neural network (ANN)-based nonlinear equalizers (NLEs) have been gaining popularity as powerful waveform equalizers for intensity-modulation (IM)/direct-detection (DD) systems. On the other hand, the M-ary pulse amplitude modulation (PAM-M) format is now widely used for high-speed IM/DD systems. For the training of ANN-NLE in PAM-M IM/DD systems, it is common to employ pseudorandom binary sequences (PRBSs) for the generation of PAM-M training sequences. However, when the PRBS orders used for training are not sufficiently high, the ANN-NLE might suffer from the overfitting problem, where the equalizer can estimate one or more of the constituent PRBSs from a part of PAM-M training sequence, and as a result, the trained ANN-NLE shows poor performance for new input sequences. In this paper, we provide a selection guideline of the PRBSs to train the ANN-NLE for PAM-M signals without experiencing the overfitting. For this purpose, we determine the minimum PRBS orders required to train the ANN-NLE for a given input size of the equalizer. Our theoretical analysis is confirmed through computer simulation. The selection guideline is applicable to training the ANN-NLE for the PAM-M signals, regardless of symbol coding. (C) 2022 Optica Publishing Group under the terms of the Optica Open Access Publishing Agreement