Nonlinear effects in optical fibers impose a capacity limit for optical communication systems. In this dissertation, the nonlinear Fourier transform (NFT) is investigated as a method to mitigate and compensate for those effects. This study consists of two parts: first a computational complexity analysis for the use of the NFT for nonlinear compensation in the normal dispersion regime, and second, an analysis of the robustness of the performance of the discrete spectrum modulation in the anomalous dispersion regime using the NFT. The first part investigates the computational complexity of the NFT based on the Zakharov-Shabat scattering problem as a nonlinear compensation technique for quadrature-phase-shift keyed (QPSK) signals with raised cosine frequency characteristic in optical fiber transmission systems with normal dispersion fibers. Results show that there are two primary sources of computational errors that arise from the use of the NFT: The computational error due to the finite eigenvalue resolution of the reflection spectra and the computational error due to the Born approximation used in the inverse NFT. In this scenario, computational costs become unacceptably large at data frame lengths and powers that are too small for this approach to be competitive with standard transmission methods. The second part investigates the robustness of a recently proposed nonlinear frequency-division multiplexing (NFDM) system comprised of two independent QPSK channels modulated in the discrete spectrum associated with two distinct eigenvalues. Among the many fiber impairments that may limit this system, we focus on determining the limits given by third-order dispersion, the Raman effect, amplified spontaneous emission (ASE) noise from erbium-doped fiber amplifiers (EDFAs), and lumped gain from EDFAs. Each of these impairments impact this system with discrete spectrum modulation and 1600 km of propagation distance in different ways: Third-order dispersion limits the maximum launch power to 13 dBm, the Raman effect limits the maximum launch power to 10.25 dBm, the ASE noise limits the maximum launch power to 9 dBm, while lumped gain limits the maximum launch power at 3.75 dBm. Additional studies are needed to investigate the effectiveness of the NFT for discrete spectrum modulation formats with three or more eigenvalues.