Spectrographs nominally contain a degree of quasistatic optical aberrations resulting from the quality of manufactured component surfaces, imperfect alignment, design residuals, thermal effects, and other other associated phenomena involved in the design and construction process. Aberrations that change over time can mimic the line centroid motion of a Doppler shift, introducing radial velocity (RV) uncertainty that increases time-series variability. Even when instrument drifts are tracked using a precise wavelength calibration source, the barycentric motion of the Earth leads to a wavelength shift of stellar light, which causes a translation of the spectrum across the focal plane array by many pixels. The wavelength shift allows absorption lines to experience different optical propagation paths and aberrations over observing epochs. We use physical optics propagation simulations to study the impact of aberrations on precise Doppler measurements made by diffraction-limited, high-resolution spectrographs. Using the optical model of the iLocater spectrograph, we quantify the uncertainties that cross-correlation techniques introduce in the presence of aberrations and barycentric RV shifts. We find that aberrations that shift the point-spread-function photocenter in the dispersion direction, in particular primary horizontal coma and trefoil, are the most concerning. To maintain aberration-induced RV errors <10  cm  /  s, phase errors for these particular aberrations must be held well below 0.05 waves at the instrument operating wavelength. Our simulations further show that wavelength calibration only partially compensates for instrumental drifts, owing to a behavioral difference between how cross-correlation techniques handle aberrations between starlight versus calibration light. Identifying subtle physical effects that influence RV errors will help to ensure that diffraction-limited planet-finding spectrographs are able to reach their full scientific potential.