Ablating surfaces with a pulsed laser system in milling processes often leads to surface changes depending on the milling depth. Especially if a constant surface roughness and evenness is essential to the process, structural degradation may advance until the process fails. The process investigated is the generation of precise thrust by laser ablation. Here, it is essential to predict or rather control the evolution of the surfaces roughness. Laser ablative milling with a short pulse laser system in vacuum (≈1 Pa) were performed over depths of several 10 µm documenting the evolution of surface roughness and unevenness with a white light interference microscope. Power spectral density analysis of the generated surface data reveals a strong influence of the crystalline structure of the solid. Furthermore, it was possible to demonstrate that this effect could be suppressed for gold.