To address the problem that a deep neural network needs a sufficient number of training samples to have a good prediction performance, this paper firstly used the Z-Map algorithm to generate a simulated profile of the milling surface and construct an optical simulation model of surface imaging to supplement the training sample size of the neural network. Then the Deep CORAL model was used to match the textures of the simulated samples and the actual samples across domains to solve the problem that the simulated samples were not in the same domain as the actual milling samples. Experimental results have shown that high texture matching could be achieved between optical simulation images and actual images, laying the foundation for expanding the actual milled workpiece images with the simulation images. The deep convolutional neural model Xception was used to predict the classification of six classes of data sets with the inclusion of simulation images, and the accuracy was improved from 86.48% to 92.79% compared with the model without the inclusion of simulation images. The proposed method solves the problem of the need for a large number of samples for deep neural networks and lays the foundation for similar methods to predict surface roughness for different machining processes.