The contradiction between the restriction of grating manufacturing technology and high-resolution measurement requirements has been the focus of attention. The precision requirement of angle calculation during the digital subdivision processing of a Moiré signal is focused on, the causes of errors in the solution of arcsine function are analysed, and an improved coordinate rotation digital computer (CORDIC)with double-rotation iteration is proposed by discussing the principle of the conventional CORDIC in detail herein. Because the iterative number and data width of the improved CORDIC are limited by the finite digital circuit resources and thus determine the calculation accuracy directly, subsequently the overall quantization error (OQE) of the improved CORDIC is analysed. The approximate error and rounding error of the algorithm are deduced, and the error models of iterative number and data width are established. The validity and application value of the improved CORDIC are proved through simulations and experiments involving a subdividing circuit. The corresponding relation between the approximate error, rounding error and iteration number, as well as the bit width are proved by quantization. The error of subdivision with the improved CORDIC, obtained through a calibration experiment, is within ±0.5′′ and the mean variance is 0.2′′. The results of the research can be applied directly to a digital subdivision system to guide the parameter setting in the iterative process, which is of crucial importance in the quantitative analysis of error separation and error synthesis.

In order to guarantee the accuracy of turntable angle measurement, a real-time compensation method for turntable positioning precision based on harmonic analysis is proposed in this paper. Firstly, the principle and feasibility of the real-time compensation method are analysed, and a detailed description of harmonic compensation is provided herein. Secondly, we analyse the relationships between the surface number of the polygon with the compensation order of the harmonic function and its corresponding compensation accuracy. The effects of the iterations number and the data width on calculation accuracy in the coordinate rotation digital computer (CORDIC) algorithm are analysed and the quantization models of the approximation error and rounding error of the CORDIC algorithm are established. Then, the calculation of the harmonic error function and real-time compensation processes are implemented on a field programmable gate array (FPGA) chip. The resource occupation and time delay of the phase angle calculation and the harmonic component calculation are discussed separately. Finally, the validity of the harmonic compensation method is proven through comparing the compensation effect with that of linear interpolation and the polynomial compensation method. The influences of the compensation order, the iterations number and the data width on the compensation results are demonstrated by simulation. A test platform with a laboratory-made FPGA circuit is built to evaluate the effect of real-time compensation with the harmonic function and the positioning error compensation can be performed within 760 ns. The results confirmed the effectiveness of the harmonic compensation method, revealing an improvement of the positioning precision from 54.21″ to 1.63″, equivalent to 96.99% reduction in positioning error.