Decisions involving comparisons of Arabic number digits often exhibit an interference between the physical size of the digit and the implied numerical magnitude, a phenomenon called the size-congruity effect. Related research over the past four decades has yielded two competing models of the phenomenon: an early interaction account, where interference between numerical and physical magnitude occurs at an early encoding stage, and a late interaction account, where the interference occurs downstream as response competition during the decision process. In the present study, we asked participants to compare the physical sizes of pairs of Arabic digits. We fit the resulting response time distributions with a shifted Wald model, a single boundary accumulator model, which gave us estimates of information accumulation rate (drift rate), response threshold, and nondecision time. We found that incongruity between physical size and numerical magnitude affected the decision-related estimates of drift rate and response threshold. Further, a Bayesian analysis confirmed a null effect of congruity on nondecision time. These results indicate that the observed interference originates from decision-related processes, lending further support for a late interaction account of the size-congruity effect.