Increasing interest, enthusiasm of sport lovers, and economics involved offer high importance to sports video recording and analysis. Being crucial for decision making, ball detection and tracking in soccer has become a challenging research area. This paper presents a novel deep learning approach for 2D ball detection and tracking (DLBT) in soccer videos posing various challenges. A new 2-stage buffer median filtering background modelling is used for moving objects blob detection. A deep learning approach for classification of an image patch into three classes, i.e. ball, player, and background is initially proposed. Probabilistic bounding box overlapping technique is proposed further for robust ball track validation. Novel full and boundary grid concepts resume tracking in ball_track_lost and ball_out_of_frame situations. DLBT does not require human intervention to identify ball from the initial frames unlike the most published algorithms. DLBT yields extraordinary accurate and robust tracking results compared to the other contemporary 2D trackers even in presence of various challenges including very small ball size and fast movements.