Shadow detection is widely considered as segmentation problem that happens to be time consuming process with lots of duplication of results. The alternative to segmentation can be the applicability of single stage detectors like YOLO (You Only Look Once) that can run on the input image once and perform the detections during single pass. The study proposes a novel deep learning-based architecture namely, Rapid-YOLO which is an extended form of YOLOv3 architecture for detection of shadows. The proposed model is an extension of YOLOv3 architecture with addition of extra YOLO detection layers and convolution layers for aiding detection accuracy. The model exploits use of spatial pyramid pooling for scaling variations, a novel nonmaximum suppression using chaotic whale optimization (CW-NMS) for overpowering false bounding box detections. We further propose tangent loss function for confidence and classification loss to improve training time and reduce errors. Drop block regularization is also used to reduce over-fitting and optimize results for shadow detection. Further, lack of proper dataset for single shot detectors encouraged us to create a novel dataset for the use of YOLO based detectors to boost further work in this direction. The model has been verified by rich experiments and results prove that the proposed methodology is superior to the state-of-the-art methods in terms of objective quantization and in subjective vision.