In this project, an efficient deep convolutional neural network (CNN) architecture is proposed based on depth wise temporal convolution along with a robust end-to-end scheme to automatically detect and classify arrhythmia from denoised electrocardiogram (ECG) signal, which is termed as `DeepArrNet’. Firstly, considering the variational pattern of wavelet denoised ECG data, a realistic augmentation scheme is designed that offers a reduction in class imbalance as well as increased data variations. A structural unit, namely PTP (Pontwise-Temporal-Pointwise Convolution) unit, is designed with its variants where depth wise temporal convolutions with varying kernel sizes are incorporated along with prior and post point wise convolution. Afterward, deep neural network architecture is constructed based on the proposed structural unit where series of such structural units are stacked together while increasing the kernel sizes for depth wise temporal convolutions in successive units along with the residual linkage between units through feature addition. Moreover, multiple depth wise temporal convolutions are introduced with varying kernel sizes in each structural unit to make the process more efficient while strided convolutions are utilized in the residual linkage between subsequent units to compensate the increased computational complexity. This architecture provides the opportunity to explore the temporal features in between convolutional layers more optimally from different perspectives utilizing diversified temporal kernels. Extensive experimentations are carried out on two publicly available datasets to validate the proposed scheme that results in outstanding performances in all traditional evaluation metrics outperforming other state of the art approaches. This project is implemented with MATLAB software.