Bridges are widely used in human life. Understanding structural performance, assessing structural conditions, and providing in-time decision are crucial components in structural health monitoring (SHM), to avoid catastrophic events and improve public safety. However, traditional SHM needs traffic closure, extensive sensor deployment, and in-contact measurements. The main purpose of this thesis is to develop a vision-based sensor of high accuracy that does not need artificial targets. When the vibration of the UAV itself is removed, the UAV is a convenient method to record video of the vibrations. Based on the recorded images and vibration data, a new deep learning method is developed and used to analyze vibrations of the structure and detect damage locations and conditions automatically. In the thesis, a non-contact vision sensor system for monitoring structural displacements with an advanced Zernike subpixel edge detection technique is first suggested. A new method to filter the effect of camera motions through background templates is proposed in the study. Several experiments on the MTS machine were performed with different frequencies and amplitudes to verify the method. The results show that filtering of vibrations of the camera significantly improves the displacement monitoring accuracy from 53.0% to 97.0%. Three translations and three rotations of the unmanned aerial vehicle (UAV) were derived through the suggested fast Normalized Cross Correlation (NCC) based template matching method, and their effect on the monitored structural displacement is analyzed. To verify the concept, a series of lab and field experiments were performed. Excellent precision and consistency were obtained for the UAV monitored displacement, the MTS piston motion, and the fixed camera derived displacement. Further in the thesis, a novel deep learning-based structural health monitoring method was developed, which could detect damages using both defects and vibration data. Two ABAQUS models on a beam and an ABAQUS model on a truss were conducted to test if the proposed CNN model could detect damages successfully. Seven transfer learning methods were compared on detecting crack images. From the outputs of the deep learning models, it is apparent that the AlexNet CNN model with defect images shows higher accuracy in estimating damage status.