Computer vision (CV)-based approaches have gained a lot of attention in recent years for objective identification of damages both at structural and material scales. In this dissertation, the metallurgical phases and the two important modes of damage in structural steel, namely fracture and corrosion, are considered. Use of CV techniques for metallurgical phase identification and fracture type identification in steel microstructure is minimal and rely on pixel intensity information. When distinct phases or fracture types possess similar pixel intensities, predictions may be erroneous. In this dissertation, various texture recognition algorithms based on an ensemble of machine learning algorithms are proposed to identify the distinct metallurgical phases and fracture types in structural steels. The existing CV-based corrosion detection techniques are efficient for the images acquired under natural daylight illumination and ignore the inherent variations in ambient lighting conditions. Further, corrosion-like hues such as bricks, surrounding vegetation, etc., present in the images yields corrosion misclassification. Furthermore, there are currently no techniques available to identify the source of corrosion (HCl, NaCl, and Na2SO4). In this dissertation, various color spaces are employed in conjunction with neural networks to identify the corrosion in real-world scenarios such as varying natural daylight illuminations, shadows, water wetting, and oil wetting. For eliminating the visual ambiguity and identifying the source of corrosion, the visible and near-infrared (VNIR) spectra are extracted to train support vector machines. Deep neural networks (DNN’s) popularly used in the field of CV are often regarded as the black box models. Sensitivity analysis (SA) is a model-agnostic explainable artificial intelligence (XAI) approach commonly employed to explain the outcome of a mathematical model. SA quantifies the variation in the model's output to the change in the input of the model. In this dissertation, a novel sensitivity analysis referred to as Complex-Step Sensitivity Analysis is developed for interpreting the DNN’s prediction. Numerical experiments are performed to demonstrate the efficacy of the proposed method in evaluating the derivatives of DNN predictions and identifying the important features. Using this newly developed method, the key wavelengths in the VNIR spectra contributing to the prediction of corrosion source corrosion are identified.