Shishir K. Shah

Video is an increasingly important resource in higher education. A key limitation of lecture video is that it is fundamentally a sequential information stream. Quickly accessing the content aligned with specific learning objectives in a video recording of a classroom lecture is challenging. Recent research has enabled automatic reorganization of a lecture video into segments discussing different subtopics. This paper explores AI generation of visual and textual summaries of lecture video segments to improve navigation. A visual summary consists of a subset of images in the video segment that are considered the most unique and important by image analysis. A textual summary consists of a set of keywords selected from the screen text in the video segment by analyzing several factors including frequency, font size, time on screen, and existence in domain and language dictionaries. Evaluation was performed against keywords and summary images selected by human experts with the following results for the most relevant formulations. AI driven keyword selection yielded an F-1 score of 0.63 versus 0.26 for keywords sampled randomly from valid keyword candidates. AI driven visual summary yielded an F-1 score of 0.70 versus 0.59 for K-medoid clustering that is often employed for similar tasks. Surveys showed that 79% (72%) of the users agreed that a visual (textual) summary made a lecture video more useful. This framework is implemented in Videopoints, a real-world lecture video portal available to educational institutions.

Traditional approaches for Person Re-identification (Re-ID) rely heavily on modeling the appearance of persons. This measure is unreliable over longer durations due to the possibility for changes in clothing or biometric information. Furthermore, viewpoint changes significantly degrade the matching ability of these methods. In this paper, we propose "Contrastive Viewpoint-aware Shape Learning for Long-term Person Re-Identification" (CVSL) to address these challenges. Our method robustly extracts local and global texture-invariant human body shape cues from 2D pose using the Relational Shape Embedding branch, which consists of a pose estimator and a shape encoder built on a Graph Attention Network. To enhance the discriminability of the shape and appearance of identities under viewpoint variations, we propose Contrastive Viewpoint-aware Losses (CVL). CVL leverages contrastive learning to simultaneously minimize the intra-class gap under different viewpoints and maximize the inter-class gap under the same viewpoint. Extensive experiments demonstrate that our proposed framework outperforms state-of-the-art methods on long-term person Re-ID benchmarks.

Person Re-Identification (Re-ID) aims at matching a person captured by a non-overlapping camera system. Real-world Re-ID presents challenges like clothing changes and occlusions, which limits the applicability of traditional appearance-based methods. Cloth-Changing Re-ID (CCRe-ID) methods that rely on cloth-invariant modalities, such as shape, gait, etc., ignore occlusions and fail to mine the complementary relationship across modalities. Meanwhile, methods that explicitly focus on occlusion management struggle with cloth-changing scenarios. To address these, we propose ACML: Attention-based Cross-Modality Learning, the first framework to tackle both clothing changes and occlusion in Re-ID. Our lightweight framework comprises a unified network with cascaded Cross-Attention Blocks that extracts appearance and shape features collaboratively, enhancing robustness under clothing changes, viewpoint variations, and poor illumination conditions. Inputs to the network are produced by our novel occlusion synthesis module, which not only helps exposing the model to occlusions but also guides the model to adaptively attend to informative cues and reduce noise. Experiments demonstrate the effectiveness of ACML on both CCRe-ID and occluded Re-ID datasets.

Semantic segmentation involves labeling each pixel in an image with a corresponding class label, enabling detailed scene understanding. In dynamic environments, where conditions change over time, incremental learning techniques are essential for updating segmentation models with newly acquired data. However, incremental segmentation faces the challenge of catastrophic forgetting, where models lose previously learned knowledge when trained on new data distributions. To address this, we propose a recall-based knowledge distillation approach for stable segmentation model training across dynamic environments. Our method combines the strengths of knowledge distillation and recall learning to enhance the model’s ability to recall information from previous data distributions while adapting to new ones. By reintroducing a small portion of the previous dataset during training and applying tailored distillation techniques, our approach mitigates catastrophic forgetting and improves the robustness of these models. Through comprehensive evaluations, we demonstrate the effectiveness of our approach in two scenarios: salt segmentation in seismic datasets and tumor segmentation in MRI datasets. Our method offers a promising solution for addressing the challenges of catastrophic forgetting in incremental semantic segmentation, facilitating the development of more adaptive and reliable computer vision systems in dynamic environments.

In recent years, Person Re-Identification (Re-ID) has seen remarkable progress in addressing the issue of clothing changes. However, in real-world scenarios, Re-ID is often further challenged by occlusions, while very little research has been conducted to explicitly tackle these two challenges simultaneously. To this end, we propose a method for Occluded Cloth-Changing Person Re-ID (OCCRe-ID) termed “OASL: Occlusion-aware Appearance and Shape Learning”. OASL introduces a plug-and-play occlusion handling strategy which can be seamlessly integrated into existing Re-ID methods, enabling them to reason discriminative appearance and shape features under occlusions. Specifically, our approach leverages occlusion type information to achieve two key objectives for occlusion-awareness: (1) guide the backbone to focus on extracting identity-aware appearance features from non-occluded image regions and reason features from occluded ones, and (2) recover pose keypoints from occluded regions for mitigating occlusions in shape encoding. Additionally, we construct E-PRCC, the first dataset for OCCRe-ID, with the aim of facilitating further research in this practical domain. Extensive experiments conducted on E-PRCC, LTCC, Occluded-REID, DeepChange, and Market-1501 datasets demonstrate that OASL achieves state-of-the-art performance, offering a robust solution to the dual challenges of occlusions and clothing changes in Person Re-ID.