Open-set Home Image Understanding : Developed vision-language models to achieve open-set image classification , object detection, and semantic segmentation capacity. Our model can recognize both seen and unseen objects in images, enhancing flexibility and compatibility for various Zillow applications.Large-scale Indoor Dataset Collection : Designed and created a large-scale indoor semantic segmentation dataset. Developed an advanced annotation tool that integrates foundational vision models (e.g., Segment Anything). This tool significantly reduces mask annotation workload, improving annotation efficiency and accuracy.Research Works : As a intern supervisor, recruited and supervised 2 research interns on their projects. * Developed a large-scale indoor description dataset via GPT4 and computer vision models with human-in-the-loop supervision. Designed and trained a baseline model based on the dataset. This pioneering work enables Zillow's model to achieve home-level understanding and evaluation capacity. * Introduced an enhanced open-set object detection model. It balances task-specific detection performance while maintaining open-set capacity for handling unexpected input. This model enhances the robustness of Zillow’s product in real-world applications.

Home Feature Extraction: Developed AI models which explores 2D and 3D home data (e.g., Zillow Indoor Dataset) in both visual and language domains to extract additional home features and insights. The learned feature improves the performances for several down-stream Zillow applications including classification, retrieval, and recommendations.Research Works: As a intern supervisor, recruited and supervised 1 research intern. We proposed a domain adaptation-based computer vision model for the Home Layout Estimation task. This project enhances Zillow’s capacity to obtain home layout information more precisely and robustly.

Multi-modal Learning : 1) Led a team in collecting a large-scale multi-modal action dataset; 2) Proposed various multi-modal methods that fully explore latent correlations across modalities; 3) Developed generative strategies to address challenges of multi-modal scenarios.Transfer Learning: 1) Explored new training strategies that adapt large models to fit specific tasks with limited data; 2) Various modules are designed for different data types (e.g., images, depth, 3D point cloud, multi-modal) and different settings (e.g., co-training, self-supervised, generative, adversarial).Multi-label Learning : 1) Proposed methods which predict multiple labels from a single instance. Modules are designed for tackling challenges such as complex label correlations and long-tail label distributions in different applications (e.g., classification, annotation, and retrieval)

Computer Vision (EECE 5639): Introduced conventional and advanced computer vision algorithms including image processing, 3D reconstruction, deep learning, classification, detection, segmentation, etc.Unsupervised Machine Learning (DS 5230): Introduced traditional and SOTA unsupervised learning strategies such as clustering, dimension reduction, auto-encoder, deep learning-based, self-supervised learning, etc.Data Visualization (EECE 5642): Introduced diverse visualization strategies in various scenarios, including presentations, reports, and research papers. Tools such as MATLAB and Tableau are introduced in assignments.

Multi-modal Saliency Detection: Explored a novel framework for multi-modal (RGB-D) saliency detection, which effectively identifies significant objects in an image. A Knowledge-Distillation strategy is implemented to considerably reduce the network's complexity and enhance its inference efficiency, even on mobile platforms.

Proposed a reinforcement learning-based NLP model which predicts sentimental polarities of a given text. It disregards task-irrelevant text and instead prioritizes identifying the most effective clues. It considerably reduces the computational resource requirements.Developed a novel mechanism for learning graph data representations. Graph structured data retains valuable connectivity information among instances (e.g., social networks and advertising). The model allows for inductive and unsupervised learning in a highly efficient and effective manner.

Developed computer vision system with the capability to capture 3D containers, classify different container types, and accurately measure their dimensions/locations. The system is able to perform high-precision localization in high-level 3D sensor noise with low computational cost (e.g., embedded platform).

Deployed human/face detection and pose estimation algorithms in a warehouse environment. These algorithms effectively tackle challenges such as low illumination, occlusion, and various interruptions.