Machine Learning Intern
Hackensack, Nj
Developed a semantic search platform designed for efficient querying of legal documents. The solution had two core components:• Query-Based Nearest Neighbor Search: Enabled rapid document retrieval.• Reranking & Summarization Model: Utilized Language Models for improved search accuracy.Technologies Used:▪️ Employed a streamlined tech stack: pypdfium2, OpenAI, numpy, and Streamlit.Achievements:🔸 Dataset Creation:• Assembled a dataset of 906 'ground-truth' questions, serving as a benchmark for evaluating embedding models. Initially 201 were manually created, so to expedite the process, over 1737 questions were generated with ChatGPT which were cut and edited to 705 questions. The criteria for the questions was context sufficiency, document relevance, and uniqueness.🔸 Model Benchmarking:• Conducted extensive evaluations on 5 different embedding models and 5 chunking methods, leading to 25 unique VectorStores.🔸 Tool Development:• Crafted a vectorized 'Top-N' function for pinpointing nearest document matches.• Devised an adaptable chunking and embedding tool, allowing configuration based on user-defined parameters.🔸 User Interface:• Built a polished, user-friendly web application using Streamlit as a showcase of the project's capabilities.🔸 Relevance Checks:• Utilized Langchain chat templates to refine ChatGPT responses, ensuring the inclusion of accurate source citation.🔸 Enhanced User Experience:• Augmented the web application to feature PDF sections that substantiated the language model's answers, thereby bolstering user trust and comprehension.