diff --git a/rag_lb/jupyter-book/images/rag_flow.png b/rag_lb/jupyter-book/images/rag_flow.png new file mode 100644 index 0000000000000000000000000000000000000000..ad34026eceba7c89489432da74d42c2301f1b7e9 Binary files /dev/null and b/rag_lb/jupyter-book/images/rag_flow.png differ diff --git "a/rag_lb/jupyter-book/notebooks/00_Einf\303\274hrung_RAG.ipynb" "b/rag_lb/jupyter-book/notebooks/00_Einf\303\274hrung_RAG.ipynb" index db1c23c7cb90be30d1ee72767453281a24654de5..baeee6e36eaac9c060dd79e632ce85a0285a9db6 100644 --- "a/rag_lb/jupyter-book/notebooks/00_Einf\303\274hrung_RAG.ipynb" +++ "b/rag_lb/jupyter-book/notebooks/00_Einf\303\274hrung_RAG.ipynb" @@ -10,14 +10,14 @@ "### Einleitung und Überblick\n", "\n", "#### Was erwartet Sie in diesem Jupyter Book?\n", - "Dieses Jupyter Book führt Sie in die Methode des Retrieval Augmented Generation (RAG) ein und zeigt deren Anwendung in der historischen Textanalyse. RAG kombiniert die Fähigkeiten von Sprachmodellen mit gezielter Informationssuche in spezifischen Dokumenten - in unserem Fall historischen Quellen.\n", + "Dieses Jupyter Book führt Sie in die Methode des Retrieval Augmented Generation (RAG) ein und zeigt deren Anwendung in der historischen Textanalyse. RAG kombiniert die Fähigkeiten von Sprachmodellen mit gezielter Informationssuche in spezifischen Dokumenten - in unserem Fall historischen Quellen. Dadurch können Einschränkungen generischer LLMs können umgangen werden, indem eine große Menge an Referenzdokumenten durch das System geleitet und auf der Grundlage der berechneten Relevanz für die Benutzerfrage abgerufen werden kann, die generierten Antworten diese abgerufenen Quellen verwenden und speziell hervorheben, welche Quellen verwendet wurden. \n", "\n", "#### Lernziele\n", "Nach diesem einführenden Notebook werden Sie:\n", "- Die Grundidee von RAG verstehen\n", "- Den Wert dieser Methode für die historische Forschung einschätzen können\n", "- Eine erste RAG-Pipeline in Aktion gesehen haben\n", - "- Die Möglichkeiten und Grenzen der Methode kennen\n", + "- Erste Möglichkeiten und Grenzen der Methode kennen\n", "\n", "#### Aufbau des Jupyter Books\n", "Das Book ist modular aufgebaut und umfasst:\n", @@ -33,7 +33,60 @@ " - Prompt-Entwicklung\n", " - Antwortgenerierung\n", "5. **Anwendungsfälle**\n", - " - Spezifische historische Analysen\n" + " - Spezifische historische Analysen\n", + "\n", + "## Voraussetzungen und benötigtes Vorwissen\n", + "\n", + "Dieses Jupyter Notebook richtet sich an Studierende und Forschende in den digitalen Geisteswissenschaften mit grundlegenden Kenntnissen in:\n", + "\n", + "### Technische Voraussetzungen\n", + "- **Python-Grundlagen**: Vertrautheit mit Variablen, Funktionen, Schleifen und der Arbeit mit Listen und Dictionaries\n", + "- **Jupyter Notebooks**: Grundlegende Kenntnis der Funktionsweise von Zellen und der Ausführung von Code\n", + "- **Pandas**: Elementare Erfahrung mit Datenrahmen (DataFrames) und einfachen Datenoperationen\n", + "\n", + "### Inhaltliche Voraussetzungen\n", + "- **Grundlegendes Verständnis historischer Quellenkritik**: Einordnung von Texten in ihren Entstehungskontext\n", + "- **Basiswissen zu Textanalyse**: Konzepte wie Tokenisierung, Vektorisierung, oder Ähnlichkeitsmaße müssen nicht im Detail bekannt sein, werden aber im Notebook erläutert\n", + "- **Interesse an KI-gestützten Methoden** für die Geisteswissenschaften\n", + "\n", + "Falls Sie mit einem der Konzepte nicht vertraut sind, bieten wir zu Beginn jedes Abschnitts kurze Einführungen und verweisen auf weiterführende Ressourcen. Das Notebook ist so konzipiert, dass es schrittweise vom Einfachen zum Komplexen führt." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Was ist RAG?\n", + "\n", + "Retrieval Augmented Generation (RAG) ist eine Methode, die es uns ermöglicht, große Textsammlungen gezielt zu durchsuchen und mit Hilfe von KI zu analysieren. Der Prozess besteht aus zwei Hauptschritten:\n", + "\n", + "1. **Retrieval**: Zuerst werden passende Informationen aus einer Datenquelle geholt. Diese Datenquelle kann eine Vektordatenbank sein, in der Dokumente (bzw. Ausschnitte von Dokumenten) mithilfe von Embeddings abgelegt sind. \n", + "\n", + "2. **Augmented Generation**: Die gefundenen (\"retrieved\") Informationen werden zusammen mit der Forschungsfrage an ein Large Language Model (LLM) übergeben. Das LLM nutzt diese Zusatzinformationen, um eine präzise und kontextbezogene Antwort zu generieren.\n", + "\n", + "### Glossar und wichtige Begriffe\n", + "\n", + "Für unsere Arbeit werden einige technische Begriffe wichtig sein. Hier sind die wichtigsten Konzepte, die wir verwenden werden:\n", + "\n", + "| Begriff | Kurzbeschreibung |\n", + "|---------|-----------------|\n", + "| Retrieval | Die Suche und Extraktion relevanter Informationen aus einer Datenbasis |\n", + "| Embedding | Die Umwandlung von Text in numerische Vektoren, die von Computern verarbeitet werden können |\n", + "| Vector Database | Eine spezielle Datenbank für die effiziente Speicherung und Suche von Embeddings |\n", + "| Chunking | Die Aufteilung langer Texte in kleinere, verarbeitbare Einheiten |\n", + "\n", + "**Übung**: Machen Sie sich während der durchführung des Jupyter-Books gedanken wie diese Begriffe ausgebaut werden können und erstellen Sie Ihre eigenen Definitionen dieser Begriffe, während wir sie in der Praxis kennenlernen.\n", + "\n", + "Weitere Ressourcen:\n", + "- [LangChain Dokumentation](https://python.langchain.com/docs/get_started/introduction)\n", + "- [Understanding Embeddings](https://www.sbert.net/)\n", + "- [Vector Database Grundlagen](https://www.pinecone.io/learn/vector-database/)\n", + "\n", + "### Systemübersicht\n", + "\n", + "Bevor wir in den Code einsteigen, schauen wir uns an, wie die einzelnen Komponenten zusammenarbeiten:\n", + "\n", + "" ] }, { @@ -43,7 +96,7 @@ "## Unser Quellenkorpus: Soldatenbriefe 1745-1872\n", "\n", "#### Beschreibung der Quelle\n", - "Das Korpus \"Soldatenbriefe\" ist eine Sammlung von 170 Briefen aus dem deutschsprachigen Raum, die mehrere wichtige historische Epochen umfasst:\n", + "Das [Korpus \"Soldatenbriefe\"](https://www.dwds.de/d/korpora/soldatenbriefe) ist eine Sammlung von 170 Briefen aus dem deutschsprachigen Raum, die mehrere wichtige historische Epochen umfasst:\n", "- Koalitions- und Befreiungskriege (1792−1815)\n", "- Deutscher Krieg (1866)\n", "- Deutsch-Französischer Krieg (1870/71)\n", @@ -63,7 +116,20 @@ "Die Briefe liegen in verschiedenen Formaten vor:\n", "- **TEI-XML**: Ursprungsformat mit detaillierten Metadaten\n", "- **CSV**: Aufbereitete Version für die Analyse\n", - "- **Plain Text**: Extrahierte Brieftexte" + "- **Plain Text**: Extrahierte Brieftexte\n", + "\n", + "### Daten Auswahl\n", + "Wir verwenden diesen Datensatz aus folgenden Gründen: \n", + "- Er ist vom Textumfang her vergleichsweise klein, so dass demonstrierbare Berechnungen in einer angemessenen Zeit durchgeführt werden können. \n", + "- Das TEI/XML-Format ist in der Geschichtswissenschaft üblich. \n", + "- Die Daten sind gut strukturiert, was es einfach macht, den Text und die Metadaten für unsere RAG-Pipeline zu organisieren." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Erste Erkundung der Daten" ] }, { @@ -75,13 +141,13 @@ "name": "stdout", "output_type": "stream", "text": [ - "Requirement already satisfied: pandas in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (2.2.3)Note: you may need to restart the kernel to use updated packages.\n", - "\n", + "Requirement already satisfied: pandas in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (2.2.3)\n", "Requirement already satisfied: numpy>=1.23.2 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from pandas) (1.26.4)\n", "Requirement already satisfied: python-dateutil>=2.8.2 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from pandas) (2.9.0.post0)\n", "Requirement already satisfied: pytz>=2020.1 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from pandas) (2024.2)\n", "Requirement already satisfied: tzdata>=2022.7 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from pandas) (2024.2)\n", - "Requirement already satisfied: six>=1.5 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from python-dateutil>=2.8.2->pandas) (1.17.0)\n" + "Requirement already satisfied: six>=1.5 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from python-dateutil>=2.8.2->pandas) (1.17.0)\n", + "Note: you may need to restart the kernel to use updated packages.\n" ] }, { @@ -100,28 +166,9 @@ "%pip install pandas" ] }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Erste Erkundung der Daten" - ] - }, { "cell_type": "code", - "execution_count": 2, - "metadata": {}, - "outputs": [], - "source": [ - "### Daten laden und erkunden\n", - "\n", - "import pandas as pd\n", - "from pathlib import Path" - ] - }, - { - "cell_type": "code", - "execution_count": 3, + "execution_count": null, "metadata": {}, "outputs": [ { @@ -208,6 +255,11 @@ } ], "source": [ + "# Daten laden und erkunden\n", + "\n", + "import pandas as pd\n", + "from pathlib import Path\n", + "\n", "# Laden der vorverarbeiteten Daten\n", "\n", "data_path = Path(\"../scripts/data/soldatenbriefe.csv\")\n", @@ -220,7 +272,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Die Daten sind bereits gut strukturiert und mit hilfreichen Metadaten erfasst. Wir bemerken aber das im Text noch Zeilenumbrüche durch \"\\n\" referenziert werden. Diese sollten wir mit einem einfachen Leerzeichen ersetzen." + "Die Daten sind bereits gut strukturiert und mit hilfreichen Metadaten erfasst. Wir bemerken aber das im Text noch Zeilenumbrüche durch \"\\n\" referenziert werden. Diese sollten wir mit einem einfachen Leerzeichen ersetzen. Dies ist ein Beispiel für den wichtigen Schritt des Preprocessing von Textdaten. " ] }, { @@ -375,7 +427,23 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "Nun haben wir eine erste quantitative Analyse unseres Datensatzes und können uns überlegen arum RAG für historische Textanalyse nützlich sein könnte gegenüber einer analogen Analyse und anderen Methoden wie zum Beispiel die welche durch [Voyant Tools](https://voyant-tools.org/) zur verfügung stehen?\n", + "### Zusammenfassung: Datengrundlage\n", + "\n", + "In diesem Abschnitt haben wir:\n", + "- Die Struktur und Umfang unseres Korpus kennengelernt (170 Briefe, 87.250 Wörter)\n", + "- Die zeitliche Verteilung der Briefe analysiert (1745-1872)\n", + "- Die soziale Struktur der Verfasser erkundet (Mannschaften, Unteroffiziere, Offiziere)\n", + "- Eine grundlegende Textbereinigung durchgeführt\n", + "\n", + "Diese Exploration bildet die Grundlage für die folgenden Schritte der RAG-Pipeline, da wir nun die Eigenschaften und Besonderheiten unserer historischen Quellen kennen und bei der Implementierung berücksichtigen können." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Jetzt haben wir eine erste quantitative Analyse unseres Datensatzes und können uns überlegen warum RAG für historische Textanalyse nützlich sein könnte gegenüber einer analogen Analyse und anderen Methoden wie zum Beispiel die welche durch [Voyant Tools](https://voyant-tools.org/) zur verfügung stehen?\n", + "-- This part has to be reviewed. I like how it flows into a discussion but it can be improved. Maybe by suggesting a exercise of inspecting the data with Voyant Tools?\n", "\n", "### Die Herausforderung historischer Textanalyse\n", "\n", @@ -396,9 +464,9 @@ "- Die gleichzeitige Berücksichtigung aller Briefe für eine Analyse ist kaum möglich\n", "- Das Auffinden spezifischer Themen oder Muster über alle Briefe hinweg ist zeitaufwändig\n", "\n", - "### Warum RAG die Lösung ist\n", + "### Warum RAG hier helfen kann\n", "\n", - "RAG (Retrieval Augmented Generation) bietet hier mehrere Vorteile:\n", + "RAG bietet hier mehrere Vorteile:\n", "\n", "1. **Effizientes Retrieval**\n", " - Automatische Identifikation relevanter Textstellen\n", @@ -420,25 +488,7 @@ " - Historische Kontexte können berücksichtigt werden\n", " - Komplexe Zusammenhänge werden erkannt\n", "\n", - "### Warum unser Korpus ideal für RAG ist\n", - "\n", - "Die Größe unseres Korpus ist aus mehreren Gründen ideal für RAG:\n", - "\n", - "1. **Chunking-Perspektive**\n", - " - Bei durchschnittlich 3.025 Zeichen pro Brief\n", - " - Und einer Chunk-Größe von 500 Zeichen\n", - " - Erhalten wir ca. 1.000-1.200 Chunks\n", - " - Dies ist optimal für präzises Retrieval\n", - "\n", - "2. **Kontextuelle Tiefe**\n", - " - Jeder Brief ist lang genug für bedeutungsvolle Analyse\n", - " - Kurz genug für präzise Chunk-Bildung\n", - " - Genug Material für Vergleiche und Muster\n", - "\n", - "3. **Metadaten-Struktur**\n", - " - Reiche Kontextinformationen (Rang, Datum, etc.)\n", - " - Ermöglicht vielfältige Analyseperspektiven\n", - " - Unterstützt historische Einordnung\n", + "-- It feels like I should be citing some literature here\n", "\n", "### Praktische Bedeutung\n", "\n", @@ -448,14 +498,16 @@ "- Entdeckung von Mustern und Zusammenhängen\n", "- Kombination quantitativer und qualitativer Analyse\n", "\n", - "Diese Vorteile machen RAG zu einem wertvollen Werkzeug für die historische Forschung, besonders bei der Analyse von Korrespondenzen und persönlichen Dokumenten." + "Diese Vorteile machen RAG zu einem wertvollen Werkzeug für die historische Forschung, zum Beispiel bei der Analyse von Korrespondenzen, Zeitungsartikeln und persönlichen Dokumenten." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ + "-- The following part has to be figured out.\n", "## Anforderungen an ein Korpus für RAG\n", + "\n", "- wann lohnt es sich, wann lohnt es sich nicht. \n", "- Welche Formate funktionieren?\n", "- Was muss man sich überlegen.\n", @@ -467,7 +519,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### Erste Schritte mit RAG\n", + "## Erste Schritte mit RAG\n", + "\n", + "Wir werden nun ein RAG-System mit unserem Datensatz aufbauen. Der Abschnitt dient dazu, dass Sie einen ersten Eindruck davon bekommen, was im System passiert. Wir werden unser Verständnis für die Schritte zum Aufbau eines Systems in den folgenden Notizbüchern entwickeln, einschließlich der Entscheidungen, die getroffen werden, und deren Auswirkungen auf die Analyse.\n", "\n", "#### Technische Voraussetzungen\n", "Für dieses Notebook benötigen wir einige Python-Bibliotheken:" @@ -475,160 +529,11 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": null, "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: langchain in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (0.3.19)\n", - "Requirement already satisfied: langchain-community in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (0.3.18)\n", - "Requirement already satisfied: langchain_openai in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (0.3.6)\n", - "Requirement already satisfied: sentence_transformers in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (3.4.1)\n", - "Requirement already satisfied: huggingface-hub in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (0.29.1)\n", - "Requirement already satisfied: chromadb in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (0.6.3)\n", - "Requirement already satisfied: langchain_huggingface in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (0.1.2)\n", - "Requirement already satisfied: langchain-core<1.0.0,>=0.3.35 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain) (0.3.37)\n", - "Requirement already satisfied: langchain-text-splitters<1.0.0,>=0.3.6 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain) (0.3.6)\n", - "Requirement already satisfied: langsmith<0.4,>=0.1.17 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain) (0.3.10)\n", - "Requirement already satisfied: pydantic<3.0.0,>=2.7.4 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain) (2.10.5)\n", - "Requirement already satisfied: SQLAlchemy<3,>=1.4 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain) (2.0.37)\n", - "Requirement already satisfied: requests<3,>=2 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain) (2.32.3)\n", - "Requirement already satisfied: PyYAML>=5.3 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain) (6.0.2)\n", - "Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain) (3.11.11)\n", - "Requirement already satisfied: tenacity!=8.4.0,<10,>=8.1.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain) (9.0.0)\n", - "Requirement already satisfied: numpy<2,>=1.26.4 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain) (1.26.4)\n", - "Requirement already satisfied: dataclasses-json<0.7,>=0.5.7 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain-community) (0.6.7)\n", - "Requirement already satisfied: pydantic-settings<3.0.0,>=2.4.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain-community) (2.8.0)\n", - "Requirement already satisfied: httpx-sse<1.0.0,>=0.4.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain-community) (0.4.0)\n", - "Requirement already satisfied: openai<2.0.0,>=1.58.1 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain_openai) (1.60.0)\n", - "Requirement already satisfied: tiktoken<1,>=0.7 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain_openai) (0.8.0)\n", - "Requirement already satisfied: transformers<5.0.0,>=4.41.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from sentence_transformers) (4.49.0)\n", - "Requirement already satisfied: tqdm in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from sentence_transformers) (4.67.1)\n", - "Requirement already satisfied: torch>=1.11.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from sentence_transformers) (2.5.1)\n", - "Requirement already satisfied: scikit-learn in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from sentence_transformers) (1.6.1)\n", - "Requirement already satisfied: scipy in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from sentence_transformers) (1.12.0)\n", - "Requirement already satisfied: Pillow in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from sentence_transformers) (11.0.0)\n", - "Requirement already satisfied: filelock in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from huggingface-hub) (3.16.1)\n", - "Requirement already satisfied: fsspec>=2023.5.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from huggingface-hub) (2024.12.0)\n", - "Requirement already satisfied: packaging>=20.9 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from huggingface-hub) (24.2)\n", - "Requirement already satisfied: typing-extensions>=3.7.4.3 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from huggingface-hub) (4.12.2)\n", - "Requirement already satisfied: build>=1.0.3 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (1.2.2.post1)\n", - "Requirement already satisfied: chroma-hnswlib==0.7.6 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (0.7.6)\n", - "Requirement already satisfied: fastapi>=0.95.2 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (0.115.8)\n", - "Requirement already satisfied: uvicorn>=0.18.3 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from uvicorn[standard]>=0.18.3->chromadb) (0.34.0)\n", - "Requirement already satisfied: posthog>=2.4.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (3.15.1)\n", - "Requirement already satisfied: onnxruntime>=1.14.1 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (1.20.1)\n", - "Requirement already satisfied: opentelemetry-api>=1.2.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (1.30.0)\n", - "Requirement already satisfied: opentelemetry-exporter-otlp-proto-grpc>=1.2.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (1.30.0)\n", - "Requirement already satisfied: opentelemetry-instrumentation-fastapi>=0.41b0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (0.51b0)\n", - "Requirement already satisfied: opentelemetry-sdk>=1.2.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (1.30.0)\n", - "Requirement already satisfied: tokenizers>=0.13.2 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (0.21.0)\n", - "Requirement already satisfied: pypika>=0.48.9 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (0.48.9)\n", - "Requirement already satisfied: overrides>=7.3.1 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (7.7.0)\n", - "Requirement already satisfied: importlib-resources in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (6.5.2)\n", - "Requirement already satisfied: grpcio>=1.58.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (1.70.0)\n", - "Requirement already satisfied: bcrypt>=4.0.1 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (4.2.1)\n", - "Requirement already satisfied: typer>=0.9.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (0.15.1)\n", - "Requirement already satisfied: kubernetes>=28.1.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (32.0.1)\n", - "Requirement already satisfied: mmh3>=4.0.1 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (5.1.0)\n", - "Requirement already satisfied: orjson>=3.9.12 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (3.10.15)\n", - "Requirement already satisfied: httpx>=0.27.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (0.28.1)\n", - "Requirement already satisfied: rich>=10.11.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (13.9.4)\n", - "Requirement already satisfied: aiohappyeyeballs>=2.3.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (2.4.4)\n", - "Requirement already satisfied: aiosignal>=1.1.2 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.3.2)\n", - "Requirement already satisfied: attrs>=17.3.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (24.3.0)\n", - "Requirement already satisfied: frozenlist>=1.1.1 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.5.0)\n", - "Requirement already satisfied: multidict<7.0,>=4.5 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (6.1.0)\n", - "Requirement already satisfied: propcache>=0.2.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (0.2.1)\n", - "Requirement already satisfied: yarl<2.0,>=1.17.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.18.3)\n", - "Requirement already satisfied: pyproject_hooks in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from build>=1.0.3->chromadb) (1.2.0)\n", - "Requirement already satisfied: colorama in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from build>=1.0.3->chromadb) (0.4.6)\n", - "Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from dataclasses-json<0.7,>=0.5.7->langchain-community) (3.26.0)\n", - "Requirement already satisfied: typing-inspect<1,>=0.4.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from dataclasses-json<0.7,>=0.5.7->langchain-community) (0.9.0)\n", - "Requirement already satisfied: starlette<0.46.0,>=0.40.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from fastapi>=0.95.2->chromadb) (0.45.3)\n", - "Requirement already satisfied: anyio in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from httpx>=0.27.0->chromadb) (4.8.0)\n", - "Requirement already satisfied: certifi in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from httpx>=0.27.0->chromadb) (2024.12.14)\n", - "Requirement already satisfied: httpcore==1.* in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from httpx>=0.27.0->chromadb) (1.0.7)\n", - "Requirement already satisfied: idna in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from httpx>=0.27.0->chromadb) (3.10)\n", - "Requirement already satisfied: h11<0.15,>=0.13 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from httpcore==1.*->httpx>=0.27.0->chromadb) (0.14.0)\n", - "Requirement already satisfied: six>=1.9.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from kubernetes>=28.1.0->chromadb) (1.17.0)\n", - "Requirement already satisfied: python-dateutil>=2.5.3 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from kubernetes>=28.1.0->chromadb) (2.9.0.post0)\n", - "Requirement already satisfied: google-auth>=1.0.1 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from kubernetes>=28.1.0->chromadb) (2.38.0)\n", - "Requirement already satisfied: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from kubernetes>=28.1.0->chromadb) (1.8.0)\n", - "Requirement already satisfied: requests-oauthlib in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from kubernetes>=28.1.0->chromadb) (2.0.0)\n", - "Requirement already satisfied: oauthlib>=3.2.2 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from kubernetes>=28.1.0->chromadb) (3.2.2)\n", - "Requirement already satisfied: urllib3>=1.24.2 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from kubernetes>=28.1.0->chromadb) (2.2.3)\n", - "Requirement already satisfied: durationpy>=0.7 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from kubernetes>=28.1.0->chromadb) (0.9)\n", - "Requirement already satisfied: jsonpatch<2.0,>=1.33 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain-core<1.0.0,>=0.3.35->langchain) (1.33)\n", - "Requirement already satisfied: requests-toolbelt<2.0.0,>=1.0.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langsmith<0.4,>=0.1.17->langchain) (1.0.0)\n", - "Requirement already satisfied: zstandard<0.24.0,>=0.23.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langsmith<0.4,>=0.1.17->langchain) (0.23.0)\n", - "Requirement already satisfied: coloredlogs in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from onnxruntime>=1.14.1->chromadb) (15.0.1)\n", - "Requirement already satisfied: flatbuffers in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from onnxruntime>=1.14.1->chromadb) (25.2.10)\n", - "Requirement already satisfied: protobuf in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from onnxruntime>=1.14.1->chromadb) (5.29.3)\n", - "Requirement already satisfied: sympy in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from onnxruntime>=1.14.1->chromadb) (1.13.1)\n", - "Requirement already satisfied: distro<2,>=1.7.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from openai<2.0.0,>=1.58.1->langchain_openai) (1.9.0)\n", - "Requirement already satisfied: jiter<1,>=0.4.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from openai<2.0.0,>=1.58.1->langchain_openai) (0.8.2)\n", - "Requirement already satisfied: sniffio in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from openai<2.0.0,>=1.58.1->langchain_openai) (1.3.1)\n", - "Requirement already satisfied: deprecated>=1.2.6 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from opentelemetry-api>=1.2.0->chromadb) (1.2.16)\n", - "Requirement already satisfied: importlib-metadata<=8.5.0,>=6.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from opentelemetry-api>=1.2.0->chromadb) (8.5.0)\n", - "Requirement already satisfied: googleapis-common-protos~=1.52 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from opentelemetry-exporter-otlp-proto-grpc>=1.2.0->chromadb) (1.68.0)\n", - "Requirement already satisfied: opentelemetry-exporter-otlp-proto-common==1.30.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from opentelemetry-exporter-otlp-proto-grpc>=1.2.0->chromadb) (1.30.0)\n", - "Requirement already satisfied: opentelemetry-proto==1.30.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from opentelemetry-exporter-otlp-proto-grpc>=1.2.0->chromadb) (1.30.0)\n", - "Requirement already satisfied: opentelemetry-instrumentation-asgi==0.51b0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from opentelemetry-instrumentation-fastapi>=0.41b0->chromadb) (0.51b0)\n", - "Requirement already satisfied: opentelemetry-instrumentation==0.51b0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from opentelemetry-instrumentation-fastapi>=0.41b0->chromadb) (0.51b0)\n", - "Requirement already satisfied: opentelemetry-semantic-conventions==0.51b0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from opentelemetry-instrumentation-fastapi>=0.41b0->chromadb) (0.51b0)\n", - "Requirement already satisfied: opentelemetry-util-http==0.51b0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from opentelemetry-instrumentation-fastapi>=0.41b0->chromadb) (0.51b0)\n", - "Requirement already satisfied: wrapt<2.0.0,>=1.0.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from opentelemetry-instrumentation==0.51b0->opentelemetry-instrumentation-fastapi>=0.41b0->chromadb) (1.17.2)\n", - "Requirement already satisfied: asgiref~=3.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from opentelemetry-instrumentation-asgi==0.51b0->opentelemetry-instrumentation-fastapi>=0.41b0->chromadb) (3.8.1)\n", - "Requirement already satisfied: monotonic>=1.5 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from posthog>=2.4.0->chromadb) (1.6)\n", - "Requirement already satisfied: backoff>=1.10.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from posthog>=2.4.0->chromadb) (2.2.1)\n", - "Requirement already satisfied: annotated-types>=0.6.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from pydantic<3.0.0,>=2.7.4->langchain) (0.7.0)\n", - "Requirement already satisfied: pydantic-core==2.27.2 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from pydantic<3.0.0,>=2.7.4->langchain) (2.27.2)\n", - "Requirement already satisfied: python-dotenv>=0.21.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from pydantic-settings<3.0.0,>=2.4.0->langchain-community) (1.0.1)\n", - "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from requests<3,>=2->langchain) (3.4.0)\n", - "Requirement already satisfied: markdown-it-py>=2.2.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from rich>=10.11.0->chromadb) (3.0.0)\n", - "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from rich>=10.11.0->chromadb) (2.18.0)\n", - "Requirement already satisfied: greenlet!=0.4.17 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from SQLAlchemy<3,>=1.4->langchain) (3.1.1)\n", - "Requirement already satisfied: regex>=2022.1.18 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from tiktoken<1,>=0.7->langchain_openai) (2024.11.6)\n", - "Requirement already satisfied: networkx in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from torch>=1.11.0->sentence_transformers) (3.4.2)\n", - "Requirement already satisfied: jinja2 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from torch>=1.11.0->sentence_transformers) (3.1.5)\n", - "Requirement already satisfied: mpmath<1.4,>=1.1.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from sympy->onnxruntime>=1.14.1->chromadb) (1.3.0)\n", - "Requirement already satisfied: safetensors>=0.4.1 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from transformers<5.0.0,>=4.41.0->sentence_transformers) (0.5.2)\n", - "Requirement already satisfied: click>=8.0.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from typer>=0.9.0->chromadb) (8.1.8)\n", - "Requirement already satisfied: shellingham>=1.3.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from typer>=0.9.0->chromadb) (1.5.4)\n", - "Requirement already satisfied: httptools>=0.6.3 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from uvicorn[standard]>=0.18.3->chromadb) (0.6.4)\n", - "Requirement already satisfied: watchfiles>=0.13 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from uvicorn[standard]>=0.18.3->chromadb) (1.0.4)\n", - "Requirement already satisfied: websockets>=10.4 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from uvicorn[standard]>=0.18.3->chromadb) (15.0)\n", - "Requirement already satisfied: joblib>=1.2.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from scikit-learn->sentence_transformers) (1.4.2)\n", - "Requirement already satisfied: threadpoolctl>=3.1.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from scikit-learn->sentence_transformers) (3.5.0)\n", - "Requirement already satisfied: cachetools<6.0,>=2.0.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from google-auth>=1.0.1->kubernetes>=28.1.0->chromadb) (5.5.2)\n", - "Requirement already satisfied: pyasn1-modules>=0.2.1 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from google-auth>=1.0.1->kubernetes>=28.1.0->chromadb) (0.4.1)\n", - "Requirement already satisfied: rsa<5,>=3.1.4 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from google-auth>=1.0.1->kubernetes>=28.1.0->chromadb) (4.9)\n", - "Requirement already satisfied: zipp>=3.20 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from importlib-metadata<=8.5.0,>=6.0->opentelemetry-api>=1.2.0->chromadb) (3.21.0)\n", - "Requirement already satisfied: jsonpointer>=1.9 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from jsonpatch<2.0,>=1.33->langchain-core<1.0.0,>=0.3.35->langchain) (3.0.0)\n", - "Requirement already satisfied: mdurl~=0.1 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from markdown-it-py>=2.2.0->rich>=10.11.0->chromadb) (0.1.2)\n", - "Requirement already satisfied: mypy-extensions>=0.3.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from typing-inspect<1,>=0.4.0->dataclasses-json<0.7,>=0.5.7->langchain-community) (1.0.0)\n", - "Requirement already satisfied: humanfriendly>=9.1 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from coloredlogs->onnxruntime>=1.14.1->chromadb) (10.0)\n", - "Requirement already satisfied: MarkupSafe>=2.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from jinja2->torch>=1.11.0->sentence_transformers) (3.0.2)\n", - "Requirement already satisfied: pyreadline3 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from humanfriendly>=9.1->coloredlogs->onnxruntime>=1.14.1->chromadb) (3.5.4)\n", - "Requirement already satisfied: pyasn1<0.7.0,>=0.4.6 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from pyasn1-modules>=0.2.1->google-auth>=1.0.1->kubernetes>=28.1.0->chromadb) (0.6.1)\n", - "Note: you may need to restart the kernel to use updated packages.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "\n", - "[notice] A new release of pip is available: 24.3.1 -> 25.0.1\n", - "[notice] To update, run: C:\\Users\\baumanoa\\AppData\\Local\\Microsoft\\WindowsApps\\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\\python.exe -m pip install --upgrade pip\n" - ] - } - ], + "outputs": [], "source": [ - "%pip install langchain langchain-community langchain_openai sentence_transformers huggingface-hub chromadb langchain_huggingface" + "%pip install langchain langchain-community huggingface-hub chromadb langchain_huggingface" ] }, { @@ -641,11 +546,6 @@ " - Ermöglicht strukturierte Arbeit mit verschiedenen LLMs\n", " - Vereinfacht die RAG-Pipeline-Entwicklung\n", "\n", - "- sentence_transformers: Bibliothek für Text-Embeddings\n", - " - Wandelt Text in numerische Vektoren um\n", - " - Unterstützt mehrsprachige Modelle\n", - " - Speziell für semantische Textähnlichkeit optimiert\n", - "\n", "- huggingface-hub & langchain_huggingface: Zugriff auf KI-Modelle\n", " - Bietet Zugang zu Open-Source-Modellen\n", " - Ermöglicht lokale Nutzung von Embedding-Modellen\n", @@ -661,198 +561,13 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "### Eine erste RAG-Pipeline\n", - "Hier demonstrieren wir die grundlegenden Schritte einer RAG-Pipeline:" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Requirement already satisfied: langchain in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (0.3.19)\n", - "Requirement already satisfied: langchain-community in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (0.3.18)\n", - "Requirement already satisfied: langchain_openai in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (0.3.6)\n", - "Requirement already satisfied: sentence_transformers in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (3.4.1)\n", - "Requirement already satisfied: huggingface-hub in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (0.29.1)\n", - "Requirement already satisfied: chromadb in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (0.6.3)\n", - "Requirement already satisfied: langchain_huggingface in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (0.1.2)\n", - "Requirement already satisfied: langchain-core<1.0.0,>=0.3.35 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain) (0.3.37)\n", - "Requirement already satisfied: langchain-text-splitters<1.0.0,>=0.3.6 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain) (0.3.6)\n", - "Requirement already satisfied: langsmith<0.4,>=0.1.17 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain) (0.3.10)\n", - "Requirement already satisfied: pydantic<3.0.0,>=2.7.4 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain) (2.10.5)\n", - "Requirement already satisfied: SQLAlchemy<3,>=1.4 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain) (2.0.37)\n", - "Requirement already satisfied: requests<3,>=2 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain) (2.32.3)\n", - "Requirement already satisfied: PyYAML>=5.3 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain) (6.0.2)\n", - "Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain) (3.11.11)\n", - "Requirement already satisfied: tenacity!=8.4.0,<10,>=8.1.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain) (9.0.0)\n", - "Requirement already satisfied: numpy<2,>=1.26.4 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain) (1.26.4)\n", - "Requirement already satisfied: dataclasses-json<0.7,>=0.5.7 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain-community) (0.6.7)\n", - "Requirement already satisfied: pydantic-settings<3.0.0,>=2.4.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain-community) (2.8.0)\n", - "Requirement already satisfied: httpx-sse<1.0.0,>=0.4.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain-community) (0.4.0)\n", - "Requirement already satisfied: openai<2.0.0,>=1.58.1 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain_openai) (1.60.0)\n", - "Requirement already satisfied: tiktoken<1,>=0.7 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain_openai) (0.8.0)\n", - "Requirement already satisfied: transformers<5.0.0,>=4.41.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from sentence_transformers) (4.49.0)\n", - "Requirement already satisfied: tqdm in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from sentence_transformers) (4.67.1)\n", - "Requirement already satisfied: torch>=1.11.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from sentence_transformers) (2.5.1)\n", - "Requirement already satisfied: scikit-learn in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from sentence_transformers) (1.6.1)\n", - "Requirement already satisfied: scipy in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from sentence_transformers) (1.12.0)\n", - "Requirement already satisfied: Pillow in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from sentence_transformers) (11.0.0)\n", - "Requirement already satisfied: filelock in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from huggingface-hub) (3.16.1)\n", - "Requirement already satisfied: fsspec>=2023.5.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from huggingface-hub) (2024.12.0)\n", - "Requirement already satisfied: packaging>=20.9 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from huggingface-hub) (24.2)\n", - "Requirement already satisfied: typing-extensions>=3.7.4.3 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from huggingface-hub) (4.12.2)\n", - "Requirement already satisfied: build>=1.0.3 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (1.2.2.post1)\n", - "Requirement already satisfied: chroma-hnswlib==0.7.6 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (0.7.6)\n", - "Requirement already satisfied: fastapi>=0.95.2 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (0.115.8)\n", - "Requirement already satisfied: uvicorn>=0.18.3 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from uvicorn[standard]>=0.18.3->chromadb) (0.34.0)\n", - "Requirement already satisfied: posthog>=2.4.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (3.15.1)\n", - "Requirement already satisfied: onnxruntime>=1.14.1 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (1.20.1)\n", - "Requirement already satisfied: opentelemetry-api>=1.2.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (1.30.0)\n", - "Requirement already satisfied: opentelemetry-exporter-otlp-proto-grpc>=1.2.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (1.30.0)\n", - "Requirement already satisfied: opentelemetry-instrumentation-fastapi>=0.41b0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (0.51b0)\n", - "Requirement already satisfied: opentelemetry-sdk>=1.2.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (1.30.0)\n", - "Requirement already satisfied: tokenizers>=0.13.2 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (0.21.0)\n", - "Requirement already satisfied: pypika>=0.48.9 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (0.48.9)\n", - "Requirement already satisfied: overrides>=7.3.1 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (7.7.0)\n", - "Requirement already satisfied: importlib-resources in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (6.5.2)\n", - "Requirement already satisfied: grpcio>=1.58.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (1.70.0)\n", - "Requirement already satisfied: bcrypt>=4.0.1 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (4.2.1)\n", - "Requirement already satisfied: typer>=0.9.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (0.15.1)\n", - "Requirement already satisfied: kubernetes>=28.1.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (32.0.1)\n", - "Requirement already satisfied: mmh3>=4.0.1 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (5.1.0)\n", - "Requirement already satisfied: orjson>=3.9.12 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (3.10.15)\n", - "Requirement already satisfied: httpx>=0.27.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (0.28.1)\n", - "Requirement already satisfied: rich>=10.11.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from chromadb) (13.9.4)\n", - "Requirement already satisfied: aiohappyeyeballs>=2.3.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (2.4.4)\n", - "Requirement already satisfied: aiosignal>=1.1.2 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.3.2)\n", - "Requirement already satisfied: attrs>=17.3.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (24.3.0)\n", - "Requirement already satisfied: frozenlist>=1.1.1 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.5.0)\n", - "Requirement already satisfied: multidict<7.0,>=4.5 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (6.1.0)\n", - "Requirement already satisfied: propcache>=0.2.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (0.2.1)\n", - "Requirement already satisfied: yarl<2.0,>=1.17.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.18.3)\n", - "Requirement already satisfied: pyproject_hooks in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from build>=1.0.3->chromadb) (1.2.0)\n", - "Requirement already satisfied: colorama in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from build>=1.0.3->chromadb) (0.4.6)\n", - "Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from dataclasses-json<0.7,>=0.5.7->langchain-community) (3.26.0)\n", - "Requirement already satisfied: typing-inspect<1,>=0.4.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from dataclasses-json<0.7,>=0.5.7->langchain-community) (0.9.0)\n", - "Requirement already satisfied: starlette<0.46.0,>=0.40.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from fastapi>=0.95.2->chromadb) (0.45.3)\n", - "Requirement already satisfied: anyio in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from httpx>=0.27.0->chromadb) (4.8.0)\n", - "Requirement already satisfied: certifi in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from httpx>=0.27.0->chromadb) (2024.12.14)\n", - "Requirement already satisfied: httpcore==1.* in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from httpx>=0.27.0->chromadb) (1.0.7)\n", - "Requirement already satisfied: idna in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from httpx>=0.27.0->chromadb) (3.10)\n", - "Requirement already satisfied: h11<0.15,>=0.13 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from httpcore==1.*->httpx>=0.27.0->chromadb) (0.14.0)\n", - "Requirement already satisfied: six>=1.9.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from kubernetes>=28.1.0->chromadb) (1.17.0)\n", - "Requirement already satisfied: python-dateutil>=2.5.3 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from kubernetes>=28.1.0->chromadb) (2.9.0.post0)\n", - "Requirement already satisfied: google-auth>=1.0.1 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from kubernetes>=28.1.0->chromadb) (2.38.0)\n", - "Requirement already satisfied: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from kubernetes>=28.1.0->chromadb) (1.8.0)\n", - "Requirement already satisfied: requests-oauthlib in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from kubernetes>=28.1.0->chromadb) (2.0.0)\n", - "Requirement already satisfied: oauthlib>=3.2.2 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from kubernetes>=28.1.0->chromadb) (3.2.2)\n", - "Requirement already satisfied: urllib3>=1.24.2 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from kubernetes>=28.1.0->chromadb) (2.2.3)\n", - "Requirement already satisfied: durationpy>=0.7 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from kubernetes>=28.1.0->chromadb) (0.9)\n", - "Requirement already satisfied: jsonpatch<2.0,>=1.33 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langchain-core<1.0.0,>=0.3.35->langchain) (1.33)\n", - "Requirement already satisfied: requests-toolbelt<2.0.0,>=1.0.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langsmith<0.4,>=0.1.17->langchain) (1.0.0)\n", - "Requirement already satisfied: zstandard<0.24.0,>=0.23.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from langsmith<0.4,>=0.1.17->langchain) (0.23.0)\n", - "Requirement already satisfied: coloredlogs in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from onnxruntime>=1.14.1->chromadb) (15.0.1)\n", - "Requirement already satisfied: flatbuffers in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from onnxruntime>=1.14.1->chromadb) (25.2.10)\n", - "Requirement already satisfied: protobuf in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from onnxruntime>=1.14.1->chromadb) (5.29.3)\n", - "Requirement already satisfied: sympy in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from onnxruntime>=1.14.1->chromadb) (1.13.1)\n", - "Requirement already satisfied: distro<2,>=1.7.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from openai<2.0.0,>=1.58.1->langchain_openai) (1.9.0)\n", - "Requirement already satisfied: jiter<1,>=0.4.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from openai<2.0.0,>=1.58.1->langchain_openai) (0.8.2)\n", - "Requirement already satisfied: sniffio in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from openai<2.0.0,>=1.58.1->langchain_openai) (1.3.1)\n", - "Requirement already satisfied: deprecated>=1.2.6 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from opentelemetry-api>=1.2.0->chromadb) (1.2.16)\n", - "Requirement already satisfied: importlib-metadata<=8.5.0,>=6.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from opentelemetry-api>=1.2.0->chromadb) (8.5.0)\n", - "Requirement already satisfied: googleapis-common-protos~=1.52 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from opentelemetry-exporter-otlp-proto-grpc>=1.2.0->chromadb) (1.68.0)\n", - "Requirement already satisfied: opentelemetry-exporter-otlp-proto-common==1.30.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from opentelemetry-exporter-otlp-proto-grpc>=1.2.0->chromadb) (1.30.0)\n", - "Requirement already satisfied: opentelemetry-proto==1.30.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from opentelemetry-exporter-otlp-proto-grpc>=1.2.0->chromadb) (1.30.0)\n", - "Requirement already satisfied: opentelemetry-instrumentation-asgi==0.51b0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from opentelemetry-instrumentation-fastapi>=0.41b0->chromadb) (0.51b0)\n", - "Requirement already satisfied: opentelemetry-instrumentation==0.51b0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from opentelemetry-instrumentation-fastapi>=0.41b0->chromadb) (0.51b0)\n", - "Requirement already satisfied: opentelemetry-semantic-conventions==0.51b0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from opentelemetry-instrumentation-fastapi>=0.41b0->chromadb) (0.51b0)\n", - "Requirement already satisfied: opentelemetry-util-http==0.51b0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from opentelemetry-instrumentation-fastapi>=0.41b0->chromadb) (0.51b0)\n", - "Requirement already satisfied: wrapt<2.0.0,>=1.0.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from opentelemetry-instrumentation==0.51b0->opentelemetry-instrumentation-fastapi>=0.41b0->chromadb) (1.17.2)\n", - "Requirement already satisfied: asgiref~=3.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from opentelemetry-instrumentation-asgi==0.51b0->opentelemetry-instrumentation-fastapi>=0.41b0->chromadb) (3.8.1)\n", - "Requirement already satisfied: monotonic>=1.5 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from posthog>=2.4.0->chromadb) (1.6)\n", - "Requirement already satisfied: backoff>=1.10.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from posthog>=2.4.0->chromadb) (2.2.1)\n", - "Requirement already satisfied: annotated-types>=0.6.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from pydantic<3.0.0,>=2.7.4->langchain) (0.7.0)\n", - "Requirement already satisfied: pydantic-core==2.27.2 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from pydantic<3.0.0,>=2.7.4->langchain) (2.27.2)\n", - "Requirement already satisfied: python-dotenv>=0.21.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from pydantic-settings<3.0.0,>=2.4.0->langchain-community) (1.0.1)\n", - "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from requests<3,>=2->langchain) (3.4.0)\n", - "Requirement already satisfied: markdown-it-py>=2.2.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from rich>=10.11.0->chromadb) (3.0.0)\n", - "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from rich>=10.11.0->chromadb) (2.18.0)\n", - "Requirement already satisfied: greenlet!=0.4.17 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from SQLAlchemy<3,>=1.4->langchain) (3.1.1)\n", - "Requirement already satisfied: regex>=2022.1.18 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from tiktoken<1,>=0.7->langchain_openai) (2024.11.6)\n", - "Requirement already satisfied: networkx in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from torch>=1.11.0->sentence_transformers) (3.4.2)\n", - "Requirement already satisfied: jinja2 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from torch>=1.11.0->sentence_transformers) (3.1.5)\n", - "Requirement already satisfied: mpmath<1.4,>=1.1.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from sympy->onnxruntime>=1.14.1->chromadb) (1.3.0)\n", - "Requirement already satisfied: safetensors>=0.4.1 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from transformers<5.0.0,>=4.41.0->sentence_transformers) (0.5.2)\n", - "Requirement already satisfied: click>=8.0.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from typer>=0.9.0->chromadb) (8.1.8)\n", - "Requirement already satisfied: shellingham>=1.3.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from typer>=0.9.0->chromadb) (1.5.4)\n", - "Requirement already satisfied: httptools>=0.6.3 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from uvicorn[standard]>=0.18.3->chromadb) (0.6.4)\n", - "Requirement already satisfied: watchfiles>=0.13 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from uvicorn[standard]>=0.18.3->chromadb) (1.0.4)\n", - "Requirement already satisfied: websockets>=10.4 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from uvicorn[standard]>=0.18.3->chromadb) (15.0)\n", - "Requirement already satisfied: joblib>=1.2.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from scikit-learn->sentence_transformers) (1.4.2)\n", - "Requirement already satisfied: threadpoolctl>=3.1.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from scikit-learn->sentence_transformers) (3.5.0)\n", - "Requirement already satisfied: cachetools<6.0,>=2.0.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from google-auth>=1.0.1->kubernetes>=28.1.0->chromadb) (5.5.2)\n", - "Requirement already satisfied: pyasn1-modules>=0.2.1 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from google-auth>=1.0.1->kubernetes>=28.1.0->chromadb) (0.4.1)\n", - "Requirement already satisfied: rsa<5,>=3.1.4 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from google-auth>=1.0.1->kubernetes>=28.1.0->chromadb) (4.9)\n", - "Requirement already satisfied: zipp>=3.20 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from importlib-metadata<=8.5.0,>=6.0->opentelemetry-api>=1.2.0->chromadb) (3.21.0)\n", - "Requirement already satisfied: jsonpointer>=1.9 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from jsonpatch<2.0,>=1.33->langchain-core<1.0.0,>=0.3.35->langchain) (3.0.0)\n", - "Requirement already satisfied: mdurl~=0.1 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from markdown-it-py>=2.2.0->rich>=10.11.0->chromadb) (0.1.2)\n", - "Requirement already satisfied: mypy-extensions>=0.3.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from typing-inspect<1,>=0.4.0->dataclasses-json<0.7,>=0.5.7->langchain-community) (1.0.0)\n", - "Requirement already satisfied: humanfriendly>=9.1 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from coloredlogs->onnxruntime>=1.14.1->chromadb) (10.0)\n", - "Requirement already satisfied: MarkupSafe>=2.0 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from jinja2->torch>=1.11.0->sentence_transformers) (3.0.2)\n", - "Requirement already satisfied: pyreadline3 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from humanfriendly>=9.1->coloredlogs->onnxruntime>=1.14.1->chromadb) (3.5.4)\n", - "Requirement already satisfied: pyasn1<0.7.0,>=0.4.6 in c:\\users\\baumanoa\\appdata\\local\\packages\\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\\localcache\\local-packages\\python311\\site-packages (from pyasn1-modules>=0.2.1->google-auth>=1.0.1->kubernetes>=28.1.0->chromadb) (0.6.1)\n", - "Note: you may need to restart the kernel to use updated packages.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "\n", - "[notice] A new release of pip is available: 24.3.1 -> 25.0.1\n", - "[notice] To update, run: C:\\Users\\baumanoa\\AppData\\Local\\Microsoft\\WindowsApps\\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\\python.exe -m pip install --upgrade pip\n" - ] - } - ], - "source": [ - "# Installation der benötigten Bibliotheken\n", - "%pip install langchain langchain-community langchain_openai sentence_transformers huggingface-hub chromadb langchain_huggingface" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "#### Erklärung der verwendeten Bibliotheken:\n", - "- langchain & langchain-community: Framework für die Entwicklung von LLM-Anwendungen\n", - " - Bietet Werkzeuge für Dokumentenverarbeitung\n", - " - Ermöglicht strukturierte Arbeit mit verschiedenen LLMs\n", - " - Vereinfacht die RAG-Pipeline-Entwicklung\n", - "\n", - "- sentence_transformers: Bibliothek für Text-Embeddings\n", - " - Wandelt Text in numerische Vektoren um\n", - " - Unterstützt mehrsprachige Modelle\n", - " - Speziell für semantische Textähnlichkeit optimiert\n", - "\n", - "- huggingface-hub & langchain_huggingface: Zugriff auf KI-Modelle\n", - " - Bietet Zugang zu Open-Source-Modellen\n", - " - Ermöglicht lokale Nutzung von Embedding-Modellen\n", - " - Integration mit LangChain\n", - "\n", - "- chromadb: Vektordatenbank für Embedding-Speicherung\n", - " - Effiziente Speicherung und Suche von Vektoren\n", - " - Unterstützt Metadaten\n", - " - Ermöglicht Ähnlichkeitssuche" + "### Chunking\n", + "-- Brief explanation of chunking" ] }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 14, "metadata": {}, "outputs": [ { @@ -894,13 +609,49 @@ "Was passiert hier?\n", "\n", "- Der Text wird in kleinere Einheiten aufgeteilt\n", - "- Die Größe (500 Zeichen) ist ein wichtiger Parameter\n", + "- Die Größe (300 Zeichen) ist ein wichtiger Parameter\n", "- Überlappung verhindert das \"Zerschneiden\" von Zusammenhängen" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Embedding-Modell konfigurieren\n", + "Die Wahl des Embedding-Modells ist entscheidend für die Qualität der Suche. \n", + "\n", + "Wir benutzen ein ein HuggingFace Modell welches Sätze und Absätze in einem 384-dimensionalen dichten Vektorraum abbildet. Es funktioniert gut für Aufgaben wie Clustering oder semantische Suche verwendet werden. Außerdem wurde das Modell gewählt, weil:\n", + "- es effektiv mit deutschen Texten arbeitet\n", + "- Eine gute Balance zwischen Geschwindigkeit und Qualität bietet\n", + "- Geringe Ressourcenanforderungen stellt" + ] + }, { "cell_type": "code", - "execution_count": 10, + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# 2. Text in Vektoren umwandeln (Embedding)\n", + "embeddings = HuggingFaceEmbeddings(\n", + " model_name=\"sentence-transformers/paraphrase-multilingual-mpnet-base-v2\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Vektordatenbank erstellen\n", + "\n", + "Die Vektordatenbank ist das Herzstück des Retrieval-Systems. Es hilft dabei das Problem zu lösen wie wir die Vektoren die beim Embedding entstehen speichern können so, dass wir sie später schnell wiederfinden können.\n", + "\n", + "Stellen Sie sich vor, Sie haben eine große Bibliothek (unsere Brief-Chunks). Mit Embeddings haben wir jedem Text eine Art \"numerischen Fingerabdruck\" gegeben. Eine Vektordatenbank ist nun wie ein hocheffizientes Ablagesystem, das diese Fingerabdrücke so organisiert, dass wir blitzschnell ähnliche Texte finden können." + ] + }, + { + "cell_type": "code", + "execution_count": null, "metadata": {}, "outputs": [ { @@ -913,11 +664,6 @@ } ], "source": [ - "# 2. Text in Vektoren umwandeln (Embedding)\n", - "embeddings = HuggingFaceEmbeddings(\n", - " model_name=\"sentence-transformers/paraphrase-multilingual-mpnet-base-v2\"\n", - ")\n", - "\n", "# 3. Vektordatenbank erstellen\n", "vectorstore = Chroma.from_texts(\n", " texts=chunks,\n", @@ -928,7 +674,7 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 9, "metadata": {}, "outputs": [ { @@ -940,7 +686,7 @@ "getreuer Man biß in den Tott. Hertz Libe frau ich bericht dir auch daß ich deinen an mir geschribnen brif von 1. Mey hab ich wol erhalten und hab auch darauß erfaren daß ihr noch alle gesund seid und ist mir ser lib geweßen. ich wolte noch gern viel mer schreiben aber die zeit will nicht leiden und\n", "\n", "Ergebnis 2:\n", - "getreuer Man biß in den Tott. Hertz Libe frau ich bericht dir auch daß ich deinen an mir geschribnen brif von 1. Mey hab ich wol erhalten und hab auch darauß erfaren daß ihr noch alle gesund seid und ist mir ser lib geweßen. ich wolte noch gern viel mer schreiben aber die zeit will nicht leiden und\n" + "si seind noch ale beide gesund. Jogen Olrich und Haferland laßen ihre Eltern auch grüßen und si sind noch gesund. Mein feder Erman schmit lest seine Muter und schwestern auch vilmals grüßen und er ist noch gesund. Mein feder Bin auß ungeburg ist auch noch gesund und der aus Werben auch. eß ist auch\n" ] } ], @@ -966,9 +712,27 @@ "- Die Datenbank findet relevante Textstellen" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### LLM und Retriever konfigurieren\n", + "\n", + "In diesem Schritt konfigurieren wir zwei zentrale Komponenten unseres RAG-Systems:\n", + "1. Der Retriever dient als Suchmaschine unseres Systems. Er nutzt die zuvor erstellte Vektordatenbank, um relevante Textpassagen zu finden. Dabei:\n", + " - Wandelt er neue Anfragen in Embeddings um\n", + " - Sucht nach ähnlichen Vektoren in der Datenbank\n", + " - Gibt die relevantesten Dokumente zurück\n", + "\n", + "2. Das Large Language Model (LLM) ist für die Generierung der Antworten zuständig. Der Parameter temperature steuert dabei das Verhalten des Models:\n", + " - temperature=0: Sehr fokussierte, deterministische Antworten\n", + " - temperature>0: Kreativere, variantenreichere Antworten\n", + " - Für Forschung empfehlen wir niedrige Temperature-Werte, um konsistente, faktenbezogene Antworten zu erhalten" + ] + }, { "cell_type": "code", - "execution_count": 12, + "execution_count": null, "metadata": {}, "outputs": [ { @@ -988,13 +752,13 @@ "Frage: Wie berichten die Soldaten über ihre Gesundheit?\n", "\n", "Antwort des Modells:\n", - "Die Soldaten berichten, dass sie und ihre Kameraden noch gesund sind. Sie melden, dass sie die Briefe der Angehörigen erhalten haben und davon profitiert haben, zu erfahren, dass ihre Lieben ebenfalls gesund sind. Die Briefe stammen von Man, Jogen Olrich, Haferland, Mein feder Erman und Mein feder Bin. Alle berichten, dass sie sich und ihre Angehörigen gut finden.\n", + "Die Soldaten berichten, dass sie und ihre Kameraden noch gesund sind. Sie melden auch die Gesundheit ihrer Eltern und Angehörigen. In einem Fall wird berichtet, dass ein Regiment 12 Tote und Etliche 30 Verwundete hat, aber die meisten Soldaten sind noch gesund. Es wird auch berichtet, dass es regnet und es Wasser auf den Feldern gibt, was vermuten lässt, dass die Lager mit Wasser versorgt sind.\n", "\n", "Verwendete Quellen:\n", "- Quelle: getreuer Man biß in den Tott. Hertz Libe frau ich bericht dir auch daß ich deinen an mir geschribnen...\n", - "- Quelle: getreuer Man biß in den Tott. Hertz Libe frau ich bericht dir auch daß ich deinen an mir geschribnen...\n", - "- Quelle: getreuer Man biß in den Tott. Hertz Libe frau ich bericht dir auch daß ich deinen an mir geschribnen...\n", - "- Quelle: si seind noch ale beide gesund. Jogen Olrich und Haferland laßen ihre Eltern auch grüßen und si sind...\n" + "- Quelle: si seind noch ale beide gesund. Jogen Olrich und Haferland laßen ihre Eltern auch grüßen und si sind...\n", + "- Quelle: gekrigt 70 fahnen 5 standarten und 5 Paar Pauken und unser regiment haben wir 12 Totte und Etliche 3...\n", + "- Quelle: es ist uf flagen felde daß waßer gekomen uf 2 Ele hoch da könt ihr leicht erdenken waß daß hir vor L...\n" ] } ], @@ -1033,7 +797,7 @@ "\"\"\"\n", "prompt = PromptTemplate(template=template, input_variables=[\"context\", \"question\"])\n", "\n", - "# RAG-Chain erstellen\n", + "# RAG-Chain erstellen. Die QA-Chain verbindet alle Komponenten. \n", "qa_chain = RetrievalQA.from_chain_type(\n", " llm=llm,\n", " chain_type=\"stuff\", # \"stuff\" kombiniert alle Dokumente in einen Prompt\n", @@ -1041,7 +805,22 @@ " chain_type_kwargs={\"prompt\": prompt},\n", " return_source_documents=True # Quellen mitliefern\n", ")\n", - "\n", + "\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Jetzt können wir eine Frage stellen:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ "# Frage stellen\n", "ergebnis = qa_chain.invoke({\"query\": \"Wie berichten die Soldaten über ihre Gesundheit?\"})\n", "\n", @@ -1054,6 +833,23 @@ " print(f\"- Quelle: {doc.page_content[:100]}...\")" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Wie funktioniert das System?\n", + "\n", + "Unser RAG-System arbeitet in drei Schritten:\n", + "1. **Vorbereitung**: Die Texte werden in eine Form gebracht, die der Computer effizient durchsuchen kann \n", + " (mehr dazu in den Kapiteln \"Chunking\" & \"Embeddings\")\n", + "2. **Suche**: Bei einer Frage sucht das System die relevantesten Textpassagen \n", + " (Details folgen im Kapitel \"Vector Databases\")\n", + "3. **Antwort**: Die gefundenen Passagen werden zusammen mit der Frage an das LLM übergeben\n", + "\n", + "Für den Moment konzentrieren wir uns darauf, wie wir das System nutzen können. Die technischen Details \n", + "der einzelnen Komponenten werden wir in den folgenden Kapiteln genau untersuchen." + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -1083,23 +879,23 @@ "- Kontextuelle Embedding-Modelle verstehen häufig den Zusammenhang trotz abweichender Schreibweise\n", "- Die semantische Ähnlichkeit bleibt meist erhalten\n", "- Multilingual trainierte Modelle sind oft robuster gegenüber Sprachvariationen\n", - "\n", + "-- Should have a source or relevant links for this.\n", "#### 2. Retrievalqualität\n", "Für optimale Ergebnisse:\n", "- Verwenden wir semantische statt lexikalischer Suche\n", "- Achten auf Kontext statt einzelne Wörter\n", "- Profitieren von der Textvektorisierung, die ähnliche Konzepte unabhängig von der exakten Schreibweise erkennt\n", - "\n", + "-- again maybe references\n", "#### 3. Forschungspotential\n", "Die originale Schreibweise bietet sogar zusätzliches Forschungspotential:\n", - "- Analyse von Bildungsunterschieden zwischen Rängen\n", + "- Analyse von Sprachstilvariationen zwischen Rängen\n", "- Untersuchung regionaler Sprachvarianten\n", "- Dokumentation historischer Sprachentwicklung\n", "\n", "### Beispiel: Robuste Suchanfragen\n", "\n", "Mit RAG können wir trotz historischer Schreibweisen effektiv suchen. Wenn wir beispielsweise nach \"Gesundheit\" suchen, werden auch Varianten wie \"Gesundheit\", \"gesuntheit\" oder \"Gesundtheit\" gefunden:\n", - "\n", + "-- How do we actually test this?\n", "```python\n", "# Beispielabfrage zur Demonstration\n", "frage = \"Wie berichten die Soldaten über ihre Gesundheit?\"\n", @@ -1149,20 +945,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "## Übungen und Reflexion\n", - "\n", - "### Diskussionsfragen\n", - "\n", - "- Welche Vorteile bietet RAG gegenüber klassischer Textsuche?\n", - "- Welche ethischen Aspekte müssen bei der Analyse persönlicher Briefe beachtet werden?\n", - "- Wie könnte die Methode auf andere historische Quellen übertragen werden?\n", - "\n", - "### Praktische Übungen\n", - "\n", - "- Erkunden Sie die Metadaten der Briefe\n", - "- Formulieren Sie eigene Suchanfragen\n", - "- Vergleichen Sie verschiedene Briefe desselben Dienstgrads\n", - "\n" + "## Praktische Übungen\n", + "-- I think the practical exercises at this stage should involve: loading the courpus into Voyant Tools and exploring it, what can we identify and what is difficult to do with this Tool that RAG might be able to help with. Another one might be to create historical research questions about soldiers experiences during the time period, pose them to a LLM and evaluate the responses, see how different types of prompts might help the ability for useful answers to be generated. And maybe a third one would be to look at a few example letters and think about how a having a complete perspective of all letters might be able to help answer specific historical research questions." ] }, {