Commit Graph

8 Commits

Author SHA1 Message Date
Tuan Anh Nguyen Dang (Tadashi_Cin)
98509f886c Update splitters + metadata extractor interface to conform with new LlamaIndex design (#81)
* change splitter to general doc parsers class to fit new llama-index desing
* moving interface of splitter
2023-11-20 10:09:30 +07:00
Nguyen Trung Duc (john)
f8b8d86d4e Move LLM-related components into LLM module (#74)
* Move splitter into indexing module
* Rename post_processing module to parsers
* Migrate LLM-specific composite pipelines into llms module

This change moves the `splitters` module into `indexing` module. The `indexing` module will be created soon, to house `indexing`-related components.

This change renames `post_processing` module into `parsers` module. Post-processing is a generic term which provides very little information. In the future, we will add other extractors into the `parser` module, like Metadata extractor...

This change migrates the composite elements into `llms` module. These elements heavily assume that the internal nodes are llm-specific. As a result, migrating these elements into `llms` module will make them more discoverable, and simplify code base structure.
2023-11-15 16:26:53 +07:00
Tuan Anh Nguyen Dang (Tadashi_Cin)
9945afdf6f Add Reranker implementation and integration in Retrieving pipeline (#77)
* Add base Reranker
* Add LLM Reranker
* Add Cohere Reranker
* Add integration of Rerankers in Retrieving pipeline
2023-11-15 16:03:51 +07:00
Nguyen Trung Duc (john)
b159897ac6 Combine docstores and vectorstores within a storages component (#72) 2023-11-14 17:50:57 +07:00
Tuan Anh Nguyen Dang (Tadashi_Cin)
640962e916 Update retrieving + agent pipeline (#71) 2023-11-14 16:40:13 +07:00
Tuan Anh Nguyen Dang (Tadashi_Cin)
4704e2c11a Add new OCRReader with PDF+OCR text merging (#66)
This change speeds up OCR extraction by allowing bypassing OCR for texts that are irrelevant (not in table).

---------

Co-authored-by: Nguyen Trung Duc (john) <trungduc1992@gmail.com>
2023-11-13 17:43:02 +07:00
Nguyen Trung Duc (john)
9035e25666 Upgrade the declarative pipeline for cleaner interface (#51) 2023-10-24 11:12:22 +07:00
Tuan Anh Nguyen Dang (Tadashi_Cin)
79cc60e6a2 [AUR-429] Add MVP pipeline with Ingestion and QA stage (#39)
* add base Tool

* minor update test_tool

* update test dependency

* update test dependency

* Fix namespace conflict

* update test

* add base Agent Interface, add ReWoo Agent

* minor update

* update test

* fix typo

* remove unneeded print

* update rewoo agent

* add LLMTool

* update BaseAgent type

* add ReAct agent

* add ReAct agent

* minor update

* minor update

* minor update

* minor update

* update base reader with BaseComponent

* add splitter

* update agent and tool

* update vectorstores

* update load/save for indexing and retrieving pipeline

* update test_agent for more use-cases

* add missing dependency for test

* update test case for in memory vectorstore

* add TextSplitter to BaseComponent

* update type hint basetool

* add insurance mvp pipeline

* update requirements

* Remove redundant plugins param

* Mock GoogleSearch

---------

Co-authored-by: trducng <trungduc1992@gmail.com>
2023-10-05 12:31:33 +07:00