Fix integrating indexing and retrieval pipelines to FileIndex (#155)

* Add docs for settings
* Add mdx_truly_sane_lists to doc requirements

docs/pages/app/index/file.md

@@ -17,11 +17,10 @@ The ktem has default indexing pipeline: `ktem.index.file.pipelines.IndexDocument
 
 This default pipeline works as follow:
 
-- Input: list of file paths
-- Output: list of nodes that are indexed into database
-- Process:
-  - Read files into texts. Different file types has different ways to read
-    texts.
+- **Input**: list of file paths
+- **Output**: list of nodes that are indexed into database
+- **Process**:
+  - Read files into texts. Different file types has different ways to read texts.
   - Split text files into smaller segments
   - Run each segments into embeddings.
   - Store the embeddings into vector store. Store the texts of each segment
@@ -55,7 +54,7 @@ You should define the following methods:
   fully-initialized pipeline, ready to be used by ktem.
   - `user_settings`: is a dictionary contains user settings (e.g. `{"pdf_mode": True, "num_retrieval": 5}`). You can declare these settings in the `get_user_settings` classmethod. ktem will collect these settings into the app Settings page, and will supply these user settings to your `get_pipeline` method.
   - `index_settings`: is a dictionary. Currently it's empty for File Index.
-- `get_user_settings`: to declare user settings, eturn a dictionary.
+- `get_user_settings`: to declare user settings, return a dictionary.
 
 By subclassing `BaseFileIndexIndexing`, You will have access to the following resources:
 
@@ -82,6 +81,30 @@ follow:
     if the user restrict.
   - Return the matched text segments
 
+### Create your own retrieval pipeline
+
+Your retrieval pipeline will subclass `BaseFileIndexRetriever`. The retriever
+has the same database, vectorstore and docstore accesses like the indexing
+pipeline.
+
+You should define the following methods:
+
+- `run(self, query, file_ids)`: retrieve relevant documents relating to the
+  query. If `file_ids` is given, you should restrict your search within these
+  `file_ids`.
+- `get_pipeline(cls, user_settings, index_settings, selected)`: return the
+  fully-initialized pipeline, ready to be used by ktem.
+  - `user_settings`: is a dictionary contains user settings (e.g. `{"pdf_mode": True, "num_retrieval": 5}`). You can declare these settings in the `get_user_settings` classmethod. ktem will collect these settings into the app Settings page, and will supply these user settings to your `get_pipeline` method.
+    - `index_settings`: is a dictionary. Currently it's empty for File Index.
+    - `selected`: a list of file ids selected by user. If user doesn't select
+      anything, this variable will be None.
+- `get_user_settings`: to declare user settings, return a dictionary.
+
+Once you build the retrieval pipeline class, you can register it in
+`flowsettings.py`: `FILE_INDEXING_RETRIEVER_PIPELIENS = ["path.to.retrieval.pipelie"]`. Because there can be
+multiple parallel pipelines within an index, this variable takes a list of
+string rather than a string.
+
 ## Software infrastructure
 
 | Infra            | Access        | Schema                                                                                                                                                                                                                                                                                             | Ref                                                            |

docs/pages/app/settings/overview.md

@@ -0,0 +1,8 @@
+# Overview
+
+There are 3 kinds of settings in `ktem`, geared towards different stakeholders
+for different use cases:
+
+- Developer settings. These settings are meant for very basic app customization, such as database URL, cloud config, logging config, which features to enable... You will be interested in the developer settings if you deploy `ktem` to your customers, or if you build extension for `ktem` for developers. These settings are declared inside `flowsettings.py`.
+- Admin settings. These settings show up in the Admin page, and are meant to allow admin-level user to customize low level features, such as which credentials to connect to data sources, which keys to use for LLM...
+- [User settings](/pages/app/settings/user-settings/). These settings are meant for run-time users to tweak ktem to their personal needs, such as which output languages the chatbot should generate, which reasoning type to use...

docs/pages/app/settings/user-settings.md

@@ -0,0 +1,52 @@
+# User settings
+
+`ktem` allows developers to extend the index and the reasoning pipeline. In
+many cases, these components can have settings that should be modified by
+users at run-time, (e.g. `topk`, `chunksize`...). These are the user settings.
+
+`ktem` allows developers to declare such user settings in their code. Once
+declared, `ktem` will render them in a Settings page.
+
+There are 2 places that `ktem` looks for declared user settings. You can
+refer to the respective pages.
+
+- In the index.
+- In the reasoning pipeline.
+
+## Syntax of a settings
+
+A collection of settings is a dictionary of type `dict[str, dict]`, where the
+key is a setting id, and the value is the description of the setting.
+
+```python
+settings = {
+    "topk": {
+        "name": "Top-k chunks",
+        "value": 10,
+        "component": "number",
+    },
+    "lang": {
+        "name": "Languages",
+        "value": "en",
+        "component": "dropdown",
+        "choices": [("en", "English"), ("cn", "Chinese")],
+    }
+}
+```
+
+Each setting description must have:
+
+- name: the human-understandable name of the settings.
+- value: the default value of the settings.
+- component: the UI component to render such setting on the UI. Available:
+
+  - "text": single-value
+  - "number": single-value
+  - "checkbox": single-value
+  - "dropdown": choices
+  - "radio": choices
+  - "checkboxgroup": choices
+
+- choices: the list of choices, if the component type allows.
+
+## Settings page structure