feat: Semantic splitter

[simjak]

• Added Semantic Splitter
• Integrated semantic splitter to with Unstructured elements grouping by title
• Include fake title detection
• Additional improvements (BaseChunk model, batch embeddings, batch uploads)

Follow the walkthrough.ipynb for how to use

[homanp]

Amazing @simjak

[homanp]

@simjak

getting the following after running poetry install

INFO:     Will watch for changes in these directories: ['/Users/ismailpelaseyed/Projects/super-rag']
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [41325] using StatReload
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/homebrew/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/ismailpelaseyed/Projects/super-rag/.venv/lib/python3.11/site-packages/uvicorn/_subprocess.py", line 78, in subprocess_started
    target(sockets=sockets)
  File "/Users/ismailpelaseyed/Projects/super-rag/.venv/lib/python3.11/site-packages/uvicorn/server.py", line 62, in run
    return asyncio.run(self.serve(sockets=sockets))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/ismailpelaseyed/Projects/super-rag/.venv/lib/python3.11/site-packages/uvicorn/server.py", line 69, in serve
    config.load()
  File "/Users/ismailpelaseyed/Projects/super-rag/.venv/lib/python3.11/site-packages/uvicorn/config.py", line 458, in load
    self.loaded_app = import_from_string(self.app)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ismailpelaseyed/Projects/super-rag/.venv/lib/python3.11/site-packages/uvicorn/importer.py", line 24, in import_from_string
    raise exc from None
  File "/Users/ismailpelaseyed/Projects/super-rag/.venv/lib/python3.11/site-packages/uvicorn/importer.py", line 21, in import_from_string
    module = importlib.import_module(module_str)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/ismailpelaseyed/Projects/super-rag/main.py", line 6, in <module>
    from router import router
  File "/Users/ismailpelaseyed/Projects/super-rag/router.py", line 3, in <module>
    from api import delete, ingest, query
  File "/Users/ismailpelaseyed/Projects/super-rag/api/delete.py", line 4, in <module>
    from service.embedding import get_encoder
  File "/Users/ismailpelaseyed/Projects/super-rag/service/embedding.py", line 11, in <module>
    from semantic_router.encoders import (
  File "/Users/ismailpelaseyed/Projects/super-rag/.venv/lib/python3.11/site-packages/semantic_router/__init__.py", line 1, in <module>
    from semantic_router.hybrid_layer import HybridRouteLayer
  File "/Users/ismailpelaseyed/Projects/super-rag/.venv/lib/python3.11/site-packages/semantic_router/hybrid_layer.py", line 6, in <module>
    from semantic_router.encoders import (
  File "/Users/ismailpelaseyed/Projects/super-rag/.venv/lib/python3.11/site-packages/semantic_router/encoders/__init__.py", line 8, in <module>
    from semantic_router.encoders.tfidf import TfidfEncoder
  File "/Users/ismailpelaseyed/Projects/super-rag/.venv/lib/python3.11/site-packages/semantic_router/encoders/tfidf.py", line 10, in <module>
    from semantic_router.route import Route
  File "/Users/ismailpelaseyed/Projects/super-rag/.venv/lib/python3.11/site-packages/semantic_router/route.py", line 5, in <module>
    from PIL.Image import Image
ModuleNotFoundError: No module named 'PIL'

[simjak]

Solved this semantic router version 0.0.25

[homanp]

Solved this semantic router version 0.0.25

Updated dependencies but still seeing the issue.

[homanp]

I did some small fixes and changes but seems to work well, good job! These changes have some braking changes with the encoder provider in the API so I need to first migrate to that before merging.

Or we should set the new splitter config as non mandatory and just set a default? Perhaps you could add that in?