use PyPi to install OpenNRE

NOTES.md

@@ -18,7 +18,7 @@ TODO:
   * also eval community detection to condense nodes using k-medoids?
   https://medium.com/neo4j/clustering-graph-data-with-k-medoids-3b6a67ea0873
 
-  * add conda packaging - but what about OpenNRE ?
+  * add conda packaging
   https://conda.github.io/grayskull/
 
 

README.md

@@ -44,17 +44,19 @@ Sample code is provided in `demo.py`
 
 ## deploy library from PyPi
 
-To install from [PyPi](https://pypi.python.org/pypi/textgraphs):
+Prepare the virtual environment:
 
 ```bash
-python3 -m pip install -u textgraphs
-python3 -m pip install git+https://github.com/thunlp/OpenNRE
+python3 -m venv venv
+source venv/bin/activate
+python3 -m pip install -U pip wheel setuptools
 ```
 
-NB: both the spaCy and PyPi teams induce packaging errors
-since they have "opinionated" views which conflict against
-each other and also don't quite follow the Python packaging
-standards.
+Install from [PyPi](https://pypi.python.org/pypi/textgraphs):
+
+```bash
+python3 -m pip install -U textgraphs
+```
 
 
 ## run demos locally
@@ -63,27 +65,22 @@ standards.
 python3 demo.py
 ```
 
-```bash
-./venv/bin/jupyter-lab
-```
-
 ```bash
 streamlit run app.py
 ```
 
 
-## install library from a source code repo locally
+## install library from source locally
 
 ```bash
 python3 -m venv venv
 source venv/bin/activate
 
 python3 -m pip install -U pip wheel setuptools
-python3 -m pip install -r requirements.txt
-python3 -m pip install git+https://github.com/thunlp/OpenNRE
+python3 -m pip install -e .
 ```
 
-to run the Streamlit or JupyterLab demos, also install:
+To run the Streamlit or JupyterLab demos, also install:
 
 ```bash
 python3 -m pip install -r requirements-dev.txt

docs/build.md

@@ -12,14 +12,17 @@ simply install based on the instructions in
 
 To set up the build environment locally:
 ```
-python3 -m pip install -r requirements.txt
+python3 -m venv venv
+source venv/bin/activate
+python3 -m pip install -U pip wheel setuptools
+
+python3 -m pip install -e .
 python3 -m pip install -r requirements-dev.txt
 ```
 
 We use *pre-commit hooks* based on [`pre-commit`](https://pre-commit.com/)
 and to configure that locally:
 ```
-python3 -m pip install -r requirements-dev.txt
 pre-commit install --hook-type pre-commit
 ```
 
@@ -96,3 +99,32 @@ To update the [release on PyPi](https://pypi.org/project/textgraphs/):
 ```
 ./bin/push_pypi.sh
 ```
+
+
+## Packaging
+
+Both the spaCy and PyPi teams induce packaging errors since they
+have "opinionated" views which conflict against each other and also
+don't quite follow the [Python packaging standards](https://peps.python.org/pep-0621/).
+
+Moreover, the various dependencies here use a wide range of approaches
+for model downloads: quite appropriately, the spaCy team does not want
+to package their language models on PyPi.
+However, they don't use more contemporary means of model download,
+such as HF transformers, either -- and that triggers logging problems.
+Overall, logging approaches used by the dependencies here for errors/warnings
+are mostly ad-hoc.
+
+These three issues (packaging, model downloads, logging) pose a small nightmare
+for managing Python library packaging downstream.
+To that point, this project implements several workarounds so that
+applications can download from PyPi.
+
+Meanwhile keep watch on developments of the following dependencies,
+if they introduce breaking changes or move toward more standard
+packaging practices:
+
+  * `spaCy` -- model downloads, logging
+  * `OpenNRE` -- PyPi packaging, logging
+  * HF `transformers` and `tokenizers` -- logging
+  * WikiMedia APIs -- SSL certificate expiry

docs/index.md

@@ -4,19 +4,24 @@
 
 Welcome to the **TextGraphs** library...
 
-Large language models (LLMs) come from research in natural language.
-How can LLMs help improve natural language work?
-In particular, how can LLMs benefit use cases that use the results of
-natural language parsing to build knowledge graphs?
+Large language models (LLMs) -- which are perhaps better described as
+_transformers_ -- come from research in natural language.
+How can transformers help improve natural language work?
+In particular, how can these models be leveraged to benefit use cases
+where NLP gets used to build _knowledge graphs_?
 
 If you've been reading the news cycle about AI, one obvious through
 rather naïve response might be
-"Just feed your data to ChatGPT and ask it to generate a knowledge graph."
-Those who've tried this approach understand its consequences:
-incredibly expensive process and poor results, relative to other methods.
+"Feed your data to ChatGPT and just ask it to generate a knowledge graph."
+Those who've tried this approach understand the consequences:
+expensive process and poor results, relative to other methods.
 
-However, are there other ways LLMs can augment natural language workflows?
+That said, are there other ways transformers might help augment
+natural language workflows?
 This project results from an ongoing pursuit of that line of inquiry.
 With sufficiently narrowed task focus and ample software engineering,
-large models can be used to augment specific _components_ of natural
+transformers can be used to augment specific _components_ of natural
 language workflows.
+Several important learnings result from this work, and a key take-away
+is that the challenges mostly involve software engineering and open
+source integration practices.

docs/tldr.md

@@ -1,18 +1,12 @@
 # TL;DR
 
-To install from [PyPi](https://pypi.python.org/pypi/textgraphs):
+Install from [PyPi](https://pypi.python.org/pypi/textgraphs):
 
 ```bash
 python3 -m pip install -u textgraphs
-python3 -m pip install git+https://github.com/thunlp/OpenNRE
 ```
 
-NB: both the spaCy and PyPi teams induce packaging errors since they
-have "opinionated" views which conflict against each other and also
-don't quite follow the [Python packaging standards](https://peps.python.org/pep-0621/).
-Ergo, `OpenNRE` must be installed separately.
-
-To run the demos locally:
+Run the demos locally:
 
 ```bash
 python3 demo.py

pyproject.toml

@@ -70,7 +70,9 @@ dependencies = [
     "markdown2 >= 2.4",
     "matplotlib >= 3.8",
     "networkx >= 3.2",
+    "open-nre >= 0.1.1",
     "pulp >= 2.7",
+    "pyinstrument >= 4.6",
     "pyvis >= 0.3",
     "qwikidata >= 0.4",
     "spacy >= 3.7",
@@ -102,7 +104,6 @@ demo = [
     "ipywidgets >= 8.1",
     "jupyterlab_execute_time >= 3.1",
     "jupyterlab >= 4.0",
-    "pyinstrument >= 4.6",
     "sense2vec >= 2.0",
     "spacy-entity-linker >= 1.0",
     "streamlit < 1.29",

requirements.txt

@@ -3,14 +3,13 @@
 spacy >= 3.7
 https://huggingface.co/spacy/en_core_web_sm/resolve/main/en_core_web_sm-any-py3-none-any.whl
 
-git+https://github.com/thunlp/OpenNRE
-
 beautifulsoup4 >= 4.12
 GitPython >= 3.1
 icecream >= 2.1
 markdown2 >= 2.4
 matplotlib >= 3.8
 networkx >= 3.2
+open-nre >= 0.1.1
 pulp >= 2.7
 pyinstrument >= 4.6
 pyvis >= 0.3

textgraphs/doc.py

@@ -54,7 +54,7 @@
 # override: `OpenNRE` uses `word2vec` which has noisy logging
 logging.disable(logging.INFO)
 
-# override: WikedMedia and others allow their SSL certs to expire
+# override: WikidMedia and others allow their SSL certs to expire
 urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)