Relationship between nlcst-normalize, retext-keywords, lexemes, and full-text search databases

[aegatlin]

Hello,

First of all, I love this ecosystem so much. You are all amazing, and thank you for your work in this space ❤️

I am researching content processing and have come across both nlcst-normalize and retext-keywords and the goals they accomplish feel similar to the goals of building full-text search databases. Also, the workflows seem similar. For example, the workflow described in postgres docs for building a full-text database of documents, which is document->token->normalized lexeme. Postgres tsvectors the outputs of retext-keywords, its stemmer dependency, and nlcst-normalize, feel like they have a lot of overlap. Borderline uncanny.

Am I correct in seeing a correlation here, between some of these unified libs and full-text search engines? If I'm not way off base, then is there some deeper relationship here? Should the next search engine be built with unified? Why or why not? If I'm missing something, what is it? Is there documentation that I missed that elaborates on this topic further?

My goal with the question-soup above is not to have each of those questions answered, but more to communicate the "direction" of my request for comment. I would understand if my question was not specific enough to be worth spending time answering, in which case, no worries! I'll just keep tinkering and reading and hopefully come back with a more coherent question! 😆

Thanks! 😄 ❤️

[wooorm]

Am I correct in seeing a correlation here

Note that the code between the two differs:

• https://github.com/syntax-tree/nlcst-normalize/blob/main/index.js
• https://github.com/retextjs/retext-keywords/blob/main/index.js (note that this also uses retext-pos, which contains a ton of data, through pos-js).

There are indeed overlapping tasks that unified/unist solve compared to text search engines.

Many NLP problems are currently solved with AI/machine learning though (“give a computer a ton of text and let it find patterns”) compared to rule based (“some humans come up with patterns”, which is close to your full-text search engines and unified/unist). So I don’t see the next big search engine being built on unified/unist. But sure you coul create things.

[aegatlin]

Thank you for the links. Now that I've read the code, there appears to be a space between the two libs where some sort of nlcst-lexemes could exists that return a list of lexemes (and maybe counts). Now I'm curious what the relationship is between full-text search "vectorization" and terminology extraction, the latter of which seems to be the spirit behind nlcst-keywords (correct me if I'm wrong or too presumptuous).

Do you see nlcst tooling as solving a particular class of problems? Or, the tooling as not solving a particular class of problems? Goals, anti-goals, etc? When syntax-trees are the core technology I could imagine the "goals" are very open ended, and so people doing whatever they want with the libs is encouraged.

Thank you for taking the time to read and interact with my question. I don't think I'll ever have to manage as many repos as you do, so I know your time is precious, and I just want to say I really appreciate your quick response <3

[wooorm]

Thanks for your kind words!

ASTs exist to allow programs to do whatever, with structured data, instead of a bunch of text. So yes, you can do anything!
It’s more clearly a thing with markup (markdown, html) and programming languages (js, css, etc) than with natural language, but things like retext-equality and retext-indefinite-article are good examples of where ASTs for natural language are useful. For other things, such as natural language detection (franc), and most other natural language processing tasks, it’s not very useful to have an AST. Yet for others, such as term extraction or spellchecking, some parsing is needed, so you can do it on an AST (but that might be more complex than needed), or with a tiny tokenizer (example)