Refactor language utility data out from utils.py

[wkyoshida]

Terms

• I have searched open and closed issues
• I agree to follow Scribe-Data's Code of Conduct

Issue

This issue is for refactoring the language data out from src/scribe_data/utils.py to a separate file, likely a JSON - as suggested by @m-charlton in #51 🙌

As proposed by @m-charlton,

This file would be loaded once on module import and then the utils functions could interrogate this loaded object.

We can use this issue to track discussion of details for the implementation and the work itself.

[wkyoshida]

I do like how this could facilitate the inclusion of new languages in the future as well 🙌 Instead of having to hunt down where all modifications would be needed in code, a lot of it can be centralized in this JSON file.

@m-charlton we can absolutely go over what format might work best too, as you've already started the discussion for. Could using a gist though make it easier perhaps to collaborate? To make comments and revisions on the file?

[m-charlton]

@wkyoshida thanks for opening this issue. The following gist contains the truncated version of the language data file (only two languages are included).

The "used by" & "description" fields are comments,the data proper are held in "languages". My main question concerns the placement of the "ignore" & "remove" fields. Is the following to be preferred?

  "remove-words": [],
  "ignore-words": []

A second question concerns the location of the resources directory, containing the JSON data file: currently it's in Scribe-Data/src/scribe_data and called resources. Is this the best location? I notice that there's a _resources directory in extract_transform

[wkyoshida]

I'm thinking just having the structure of

  "remove-words": [],
  "ignore-words": []

as you mentioned makes sense actually. Tying them under a "words" property maybe feels like some unnecessary nesting perhaps? The common denominator is really only that both lists are of, well.., words hahah 😆 Do others feel the same? Definitely open to discussion if there's benefit to the current structure that I'm overlooking.

[andrewtavis]

Makes sense to me, @wkyoshida :)

[andrewtavis]

951bf86 sent along some minor changes to fix what we discussed in the sync and also improved the docstrings in a few places. Thanks for the note that one was copied and not updated, @m-charlton! I think the next thing to do here would be #55 so we simplify this process even more. I noted in #54 what parts would be able to be removed :)