CLDR-16941 Improve coverage for non-latn default/native number symbols/patterns #3690

pedberg-icu · 2024-05-06T22:18:56Z

This PR completes the ticket.

Improve coverage for number system symbols and patterns belonging to non-latn default, native, and finance numbering systems in all locales. In some cases we select coverage by script instead of or in addition to language, to disambiguate cases where languages are written in multiple scripts.
Add a unit test to verify that various number symbols and patterns are at the desired coverage level for each of the two or three number systems (default, native, and possibly finance) specified in a locale, for all locales. This currently takes some time (25 sec on my machine), maybe could be sped up.
Whet this does not do is group collections of number symbols or patterns into sets. That would require coverage variables with alternate values for elements in paths, which is currently not supported. We can discuss for a future enhancement, this would have performance impact.

ALLOW_MANY_COMMITS=true

…s/patterns

AEApple · 2024-05-06T22:32:25Z

Are we also tracking whether the equivalent number formats to those currently in basic where the locale uses native digits by default in a separate ticket for resolution in esub or a future release instead (since they are available at a higher coverage level if we do need to change them in v46)?

pedberg-icu · 2024-05-07T00:05:55Z

Are we also tracking whether the equivalent number formats to those currently in basic where the locale uses native digits by default in a separate ticket for resolution in esub or a future release instead (since they are available at a higher coverage level if we do need to change them in v46)?

@AEApple Not sure I understand the question. We had coverage in certain locales for the number symbols and patterns for default and native number systems, at either moderate or modern coverage depending on the item. What this ticket does is ensure that we do that across all locales for all of the number systems that are default, native, or finance systems in those locales.

pedberg-icu · 2024-05-07T00:06:59Z

common/dtd/ldmlSupplemental.dtd

@@ -1278,7 +1278,7 @@ CLDR data files are interpreted according to the LDML specification (http://unic
 <!ATTLIST coverageLevel inLanguage CDATA #IMPLIED >
    <!--@MATCH:any-->
 <!ATTLIST coverageLevel inScript CDATA #IMPLIED >
-    <!--@MATCH:validity/script-->
+    <!--@MATCH:any-->


This is so we can use coverage variables for inScript, as we already do for inLanguage and inTerritory.

ideally @MATCH:validity/script-with-variables or something

Larger question is whether coverage level belongs in "supplemental data" when, like casing, it's primarily inward tool-focused data. Perhaps we should split some of these out -- and give them their own DTD.

Filed https://unicode-org.atlassian.net/browse/CLDR-17618

macchiati

looks great. A lot of work, though! Longer term, I think we need to set aside some time to review the structure of the coverage levels, and some up with a more concise format to lower the maintenance burden. We already talked about expanding elements; we also might have more predefined variables (that we can construct at boot time based on what's in a particular locale). And other ideas.

srl295 · 2024-05-07T17:53:44Z

common/supplemental/coverageLevels.xml

-		<coverageVariable key="%guruNativeLanguages" value="(pa)"/>
-		<coverageVariable key="%hanidecNativeLanguages" value="(yue|zh)"/>
+		<!-- Number system coverage by language and/or script; default number system items at various levels, native & finance system items all at modern -->
+		<coverageVariable key="%adlmDefaultScripts" value="(Adlm)"/>


I still don't think any of this belongs in data at all. It should be read from the locale files themselves and cached. But.. it's an improvement.

Covered by https://unicode-org.atlassian.net/browse/CLDR-17618

srl295 · 2024-05-07T17:54:55Z

common/supplemental/coverageLevels.xml

@@ -265,29 +283,34 @@ For terms of use, see http://www.unicode.org/copyright.html
 		<coverageLevel	value="moderate"	inLanguage="%traditionalCollationLanguages"	match="localeDisplayNames/types/type[@key='collation'][@type='traditional']"/>
 		<coverageLevel	value="moderate"	inLanguage="%CJK_Languages"	match="localeDisplayNames/types/type[@key='collation'][@type='unihan']"/>
 		<coverageLevel	value="moderate"	 	match="localeDisplayNames/types/type[@key='numbers'][@type='latn']"/>
-		<coverageLevel	value="moderate"	inLanguage="zh"	match="localeDisplayNames/types/type[@key='numbers'][@type='(han(s|t)(fin)?|hanidec)']"/>
+		<coverageLevel	value="moderate"	inLanguage="%hanidecNativeLanguages"	match="localeDisplayNames/types/type[@key='numbers'][@type='(han(s|t)(fin)?|hanidec)']"/>


Suggested change

<coverageLevel value="moderate" inLanguage="%hanidecNativeLanguages" match="localeDisplayNames/types/type[@key='numbers'][@type='(han(s|t)(fin)?|hanidec)']"/>

<coverageLevel value="moderate" inLanguage="%hanidecNativeLanguages" match="localeDisplayNames/types/type[@key='numbers'][@type='(hans|hant)(fin)?|hanidec)']"/>

more readable.. or, spell all of them out.

Addressed this in the follow-on PR.

srl295 · 2024-05-07T17:55:25Z

common/supplemental/coverageLevels.xml

+		<coverageLevel	value="moderate"	inScript="%mteiDefaultScripts"	match="numbers/symbols[@numberSystem='mtei']/group"/>
+		<coverageLevel	value="moderate"	inScript="%mteiDefaultScripts"	match="numbers/symbols[@numberSystem='mtei']/minusSign"/>
+		<coverageLevel	value="moderate"	inScript="%mteiDefaultScripts"	match="numbers/symbols[@numberSystem='mtei']/percentSign"/>
+		<coverageLevel	value="moderate"	inScript="%mteiDefaultScripts"	match="numbers/symbols[@numberSystem='mtei']/plusSign"/>


i'll say again, I appreciate this being done.

srl295 · 2024-05-07T17:57:40Z

looks great. A lot of work, though! Longer term, I think we need to set aside some time to review the structure of the coverage levels, and some up with a more concise format to lower the maintenance burden. We already talked about expanding elements; we also might have more predefined variables (that we can construct at boot time based on what's in a particular locale). And other ideas.

drop the data from coverage*.xml entirely and derive it from the locale files

pedberg-icu · 2024-05-08T20:57:01Z

looks great. A lot of work, though! Longer term, I think we need to set aside some time to review the structure of the coverage levels, and some up with a more concise format to lower the maintenance burden. We already talked about expanding elements; we also might have more predefined variables (that we can construct at boot time based on what's in a particular locale). And other ideas.

drop the data from coverage*.xml entirely and derive it from the locale files

Covered by https://unicode-org.atlassian.net/browse/CLDR-17618

CLDR-16941 Improve coverage for non-latn default/native number symbol…

e50cb02

…s/patterns

pedberg-icu requested review from srl295, macchiati, AEApple and DraganBesevic May 6, 2024 22:18

pedberg-icu self-assigned this May 6, 2024

pedberg-icu commented May 7, 2024

View reviewed changes

macchiati approved these changes May 7, 2024

View reviewed changes

srl295 approved these changes May 7, 2024

View reviewed changes

pedberg-icu merged commit d2123f4 into unicode-org:main May 7, 2024
10 checks passed

pedberg-icu deleted the CLDR-16941-better-coverage-for-number-systems branch May 7, 2024 18:07

pedberg-icu mentioned this pull request May 8, 2024

CLDR-16941 Further number symbols/formats coverage fixes per TC discussion #3702

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CLDR-16941 Improve coverage for non-latn default/native number symbols/patterns #3690

CLDR-16941 Improve coverage for non-latn default/native number symbols/patterns #3690

pedberg-icu commented May 6, 2024 •

edited

Loading

AEApple commented May 6, 2024

pedberg-icu commented May 7, 2024

pedberg-icu May 7, 2024 •

edited

Loading

srl295 May 7, 2024

pedberg-icu May 8, 2024

macchiati left a comment

srl295 May 7, 2024

pedberg-icu May 8, 2024

srl295 May 7, 2024

pedberg-icu May 8, 2024

srl295 May 7, 2024

srl295 commented May 7, 2024

pedberg-icu commented May 8, 2024

	<coverageLevel value="moderate" inLanguage="%hanidecNativeLanguages" match="localeDisplayNames/types/type[@key='numbers'][@type='(han(s\|t)(fin)?\|hanidec)']"/>
	<coverageLevel value="moderate" inLanguage="%hanidecNativeLanguages" match="localeDisplayNames/types/type[@key='numbers'][@type='(hans\|hant)(fin)?\|hanidec)']"/>

CLDR-16941 Improve coverage for non-latn default/native number symbols/patterns #3690

CLDR-16941 Improve coverage for non-latn default/native number symbols/patterns #3690

Conversation

pedberg-icu commented May 6, 2024 • edited Loading

AEApple commented May 6, 2024

pedberg-icu commented May 7, 2024

pedberg-icu May 7, 2024 • edited Loading

Choose a reason for hiding this comment

srl295 May 7, 2024

Choose a reason for hiding this comment

pedberg-icu May 8, 2024

Choose a reason for hiding this comment

macchiati left a comment

Choose a reason for hiding this comment

srl295 May 7, 2024

Choose a reason for hiding this comment

pedberg-icu May 8, 2024

Choose a reason for hiding this comment

srl295 May 7, 2024

Choose a reason for hiding this comment

pedberg-icu May 8, 2024

Choose a reason for hiding this comment

srl295 May 7, 2024

Choose a reason for hiding this comment

srl295 commented May 7, 2024

pedberg-icu commented May 8, 2024

pedberg-icu commented May 6, 2024 •

edited

Loading

pedberg-icu May 7, 2024 •

edited

Loading