Lots of issues with tinytex complaining about tlmgr being out of date and not being able to find and install packages

[dmkaplan2000]

I have tried clean installs of tinytex on two different machines running Ubuntu 20 and 22 and I have had lots of issues with tinytex not being able to install necessary ctan packages, with it complaining that the tlmgr remote database is out of date and then it not being able to find and install the necessary ctan packages even though I am able to do so from the command line (though sometimes requiring multiple attempts to update tlmgr). An example of the error messages is copied below and I put the standard debugging info at the end of this report.

R commands to reproduce my error are:

tinytex::install_tinytex()
rmarkdown::draft("test-oup.Rmd", template = "oup_v1", package = "rticles")
rmarkdown::render("test-oup/test-oup.Rmd")

Example output of the knit process is:

processing file: test-oup.Rmd
                                                                                                        
/usr/lib/rstudio/resources/app/bin/quarto/bin/tools/pandoc +RTS -K512m -RTS test-oup.knit.md --to latex --from markdown+tex_math_single_backslash-autolink_bare_uris --output test-oup.tex --lua-filter /home/dmk/R/x86_64-pc-linux-gnu-library/4.3/rmarkdown/rmarkdown/lua/pagebreak.lua --lua-filter /home/dmk/R/x86_64-pc-linux-gnu-library/4.3/rmarkdown/rmarkdown/lua/latex-div.lua --embed-resources --standalone --template /home/dmk/R/x86_64-pc-linux-gnu-library/4.3/rticles/rmarkdown/templates/oup_v1/resources/template.tex --number-sections --highlight-style tango --pdf-engine pdflatex --natbib --variable 'journal=Journal Name' --variable papersize=large --variable document_style=contemporary --variable namedate --variable pandoc3 --include-in-header /tmp/RtmprnKfKb/rmarkdown-str5efb82fd21351.html --include-in-header /tmp/RtmprnKfKb/rmarkdown-str5efb81734e1f5.html 
output file: test-oup.knit.md


tlmgr: Remote database (revision 70524 of the texlive-scripts package)
seems to be older than the local installation (rev 70546 of
texlive-scripts); please use a different mirror and/or wait a day or two.

tlmgr update --self
tlmgr: package repository https://distrib-coffee.ipsl.jussieu.fr/pub/mirrors/ctan/systems/texlive/tlnet (verified)
[1/1, ??:??/??:??] install: subfloat [2k]
running mktexlsr ...
done running mktexlsr.
tlmgr: package log updated: /home/dmk/.TinyTeX/texmf-var/web2c/tlmgr.log
tlmgr: command log updated: /home/dmk/.TinyTeX/texmf-var/web2c/tlmgr-commands.log

tlmgr: Remote database (revision 70524 of the texlive-scripts package)
seems to be older than the local installation (rev 70546 of
texlive-scripts); please use a different mirror and/or wait a day or two.

! LaTeX Error: File `anyfontsize.sty' not found.

! Emergency stop.
<read *> 

Error: LaTeX failed to compile test-oup.tex. See https://yihui.org/tinytex/r/#debugging for debugging tips. See test-oup.log for more info.
In addition: Warning message:
In system2("tlmgr", args, ...) :
  running command ''tlmgr' search --file --global '/anyfontsize.sty'' had status 1
Execution halted

Here it complains about not being able to find anyfontsize.sty, but if I go to the shell and execute:

'tlmgr' search --file --global '/anyfontsize.sty'
'tlmgr' install anyfontsize

It works just fine. Doing this I can slowly walk my way through all the CTAN dependencies, but my understanding is that tinytex is supposed to figure this out all by itself...

The debugging info (for the Ubuntu 20 machine; the newer Ubuntu 22 machine is elsewhere) is:

> xfun::session_info('rmarkdown')
R version 4.3.3 (2024-02-29)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04.6 LTS, RStudio 2023.12.1.402

Locale:
  LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_GB.UTF-8        LC_COLLATE=en_US.UTF-8    
  LC_MONETARY=en_GB.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_GB.UTF-8       LC_NAME=C                 
  LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_GB.UTF-8 LC_IDENTIFICATION=C       

Package version:
  base64enc_0.1.3   bslib_0.5.1       cachem_1.0.8      digest_0.6.33     ellipsis_0.3.2    evaluate_0.21     fastmap_1.1.1    
  fontawesome_0.5.2 fs_1.6.3          graphics_4.3.3    grDevices_4.3.3   highr_0.10        htmltools_0.5.6   jquerylib_0.1.4  
  jsonlite_1.8.7    knitr_1.45        memoise_2.0.1     methods_4.3.3     mime_0.12         R6_2.5.1          rappdirs_0.3.3   
  rlang_1.1.1       rmarkdown_2.26    sass_0.4.7        stats_4.3.3       tinytex_0.49      tools_4.3.3       utils_4.3.3      
  xfun_0.42         yaml_2.3.7       

Pandoc version: 3.1.1

---

By filing an issue to this repo, I promise that

• [ X] I have fully read the issue guide at https://yihui.org/issue/.
• [ X] I have provided the necessary information about my issue.
  • If I'm asking a question, I have already asked it on Stack Overflow or RStudio Community, waited for at least 24 hours, and included a link to my question there.
  • If I'm filing a bug report, I have included a minimal, self-contained, and reproducible example, and have also included xfun::session_info('tinytex'). I have upgraded all my packages to their latest versions (e.g., R, RStudio, and R packages), and also tried the development version: remotes::install_github('rstudio/tinytex').
  • If I have posted the same issue elsewhere, I have also mentioned it in this issue.
• [ X] I have learned the Github Markdown syntax, and formatted my issue correctly.

I understand that my issue may be closed if I don't fulfill my promises.

[remlapmot]

This might be because TeXLive 2024 is about to come out https://tug.org/texlive/

[dmkaplan2000]

Could be, but it has been going on for at least a few days already...

[cderv]

tlmgr: Remote database (revision 70524 of the texlive-scripts package)
seems to be older than the local installation (rev 70546 of
texlive-scripts); please use a different mirror and/or wait a day or two.

This is usually when the CTAN mirror used is outdated compared to the local version you have installed. Did your mirror changed recently ?

Sometimes there could be issue and some mirrors could be not synced correctly. See https://ctan.org/mirrors/mirmon
For example there is currently one in Australia with is 5.9 days old:

This is unfortunate but this could happen

tlmgr: package repository https://distrib-coffee.ipsl.jussieu.fr/pub/mirrors/ctan/systems/texlive/tlnet (verified)

It seems this is the one used currently. You can change with tinytex::tlmgr_repo()
Calling without a URL will give you your current setup. And passing a URL you can change the option. It will call tlmgr options repository

It works just fine. Doing this I can slowly walk my way through all the CTAN dependencies, but my understanding is that tinytex is supposed to figure this out all by itself...

Is this tlmgr you are calling the one from TinyTeX or is it another install (system-wide maybe) ? If this is another, I wonder if it is not using another CTAN repo.

This might be because TeXLive 2024 is about to come out

Could be too! We are in the period where there will be a day or two with possible broken install and broken CTAN mirror that are yet to be sync up with TeX Live 2024.

[dmkaplan2000]

tlmgr: Remote database (revision 70524 of the texlive-scripts package)
seems to be older than the local installation (rev 70546 of
texlive-scripts); please use a different mirror and/or wait a day or two.

This is usually when the CTAN mirror used is outdated compared to the local version you have installed. Did your mirror changed recently ?

I have never touched the mirror used by tlmgr, it is whatever the default is upon a clean install.

This is unfortunate but this could happen

tlmgr: package repository https://distrib-coffee.ipsl.jussieu.fr/pub/mirrors/ctan/systems/texlive/tlnet (verified)

It seems this is the one used currently. You can change with tinytex::tlmgr_repo() Calling without a URL will give you your current setup. And passing a URL you can change the option. It will call tlmgr options repository

> tinytex::tlmgr_repo()
tlmgr option repository
Default package repository (repository): https://mirror.ctan.org/systems/texlive/tlnet

It works just fine. Doing this I can slowly walk my way through all the CTAN dependencies, but my understanding is that tinytex is supposed to figure this out all by itself...

Is this tlmgr you are calling the one from TinyTeX or is it another install (system-wide maybe) ? If this is another, I wonder if it is not using another CTAN repo.

I have only one tlmgr - the one installed by tinytex in $HOME/bin. I should say that "works fine" should be in quotes. Very regularly (i.e., every minute...) it requires me to execute tlmgr update --all --self. If this works, then I have a few seconds to install the package I am interested in. This suggests that on the server side they are doing updates that puts my database out of date every few minutes or seconds.

This might be because TeXLive 2024 is about to come out

Could be too! We are in the period where there will be a day or two with possible broken install and broken CTAN mirror that are yet to be sync up with TeX Live 2024.

This sounds like the likely explanation. It seems there should be a cleaner way to do this...

[cderv]

Default package repository (repository): https://mirror.ctan.org/systems/texlive/tlnet

So this is the redirection mirror - See https://ctan.org/mirrors/

In order to reduce network load, it is recommended that you use the Comprehensive TEX Archive Network (CTAN) host which is located in the closest network proximity to your site. Alternatively, you may wish to obtain a copy of the CTAN via DVD.

CTAN has a mirror multiplexor service. Point your browser to https://mirrors.ctan.org/
for automatic redirection to a mirror in or close to your country.
Note that the multiplexor always redirects to a https mirror. If you intend to us another protocol then you have to do it manually. For this the list also contains non-https URLs.

So this will redirect to nearest mirror - this could change from time to time maybe. I do think the fact your ended up with a outdated mirror was bad timing - or current problem / slowness on mirror syncing.

Not sure we can do something about it in TinyTeX. You can configure your repo to another one you think is more reliable.

Very regularly (i.e., every minute...) it requires me to execute tlmgr update --all --self.

This is really odd behavior. We do run tlmgr update --all --self for you automatically

tinytex/R/tlmgr.R

Lines 114 to 115 in 54eb1fd

	# if any packages have been installed, update packages first
	if (getOption('tinytex.tlmgr_update', TRUE) && any(check_installed(pkgs))) update_pkgs()

tinytex/R/tlmgr.R

Lines 118 to 121 in 54eb1fd

	if (res != 0 || any(!check_installed(pkgs))) {
	update_pkgs(all = FALSE) # update tlmgr itself since it might be outdated
	res = tlmgr(c('install', pkgs), usermode, ...)
	}

TeX Live does update often so it requires updating regularly, but I don't recall for it to be every minute.

@yihui did you encounter some behavior as described by @dmkaplan2000 ? Do you have experience of something that would explain it ?

[dmkaplan2000]

To be absolutely clear about the behavior, sometimes when I execute tlmgr update --all --self, the output of that says that it is out of data (tlmgr: Remote database (revision 70524 of the texlive-scripts package) seems to be older than the local installation (rev 70546 of...). In those case, I rerun the comment until it magically says it is up to date and then I immediately execute tlmgr install and sometimes it works, whereas other times it says that tlmgr is out of date, in which case I start the process over again...

[cderv]

I think you should try setup another mirror as your default config. It seems the redirection will send you quite often to a mirror that is not yet synced. tinytex::install_tinytex() has a repository argument to allow changing the default at install time.

[yihui]

I think you should try setup another mirror as your default config. It seems the redirection will send you quite often to a mirror that is not yet synced. tinytex::install_tinytex() has a repository argument to allow changing the default at install time.

That's my guess too. CTAN's default choice of mirrors can be problematic because the choice is random to some degree (i.e., the "fastest" mirror is not a fixed one). You could end up using different mirrors each time you call tlmgr, which means you might be using an older mirror last time but a newer one next time.

I have been aware of this problem for a few years now. I just thought more about it, and I think it may be a good idea to pin the CTAN repo after TinyTeX installation. FWIW, I have done that only for CI servers (such as Github Action runners) in ca4a834.

The tricky thing is to decide which CTAN repo to use. One option is CTAN's automatic redirect:

# in R
curlGetHeaders('https://mirror.ctan.org/systems/texlive/tlnet')

# in shell
curl -I https://mirror.ctan.org/systems/texlive/tlnet

But this repo could be older than the repo that we use to build TinyTeX, i.e., the Illinois mirror:

tinytex/.github/workflows/build.yaml

Line 38 in 54eb1fd

CTAN_REPO: https://ctan.math.illinois.edu/systems/texlive/tlnet

Then users will be in the same trouble (remote older than local). I think it may be still worth doing, though. If the problem arises, at least it will arise consistently...

[yihui]

@dmkaplan2000 For now, I think the workaround for you is to find a mirror that looks up-to-date at https://ctan.org/mirrors/mirmon, right-click on the URL to copy it, and paste the URL to tinytex::tlmgr_repo(). That will lock TinyTeX to a fixed CTAN repo.

[dmkaplan2000]

Thanks, that explains a lot - I didn't understand their system of randomly selecting a mirror. I will set the mirror as you suggest.

[dmkaplan2000]

Would always using the mirror that tinytex is built with (https://ctan.math.illinois.edu/systems/texlive/tlnet) mean that we never have this problem? Perhaps not as it seems that "up to date" is a function of a central server that is different from the mirrors where things are downloaded. But if this were the case, this would seem like a simple fix. Downloads might take a bit longer as I am in France whereas the server is in Illinois, but this seems like a relatively small price to pay for stability...

[yihui]

That's what I did for a while in the past. I don't remember clearly why I stopped doing so, but there are a few possible reasons:

1. It could be relatively slow to some users.
2. I'm not sure if pinning to the Illinois mirror will route too much traffic to their server.
3. The server could be down, although I guess it should be rare.

The ideal solution is a CTAN mirror based on CDN (like the RStudio/Posit CRAN mirror), and let CDN take care of routing globally, instead of letting mirror.ctan.org choose a physical server. Some years ago, I actually asked RStudio about the possibility of hosting CTAN on CDN (precisely because of this problem), but it didn't work out.

---

Oh wait, stupid me. I have actually implemented the idea mentioned in my previous reply via 626949f two years ago! I have totally forgotten it... Thanks to this thread, I just discovered two subtle bugs there upon a close inspection. I also discovered that actually there do exist CDN mirrors, such as https://www.matthewthom.as/mirrors/ Then I feel it may be possible default to a certain CDN mirror that looks reliable enough.

[yihui]

All mirrors based on CloudFlare:

x = readLines('https://ctan.org/mirrors/mirmon')
urls = xfun::grep_sub('.*<TD ALIGN=RIGHT><A HREF="(https://[^"]+)".*', '\\1', x)
s = unlist(lapply(urls, function(u) {
  h = tryCatch(curlGetHeaders(u), error = function(e) character())
  h = tolower(tail(grep('^server: ', h, value = TRUE, ignore.case = TRUE), 1))
  if (length(h)) h else NA
}))

cat(urls[grep('cloudflare', s)], sep = '\n')

https://eu.mirrors.cicku.me/ctan/
https://au.mirrors.cicku.me/ctan/
https://ca.mirrors.cicku.me/ctan/
https://de.mirrors.cicku.me/ctan/
https://in.mirrors.cicku.me/ctan/
https://jp.mirrors.cicku.me/ctan/
https://kr.mirrors.cicku.me/ctan/
https://sg.mirrors.cicku.me/ctan/
https://ctan.javinator9889.com/
https://us.mirrors.cicku.me/ctan/
https://mirror.mwt.me/ctan/

I'm a little hesitant to default to any of these mirrors since they seem to be backed by individual contributors. If there were a .edu mirror based on CloudFlare, I'd be more confident to make it the default.

Anyway, I have fixed the bugs in the default selection of mirrors. TinyTeX installed via tinytex::install_tinytex() shouldn't use a random mirror any more.

remotes::install_github('rstudio/tinytex')

[cderv]

All mirrors based on CloudFlare:

@yihui do you think that would be useful to provide a helper function to get a list of these if someone is willing to set manually to one of them ? We would set by default, but we could provide helpers to help set to one of them 🤷

Maybe the fix is enough though - good catch !!

Thanks for asking and reporting @dmkaplan2000 ! Your contributions are always relevant and very welcome ! Thanks !

[dmkaplan2000]

Thanks for asking and reporting @dmkaplan2000 ! Your contributions are always relevant and very welcome ! Thanks !

@cderv that is a really nice thing to say. Sometimes it is hard to find the time and energy to file a proper bug report, so it is nice to see that it is worth something.

[yihui]

@cderv I added tinytex:::ctan_mirrors() yesterday: bc8fda0. For the code that finds CloudFlare-based servers, it takes some time to run (it needs to query about 120 servers), and the servers returned may not really be faster than the one returned by the automatic redirect of mirror.ctan.org, so I'm not sure if this code is really useful.

Sometimes it is hard to find the time and energy to file a proper bug report, so it is nice to see that it is worth something.

@dmkaplan2000 Yes, it definitely requires time and energy to file a good bug report. We really appreciate your effort!

[cicku]

@yihui If you have any issues with my mirror, feel free to reach out and clarify. I handle TB/PB levels of traffic everyday for many projects.