Faster preview

[YodaEmbedding]

Telescope freezes when attempting to preview large files. For instance, previewing a 100000 line CSV file causes a 10 second long freeze.

Possible solutions, in order of easiest to most difficult to implement (I presume):

• Disable preview for files >1MB.
• cat file.csv | head -c 1000000 | head -n 1000 > file.csv.preview so that a preview does not need to be generated for too much data. Here, file.csv.preview is a smaller file (<1MB and <1000 lines) that can read by telescope.
• Async preview. The preview will still take time and CPU to generate, but at least telescope will not freeze.

[Conni2461]

We have considered all the above and we found a pretty good solution, (in my opinion, but i also designed them sooooooo 🤣 ) already.
In fact previewers are already async :) File reading happens off thread, but neovim does not allow to write to the buffer async as well as highlighting which still happens in main thread. And the last thing is mostly what makes it slow. But i have never displayed a 100000 line CSV file 😆

So we decided to make buffer_previewer_maker overridable. Its the heart of the buffer previewer and if you want some information about how previewers work :help telescope.previewers.

I wanted to make an extension that lets you not have to override the function but i never found the interest in doing that. So here are some snippets:

Disable highlighting for certain files:

local previewers = require('telescope.previewers')

local _bad = { '.*%.csv', '.*%.lua' } -- Put all filetypes that slow you down in this array
local bad_files = function(filepath)
  for _, v in ipairs(_bad) do
    if filepath:match(v) then
      return false
    end
  end

  return true
end

local new_maker = function(filepath, bufnr, opts)
  opts = opts or {}
  if opts.use_ft_detect == nil then opts.use_ft_detect = true end
  opts.use_ft_detect = opts.use_ft_detect == false and false or bad_files(filepath)
  previewers.buffer_previewer_maker(filepath, bufnr, opts)
end

require('telescope').setup {
  defaults = {
    buffer_previewer_maker = new_maker,
  }
}

Ignore files bigger than a threshold. Make sure you are on current master, i pushed a fix to make this work:

local previewers = require('telescope.previewers')

local new_maker = function(filepath, bufnr, opts)
  opts = opts or {}

  filepath = vim.fn.expand(filepath)
  vim.loop.fs_stat(filepath, function(_, stat)
    if not stat then return end
    if stat.size > 100000 then
      return
    else
      previewers.buffer_previewer_maker(filepath, bufnr, opts)
    end
  end)
end

require('telescope').setup {
  defaults = {
    buffer_previewer_maker = new_maker,
  }
}

[zetashift]

I had a file with that's basically one big line, that made Telescope freeze too, it's quite a small file, so Id have to write a custom previewer function for that too?

[marcusbuffett]

This may be understood already, since the issue is still open, but I think some better default handling of this would be great. I was going to switch back to clap before I found this workaround, would be nicer IMO if there was some sensible default, like timeout after 200ms or something.

[Akeboshiwind]

In case others find this useful:
I've modified Conni2461 example to truncate files instead of skipping them.

local previewers = require('telescope.previewers')
local previewers_utils = require('telescope.previewers.utils')

local max_size = 100000
local truncate_large_files = function(filepath, bufnr, opts)
    opts = opts or {}
    
    filepath = vim.fn.expand(filepath)
    vim.loop.fs_stat(filepath, function(_, stat)
        if not stat then return end
        if stat.size > max_size then
            local cmd = {"head", "-c", max_size, filepath}
            previewers_utils.job_maker(cmd, bufnr, opts)
        else
            previewers.buffer_previewer_maker(filepath, bufnr, opts)
        end
    end)
end

[fdschmidt93]

I've added this to mentored projects. I'm really happy to guide a PR here. It's a bit of a question also, as to what Telescope can do here since it basically boils down to an upstream problem vim.treesitter should hopefully eventually handle.

fzf-lua tackles this problem by switching to bat if the number of lines exceeds a threshold, which would be straightforward to implement I suppose though it's of course a hack. Further, this hack is pretty much the same solution as posted here.

[Akeboshiwind]

Another option could be to add a timeout to rendering previewers somehow.

This would be a nice option because it doesn't have to be specific to what's causing this issue here.
If a user provides their own previewer and that's slow sometimes, there's a graceful way handle that.

EDIT: Mfw I should have re-read the thread 🤦

[fdschmidt93]

Yeah, I haven't fully thought the timeout idea through; even if treesitter was fast there's enough issues where sluggishness probably would persist. But yeah, probably even more challenging ;)

[fdschmidt93]

I've written up a quite naive PR. If anyone here can confirm that this works for you just as well, that would be great. Then I'd bring this in better shape. Thanks in advance.

[fdschmidt93]

Ok, the PR is in pretty good shape. Happy to get comments from anyone who's subscribed here. Any changes to the PR are hopefully primarily of cosmetic nature.

[fdschmidt93]

I hope this is now sufficiently addressed with #1231. Please see :h telescope.defaults.preview on how to configure the previewers to your liking.

We also enable hooks for varying scenarios such that you can quite easily provide functions that display what you'd like to see in case default previewer is not possible, for instance with some of the functions as laid out here. Please comment if you feel we should provide default hooks, then I'd maybe look into it (but really hope somebody steps up, it's not that hard :). We can then reopen this or make a new issue or something. Other than that, I'll close this for the time being.

[Akeboshiwind]

Here's how I'm truncating the files in the preview if they're over the file size limit:

telescope.setup {
    defaults = {
        preview = {
            filesize_hook = function(filepath, bufnr, opts)
                local max_bytes = 10000
                local cmd = {"head", "-c", max_bytes, filepath}
                require('telescope.previewers.utils').job_maker(cmd, bufnr, opts)
            end
        }
    }
}

Much neater :D

(Although I discovered #1252 😭)

[spellman]

Moreoever, filesize_hook is only run when the size of the file exceeds <telescope_config>.defaults.preview.filesize_limit:

telescope.nvim/lua/telescope/previewers/buffer_previewer.lua

Lines 187 to 194 in 2d92125

	if mb_filesize > opts.preview.filesize_limit then
	if type(opts.preview.filesize_hook) == "function" then
	opts.preview.filesize_hook(filepath, bufnr, opts)
	else
	putils.set_preview_message(bufnr, opts.winid, "File exceeds preview size limit", opts.preview.msg_bg_fillchar)
	end
	return
	end

(I don't see that in the documentation.)

[kkoomen]

I don't know why, but for large minified (javascript/typescript) files, if I do specifically a filesize_limit strictly lower than 1, I get immediately "File exceeds preview size limit" but when I do 1 or higher, my vim freezes for several seconds. Not sure why but the below fixes the issues. Does anyone know why?

telescope.setup {
  defaults = {
    preview = {
      filesize_limit = 0.9999,
      timeout = 250,
    },
  },
}

[jamestrew]

@kkoomen how large are these files?

[kkoomen]

@jamestrew 2MB

[jamestrew]

@kkoomen
Hard to debug what's happening as I don't really have a minified js file that big. My initial guess is that tree-sitter is causing the slowdown but tree-sitter shouldn't be highlighting any files above 1MB.

The previewer timeout mechanism can definitely be improved...

[kkoomen]

@jamestrew https://cdn.jsdelivr.net/pyodide/v0.24.1/full/pyodide.asm.js here is a 1.1MB js file.

My initial guess is that tree-sitter is causing the slowdown but tree-sitter shouldn't be highlighting any files above 1MB.

Oh right, totally forgot about that. I'll have a look at this as well.

[kkoomen]

@jamestrew When I disable all my plugins except for telescope, I still get the same problem that it yet again fixed with 0.9999 as a value.

[jamestrew]

Ah. I assume you're on 0.1.x branch or one of the tags.
There's a bunch of funky behaviors older versions that aren't present on the master branch. Probably easiest to just stick with the config you have working now or bump up your telescope version.

[Conni2461]

i am gonna stop this conversation on this almost 3 year old issue now! If you @kkoomen need help, found a bug with a feature, please open a new issue, fill out the issue form and then we will look into this. not here on an old issue!

It is not helpful and its hard to debug stuff without much information