live_grep makes linux computer run out of memory and freeze completely

[jweckman]

Description

I know there are a number of other performance related issues open related to live_grep using ripgrep, but i could not find instances of people reporting freezing of the entire OS making them lose all their work.

I know my use case is very crazy with say 10GB of code and i can see from journalctl that the ripgrep process is what causes the machine to run out of 16GB of RAM and 8GB of swap (fedora 38 with latest updates).

I can reproduce 100% if i go to a large directory and simply pless "k" to go through the live_grep results and i can see from my task manager that pressing "k" even once or twice makes the swap fill up and i have a frozen computer.

Running ripgrep on its own with the exact same scenario chugs along fine using all 8 cores and finishes without crashing. A single instance of ripgrep can handle itself no matter how heavy the load (100GB of code)

Would it be possible to maybe automatically slow things down a bit when a heavy load is detected to avoid freezing the operating system?

It seems like the memory issue is specifically related to cycling through results in the UI. I've had freezes happen even when there were 5 results in total and i quickly went through them.

My concrete question is then: Can the logic that is triggered when pressing "j" and "k" be adapted to refresh less or something when the queries get crazy? Maybe it could ask the OS how its doing before performing all the updates on every press of "j" or "k"?

Neovim version

NVIM v0.9.0
Build type: RelWithDebInfo
LuaJIT 2.1.0-beta3

Operating system and version

Fedora 38

Telescope version / branch / rev

latest 0.1.x

checkhealth telescope

==============================================================================
telescope: require("telescope.health").check()

Checking for required plugins ~
- OK plenary installed.
- OK nvim-treesitter installed.

Checking external dependencies ~
- OK rg: found ripgrep 13.0.0
- OK fd: found fd 8.7.0

===== Installed extensions ===== ~

Telescope Extension: `file_browser` ~
- No healthcheck provided

Telescope Extension: `fzf` ~
- OK lib working as expected
- OK file_sorter correctly configured
- OK generic_sorter correctly configured

Steps to reproduce

1. Download e.g. 50GB of code (duplicate a large project like linux kernel a few times)
2. open live_grep
3. press "k" a few times
4. If on a linux system you can easily run out of RAM and swap if you quickly cycle through results, freezing the whole OS

Expected behavior

Either slow down or kill neovim instead of freezing the OS.

Actual behavior

Frozen operating system (Fedora 38)

Also had the same issues on Fedora 36

Minimal config

vim.cmd [[set runtimepath=$VIMRUNTIME]]
vim.cmd [[set packpath=/tmp/nvim/site]]
local package_root = '/tmp/nvim/site/pack'
local install_path = package_root .. '/packer/start/packer.nvim'
local function load_plugins()
  require('packer').startup {
    {
      'wbthomason/packer.nvim',
      {
        'nvim-telescope/telescope.nvim',
        requires = {
          'nvim-lua/plenary.nvim',
          { 'nvim-telescope/telescope-fzf-native.nvim', run = 'make' },
        },
      },
      -- ADD PLUGINS THAT ARE _NECESSARY_ FOR REPRODUCING THE ISSUE
    },
    config = {
      package_root = package_root,
      compile_path = install_path .. '/plugin/packer_compiled.lua',
      display = { non_interactive = true },
    },
  }
end
_G.load_config = function()
  require('telescope').setup()
  require('telescope').load_extension('fzf')
  -- ADD INIT.LUA SETTINGS THAT ARE _NECESSARY_ FOR REPRODUCING THE ISSUE
end
if vim.fn.isdirectory(install_path) == 0 then
  print("Installing Telescope and dependencies.")
  vim.fn.system { 'git', 'clone', '--depth=1', 'https://github.com/wbthomason/packer.nvim', install_path }
end
load_plugins()
require('packer').sync()
vim.cmd [[autocmd User PackerComplete ++once echo "Ready!" | lua load_config()]]

[jweckman]

Actually sorry i can not reproduce using the provided minimal config. I opened my entire home folder with over 100GB of data and cycled through quickly and it seemed that ripgrep was throttling at 30% CPU. I must have something oddly set up in my config.

I will track down what is causing it and come up with a suggestion for how to fix.

[jweckman]

Issues was with either of these packages:
use("nvim-telescope/telescope-fzy-native.nvim")
use("nvim-telescope/telescope-rg.nvim")

Using only telescope-fzf-native.nvim avoids any problems for me.

Moral of the story seems to be that fzf-native.nvim has the best support and should be used. Closing

[nmrtv]

There is a bug with ripgrep introduced in #2440.

[nmrtv]

If you want to search with ripgrep, just use the old style arguments:

telescope.setup({
    defaults = {
        vimgrep_arguments = {
	    "rg",
	    "--color=never",
	    "--no-heading",
	    "--with-filename",
	    "--line-number",
	    "--column",
	    "--smart-case",
        }
    }
})

[soyuka]

can confirm also have the bug, and I confirm it has to do with #2440 I suggest we reopen this issue

[jamestrew]

Can you guys try using rg --vimgrep outside of neovim and see if it still causes an issue? I'm wondering if this is an rg bug or a telescope bug.

[nmrtv]

@jamestrew, this is a telescope bug. And you can test it with the repository I mentioned in #2440.

[jamestrew]

Alright I'll look into this. I can create a PR to revert the --vimgrep at least initially tomorrow.

[soyuka]

np @jamestrew the temp fix works, I'll follow the repo to see when this gets fixed no rush, maybe just reopen this issue while it gets solved?

[jweckman]

Please also fix nvim-telescope/telescope-live-grep-args.nvim if it also needs separate attention. live_grep without it is pretty useless to me as i have e.g. translation files that fill my entire telescope if i don't filter by file type/globs.

[matu3ba]

I dont think the revert fixes the core of the problem: Memory is not properly freed, for whatever reason.

I think an accurate description how memory ought to be managed and optional tracing with plenary.log is necessary besides manually running gc to get used memory. See #647 (comment).

[jamestrew]

@matu3ba I'm not sure what the root of the issue is, but I do believe the revert will fix some causes / use cases.
The number of issues related to live_grep or grep_string performance/memory usage saw a noticeable increase since that --vimgrep commit.

I'll continue to monitor issues and do a little digging myself.

[eduardoarandah]

Same issue for me, with a not so big JavaScript bundled file. Less than 500k.
Navigating results for live_grep.
tresitter disabled.
macos with 16G ram died twice.

My config:

require("telescope").setup({
				defaults = {
					preview = {
						treesitter = false, -- treesitter freezes on big files
					},
				},
			})

https://github.com/eduardoarandah/nvimrc-public/blob/master/lua/plugins/telescope.lua

[jamestrew]

@eduardoarandah try this #2482 (comment)

[voidus]

I've seen problematic behaviour with rg alone as well: BurntSushi/ripgrep#2505
The proposed fix is to add -j1 to the ripgrep args, maybe that should be included by default?

[voidus]

This seems to fix it for me:

nnoremap <leader>fg <cmd>lua require("telescope.builtin").live_grep({ additional_args = { "-j1" }})<CR>

[jamestrew]

@voidus thanks for linking that issue, that was very informative. I think a quick fix is still to simply revert back to the old vimgrep_arguments settings (no using --vimgrep) as I've done in #2488.

The recommendation to use --json is interesting though. Separate issue (#2272) but I've briefly considered this as well. Switching over to --json would require a good amount of rework though.

[Legomegger]

hello, having the issue with complete livegrep ram usage (100%) of ram on
release-1.3/neovim-0.9-d1c1bace

○ telescope.nvim   Telescope
        dir    /home/coolcat/.local/share/lunarvim/site/pack/lazy/opt/telescope.nvim
        url    https://github.com/nvim-telescope/telescope.nvim
        branch 0.1.x
        commit 9de317b
        readme README.md
        help   |telescope.theprimeagen|
        help   |telescope.nvim|
        help   |telescope.changelog|

@voidus suggestion helps, but would like to have correct behaviour out of the box

UPD: I was using LunarVim which is not being maintained now
and it was locking plugins version. So my solution is to use something new. (updating, because maybe some other lunarvim users will face similar issue)

[jamestrew]

@Legomegger you're on a super old version of telescope. Just update it.