Crashes when comparing large files

[vertigo220]

I initially mentioned this in issue #318, but it's a serious enough problem I feel it needs its own issue.

First, I need to state that this issue is HUGE, and until it's fixed, I will have to limit my use of this program to the 32-bit version and to non-video files, i.e. relatively small files. I've quoted the pertinent info from the other issue first, then provided additional info regarding the more serious crashes experienced with the 64-bit version. Both versions are 2.16.6.0 on Win10x64 1909.18363.592.

Perhaps something changed in a recent version, because it used to crash, but now that I've updated to the latest version, it hasn't done so (yet), but it also fails to do the comparison. If I select a moderately large mkv file--the one I tested is 160MB--it sits there for a while after clicking the compare button before popping up with a message saying "An error occurred while comparing the files" and the comparison screen with the content of the two files is in the background. When I click ok, it closes them. If I try a larger file--756MB tested, and of course the same happens with larger files--I instantly get the error

Cannot open file [Path]\[filename1]
System exception: Cannot map shared memory object: [Path]\[filename1]

and clicking ok I get the same error for the second file as well.

I've successfully compared 30+ GB files with HxD hex editor many times, yet I can't even compare a few hundred MB file with WinMerge, and have never been able to.

I went through and attempted comparisons on a bunch of various sized mkv files. The largest one that successfully completed the comparison was 58MB. The smallest one to fail was 94MB. The largest to fail while trying to do a comparison was 187MB, and the smallest to fail immediately was 198MB. IOW, the breakdown is as follows:

Limit for successful comparison: approximately 58-94MB
Range for comparison attempt, though comparison fails: approximately (58-94)-(187-198)MB
Limit for even attempting comparison, beyond which it results in system exception error: between 187-198

I realized the other day while trying to help another user with a problem they were having that I was running the 32-bit version of WinMerge. I have no idea which version, 32 or 64-bit, I was running previously, but after uninstalling them the other day and reinstalling, trying to resolve another issue, I ended up installing the 32-bit version without even realizing there were separate ones. I have to say that the available versions are a bit odd and confusing, with the only 64-bit ones being a special version that can't be installed per user for some reason and a portable version. Anyways, I downloaded the portable 64-bit version to test it on larger files, thinking maybe that's why I was having issues before, due to the inability of a 32-bit program to address larger files (though it should still be able to handle ones a few hundred and several hundred MB, which it couldn't).

I did find that the 64-bit version can handle larger files though, as mentioned, many of the files should be able to be handled with the 32-bit version based on size. However, while I was able to successfully compare two files that I couldn't compare with the 32-bit version (few to several hundred MB maybe, don't remember exactly, but still relatively small), it took a LOT longer than it takes to compare files the same size, or even several times larger, with HxD (hex editor I use to compare large files).

What's worse, and the reason I'm now wary of even using this program, is that when I tried comparing a couple files that are each ~16GB, it ended in disaster. The first time I tried, after waiting quite a while for the comparison to run, in the middle of watching a video while waiting I heard a couple error sounds, the video (and maybe the whole screen, but I don't remember) went blank, and after a bit I got a few different error messages from different programs, as well as one from WinMerge saying out of memory. I saw that my memory usage was very high and realized that the comparison had used up all my memory, which is what caused the other programs to error as well. I then checked task manager, which only showed WinMerge using a few (or several, don't remember exactly, but not THAT much) GB of memory, yet my usage was still very high. By the way, this was after I closed WinMerge, so despite closing it, it remained running in the background using a bunch of memory. Once I killed it in task manager, my memory dropped back down to normal.

My system has 64GB, and before running WinMerge, I was using ~13% (~8.5GB), leaving ~55GB free. That means that even if WinMerge loads the entire contents of the files it's comparing into memory, I had enough free memory to do so 3.5x for the files I was comparing, so it shouldn't have been a problem. And if it does have to load the files into memory, that's not a good way of doing it, for obvious reasons. In comparison, I just compared two 39GB files with HxD, and my memory usage didn't increase 1% (not to mention it finished in at least the same, if not less, time for files over twice the size). Nor does it go up when comparing directories containing multiple large files with FreeFileSync. For some reason, neither of those comparison programs use more than a miniscule amount of memory to do the job, and they do it faster, whereas WinMerge uses far more memory than it should possibly need, and is very slow.

I ran the comparison again, planning to keep closer tabs on things this time, and I noticed that the memory usage increased gradually, though fairly quickly, to the 90s, then slowly crept up until it got to 99%, then it dropped down to the mid-70s, then started creeping up again. This time, when it got near 100%, the screen went blank, I heard the error sounds again, and I couldn't do anything. The most I could get, and even then only sometimes, was to get the taskbar to show and to somewhat interact with it, but nothing above the taskbar was visible. I couldn't kill WinMerge (couldn't see task manager) or anything. I kept trying and trying until finally, I had to restart the computer. Granted, Windows deserves a lot of blame for that, being completely inept at handling OOM situations, and this is just another of many examples of how crappy W10 is. But WinMerge should never have let it get to that point, both because it shouldn't be using so much memory, and because it should self-terminate before it takes down the whole OS.

I don't know if the comparison method is just not very good (and I mean no disrespect to the developers, as this is a very useful and otherwise incredible program) or if there's a memory leak, but it clearly has a serious problem.

[sdottaka]

I thought the crash in issue #318 was during folder comparison. For file comparison, "Quick contents compare method" and "Binary contents compare method" have nothing to do with it.

When comparing files, WinMerge reads all the contents of the files into memory for comparison.

Because of this, it uses a very large amount of memory when compared to files with large file sizes. (When comparing folders, if the files are large, it will switch to "Quick contents compare method" or "Binary contents compare mehtod" to compare them, so it doesn't use a lot of memory)

The reason why HxD has no problem opening a large file is probably that the implementation does not read all the contents into memory, but only partially reads them.

It's too much for me to change to such an implementation, so I introduced a workaround in 330fa44.

When comparing large files, it displays a message box that asks if you want to see only the results of the comparison, not the contents of the file.

If you only want to see the comparison results, WinMerge doesn't use as much memory as it does for folder comparisons.

[vertigo220]

Ah, yeah, I see how I was misleading on that, sorry. It's definitely when comparing large files. Still, it doesn't make sense, even if it reads the entire file into memory, for it to run out of memory and crash when comparing two files that, combined, only use a portion of my free memory. Also, HxD shows the file contents, so I'm not sure why it can do so without using any memory while WinMerge uses so much.

[Warfront1]

I had the same problem on 2.16.18; I then fixed my issue by downgrading.

If you really just need to access the most basic of WinMerge functionality just downgrade to: 2.15.5

Works perfectly with large files..
Best of luck,
Warfront1

[sdottaka]

@Warfront1

Could you tell me under what conditions the crash occurred?

• During file or folder comparisons?
• WinMerge 32bit or 64bit version?
• OS version
• The size or type of file causing the crash

[hegyre]

I have the same issue with 2.16.22 (x64)
Downgrading to 2.15.5 works fine for comparing large files.
Thanks @Warfront1 for the hint!

To answer your question @sdottaka , here's the details:
During file or folder comparisons? --> file comparison
WinMerge 32bit or 64bit version? --> 2.16.22 x64
OS version --> Windows 10 Entreprise 21H2 x64
The size or type of file causing the crash --> 2 CSV files of about 50 MB each, with ~179000 lines

[sdottaka]

@hegyre
Thank you for the info.
However, I have created the attached files and compared them and could not reproduce the problem.
It would be great if you could attach those files that can reproduce the crash.

test.zip

[hegyre]

Hey @sdottaka ,
I think I know where the issue is.
One csv file has comma delimited, the other file has semicolon delimited.

With 2.15.5, there's no CSV file handling, and I guess that's why it's working fine
On 2.16.22, it does CSV handling by default and tries to display the comparison in a table.
This is where the issue here.
If I try to manually compare the 2 files as "text" with 2.16.22, then it works fine.

[sdottaka]

Changing one side to semicolon-separated instead of comma-separated caused the comparison to be very slow, but did not crash.
Do you call a very slow thing a crash? Or did WinMerge actually crash?

[hegyre]

There's no crash, however it takes an infinite time to load the comparison.

[sdottaka]

Thank you for clarifying
I'll look into it later.

[sdottaka]

@hegyre

I thought that replacing the comma with a semicolon in one of the files slowed down the comparison, but since it was just as slow in the text mode comparison, perhaps I am not reproducing the problem you are experiencing.

Would it be very slow when you compare the attached files? If it is slow, could you please attach the text file that is created by the [Help]->[Configuration] menu item?
(If you attach the file, please remove any private information that may be contained in the file.)
test2.zip

[MichelMerlin]

My Winmerge, that I am using many times daily (Winmerge 2.16.4.0 x86, Windows 7 Pro FR), had a problem, but before updating I googled for "Is 64 bit Winmerge really better than 32 bit", which landed me here and made me NOT UPdating to the last stable version (https://github.com/WinMerge/winmerge/releases = WinMerge 2.16.26 of 26 jan 2023) BUT RETROdating to 2.15.5 32-bit of 28 oct 2018, recommended on this page.
Before I finally update to 2.16.26 or newer, and eventually to 64-bit, could someone report here about the state of the (apparently somewhat CSV-related) big files comparison bug, and BTW about the eventual switch to 64-bit (for a start, is it still impossible to transfer the settings but by tedious registry editing)? TIA,
Versailles, Sun 29 Jan 2023 15:24:35 +0100

[sdottaka]

Is your Windows 7 64bit version? If it is 32bit version, you cannot use WinMerge 64bit version.

could someone report here about the state of the (apparently somewhat CSV-related) big files comparison bug,

The latest version does not improve the reading speed of large CSV files.

Note that if you uninstall the 32-bit version and install the 64-bit version, the configuration information will be lost.
You can export the following paths with a registry editor (regedit.exe) and import them after updating to retain your settings.

HKEY_CURRENT_USER\Software\Thingamahoochie\WinMerge

The 64-bit version of WinMerge can do the following compared to the 32-bit version.

• PDF and SVG files can be compared as images.
• Text can be extracted using OCR in image comparisons.
• Add columns for properties such as MD5 hash values, image size, etc. in folder comparisons.
• You can display colored emoji by selecting DirectWrite in the Rendering Mode of the Editor category in the Options dialog.
• You can compare larger files than the 32-bit version.

[MichelMerlin]

Takashi Sawanaka Mon 30 jan 2023 14h08m16s GMT, Thanks for replying.
Is your Windows 7 64bit version?
64bit (as everyone in 2023 I assume)
The latest version does not improve the reading speed of large CSV files.
So the bug is NOT fixed yet. However I am personally NOT directly impacted. Anyway Thx for this useful info.
Note that if you uninstall the 32-bit version and install the 64-bit version, the configuration information will be lost. You can export the following paths with a registry editor (regedit.exe) and import them after updating to retain your settings.
HKEY_CURRENT_USER\Software\Thingamahoochie\WinMerge
Thx for recalling this. Too bad the interface still doesn't, as most other programs, the transfer more conveniently, maybe they (you?) can find the time to do it later.
On 32bit vs 64bit in general:
As soon as the market massively switched to 64bit OSes it appeared that the drawbacks were much smaller than anticipated (let's remember Bill Gates pointing that returning to 8bit OSes was easy even with managing long addresses and could be useful in some cases), but it also appeared that for Applications the balance was different: the issue of longer memory addresses and larger addressable space is smaller, and oppositely the one of software translation or development was bigger. It appears that in many programs, many functions, in the program itself or in plugins or other derivatives, are NOT as completely, carefully, and reliably written in the (often translated) 64bit version as in the 32bit one. Personally there are a couple apps where I, long ago, switched from 64bit back to 32bit, e.g. Notepad++ and MPC-HC. So I remain somewhat vigilant when some orgs or people seem feeling forced to join the official stances systematically pushing (most often out of lack of technical knowledge in the political circles) towards bigger, newer etc. So, while remaining ready to upgrade as soon as the situation changes, I am so far keeping Winmerge 2.15.5 32-bit of 28 oct 2018, but being more re-ensured thanks to your infos.
Versailles, Tue 31 Jan 2023 10:27:10 +0100

[xerces8]

Also, HxD shows the file contents, so I'm not sure why it can do so without using any memory while WinMerge uses so much.

This was already partially answered, but: HxD loads only the part of the file. The part that contains what it is currently displaying.

WinMerge could be modified to do something similar, but is probably not trivial.