Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HIVE-15221: Improvement for MapJoin checkMemoryStatus, adding gc befor… #5562

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

HIVE-15221: Improvement for MapJoin checkMemoryStatus, adding gc befor… #5562

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion ql/src/java/org/apache/hadoop/hive/ql/exec/mapjoin/MapJoinMemoryExhaustionHandler.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,12 @@ public void checkMemoryStatus(long tableContainerSize, long numRows)
+ tableContainerSize + "\tMemory usage:\t" + usedMemory + "\tpercentage:\t" + percentageNumberFormat.format(percentage);
console.printInfo(msg);
if(percentage > maxMemoryUsage) {
throw new MapJoinMemoryExhaustionError(msg);
System.gc();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure how to test this code. Do you have some ut or local test to show that this code work well?

btw, I think adding a configuration will be good here & Keep the conf disabled.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this safe or desirable with LLAP?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zhangbutao @okumin
As a whole, this patch doesn't change any execution logic, it just rechecks the memory space after trying to call system.gc() once in case of out-of-memory.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wondered if we can use System.gc, a JVM global method, for a task-local problem. I basically don't know why we implemented MapJoinMemoryExhaustionHandler. I might be saying something strange

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've used this patch in our production environment, and the OOM and MapJoinMemoryExhaustionError related issues have really been reduced. However, I do agree with you to some extent that the current implementation is a bit problematic.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First, I might not have sufficient knowledge to give this +1. I've never faced this error.
Reading the source code, I found that only Hive on MapReduce uses MemoryExhaustionChecker. So, not only LLAP but also Tez are not involved with this error. My original concern, i.e., the entire LLAP daemon could get degraded, is gone.
But I am still wondering why Hive on MapReduce needs MemoryExhaustionChecker. We may remove the class and let MapReduce crush with OOM when creating a giant hash table.

Copy link
Contributor Author

@BsoBird BsoBird Dec 31, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@okumin Hello. Although I haven't carefully reviewed this part of the code yet, using the TEZ engine will still trigger the MapJoinMemoryExhaustionError.

usedMemory = memoryMXBean.getHeapMemoryUsage().getUsed();
percentage = (double) usedMemory / (double) maxHeapSize;
if(percentage > maxMemoryUsage) {
throw new MapJoinMemoryExhaustionError(msg);
}
}
}
}
Loading