MemoryConsumer 是内存的使用者,如 sort, shuffle。MemoryConsumer 的 spill 方法是抽象方法,由子类提供。
MemoryConsumer 的字段和构造方法如下:
public abstract class MemoryConsumer {
protected final TaskMemoryManager taskMemoryManager;
private final long pageSize;
private final MemoryMode mode;
protected long used;
protected MemoryConsumer(TaskMemoryManager taskMemoryManager, long pageSize, MemoryMode mode) {
this.taskMemoryManager = taskMemoryManager;
this.pageSize = pageSize;
this.mode = mode;
}
spill
释放内存
/*
* @param size the amount of memory should be released
* @param trigger the MemoryConsumer that trigger this spilling
* @return the amount of released memory in bytes
*/
public abstract long spill(long size, MemoryConsumer trigger) throws IOException;
分配内存
分配内存有3种,allocateArray 和 allocatePage 包括具体的占用这些内存的对象。acquireMemory 仅分配指定大小的内存。
public LongArray allocateArray(long size) {
long required = size * 8L;
MemoryBlock page = taskMemoryManager.allocatePage(required, this);
if (page == null || page.size() < required) {
throwOom(page, required);
}
used += required;
return new LongArray(page);
}
释放内存
同理,释放内存有 freeArray, freePage 和 freeMemory。
MemoryConsumer 的子类
BytesToBytesMap
HybridRowQueue
LongToUnsafeRowMap