From c5dd1fcd062a7eef9537302b931a8ab5a56c52a8 Mon Sep 17 00:00:00 2001 From: David Rowley Date: Thu, 9 Jun 2022 20:09:22 +1200 Subject: [PATCH v1] WIP: Reduce memory context overheads Whenever we palloc a chunk of memory, traditionally, we prefix the returned pointer with a pointer to the memory context which the chunk belongs to. This is required so that we're able to easily determine the owning context when performing operations such as pfree() and repalloc(). The AllocSet context additionally prefixes the pointer to the owning context with the size of the chunk which makes the header 16 bytes in size. For generation contexts, the problem was worse. In addition to the pointer to the owning context, we also stored a pointer to the owning block then prefixed the chunk size, which resulted in a 24-byte header. Here we completely change the rule that pointers must be directly prefixed by the owning memory context and instead, insist that they're directly prefixed by a uint64 variable where the least significant 3-bits are set to a value to indicate which type of memory context the pointer belongs to. We're then able to pass the pointer to the function specific to that context implementation to allow it to determine the actual memory context to which the pointer belongs. Here we implement a general-purpose way of encoding the remaining bits in the prefixing uint64 value which allows encoding of the chunk size and also the number of bytes to subtract from the pointer to get a pointer to the block the pointer belongs to. All memory contexts now subtract the block offset bytes to obtain a pointer to the block and that block contains a pointer to the owning context. This means we only store a pointer to the context once per block, rather than once per chunk as we did previously. Because the remaining space in the prefixing uint64 value after storing the 3-bit value to indicate the memory context type is only 61 bits, we only have 30 bits each to store the chunk size and the block offset values. We do allow much larger allocations than what can be stored in 30 bits, so we also have a concept of a "large" chunk and we mark if the chunk is small or large with the 1 remaining bit in the uint64 header. When a chunk is large, the uint64 header value is immediately preceded by two Size values, one to indicate the chunk size and one for the block offset. For MEMORY_CONTEXT_CHECKING builds, each chunk header is prefixed by the number of bytes in the memory request. This allows us to perform all of the memory context checking still. --- src/backend/utils/mmgr/README | 59 ++- src/backend/utils/mmgr/aset.c | 618 ++++++++++++------------ src/backend/utils/mmgr/generation.c | 407 +++++++--------- src/backend/utils/mmgr/mcxt.c | 94 +++- src/backend/utils/mmgr/slab.c | 281 +++++------ src/include/nodes/memnodes.h | 7 +- src/include/utils/memutils.h | 48 +- src/include/utils/memutils_generichdr.h | 265 ++++++++++ src/include/utils/memutils_internal.h | 117 +++++ 9 files changed, 1140 insertions(+), 756 deletions(-) create mode 100644 src/include/utils/memutils_generichdr.h create mode 100644 src/include/utils/memutils_internal.h diff --git a/src/backend/utils/mmgr/README b/src/backend/utils/mmgr/README index 221b4bd343..58a2c5fc3d 100644 --- a/src/backend/utils/mmgr/README +++ b/src/backend/utils/mmgr/README @@ -378,35 +378,54 @@ MemoryContext like the parent and child contexts, and the name of the context. This is essentially an abstract superclass, and the behavior is -determined by the "methods" pointer is its virtual function table -(struct MemoryContextMethods). Specific memory context types will use -derived structs having these fields as their first fields. All the +determined by the "methods" pointer which references which set of +MemoryContextMethods are to be used. Specific memory context types will +use derived structs having these fields as their first fields. All the contexts of a specific type will have methods pointers that point to -the same static table of function pointers. +the corresponding element in the mcxt_methods[] array. While operations like allocating from and resetting a context take the relevant MemoryContext as a parameter, operations like free and realloc are trickier. To make those work, we require all memory context types to produce allocated chunks that are immediately, -without any padding, preceded by a pointer to the corresponding -MemoryContext. +without any padding, preceded by a uint64 value of which the least +significant 3 bits are set to the owning context's MemoryContextMethodID. +This allows the code to determine the correct MemoryContextMethods to +use by looking up the mcxt_methods[] array using the 3 bits as an index +into that array. If a type of allocator needs additional information about its chunks, like e.g. the size of the allocation, that information can in turn -precede the MemoryContext. This means the only overhead implied by -the memory context mechanism is a pointer to its context, so we're not -constraining context-type designers very much. - -Given this, routines like pfree determine their corresponding context -with an operation like (although that is usually encapsulated in -GetMemoryChunkContext()) - - MemoryContext context = *(MemoryContext*) (((char *) pointer) - sizeof(void *)); - -and then invoke the corresponding method for the context - - context->methods->free_p(pointer); - +either be encoded into the remaining 61 bits of the preceding uint64 value +or if more space is required, additional values may be stored directly prior +to the uint64 value. It is up to the context implementation to manage this. + +Given this, routines like pfree can determine which set of +MemoryContextMethods to call the free_p function for by calling +GetMemoryChunkMethodID() and finding the corresponding MemoryContextMethods +in the mcxt_methods array[]. For convenience, the MCXT_METHOD() macro is +provided, making the code as simple as: + +void +pfree(void *pointer) +{ + MCXT_METHOD(pointer, free_p)(pointer); +} + +All of the current memory contexts make use of a generic chunk header type +which is defined in memutils_generichdr.h. This suits all of the existing +context types well as it makes use of the remaining 61-bits of the uint64 +header to efficiently encode the size of the chunk of memory and the number +of bytes which must be subtracted from the pointer in order to obtain a +reference to the block that the pointer belongs to. 30 bits are used for each +of these. If more than 30 bits are required (i.e. > 1GB) then a "large" chunk +header is used which uses an additional two Size fields in the preceding +memory. All memory context implementations store a pointer back to the owning +context within the block and we can find that block by subtracting the block +offset bytes, as is stored in the generic header. During operations like +pfree, this allows memory contexts such as the generation context to track how +many chunks have been freed from a given block and then release the memory for +the block when all chunks have been freed. More Control Over aset.c Behavior --------------------------------- diff --git a/src/backend/utils/mmgr/aset.c b/src/backend/utils/mmgr/aset.c index ec3e264a73..f88d506cc1 100644 --- a/src/backend/utils/mmgr/aset.c +++ b/src/backend/utils/mmgr/aset.c @@ -49,6 +49,8 @@ #include "port/pg_bitutils.h" #include "utils/memdebug.h" #include "utils/memutils.h" +#include "utils/memutils_generichdr.h" +#include "utils/memutils_internal.h" /*-------------------- * Chunk freelist k holds chunks of size 1 << (k + ALLOC_MINBITS), @@ -66,7 +68,9 @@ * CAUTION: ALLOC_MINBITS must be large enough so that * 1<= 1024 && minContextSize <= maxBlockSize)); + Assert(maxBlockSize <= SMALL_CHUNK_LIMIT); /* * Check whether the parameters match either available freelist. We do @@ -443,7 +355,7 @@ AllocSetContextCreateInternal(MemoryContext parent, /* Reinitialize its header, installing correct name and parent */ MemoryContextCreate((MemoryContext) set, T_AllocSetContext, - &AllocSetMethods, + MCTX_ASET_ID, parent, name); @@ -456,7 +368,7 @@ AllocSetContextCreateInternal(MemoryContext parent, /* Determine size of initial block */ firstBlockSize = MAXALIGN(sizeof(AllocSetContext)) + - ALLOC_BLOCKHDRSZ + ALLOC_CHUNKHDRSZ; + ALLOC_BLOCKHDRSZ + SMALL_CHUNK_SIZE; if (minContextSize != 0) firstBlockSize = Max(firstBlockSize, minContextSize); else @@ -486,6 +398,7 @@ AllocSetContextCreateInternal(MemoryContext parent, /* Fill in the initial block's block header */ block = (AllocBlock) (((char *) set) + MAXALIGN(sizeof(AllocSetContext))); block->aset = set; + block->large_chunk = false; block->freeptr = ((char *) block) + ALLOC_BLOCKHDRSZ; block->endptr = ((char *) set) + firstBlockSize; block->prev = NULL; @@ -519,22 +432,23 @@ AllocSetContextCreateInternal(MemoryContext parent, * * We have to have allocChunkLimit a power of two, because the requested * and actually-allocated sizes of any chunk must be on the same side of - * the limit, else we get confused about whether the chunk is "big". + * the limit, else we get confused about whether the chunk is large. * * Also, allocChunkLimit must not exceed ALLOCSET_SEPARATE_THRESHOLD. */ StaticAssertStmt(ALLOC_CHUNK_LIMIT == ALLOCSET_SEPARATE_THRESHOLD, "ALLOC_CHUNK_LIMIT != ALLOCSET_SEPARATE_THRESHOLD"); - set->allocChunkLimit = ALLOC_CHUNK_LIMIT; - while ((Size) (set->allocChunkLimit + ALLOC_CHUNKHDRSZ) > + /* We always want to allocate non-dedicated blocks as small chunks */ + set->allocChunkLimit = Min(ALLOC_CHUNK_LIMIT, SMALL_CHUNK_LIMIT); + while ((Size) (set->allocChunkLimit + SMALL_CHUNK_SIZE) > (Size) ((maxBlockSize - ALLOC_BLOCKHDRSZ) / ALLOC_CHUNK_FRACTION)) set->allocChunkLimit >>= 1; /* Finally, do the type-independent part of context creation */ MemoryContextCreate((MemoryContext) set, T_AllocSetContext, - &AllocSetMethods, + MCTX_ASET_ID, parent, name); @@ -555,7 +469,7 @@ AllocSetContextCreateInternal(MemoryContext parent, * thrash malloc() when a context is repeatedly reset after small allocations, * which is typical behavior for per-tuple contexts. */ -static void +void AllocSetReset(MemoryContext context) { AllocSet set = (AllocSet) context; @@ -623,7 +537,7 @@ AllocSetReset(MemoryContext context) * * Unlike AllocSetReset, this *must* free all resources of the set. */ -static void +void AllocSetDelete(MemoryContext context) { AllocSet set = (AllocSet) context; @@ -717,15 +631,15 @@ AllocSetDelete(MemoryContext context) * is marked, as mcxt.c will set it to UNDEFINED. In some paths we will * return space that is marked NOACCESS - AllocSetRealloc has to beware! */ -static void * +void * AllocSetAlloc(MemoryContext context, Size size) { AllocSet set = (AllocSet) context; AllocBlock block; - AllocChunk chunk; int fidx; Size chunk_size; Size blksize; + void *pointer; AssertArg(AllocSetIsValid(set)); @@ -735,8 +649,12 @@ AllocSetAlloc(MemoryContext context, Size size) */ if (size > set->allocChunkLimit) { + Size genhdrsize; + chunk_size = MAXALIGN(size); - blksize = chunk_size + ALLOC_BLOCKHDRSZ + ALLOC_CHUNKHDRSZ; + /* err on the side of caution and assume it'll be a large chunk */ + genhdrsize = GenericChunkHeaderSize(ALLOC_BLOCKHDRSZ + LARGE_CHUNK_SIZE, chunk_size); + blksize = ALLOC_BLOCKHDRSZ + genhdrsize + chunk_size; block = (AllocBlock) malloc(blksize); if (block == NULL) return NULL; @@ -744,20 +662,25 @@ AllocSetAlloc(MemoryContext context, Size size) context->mem_allocated += blksize; block->aset = set; + /* mark if this is a large chunk or not */ + block->large_chunk = (genhdrsize == LARGE_CHUNK_SIZE); block->freeptr = block->endptr = ((char *) block) + blksize; - chunk = (AllocChunk) (((char *) block) + ALLOC_BLOCKHDRSZ); - chunk->aset = set; - chunk->size = chunk_size; + pointer = (void *) (((char *) block) + ALLOC_BLOCKHDRSZ + genhdrsize); + #ifdef MEMORY_CONTEXT_CHECKING - chunk->requested_size = size; + GenericChunkHeaderEncode(pointer, block, chunk_size, size, MCTX_ASET_ID); + /* set mark to catch clobber of "unused" space */ if (size < chunk_size) - set_sentinel(AllocChunkGetPointer(chunk), size); + set_sentinel(pointer, size); +#else + GenericChunkHeaderEncode(pointer, block, chunk_size, MCTX_ASET_ID); #endif + #ifdef RANDOMIZE_ALLOCATED_MEMORY /* fill the allocated space with junk */ - randomize_mem((char *) AllocChunkGetPointer(chunk), size); + randomize_mem((char *) pointer, size); #endif /* @@ -780,13 +703,12 @@ AllocSetAlloc(MemoryContext context, Size size) } /* Ensure any padding bytes are marked NOACCESS. */ - VALGRIND_MAKE_MEM_NOACCESS((char *) AllocChunkGetPointer(chunk) + size, + VALGRIND_MAKE_MEM_NOACCESS(((char *) pointer) + size, chunk_size - size); - /* Disallow external access to private part of chunk header. */ - VALGRIND_MAKE_MEM_NOACCESS(chunk, ALLOCCHUNK_PRIVATE_LEN); + Assert(GetMemoryChunkMethodID(pointer) == MCTX_ASET_ID); - return AllocChunkGetPointer(chunk); + return pointer; } /* @@ -796,42 +718,59 @@ AllocSetAlloc(MemoryContext context, Size size) * of the alloc set and return its data address. */ fidx = AllocSetFreeIndex(size); - chunk = set->freelist[fidx]; - if (chunk != NULL) - { - Assert(chunk->size >= size); - set->freelist[fidx] = (AllocChunk) chunk->aset; + /* + * Choose the actual chunk size to allocate. + */ + chunk_size = (1 << ALLOC_MINBITS) << fidx; + Assert(chunk_size >= size); - chunk->aset = (void *) set; + pointer = set->freelist[fidx]; + if (pointer != NULL) + { + /* + * Point the freelist head at the next free chunk which we stored in + * the allocated memory + */ + set->freelist[fidx] = *((void **) pointer); #ifdef MEMORY_CONTEXT_CHECKING - chunk->requested_size = size; + { + /* + * The chunk header is already set correctly when not doing + * MEMORY_CONTEXT_CHECKING. Here we just need to update the + * requested size. We must decode it to get the block offset + * before we reencode it again. + */ + AllocBlock block; + Size chksz, blkoff, reqsz; + + GenericChunkHeaderDecode(pointer, &chksz, &blkoff, &reqsz); + + block = (AllocBlock) ((char *) pointer - blkoff); + + GenericChunkHeaderEncode(pointer, block, chunk_size, size, MCTX_ASET_ID); + } + /* set mark to catch clobber of "unused" space */ - if (size < chunk->size) - set_sentinel(AllocChunkGetPointer(chunk), size); + if (size < chunk_size) + set_sentinel(pointer, size); #endif + #ifdef RANDOMIZE_ALLOCATED_MEMORY /* fill the allocated space with junk */ - randomize_mem((char *) AllocChunkGetPointer(chunk), size); + randomize_mem((char *) pointer, size); #endif /* Ensure any padding bytes are marked NOACCESS. */ - VALGRIND_MAKE_MEM_NOACCESS((char *) AllocChunkGetPointer(chunk) + size, - chunk->size - size); + VALGRIND_MAKE_MEM_NOACCESS(((char *) pointer) + size, + chunk_size - size); - /* Disallow external access to private part of chunk header. */ - VALGRIND_MAKE_MEM_NOACCESS(chunk, ALLOCCHUNK_PRIVATE_LEN); + Assert(GetMemoryChunkMethodID(pointer) == MCTX_ASET_ID); - return AllocChunkGetPointer(chunk); + return pointer; } - /* - * Choose the actual chunk size to allocate. - */ - chunk_size = (1 << ALLOC_MINBITS) << fidx; - Assert(chunk_size >= size); - /* * If there is enough room in the active allocation block, we will put the * chunk into that block. Else must start a new one. @@ -840,7 +779,9 @@ AllocSetAlloc(MemoryContext context, Size size) { Size availspace = block->endptr - block->freeptr; - if (availspace < (chunk_size + ALLOC_CHUNKHDRSZ)) + Assert(availspace <= SMALL_CHUNK_LIMIT); + + if (availspace < (SMALL_CHUNK_SIZE + chunk_size)) { /* * The existing active (top) block does not have enough room for @@ -854,9 +795,9 @@ AllocSetAlloc(MemoryContext context, Size size) * ALLOC_CHUNK_LIMIT left in the block, this loop cannot iterate * more than ALLOCSET_NUM_FREELISTS-1 times. */ - while (availspace >= ((1 << ALLOC_MINBITS) + ALLOC_CHUNKHDRSZ)) + while (availspace >= ((1 << ALLOC_MINBITS) + SMALL_CHUNK_SIZE + sizeof(void *))) { - Size availchunk = availspace - ALLOC_CHUNKHDRSZ; + Size availchunk = availspace - SMALL_CHUNK_SIZE; int a_fidx = AllocSetFreeIndex(availchunk); /* @@ -871,22 +812,21 @@ AllocSetAlloc(MemoryContext context, Size size) availchunk = ((Size) 1 << (a_fidx + ALLOC_MINBITS)); } - chunk = (AllocChunk) (block->freeptr); - - /* Prepare to initialize the chunk header. */ - VALGRIND_MAKE_MEM_UNDEFINED(chunk, ALLOC_CHUNKHDRSZ); + pointer = (void *) (block->freeptr + SMALL_CHUNK_SIZE); - block->freeptr += (availchunk + ALLOC_CHUNKHDRSZ); - availspace -= (availchunk + ALLOC_CHUNKHDRSZ); + block->freeptr += (availchunk + SMALL_CHUNK_SIZE); + availspace -= (availchunk + SMALL_CHUNK_SIZE); - chunk->size = availchunk; #ifdef MEMORY_CONTEXT_CHECKING - chunk->requested_size = 0; /* mark it free */ + /* mark it free */ + GenericChunkHeaderEncode(pointer, block, availchunk, 0, MCTX_ASET_ID); +#else + GenericChunkHeaderEncode(pointer, block, availchunk, MCTX_ASET_ID); #endif - chunk->aset = (void *) set->freelist[a_fidx]; - set->freelist[a_fidx] = chunk; + /* push this chunk onto the freelist */ + *((void **) pointer) = set->freelist[a_fidx]; + set->freelist[a_fidx] = pointer; } - /* Mark that we need to create a new block */ block = NULL; } @@ -912,7 +852,7 @@ AllocSetAlloc(MemoryContext context, Size size) * If initBlockSize is less than ALLOC_CHUNK_LIMIT, we could need more * space... but try to keep it a power of 2. */ - required_size = chunk_size + ALLOC_BLOCKHDRSZ + ALLOC_CHUNKHDRSZ; + required_size = ALLOC_BLOCKHDRSZ + SMALL_CHUNK_SIZE + chunk_size; while (blksize < required_size) blksize <<= 1; @@ -937,6 +877,7 @@ AllocSetAlloc(MemoryContext context, Size size) context->mem_allocated += blksize; block->aset = set; + block->large_chunk = false; block->freeptr = ((char *) block) + ALLOC_BLOCKHDRSZ; block->endptr = ((char *) block) + blksize; @@ -954,76 +895,83 @@ AllocSetAlloc(MemoryContext context, Size size) /* * OK, do the allocation */ - chunk = (AllocChunk) (block->freeptr); - - /* Prepare to initialize the chunk header. */ - VALGRIND_MAKE_MEM_UNDEFINED(chunk, ALLOC_CHUNKHDRSZ); + pointer = (void *) (block->freeptr + SMALL_CHUNK_SIZE); - block->freeptr += (chunk_size + ALLOC_CHUNKHDRSZ); + block->freeptr += (SMALL_CHUNK_SIZE + chunk_size); Assert(block->freeptr <= block->endptr); - chunk->aset = (void *) set; - chunk->size = chunk_size; + #ifdef MEMORY_CONTEXT_CHECKING - chunk->requested_size = size; + GenericChunkHeaderEncode(pointer, block, chunk_size, size, MCTX_ASET_ID); + /* set mark to catch clobber of "unused" space */ - if (size < chunk->size) - set_sentinel(AllocChunkGetPointer(chunk), size); + if (size < chunk_size) + set_sentinel(pointer, size); +#else + GenericChunkHeaderEncode(pointer, block, chunk_size, MCTX_ASET_ID); #endif + #ifdef RANDOMIZE_ALLOCATED_MEMORY /* fill the allocated space with junk */ - randomize_mem((char *) AllocChunkGetPointer(chunk), size); + randomize_mem((char *) pointer, size); #endif /* Ensure any padding bytes are marked NOACCESS. */ - VALGRIND_MAKE_MEM_NOACCESS((char *) AllocChunkGetPointer(chunk) + size, + VALGRIND_MAKE_MEM_NOACCESS((char *) pointer + size, chunk_size - size); - /* Disallow external access to private part of chunk header. */ - VALGRIND_MAKE_MEM_NOACCESS(chunk, ALLOCCHUNK_PRIVATE_LEN); + Assert(GetMemoryChunkMethodID(pointer) == MCTX_ASET_ID); - return AllocChunkGetPointer(chunk); + return pointer; } /* * AllocSetFree * Frees allocated memory; memory is removed from the set. */ -static void -AllocSetFree(MemoryContext context, void *pointer) +void +AllocSetFree(void *pointer) { - AllocSet set = (AllocSet) context; - AllocChunk chunk = AllocPointerGetChunk(pointer); + AllocBlock block; + AllocSet set; + Size chunksize; + Size blockoffset; + Size hdrsize; +#ifdef MEMORY_CONTEXT_CHECKING + Size reqsize; + + hdrsize = GenericChunkHeaderDecode(pointer, &chunksize, &blockoffset, &reqsize); +#else + hdrsize = GenericChunkHeaderDecode(pointer, &chunksize, &blockoffset); +#endif - /* Allow access to private part of chunk header. */ - VALGRIND_MAKE_MEM_DEFINED(chunk, ALLOCCHUNK_PRIVATE_LEN); + block = (AllocBlock) ((char *) pointer - blockoffset); + set = block->aset; #ifdef MEMORY_CONTEXT_CHECKING /* Test for someone scribbling on unused space in chunk */ - if (chunk->requested_size < chunk->size) - if (!sentinel_ok(pointer, chunk->requested_size)) + if (reqsize < chunksize) + if (!sentinel_ok(pointer, reqsize)) elog(WARNING, "detected write past chunk end in %s %p", - set->header.name, chunk); + set->header.name, pointer); #endif - if (chunk->size > set->allocChunkLimit) + if (chunksize > set->allocChunkLimit) { /* * Big chunks are certain to have been allocated as single-chunk * blocks. Just unlink that block and return it to malloc(). */ - AllocBlock block = (AllocBlock) (((char *) chunk) - ALLOC_BLOCKHDRSZ); /* * Try to verify that we have a sane block pointer: it should * reference the correct aset, and freeptr and endptr should point * just past the chunk. */ - if (block->aset != set || - block->freeptr != block->endptr || + if (block->freeptr != block->endptr || block->freeptr != ((char *) block) + - (chunk->size + ALLOC_BLOCKHDRSZ + ALLOC_CHUNKHDRSZ)) - elog(ERROR, "could not find block containing chunk %p", chunk); + (ALLOC_BLOCKHDRSZ + hdrsize + chunksize)) + elog(ERROR, "could not find block containing pointer %p", pointer); /* OK, remove block from aset's list and free it */ if (block->prev) @@ -1033,7 +981,7 @@ AllocSetFree(MemoryContext context, void *pointer) if (block->next) block->next->prev = block->prev; - context->mem_allocated -= block->endptr - ((char *) block); + set->header.mem_allocated -= block->endptr - ((char *) block); #ifdef CLOBBER_FREED_MEMORY wipe_mem(block, block->freeptr - ((char *) block)); @@ -1043,19 +991,22 @@ AllocSetFree(MemoryContext context, void *pointer) else { /* Normal case, put the chunk into appropriate freelist */ - int fidx = AllocSetFreeIndex(chunk->size); + int fidx = AllocSetFreeIndex(chunksize); - chunk->aset = (void *) set->freelist[fidx]; + /* Use the pointer to store the freelist link */ + *((void **) pointer) = set->freelist[fidx]; #ifdef CLOBBER_FREED_MEMORY - wipe_mem(pointer, chunk->size); + /* Wipe the memory, but not the freelist link we just set above */ + wipe_mem(((char *) pointer) + sizeof(void *), chunksize - sizeof(void *)); #endif #ifdef MEMORY_CONTEXT_CHECKING /* Reset requested_size to 0 in chunks that are on freelist */ - chunk->requested_size = 0; + GenericChunkHeaderEncode(pointer, block, chunksize, 0, MCTX_ASET_ID); #endif - set->freelist[fidx] = chunk; + + set->freelist[fidx] = pointer; } } @@ -1071,24 +1022,32 @@ AllocSetFree(MemoryContext context, void *pointer) * (In principle, we could use VALGRIND_GET_VBITS() to rediscover the old * request size.) */ -static void * -AllocSetRealloc(MemoryContext context, void *pointer, Size size) +void * +AllocSetRealloc(void *pointer, Size size) { - AllocSet set = (AllocSet) context; - AllocChunk chunk = AllocPointerGetChunk(pointer); + AllocBlock block; + AllocSet set; Size oldsize; + Size blockoffset; + Size hdrsize; +#ifdef MEMORY_CONTEXT_CHECKING + Size oldreqsize; - /* Allow access to private part of chunk header. */ - VALGRIND_MAKE_MEM_DEFINED(chunk, ALLOCCHUNK_PRIVATE_LEN); + hdrsize = GenericChunkHeaderDecode(pointer, &oldsize, &blockoffset, &oldreqsize); +#else - oldsize = chunk->size; + hdrsize = GenericChunkHeaderDecode(pointer, &oldsize, &blockoffset); +#endif + + block = (AllocBlock) ((char *) pointer - blockoffset); + set = block->aset; #ifdef MEMORY_CONTEXT_CHECKING /* Test for someone scribbling on unused space in chunk */ - if (chunk->requested_size < oldsize) - if (!sentinel_ok(pointer, chunk->requested_size)) + if (oldreqsize < oldsize) + if (!sentinel_ok(pointer, oldreqsize)) elog(WARNING, "detected write past chunk end in %s %p", - set->header.name, chunk); + set->header.name, pointer); #endif if (oldsize > set->allocChunkLimit) @@ -1098,21 +1057,20 @@ AllocSetRealloc(MemoryContext context, void *pointer, Size size) * realloc() to make the containing block bigger, or smaller, with * minimum space wastage. */ - AllocBlock block = (AllocBlock) (((char *) chunk) - ALLOC_BLOCKHDRSZ); Size chksize; Size blksize; Size oldblksize; + Size newhdrsize; /* * Try to verify that we have a sane block pointer: it should * reference the correct aset, and freeptr and endptr should point * just past the chunk. */ - if (block->aset != set || - block->freeptr != block->endptr || + if (block->freeptr != block->endptr || block->freeptr != ((char *) block) + - (oldsize + ALLOC_BLOCKHDRSZ + ALLOC_CHUNKHDRSZ)) - elog(ERROR, "could not find block containing chunk %p", chunk); + (ALLOC_BLOCKHDRSZ + hdrsize + oldsize)) + elog(ERROR, "could not find block containing pointer %p", pointer); /* * Even if the new request is less than set->allocChunkLimit, we stick @@ -1123,41 +1081,66 @@ AllocSetRealloc(MemoryContext context, void *pointer, Size size) chksize = Max(size, set->allocChunkLimit + 1); chksize = MAXALIGN(chksize); + /* Assume big chunk size */ + newhdrsize = GenericChunkHeaderSize(ALLOC_BLOCKHDRSZ + LARGE_CHUNK_SIZE, chksize); + /* Do the realloc */ - blksize = chksize + ALLOC_BLOCKHDRSZ + ALLOC_CHUNKHDRSZ; + blksize = ALLOC_BLOCKHDRSZ + newhdrsize + chksize; oldblksize = block->endptr - ((char *) block); - block = (AllocBlock) realloc(block, blksize); - if (block == NULL) + if (newhdrsize == hdrsize) + block = (AllocBlock) realloc(block, blksize); + else { - /* Disallow external access to private part of chunk header. */ - VALGRIND_MAKE_MEM_NOACCESS(chunk, ALLOCCHUNK_PRIVATE_LEN); - return NULL; + /* + * When the size of the chunk header changes, we're unable to + * simply call realloc as the offset between the block and the + * pointer will have changed. For this case we resort to just + * allocating new memory with AllocSetAlloc then copying the old + * memory over before finally freeing the old memory and block + * with AllocSetFree. + */ + void *newPointer = AllocSetAlloc((MemoryContext) set, size); + + if (newPointer == NULL) + return NULL; + + memcpy(newPointer, pointer, Min(oldsize, size)); + + AllocSetFree(pointer); + + return newPointer; } + if (block == NULL) + return NULL; + /* updated separately, not to underflow when (oldblksize > blksize) */ - context->mem_allocated -= oldblksize; - context->mem_allocated += blksize; + set->header.mem_allocated -= oldblksize; + set->header.mem_allocated += blksize; block->freeptr = block->endptr = ((char *) block) + blksize; /* Update pointers since block has likely been moved */ - chunk = (AllocChunk) (((char *) block) + ALLOC_BLOCKHDRSZ); - pointer = AllocChunkGetPointer(chunk); + pointer = (void *) ((char *) block + ALLOC_BLOCKHDRSZ + newhdrsize); if (block->prev) block->prev->next = block; else set->blocks = block; if (block->next) block->next->prev = block; - chunk->size = chksize; + +#ifdef MEMORY_CONTEXT_CHECKING + GenericChunkHeaderEncode(pointer, block, chksize, size, MCTX_ASET_ID); +#else + GenericChunkHeaderEncode(pointer, block, chksize, MCTX_ASET_ID); +#endif #ifdef MEMORY_CONTEXT_CHECKING #ifdef RANDOMIZE_ALLOCATED_MEMORY - /* We can only fill the extra space if we know the prior request */ - if (size > chunk->requested_size) - randomize_mem((char *) pointer + chunk->requested_size, - size - chunk->requested_size); + /* for enlarged chunks, randomize the enlarged portion. */ + if (size > oldreqsize) + randomize_mem(((char *) pointer) + oldreqsize, size - oldreqsize); #endif /* @@ -1166,15 +1149,12 @@ AllocSetRealloc(MemoryContext context, void *pointer, Size size) * old allocation. */ #ifdef USE_VALGRIND - if (oldsize > chunk->requested_size) - VALGRIND_MAKE_MEM_UNDEFINED((char *) pointer + chunk->requested_size, - oldsize - chunk->requested_size); + if (oldsize > oldreqsize) + VALGRIND_MAKE_MEM_UNDEFINED((char *) pointer + oldreqsize, + oldsize - oldreqsize); #endif - - chunk->requested_size = size; - /* set mark to catch clobber of "unused" space */ - if (size < chunk->size) + if (size < chksize) set_sentinel(pointer, size); #else /* !MEMORY_CONTEXT_CHECKING */ @@ -1189,8 +1169,7 @@ AllocSetRealloc(MemoryContext context, void *pointer, Size size) /* Ensure any padding bytes are marked NOACCESS. */ VALGRIND_MAKE_MEM_NOACCESS((char *) pointer + size, chksize - size); - /* Disallow external access to private part of chunk header. */ - VALGRIND_MAKE_MEM_NOACCESS(chunk, ALLOCCHUNK_PRIVATE_LEN); + Assert(GetMemoryChunkMethodID(pointer) == MCTX_ASET_ID); return pointer; } @@ -1203,24 +1182,19 @@ AllocSetRealloc(MemoryContext context, void *pointer, Size size) else if (oldsize >= size) { #ifdef MEMORY_CONTEXT_CHECKING - Size oldrequest = chunk->requested_size; - #ifdef RANDOMIZE_ALLOCATED_MEMORY /* We can only fill the extra space if we know the prior request */ - if (size > oldrequest) - randomize_mem((char *) pointer + oldrequest, - size - oldrequest); + if (size > oldreqsize) + randomize_mem((char *) pointer + oldreqsize, + size - oldreqsize); #endif - - chunk->requested_size = size; - /* * If this is an increase, mark any newly-available part UNDEFINED. * Otherwise, mark the obsolete part NOACCESS. */ - if (size > oldrequest) - VALGRIND_MAKE_MEM_UNDEFINED((char *) pointer + oldrequest, - size - oldrequest); + if (size > oldreqsize) + VALGRIND_MAKE_MEM_UNDEFINED((char *) pointer + oldreqsize, + size - oldreqsize); else VALGRIND_MAKE_MEM_NOACCESS((char *) pointer + size, oldsize - size); @@ -1228,6 +1202,9 @@ AllocSetRealloc(MemoryContext context, void *pointer, Size size) /* set mark to catch clobber of "unused" space */ if (size < oldsize) set_sentinel(pointer, size); + + /* Re-encode the chunk to apply new request size */ + GenericChunkHeaderEncode(pointer, block, oldsize, size, MCTX_ASET_ID); #else /* !MEMORY_CONTEXT_CHECKING */ /* @@ -1239,8 +1216,7 @@ AllocSetRealloc(MemoryContext context, void *pointer, Size size) VALGRIND_MAKE_MEM_DEFINED(pointer, size); #endif - /* Disallow external access to private part of chunk header. */ - VALGRIND_MAKE_MEM_NOACCESS(chunk, ALLOCCHUNK_PRIVATE_LEN); + Assert(GetMemoryChunkMethodID(pointer) == MCTX_ASET_ID); return pointer; } @@ -1264,11 +1240,7 @@ AllocSetRealloc(MemoryContext context, void *pointer, Size size) /* leave immediately if request was not completed */ if (newPointer == NULL) - { - /* Disallow external access to private part of chunk header. */ - VALGRIND_MAKE_MEM_NOACCESS(chunk, ALLOCCHUNK_PRIVATE_LEN); return NULL; - } /* * AllocSetAlloc() may have returned a region that is still NOACCESS. @@ -1279,8 +1251,9 @@ AllocSetRealloc(MemoryContext context, void *pointer, Size size) * trailing bytes. */ VALGRIND_MAKE_MEM_UNDEFINED(newPointer, size); + #ifdef MEMORY_CONTEXT_CHECKING - oldsize = chunk->requested_size; + oldsize = oldreqsize; #else VALGRIND_MAKE_MEM_DEFINED(pointer, oldsize); #endif @@ -1289,34 +1262,67 @@ AllocSetRealloc(MemoryContext context, void *pointer, Size size) memcpy(newPointer, pointer, oldsize); /* free old chunk */ - AllocSetFree((MemoryContext) set, pointer); + AllocSetFree(pointer); + + Assert(GetMemoryChunkMethodID(newPointer) == MCTX_ASET_ID); return newPointer; } } +/* + * AllocSetGetChunkContext + * Return the MemoryContext that 'pointer' belongs to. + */ +MemoryContext +AllocSetGetChunkContext(void *pointer) +{ + AllocBlock block; + AllocSet set; + Size chunksize; + Size blockoffset; +#ifdef MEMORY_CONTEXT_CHECKING + Size reqsize; + + GenericChunkHeaderDecode(pointer, &chunksize, &blockoffset, &reqsize); +#else + GenericChunkHeaderDecode(pointer, &chunksize, &blockoffset); +#endif + + block = (AllocBlock) ((char *) pointer - blockoffset); + + set = block->aset; + + return &set->header; +} + /* * AllocSetGetChunkSpace * Given a currently-allocated chunk, determine the total space * it occupies (including all memory-allocation overhead). */ -static Size -AllocSetGetChunkSpace(MemoryContext context, void *pointer) +Size +AllocSetGetChunkSpace(void *pointer) { - AllocChunk chunk = AllocPointerGetChunk(pointer); - Size result; + Size chunksize; + Size blockoffset; + Size hdrsize; +#ifdef MEMORY_CONTEXT_CHECKING + Size reqsize; - VALGRIND_MAKE_MEM_DEFINED(chunk, ALLOCCHUNK_PRIVATE_LEN); - result = chunk->size + ALLOC_CHUNKHDRSZ; - VALGRIND_MAKE_MEM_NOACCESS(chunk, ALLOCCHUNK_PRIVATE_LEN); - return result; + hdrsize = GenericChunkHeaderDecode(pointer, &chunksize, &blockoffset, &reqsize); +#else + hdrsize = GenericChunkHeaderDecode(pointer, &chunksize, &blockoffset); +#endif + + return hdrsize + chunksize; } /* * AllocSetIsEmpty * Is an allocset empty of any allocated space? */ -static bool +bool AllocSetIsEmpty(MemoryContext context) { /* @@ -1339,7 +1345,7 @@ AllocSetIsEmpty(MemoryContext context) * totals: if not NULL, add stats about this context into *totals. * print_to_stderr: print stats to stderr if true, elog otherwise. */ -static void +void AllocSetStats(MemoryContext context, MemoryStatsPrintFunc printfunc, void *passthru, MemoryContextCounters *totals, bool print_to_stderr) @@ -1363,13 +1369,19 @@ AllocSetStats(MemoryContext context, } for (fidx = 0; fidx < ALLOCSET_NUM_FREELISTS; fidx++) { - AllocChunk chunk; + void *pointer = set->freelist[fidx]; + Size chunk_size = (1 << ALLOC_MINBITS) << fidx; - for (chunk = set->freelist[fidx]; chunk != NULL; - chunk = (AllocChunk) chunk->aset) + while (pointer != NULL) { freechunks++; - freespace += chunk->size + ALLOC_CHUNKHDRSZ; + freespace += chunk_size + SMALL_CHUNK_SIZE; + + /* + * Set pointer to the next freelist item which is stored in the + * pointed to memory. + */ + pointer = *((void **) pointer); } } @@ -1404,7 +1416,7 @@ AllocSetStats(MemoryContext context, * find yourself in an infinite loop when trouble occurs, because this * routine will be entered again when elog cleanup tries to release memory! */ -static void +void AllocSetCheck(MemoryContext context) { AllocSet set = (AllocSet) context; @@ -1421,6 +1433,7 @@ AllocSetCheck(MemoryContext context) long blk_used = block->freeptr - bpoz; long blk_data = 0; long nchunks = 0; + Size expected_hdrsize; if (set->keeper == block) total_allocated += block->endptr - ((char *) set); @@ -1447,69 +1460,64 @@ AllocSetCheck(MemoryContext context) elog(WARNING, "problem in alloc set %s: corrupt header in block %p", name, block); + if (block->large_chunk) + expected_hdrsize = LARGE_CHUNK_SIZE; + else + expected_hdrsize = SMALL_CHUNK_SIZE; + /* * Chunk walker */ while (bpoz < block->freeptr) { - AllocChunk chunk = (AllocChunk) bpoz; - Size chsize, - dsize; + void *pointer; + Size chsize; + Size blkoff; + Size hdrsize; + Size reqsize; - /* Allow access to private part of chunk header. */ - VALGRIND_MAKE_MEM_DEFINED(chunk, ALLOCCHUNK_PRIVATE_LEN); - - chsize = chunk->size; /* aligned chunk size */ - dsize = chunk->requested_size; /* real data */ + pointer = bpoz + expected_hdrsize; + hdrsize = GenericChunkHeaderDecode(pointer, &chsize, &blkoff, &reqsize); /* * Check chunk size */ - if (dsize > chsize) - elog(WARNING, "problem in alloc set %s: req size > alloc size for chunk %p in block %p", - name, chunk, block); + if (expected_hdrsize != hdrsize) + elog(WARNING, "problem in alloc set %s: expected chunk header size is %zu but actual size is %zu", + name, expected_hdrsize, hdrsize); + if (reqsize > chsize) + elog(WARNING, "problem in alloc set %s: req size > alloc size for pointer %p in block %p", + name, pointer, block); if (chsize < (1 << ALLOC_MINBITS)) - elog(WARNING, "problem in alloc set %s: bad size %zu for chunk %p in block %p", - name, chsize, chunk, block); + elog(WARNING, "problem in alloc set %s: bad size %zu for pointer %p in block %p", + name, chsize, pointer, block); /* single-chunk block? */ if (chsize > set->allocChunkLimit && - chsize + ALLOC_CHUNKHDRSZ != blk_used) - elog(WARNING, "problem in alloc set %s: bad single-chunk %p in block %p", - name, chunk, block); + chsize + expected_hdrsize != blk_used) + elog(WARNING, "problem in alloc set %s: bad single-chunk pointer %p in block %p", + name, pointer, block); - /* - * If chunk is allocated, check for correct aset pointer. (If it's - * free, the aset is the freelist pointer, which we can't check as - * easily...) Note this is an incomplete test, since palloc(0) - * produces an allocated chunk with requested_size == 0. - */ - if (dsize > 0 && chunk->aset != (void *) set) - elog(WARNING, "problem in alloc set %s: bogus aset link in block %p, chunk %p", - name, block, chunk); + /* Check the block offset correctly matches this block */ + if ((AllocBlock) ((char *) pointer - blkoff) != block) + elog(WARNING, "problem in alloc set %s: bad block offset for pointer %p in block %p", + name, pointer, block); /* * Check for overwrite of padding space in an allocated chunk. */ - if (chunk->aset == (void *) set && dsize < chsize && - !sentinel_ok(chunk, ALLOC_CHUNKHDRSZ + dsize)) - elog(WARNING, "problem in alloc set %s: detected write past chunk end in block %p, chunk %p", - name, block, chunk); - - /* - * If chunk is allocated, disallow external access to private part - * of chunk header. - */ - if (chunk->aset == (void *) set) - VALGRIND_MAKE_MEM_NOACCESS(chunk, ALLOCCHUNK_PRIVATE_LEN); + if (reqsize > 0 && reqsize < chsize && + !sentinel_ok(pointer, reqsize)) + elog(WARNING, "problem in alloc set %s: detected write past chunk end in block %p, pointer %p", + name, block, pointer); blk_data += chsize; nchunks++; - bpoz += ALLOC_CHUNKHDRSZ + chsize; + bpoz += expected_hdrsize + chsize; } - if ((blk_data + (nchunks * ALLOC_CHUNKHDRSZ)) != blk_used) + if (bpoz != block->freeptr) elog(WARNING, "problem in alloc set %s: found inconsistent memory block %p", name, block); } diff --git a/src/backend/utils/mmgr/generation.c b/src/backend/utils/mmgr/generation.c index e530e272e0..3a4e7588c6 100644 --- a/src/backend/utils/mmgr/generation.c +++ b/src/backend/utils/mmgr/generation.c @@ -39,15 +39,15 @@ #include "port/pg_bitutils.h" #include "utils/memdebug.h" #include "utils/memutils.h" +#include "utils/memutils_generichdr.h" +#include "utils/memutils_internal.h" #define Generation_BLOCKHDRSZ MAXALIGN(sizeof(GenerationBlock)) -#define Generation_CHUNKHDRSZ sizeof(GenerationChunk) #define Generation_CHUNK_FRACTION 8 typedef struct GenerationBlock GenerationBlock; /* forward reference */ -typedef struct GenerationChunk GenerationChunk; typedef void *GenerationPointer; @@ -89,112 +89,29 @@ typedef struct GenerationContext struct GenerationBlock { dlist_node node; /* doubly-linked list of blocks */ + GenerationContext *context; Size blksize; /* allocated size of this block */ int nchunks; /* number of chunks in the block */ int nfree; /* number of free chunks */ + bool large_chunk; /* true if the only chunk is a large chunk */ char *freeptr; /* start of free space in this block */ char *endptr; /* end of space in this block */ }; -/* - * GenerationChunk - * The prefix of each piece of memory in a GenerationBlock - * - * Note: to meet the memory context APIs, the payload area of the chunk must - * be maxaligned, and the "context" link must be immediately adjacent to the - * payload area (cf. GetMemoryChunkContext). We simplify matters for this - * module by requiring sizeof(GenerationChunk) to be maxaligned, and then - * we can ensure things work by adding any required alignment padding before - * the pointer fields. There is a static assertion below that the alignment - * is done correctly. - */ -struct GenerationChunk -{ - /* size is always the size of the usable space in the chunk */ - Size size; -#ifdef MEMORY_CONTEXT_CHECKING - /* when debugging memory usage, also store actual requested size */ - /* this is zero in a free chunk */ - Size requested_size; - -#define GENERATIONCHUNK_RAWSIZE (SIZEOF_SIZE_T * 2 + SIZEOF_VOID_P * 2) -#else -#define GENERATIONCHUNK_RAWSIZE (SIZEOF_SIZE_T + SIZEOF_VOID_P * 2) -#endif /* MEMORY_CONTEXT_CHECKING */ - - /* ensure proper alignment by adding padding if needed */ -#if (GENERATIONCHUNK_RAWSIZE % MAXIMUM_ALIGNOF) != 0 - char padding[MAXIMUM_ALIGNOF - GENERATIONCHUNK_RAWSIZE % MAXIMUM_ALIGNOF]; -#endif - - GenerationBlock *block; /* block owning this chunk */ - GenerationContext *context; /* owning context, or NULL if freed chunk */ - /* there must not be any padding to reach a MAXALIGN boundary here! */ -}; - -/* - * Only the "context" field should be accessed outside this module. - * We keep the rest of an allocated chunk's header marked NOACCESS when using - * valgrind. But note that freed chunk headers are kept accessible, for - * simplicity. - */ -#define GENERATIONCHUNK_PRIVATE_LEN offsetof(GenerationChunk, context) - /* * GenerationIsValid * True iff set is valid allocation set. */ #define GenerationIsValid(set) PointerIsValid(set) -#define GenerationPointerGetChunk(ptr) \ - ((GenerationChunk *)(((char *)(ptr)) - Generation_CHUNKHDRSZ)) -#define GenerationChunkGetPointer(chk) \ - ((GenerationPointer *)(((char *)(chk)) + Generation_CHUNKHDRSZ)) - /* Inlined helper functions */ -static inline void GenerationBlockInit(GenerationBlock *block, Size blksize); +static inline void GenerationBlockInit(GenerationContext *context, GenerationBlock *block, Size blksize); static inline bool GenerationBlockIsEmpty(GenerationBlock *block); static inline void GenerationBlockMarkEmpty(GenerationBlock *block); static inline Size GenerationBlockFreeBytes(GenerationBlock *block); static inline void GenerationBlockFree(GenerationContext *set, GenerationBlock *block); -/* - * These functions implement the MemoryContext API for Generation contexts. - */ -static void *GenerationAlloc(MemoryContext context, Size size); -static void GenerationFree(MemoryContext context, void *pointer); -static void *GenerationRealloc(MemoryContext context, void *pointer, Size size); -static void GenerationReset(MemoryContext context); -static void GenerationDelete(MemoryContext context); -static Size GenerationGetChunkSpace(MemoryContext context, void *pointer); -static bool GenerationIsEmpty(MemoryContext context); -static void GenerationStats(MemoryContext context, - MemoryStatsPrintFunc printfunc, void *passthru, - MemoryContextCounters *totals, - bool print_to_stderr); - -#ifdef MEMORY_CONTEXT_CHECKING -static void GenerationCheck(MemoryContext context); -#endif - -/* - * This is the virtual function table for Generation contexts. - */ -static const MemoryContextMethods GenerationMethods = { - GenerationAlloc, - GenerationFree, - GenerationRealloc, - GenerationReset, - GenerationDelete, - GenerationGetChunkSpace, - GenerationIsEmpty, - GenerationStats -#ifdef MEMORY_CONTEXT_CHECKING - ,GenerationCheck -#endif -}; - /* * Public routines @@ -223,13 +140,6 @@ GenerationContextCreate(MemoryContext parent, GenerationContext *set; GenerationBlock *block; - /* Assert we padded GenerationChunk properly */ - StaticAssertStmt(Generation_CHUNKHDRSZ == MAXALIGN(Generation_CHUNKHDRSZ), - "sizeof(GenerationChunk) is not maxaligned"); - StaticAssertStmt(offsetof(GenerationChunk, context) + sizeof(MemoryContext) == - Generation_CHUNKHDRSZ, - "padding calculation in GenerationChunk is wrong"); - /* * First, validate allocation parameters. Asserts seem sufficient because * nobody varies their parameters at runtime. We somewhat arbitrarily @@ -244,10 +154,11 @@ GenerationContextCreate(MemoryContext parent, (minContextSize == MAXALIGN(minContextSize) && minContextSize >= 1024 && minContextSize <= maxBlockSize)); + Assert(maxBlockSize <= SMALL_CHUNK_LIMIT); /* Determine size of initial block */ allocSize = MAXALIGN(sizeof(GenerationContext)) + - Generation_BLOCKHDRSZ + Generation_CHUNKHDRSZ; + Generation_BLOCKHDRSZ + LARGE_CHUNK_SIZE; if (minContextSize != 0) allocSize = Max(allocSize, minContextSize); else @@ -278,7 +189,7 @@ GenerationContextCreate(MemoryContext parent, block = (GenerationBlock *) (((char *) set) + MAXALIGN(sizeof(GenerationContext))); /* determine the block size and initialize it */ firstBlockSize = allocSize - MAXALIGN(sizeof(GenerationContext)); - GenerationBlockInit(block, firstBlockSize); + GenerationBlockInit(set, block, firstBlockSize); /* add it to the doubly-linked list of blocks */ dlist_push_head(&set->blocks, &block->node); @@ -302,15 +213,15 @@ GenerationContextCreate(MemoryContext parent, * * Follows similar ideas as AllocSet, see aset.c for details ... */ - set->allocChunkLimit = maxBlockSize; - while ((Size) (set->allocChunkLimit + Generation_CHUNKHDRSZ) > + set->allocChunkLimit = Min(maxBlockSize, SMALL_CHUNK_LIMIT); + while ((Size) (set->allocChunkLimit + SMALL_CHUNK_SIZE) > (Size) ((Size) (maxBlockSize - Generation_BLOCKHDRSZ) / Generation_CHUNK_FRACTION)) set->allocChunkLimit >>= 1; /* Finally, do the type-independent part of context creation */ MemoryContextCreate((MemoryContext) set, T_GenerationContext, - &GenerationMethods, + MCTX_GENERATION_ID, parent, name); @@ -326,7 +237,7 @@ GenerationContextCreate(MemoryContext parent, * The code simply frees all the blocks in the context - we don't keep any * keeper blocks or anything like that. */ -static void +void GenerationReset(MemoryContext context) { GenerationContext *set = (GenerationContext *) context; @@ -371,7 +282,7 @@ GenerationReset(MemoryContext context) * GenerationDelete * Free all memory which is allocated in the given context. */ -static void +void GenerationDelete(MemoryContext context) { /* Reset to release all releasable GenerationBlocks */ @@ -393,19 +304,23 @@ GenerationDelete(MemoryContext context) * is marked, as mcxt.c will set it to UNDEFINED. In some paths we will * return space that is marked NOACCESS - GenerationRealloc has to beware! */ -static void * +void * GenerationAlloc(MemoryContext context, Size size) { GenerationContext *set = (GenerationContext *) context; GenerationBlock *block; - GenerationChunk *chunk; Size chunk_size = MAXALIGN(size); - Size required_size = chunk_size + Generation_CHUNKHDRSZ; + Size genhdrsize; + Size required_size; + void *pointer; /* is it an over-sized chunk? if yes, allocate special block */ if (chunk_size > set->allocChunkLimit) { - Size blksize = required_size + Generation_BLOCKHDRSZ; + Size blksize; + + genhdrsize = GenericChunkHeaderSize(Generation_BLOCKHDRSZ + LARGE_CHUNK_SIZE, chunk_size); + blksize = Generation_BLOCKHDRSZ + genhdrsize + chunk_size; block = (GenerationBlock *) malloc(blksize); if (block == NULL) @@ -414,24 +329,27 @@ GenerationAlloc(MemoryContext context, Size size) context->mem_allocated += blksize; /* block with a single (used) chunk */ + block->context = set; block->blksize = blksize; block->nchunks = 1; block->nfree = 0; + block->large_chunk = (genhdrsize == LARGE_CHUNK_SIZE); /* the block is completely full */ block->freeptr = block->endptr = ((char *) block) + blksize; - chunk = (GenerationChunk *) (((char *) block) + Generation_BLOCKHDRSZ); - chunk->block = block; - chunk->context = set; - chunk->size = chunk_size; + pointer = (void *) ((char *) block + Generation_BLOCKHDRSZ + genhdrsize); #ifdef MEMORY_CONTEXT_CHECKING - chunk->requested_size = size; + GenericChunkHeaderEncode(pointer, block, chunk_size, size, MCTX_GENERATION_ID); + /* set mark to catch clobber of "unused" space */ if (size < chunk_size) - set_sentinel(GenerationChunkGetPointer(chunk), size); + set_sentinel(pointer, size); +#else + GenericChunkHeaderEncode(pointer, block, chunk_size, MCTX_GENERATION_ID); #endif + #ifdef RANDOMIZE_ALLOCATED_MEMORY /* fill the allocated space with junk */ randomize_mem((char *) GenerationChunkGetPointer(chunk), size); @@ -441,13 +359,10 @@ GenerationAlloc(MemoryContext context, Size size) dlist_push_head(&set->blocks, &block->node); /* Ensure any padding bytes are marked NOACCESS. */ - VALGRIND_MAKE_MEM_NOACCESS((char *) GenerationChunkGetPointer(chunk) + size, + VALGRIND_MAKE_MEM_NOACCESS((char *) pointer + size, chunk_size - size); - /* Disallow external access to private part of chunk header. */ - VALGRIND_MAKE_MEM_NOACCESS(chunk, GENERATIONCHUNK_PRIVATE_LEN); - - return GenerationChunkGetPointer(chunk); + return pointer; } /* @@ -466,6 +381,7 @@ GenerationAlloc(MemoryContext context, Size size) * that would compound over time. */ block = set->block; + required_size = SMALL_CHUNK_SIZE + chunk_size; if (block == NULL || GenerationBlockFreeBytes(block) < required_size) @@ -492,6 +408,7 @@ GenerationAlloc(MemoryContext context, Size size) } else { + /* * The first such block has size initBlockSize, and we double the * space in each succeeding block, but not more than maxBlockSize. @@ -501,12 +418,9 @@ GenerationAlloc(MemoryContext context, Size size) if (set->nextBlockSize > set->maxBlockSize) set->nextBlockSize = set->maxBlockSize; - /* we'll need a block hdr too, so add that to the required size */ - required_size += Generation_BLOCKHDRSZ; - /* round the size up to the next power of 2 */ - if (blksize < required_size) - blksize = pg_nextpower2_size_t(required_size); + if (blksize < required_size + Generation_BLOCKHDRSZ) + blksize = pg_nextpower2_size_t(required_size + Generation_BLOCKHDRSZ); block = (GenerationBlock *) malloc(blksize); @@ -516,7 +430,7 @@ GenerationAlloc(MemoryContext context, Size size) context->mem_allocated += blksize; /* initialize the new block */ - GenerationBlockInit(block, blksize); + GenerationBlockInit(set, block, blksize); /* add it to the doubly-linked list of blocks */ dlist_push_head(&set->blocks, &block->node); @@ -531,41 +445,34 @@ GenerationAlloc(MemoryContext context, Size size) /* we're supposed to have a block with enough free space now */ Assert(block != NULL); - Assert((block->endptr - block->freeptr) >= Generation_CHUNKHDRSZ + chunk_size); - - chunk = (GenerationChunk *) block->freeptr; + Assert((block->endptr - block->freeptr) >= required_size); - /* Prepare to initialize the chunk header. */ - VALGRIND_MAKE_MEM_UNDEFINED(chunk, Generation_CHUNKHDRSZ); + pointer = (void *) (block->freeptr + SMALL_CHUNK_SIZE); block->nchunks += 1; - block->freeptr += (Generation_CHUNKHDRSZ + chunk_size); + block->freeptr += required_size; Assert(block->freeptr <= block->endptr); - chunk->block = block; - chunk->context = set; - chunk->size = chunk_size; - #ifdef MEMORY_CONTEXT_CHECKING - chunk->requested_size = size; + GenericChunkHeaderEncode(pointer, block, chunk_size, size, MCTX_GENERATION_ID); /* set mark to catch clobber of "unused" space */ - if (size < chunk->size) - set_sentinel(GenerationChunkGetPointer(chunk), size); + if (size < chunk_size) + set_sentinel(pointer, size); +#else + GenericChunkHeaderEncode(pointer, block, chunk_size, MCTX_GENERATION_ID); #endif + #ifdef RANDOMIZE_ALLOCATED_MEMORY /* fill the allocated space with junk */ - randomize_mem((char *) GenerationChunkGetPointer(chunk), size); + randomize_mem(pointer, size); #endif /* Ensure any padding bytes are marked NOACCESS. */ - VALGRIND_MAKE_MEM_NOACCESS((char *) GenerationChunkGetPointer(chunk) + size, + VALGRIND_MAKE_MEM_NOACCESS((char *) pointer + size, chunk_size - size); - /* Disallow external access to private part of chunk header. */ - VALGRIND_MAKE_MEM_NOACCESS(chunk, GENERATIONCHUNK_PRIVATE_LEN); - - return GenerationChunkGetPointer(chunk); + return pointer; } /* @@ -574,12 +481,14 @@ GenerationAlloc(MemoryContext context, Size size) * mem_allocated field. */ static inline void -GenerationBlockInit(GenerationBlock *block, Size blksize) +GenerationBlockInit(GenerationContext *context, GenerationBlock *block, Size blksize) { + block->context = context; block->blksize = blksize; block->nchunks = 0; block->nfree = 0; + block->large_chunk = false; block->freeptr = ((char *) block) + Generation_BLOCKHDRSZ; block->endptr = ((char *) block) + blksize; @@ -661,36 +570,41 @@ GenerationBlockFree(GenerationContext *set, GenerationBlock *block) * Update number of chunks in the block, and if all chunks in the block * are now free then discard the block. */ -static void -GenerationFree(MemoryContext context, void *pointer) +void +GenerationFree(void *pointer) { - GenerationContext *set = (GenerationContext *) context; - GenerationChunk *chunk = GenerationPointerGetChunk(pointer); GenerationBlock *block; + GenerationContext *set; + Size chunksize; + Size blockoffset; +#ifdef MEMORY_CONTEXT_CHECKING + Size reqsize; + + GenericChunkHeaderDecode(pointer, &chunksize, &blockoffset, &reqsize); + +#else + GenericChunkHeaderDecode(pointer, &chunksize, &blockoffset); +#endif - /* Allow access to private part of chunk header. */ - VALGRIND_MAKE_MEM_DEFINED(chunk, GENERATIONCHUNK_PRIVATE_LEN); + block = (GenerationBlock *) (((char *) pointer) - blockoffset); - block = chunk->block; + set = block->context; #ifdef MEMORY_CONTEXT_CHECKING /* Test for someone scribbling on unused space in chunk */ - if (chunk->requested_size < chunk->size) - if (!sentinel_ok(pointer, chunk->requested_size)) + if (reqsize < chunksize) + if (!sentinel_ok(pointer, reqsize)) elog(WARNING, "detected write past chunk end in %s %p", - ((MemoryContext) set)->name, chunk); + ((MemoryContext) set)->name, pointer); #endif #ifdef CLOBBER_FREED_MEMORY - wipe_mem(pointer, chunk->size); + wipe_mem(pointer, chunksize); #endif - /* Reset context to NULL in freed chunks */ - chunk->context = NULL; - #ifdef MEMORY_CONTEXT_CHECKING /* Reset requested_size to 0 in freed chunks */ - chunk->requested_size = 0; + GenericChunkHeaderEncode(pointer, (void *) block, chunksize, 0, MCTX_GENERATION_ID); #endif block->nfree += 1; @@ -732,7 +646,7 @@ GenerationFree(MemoryContext context, void *pointer) */ dlist_delete(&block->node); - context->mem_allocated -= block->blksize; + set->header.mem_allocated -= block->blksize; free(block); } @@ -742,25 +656,32 @@ GenerationFree(MemoryContext context, void *pointer) * and discard the old one. The only exception is when the new size fits * into the old chunk - in that case we just update chunk header. */ -static void * -GenerationRealloc(MemoryContext context, void *pointer, Size size) +void * +GenerationRealloc(void *pointer, Size size) { - GenerationContext *set = (GenerationContext *) context; - GenerationChunk *chunk = GenerationPointerGetChunk(pointer); + GenerationBlock *block; + GenerationContext *set; GenerationPointer newPointer; Size oldsize; + Size blockoffset; +#ifdef MEMORY_CONTEXT_CHECKING + Size oldreqsize; - /* Allow access to private part of chunk header. */ - VALGRIND_MAKE_MEM_DEFINED(chunk, GENERATIONCHUNK_PRIVATE_LEN); + GenericChunkHeaderDecode(pointer, &oldsize, &blockoffset, &oldreqsize); - oldsize = chunk->size; +#else + GenericChunkHeaderDecode(pointer, &oldsize, &blockoffset); +#endif + + block = (GenerationBlock *) (((char *) pointer) - blockoffset); + set = block->context; #ifdef MEMORY_CONTEXT_CHECKING /* Test for someone scribbling on unused space in chunk */ - if (chunk->requested_size < oldsize) - if (!sentinel_ok(pointer, chunk->requested_size)) + if (oldreqsize < oldsize) + if (!sentinel_ok(pointer, oldreqsize)) elog(WARNING, "detected write past chunk end in %s %p", - ((MemoryContext) set)->name, chunk); + ((MemoryContext) set)->name, pointer); #endif /* @@ -776,24 +697,20 @@ GenerationRealloc(MemoryContext context, void *pointer, Size size) if (oldsize >= size) { #ifdef MEMORY_CONTEXT_CHECKING - Size oldrequest = chunk->requested_size; #ifdef RANDOMIZE_ALLOCATED_MEMORY /* We can only fill the extra space if we know the prior request */ - if (size > oldrequest) - randomize_mem((char *) pointer + oldrequest, - size - oldrequest); + if (size > oldreqsize) + randomize_mem((char *) pointer + oldreqsize, + size - oldreqsize); #endif - - chunk->requested_size = size; - /* * If this is an increase, mark any newly-available part UNDEFINED. * Otherwise, mark the obsolete part NOACCESS. */ - if (size > oldrequest) - VALGRIND_MAKE_MEM_UNDEFINED((char *) pointer + oldrequest, - size - oldrequest); + if (size > oldreqsize) + VALGRIND_MAKE_MEM_UNDEFINED((char *) pointer + oldreqsize, + size - oldreqsize); else VALGRIND_MAKE_MEM_NOACCESS((char *) pointer + size, oldsize - size); @@ -801,6 +718,9 @@ GenerationRealloc(MemoryContext context, void *pointer, Size size) /* set mark to catch clobber of "unused" space */ if (size < oldsize) set_sentinel(pointer, size); + + /* Re-encode the chunk to apply new request size */ + GenericChunkHeaderEncode(pointer, block, oldsize, size, MCTX_GENERATION_ID); #else /* !MEMORY_CONTEXT_CHECKING */ /* @@ -812,9 +732,6 @@ GenerationRealloc(MemoryContext context, void *pointer, Size size) VALGRIND_MAKE_MEM_DEFINED(pointer, size); #endif - /* Disallow external access to private part of chunk header. */ - VALGRIND_MAKE_MEM_NOACCESS(chunk, GENERATIONCHUNK_PRIVATE_LEN); - return pointer; } @@ -824,8 +741,6 @@ GenerationRealloc(MemoryContext context, void *pointer, Size size) /* leave immediately if request was not completed */ if (newPointer == NULL) { - /* Disallow external access to private part of chunk header. */ - VALGRIND_MAKE_MEM_NOACCESS(chunk, GENERATIONCHUNK_PRIVATE_LEN); return NULL; } @@ -839,7 +754,7 @@ GenerationRealloc(MemoryContext context, void *pointer, Size size) */ VALGRIND_MAKE_MEM_UNDEFINED(newPointer, size); #ifdef MEMORY_CONTEXT_CHECKING - oldsize = chunk->requested_size; + oldsize = oldreqsize; #else VALGRIND_MAKE_MEM_DEFINED(pointer, oldsize); #endif @@ -848,7 +763,7 @@ GenerationRealloc(MemoryContext context, void *pointer, Size size) memcpy(newPointer, pointer, oldsize); /* free old chunk */ - GenerationFree((MemoryContext) set, pointer); + GenerationFree(pointer); return newPointer; } @@ -858,23 +773,52 @@ GenerationRealloc(MemoryContext context, void *pointer, Size size) * Given a currently-allocated chunk, determine the total space * it occupies (including all memory-allocation overhead). */ -static Size -GenerationGetChunkSpace(MemoryContext context, void *pointer) +MemoryContext +GenerationGetChunkContext(void *pointer) +{ + GenerationBlock *block; + Size chunksize; + Size blockoffset; +#ifdef MEMORY_CONTEXT_CHECKING + Size reqsize; + + GenericChunkHeaderDecode(pointer, &chunksize, &blockoffset, &reqsize); +#else + GenericChunkHeaderDecode(pointer, &chunksize, &blockoffset); +#endif + + block = (GenerationBlock *) (((char *) pointer) - blockoffset); + + return &block->context->header; +} + +/* + * GenerationGetChunkSpace + * Given a currently-allocated chunk, determine the total space + * it occupies (including all memory-allocation overhead). + */ +Size +GenerationGetChunkSpace(void *pointer) { - GenerationChunk *chunk = GenerationPointerGetChunk(pointer); - Size result; + Size chunksize; + Size blockoffset; + Size hdrsize; +#ifdef MEMORY_CONTEXT_CHECKING + Size reqsize; - VALGRIND_MAKE_MEM_DEFINED(chunk, GENERATIONCHUNK_PRIVATE_LEN); - result = chunk->size + Generation_CHUNKHDRSZ; - VALGRIND_MAKE_MEM_NOACCESS(chunk, GENERATIONCHUNK_PRIVATE_LEN); - return result; + hdrsize = GenericChunkHeaderDecode(pointer, &chunksize, &blockoffset, &reqsize); +#else + hdrsize = GenericChunkHeaderDecode(pointer, &chunksize, &blockoffset); +#endif + + return hdrsize + chunksize; } /* * GenerationIsEmpty * Is a GenerationContext empty of any allocated space? */ -static bool +bool GenerationIsEmpty(MemoryContext context) { GenerationContext *set = (GenerationContext *) context; @@ -903,7 +847,7 @@ GenerationIsEmpty(MemoryContext context) * XXX freespace only accounts for empty space at the end of the block, not * space of freed chunks (which is unknown). */ -static void +void GenerationStats(MemoryContext context, MemoryStatsPrintFunc printfunc, void *passthru, MemoryContextCounters *totals, bool print_to_stderr) @@ -961,7 +905,7 @@ GenerationStats(MemoryContext context, * find yourself in an infinite loop when trouble occurs, because this * routine will be entered again when elog cleanup tries to release memory! */ -static void +void GenerationCheck(MemoryContext context) { GenerationContext *gen = (GenerationContext *) context; @@ -973,6 +917,7 @@ GenerationCheck(MemoryContext context) dlist_foreach(iter, &gen->blocks) { GenerationBlock *block = dlist_container(GenerationBlock, node, iter.cur); + Size expected_hdrsize; int nfree, nchunks; char *ptr; @@ -987,6 +932,16 @@ GenerationCheck(MemoryContext context) elog(WARNING, "problem in Generation %s: number of free chunks %d in block %p exceeds %d allocated", name, block->nfree, block, block->nchunks); + /* check block belongs to the correct context */ + if (block->context != gen) + elog(WARNING, "problem in Generation %s: bogus context link in block %p", + name, block); + + if (block->large_chunk) + expected_hdrsize = LARGE_CHUNK_SIZE; + else + expected_hdrsize = SMALL_CHUNK_SIZE; + /* Now walk through the chunks and count them. */ nfree = 0; nchunks = 0; @@ -994,55 +949,43 @@ GenerationCheck(MemoryContext context) while (ptr < block->freeptr) { - GenerationChunk *chunk = (GenerationChunk *) ptr; + void *pointer = ptr + expected_hdrsize; + Size chsize; + Size blkoff; + Size hdrsize; + Size reqsize; + - /* Allow access to private part of chunk header. */ - VALGRIND_MAKE_MEM_DEFINED(chunk, GENERATIONCHUNK_PRIVATE_LEN); + hdrsize = GenericChunkHeaderDecode(pointer, &chsize, &blkoff, &reqsize); /* move to the next chunk */ - ptr += (chunk->size + Generation_CHUNKHDRSZ); + ptr += hdrsize + chsize; nchunks += 1; - /* chunks have both block and context pointers, so check both */ - if (chunk->block != block) - elog(WARNING, "problem in Generation %s: bogus block link in block %p, chunk %p", - name, block, chunk); + if (expected_hdrsize != hdrsize) + elog(WARNING, "problem in Generation %s: expected chunk header size is %zu but actual size is %zu", + name, expected_hdrsize, hdrsize); - /* - * Check for valid context pointer. Note this is an incomplete - * test, since palloc(0) produces an allocated chunk with - * requested_size == 0. - */ - if ((chunk->requested_size > 0 && chunk->context != gen) || - (chunk->context != gen && chunk->context != NULL)) - elog(WARNING, "problem in Generation %s: bogus context link in block %p, chunk %p", - name, block, chunk); + /* Check the block offset correctly matches this block */ + if ((GenerationBlock *) ((char *) pointer - blkoff) != block) + elog(WARNING, "problem in Generation %s: bogus block link in block %p, pointer %p", + name, block, pointer); /* now make sure the chunk size is correct */ - if (chunk->size < chunk->requested_size || - chunk->size != MAXALIGN(chunk->size)) - elog(WARNING, "problem in Generation %s: bogus chunk size in block %p, chunk %p", - name, block, chunk); + if (chsize < reqsize || chsize != MAXALIGN(chsize)) + elog(WARNING, "problem in Generation %s: bogus chunk size in block %p, pointer %p", + name, block, pointer); - /* is chunk allocated? */ - if (chunk->context != NULL) + if (reqsize > 0) { - /* check sentinel, but only in allocated blocks */ - if (chunk->requested_size < chunk->size && - !sentinel_ok(chunk, Generation_CHUNKHDRSZ + chunk->requested_size)) - elog(WARNING, "problem in Generation %s: detected write past chunk end in block %p, chunk %p", - name, block, chunk); + /* check sentinel */ + if (reqsize < chsize && !sentinel_ok(pointer, reqsize)) + elog(WARNING, "problem in Generation %s: detected write past chunk end in block %p, pointer %p", + name, block, pointer); } else nfree += 1; - - /* - * If chunk is allocated, disallow external access to private part - * of chunk header. - */ - if (chunk->context != NULL) - VALGRIND_MAKE_MEM_NOACCESS(chunk, GENERATIONCHUNK_PRIVATE_LEN); } /* diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c index e12be1b9bd..f7cd413eea 100644 --- a/src/backend/utils/mmgr/mcxt.c +++ b/src/backend/utils/mmgr/mcxt.c @@ -29,12 +29,60 @@ #include "utils/fmgrprotos.h" #include "utils/memdebug.h" #include "utils/memutils.h" +#include "utils/memutils_internal.h" /***************************************************************************** * GLOBAL MEMORY * *****************************************************************************/ +static const MemoryContextMethods mcxt_methods[] = { + [MCTX_ASET_ID] = { + AllocSetAlloc, + AllocSetFree, + AllocSetRealloc, + AllocSetReset, + AllocSetDelete, + AllocSetGetChunkContext, + AllocSetGetChunkSpace, + AllocSetIsEmpty, + AllocSetStats +#ifdef MEMORY_CONTEXT_CHECKING + ,AllocSetCheck +#endif + }, + + [MCTX_GENERATION_ID] = { + GenerationAlloc, + GenerationFree, + GenerationRealloc, + GenerationReset, + GenerationDelete, + GenerationGetChunkContext, + GenerationGetChunkSpace, + GenerationIsEmpty, + GenerationStats +#ifdef MEMORY_CONTEXT_CHECKING + ,GenerationCheck +#endif + }, + + [MCTX_SLAB_ID] = { + SlabAlloc, + SlabFree, + SlabRealloc, + SlabReset, + SlabDelete, + SlabGetChunkContext, + SlabGetChunkSpace, + SlabIsEmpty, + SlabStats +#ifdef MEMORY_CONTEXT_CHECKING + ,SlabCheck +#endif + }, +}; + /* * CurrentMemoryContext * Default memory context for allocations. @@ -74,6 +122,9 @@ static void MemoryContextStatsPrint(MemoryContext context, void *passthru, Assert(CritSectionCount == 0 || (context)->allowInCritSection) +#define MCXT_METHOD(pointer, method) mcxt_methods[GetMemoryChunkMethodID(pointer)].method + + /***************************************************************************** * EXPORTED ROUTINES * *****************************************************************************/ @@ -422,6 +473,17 @@ MemoryContextAllowInCriticalSection(MemoryContext context, bool allow) context->allowInCritSection = allow; } +/* + * GetMemoryChunkContext + * Given a currently-allocated chunk, determine the MemoryContext that + * the chunk belongs to. + */ +MemoryContext +GetMemoryChunkContext(void *pointer) +{ + return MCXT_METHOD(pointer, get_chunk_context)(pointer); +} + /* * GetMemoryChunkSpace * Given a currently-allocated chunk, determine the total space @@ -433,9 +495,7 @@ MemoryContextAllowInCriticalSection(MemoryContext context, bool allow) Size GetMemoryChunkSpace(void *pointer) { - MemoryContext context = GetMemoryChunkContext(pointer); - - return context->methods->get_chunk_space(context, pointer); + return MCXT_METHOD(pointer, get_chunk_space)(pointer); } /* @@ -814,7 +874,7 @@ MemoryContextContains(MemoryContext context, void *pointer) void MemoryContextCreate(MemoryContext node, NodeTag tag, - const MemoryContextMethods *methods, + MemoryContextMethodID method_id, MemoryContext parent, const char *name) { @@ -824,7 +884,7 @@ MemoryContextCreate(MemoryContext node, /* Initialize all standard fields of memory context header */ node->type = tag; node->isReset = true; - node->methods = methods; + node->methods = &mcxt_methods[method_id]; node->parent = parent; node->firstchild = NULL; node->mem_allocated = 0; @@ -1174,10 +1234,8 @@ palloc_extended(Size size, int flags) void pfree(void *pointer) { - MemoryContext context = GetMemoryChunkContext(pointer); - - context->methods->free_p(context, pointer); - VALGRIND_MEMPOOL_FREE(context, pointer); + MCXT_METHOD(pointer, free_p)(pointer); + VALGRIND_MEMPOOL_FREE(GetMemoryChunkContext(pointer), pointer); } /* @@ -1187,9 +1245,12 @@ pfree(void *pointer) void * repalloc(void *pointer, Size size) { - MemoryContext context = GetMemoryChunkContext(pointer); void *ret; +#if defined(USE_ASSERT_CHECKING) || defined(USE_VALGRIND) + MemoryContext context = GetMemoryChunkContext(pointer); +#endif + if (!AllocSizeIsValid(size)) elog(ERROR, "invalid memory alloc request size %zu", size); @@ -1198,9 +1259,11 @@ repalloc(void *pointer, Size size) /* isReset must be false already */ Assert(!context->isReset); - ret = context->methods->realloc(context, pointer, size); + ret = MCXT_METHOD(pointer, realloc)(pointer, size); if (unlikely(ret == NULL)) { + MemoryContext context = GetMemoryChunkContext(pointer); + MemoryContextStats(TopMemoryContext); ereport(ERROR, (errcode(ERRCODE_OUT_OF_MEMORY), @@ -1257,9 +1320,12 @@ MemoryContextAllocHuge(MemoryContext context, Size size) void * repalloc_huge(void *pointer, Size size) { - MemoryContext context = GetMemoryChunkContext(pointer); void *ret; +#if defined(USE_ASSERT_CHECKING) || defined(USE_VALGRIND) + MemoryContext context = GetMemoryChunkContext(pointer); +#endif + if (!AllocHugeSizeIsValid(size)) elog(ERROR, "invalid memory alloc request size %zu", size); @@ -1268,9 +1334,11 @@ repalloc_huge(void *pointer, Size size) /* isReset must be false already */ Assert(!context->isReset); - ret = context->methods->realloc(context, pointer, size); + ret = MCXT_METHOD(pointer, realloc)(pointer, size); if (unlikely(ret == NULL)) { + MemoryContext context = GetMemoryChunkContext(pointer); + MemoryContextStats(TopMemoryContext); ereport(ERROR, (errcode(ERRCODE_OUT_OF_MEMORY), diff --git a/src/backend/utils/mmgr/slab.c b/src/backend/utils/mmgr/slab.c index 67d97b22e5..a1d9326c66 100644 --- a/src/backend/utils/mmgr/slab.c +++ b/src/backend/utils/mmgr/slab.c @@ -55,6 +55,8 @@ #include "lib/ilist.h" #include "utils/memdebug.h" #include "utils/memutils.h" +#include "utils/memutils_generichdr.h" +#include "utils/memutils_internal.h" /* * SlabContext is a specialized implementation of MemoryContext. @@ -90,74 +92,15 @@ typedef struct SlabBlock dlist_node node; /* doubly-linked list */ int nfree; /* number of free chunks */ int firstFreeChunk; /* index of the first free chunk in the block */ -} SlabBlock; - -/* - * SlabChunk - * The prefix of each piece of memory in a SlabBlock - * - * Note: to meet the memory context APIs, the payload area of the chunk must - * be maxaligned, and the "slab" link must be immediately adjacent to the - * payload area (cf. GetMemoryChunkContext). Since we support no machines on - * which MAXALIGN is more than twice sizeof(void *), this happens without any - * special hacking in this struct declaration. But there is a static - * assertion below that the alignment is done correctly. - */ -typedef struct SlabChunk -{ - SlabBlock *block; /* block owning this chunk */ SlabContext *slab; /* owning context */ - /* there must not be any padding to reach a MAXALIGN boundary here! */ -} SlabChunk; - - -#define SlabPointerGetChunk(ptr) \ - ((SlabChunk *)(((char *)(ptr)) - sizeof(SlabChunk))) -#define SlabChunkGetPointer(chk) \ - ((void *)(((char *)(chk)) + sizeof(SlabChunk))) -#define SlabBlockGetChunk(slab, block, idx) \ - ((SlabChunk *) ((char *) (block) + sizeof(SlabBlock) \ - + (idx * slab->fullChunkSize))) -#define SlabBlockStart(block) \ - ((char *) block + sizeof(SlabBlock)) -#define SlabChunkIndex(slab, block, chunk) \ - (((char *) chunk - SlabBlockStart(block)) / slab->fullChunkSize) - -/* - * These functions implement the MemoryContext API for Slab contexts. - */ -static void *SlabAlloc(MemoryContext context, Size size); -static void SlabFree(MemoryContext context, void *pointer); -static void *SlabRealloc(MemoryContext context, void *pointer, Size size); -static void SlabReset(MemoryContext context); -static void SlabDelete(MemoryContext context); -static Size SlabGetChunkSpace(MemoryContext context, void *pointer); -static bool SlabIsEmpty(MemoryContext context); -static void SlabStats(MemoryContext context, - MemoryStatsPrintFunc printfunc, void *passthru, - MemoryContextCounters *totals, - bool print_to_stderr); -#ifdef MEMORY_CONTEXT_CHECKING -static void SlabCheck(MemoryContext context); -#endif - -/* - * This is the virtual function table for Slab contexts. - */ -static const MemoryContextMethods SlabMethods = { - SlabAlloc, - SlabFree, - SlabRealloc, - SlabReset, - SlabDelete, - SlabGetChunkSpace, - SlabIsEmpty, - SlabStats -#ifdef MEMORY_CONTEXT_CHECKING - ,SlabCheck -#endif -}; +} SlabBlock; +#define SlabBlockGetPointer(slab, block, idx) \ + (void *) (((char *) block + sizeof(SlabBlock)) + SMALL_CHUNK_SIZE + ((idx) * (slab)->fullChunkSize)) +#define SlabFirstChunkStart(block) \ + ((char *) block + sizeof(SlabBlock) + SMALL_CHUNK_SIZE) +#define SlabPointerGetIdx(slab, block, pointer) \ + (((char *) pointer - SlabFirstChunkStart(block)) / slab->fullChunkSize) /* * SlabContextCreate @@ -168,8 +111,7 @@ static const MemoryContextMethods SlabMethods = { * blockSize: allocation block size * chunkSize: allocation chunk size * - * The chunkSize may not exceed: - * MAXALIGN_DOWN(SIZE_MAX) - MAXALIGN(sizeof(SlabBlock)) - sizeof(SlabChunk) + * The chunkSize may not exceed SMALL_CHUNK_LIMIT */ MemoryContext SlabContextCreate(MemoryContext parent, @@ -184,19 +126,14 @@ SlabContextCreate(MemoryContext parent, SlabContext *slab; int i; - /* Assert we padded SlabChunk properly */ - StaticAssertStmt(sizeof(SlabChunk) == MAXALIGN(sizeof(SlabChunk)), - "sizeof(SlabChunk) is not maxaligned"); - StaticAssertStmt(offsetof(SlabChunk, slab) + sizeof(MemoryContext) == - sizeof(SlabChunk), - "padding calculation in SlabChunk is wrong"); + Assert(chunkSize <= SMALL_CHUNK_LIMIT); /* Make sure the linked list node fits inside a freed chunk */ if (chunkSize < sizeof(int)) chunkSize = sizeof(int); - /* chunk, including SLAB header (both addresses nicely aligned) */ - fullChunkSize = sizeof(SlabChunk) + MAXALIGN(chunkSize); + /* chunk, including small chunk header (both addresses nicely aligned) */ + fullChunkSize = SMALL_CHUNK_SIZE + MAXALIGN(chunkSize); /* Make sure the block can store at least one chunk. */ if (blockSize < fullChunkSize + sizeof(SlabBlock)) @@ -265,7 +202,7 @@ SlabContextCreate(MemoryContext parent, /* Finally, do the type-independent part of context creation */ MemoryContextCreate((MemoryContext) slab, T_SlabContext, - &SlabMethods, + MCTX_SLAB_ID, parent, name); @@ -279,7 +216,7 @@ SlabContextCreate(MemoryContext parent, * The code simply frees all the blocks in the context - we don't keep any * keeper blocks or anything like that. */ -static void +void SlabReset(MemoryContext context) { int i; @@ -322,7 +259,7 @@ SlabReset(MemoryContext context) * SlabDelete * Free all memory which is allocated in the given context. */ -static void +void SlabDelete(MemoryContext context) { /* Reset to release all the SlabBlocks */ @@ -336,12 +273,12 @@ SlabDelete(MemoryContext context) * Returns pointer to allocated memory of given size or NULL if * request could not be completed; memory is added to the slab. */ -static void * +void * SlabAlloc(MemoryContext context, Size size) { SlabContext *slab = castNode(SlabContext, context); SlabBlock *block; - SlabChunk *chunk; + void *pointer; int idx; Assert(slab); @@ -370,16 +307,14 @@ SlabAlloc(MemoryContext context, Size size) block->nfree = slab->chunksPerBlock; block->firstFreeChunk = 0; + block->slab = slab; /* * Put all the chunks on a freelist. Walk the chunks and point each * one to the next one. */ for (idx = 0; idx < slab->chunksPerBlock; idx++) - { - chunk = SlabBlockGetChunk(slab, block, idx); - *(int32 *) SlabChunkGetPointer(chunk) = (idx + 1); - } + *(int32 *) SlabBlockGetPointer(slab, block, idx) = (idx + 1); /* * And add it to the last freelist with all chunks empty. @@ -411,8 +346,8 @@ SlabAlloc(MemoryContext context, Size size) /* make sure the chunk index is valid, and that it's marked as empty */ Assert((idx >= 0) && (idx < slab->chunksPerBlock)); - /* compute the chunk location block start (after the block header) */ - chunk = SlabBlockGetChunk(slab, block, idx); + /* compute the pointer */ + pointer = SlabBlockGetPointer(slab, block, idx); /* * Update the block nfree count, and also the minFreeChunks as we've @@ -426,8 +361,8 @@ SlabAlloc(MemoryContext context, Size size) * Remove the chunk from the freelist head. The index of the next free * chunk is stored in the chunk itself. */ - VALGRIND_MAKE_MEM_DEFINED(SlabChunkGetPointer(chunk), sizeof(int32)); - block->firstFreeChunk = *(int32 *) SlabChunkGetPointer(chunk); + VALGRIND_MAKE_MEM_DEFINED(pointer, sizeof(int32)); + block->firstFreeChunk = *(int32 *) pointer; Assert(block->firstFreeChunk >= 0); Assert(block->firstFreeChunk <= slab->chunksPerBlock); @@ -464,54 +399,65 @@ SlabAlloc(MemoryContext context, Size size) slab->minFreeChunks = 0; /* Prepare to initialize the chunk header. */ - VALGRIND_MAKE_MEM_UNDEFINED(chunk, sizeof(SlabChunk)); - - chunk->block = block; - chunk->slab = slab; + VALGRIND_MAKE_MEM_UNDEFINED((char *) pointer - SMALL_CHUNK_SIZE, SMALL_CHUNK_SIZE); #ifdef MEMORY_CONTEXT_CHECKING + GenericChunkHeaderEncode(pointer, block, slab->chunkSize, slab->chunkSize, MCTX_SLAB_ID); + /* slab mark to catch clobber of "unused" space */ - if (slab->chunkSize < (slab->fullChunkSize - sizeof(SlabChunk))) + if (slab->chunkSize < (slab->fullChunkSize - SMALL_CHUNK_SIZE)) { - set_sentinel(SlabChunkGetPointer(chunk), size); - VALGRIND_MAKE_MEM_NOACCESS(((char *) chunk) + - sizeof(SlabChunk) + slab->chunkSize, - slab->fullChunkSize - - (slab->chunkSize + sizeof(SlabChunk))); + set_sentinel(pointer, size); + VALGRIND_MAKE_MEM_NOACCESS(pointer + slab->chunkSize, + slab->fullChunkSize - slab->chunkSize); } +#else + GenericChunkHeaderEncode(pointer, block, slab->chunkSize, MCTX_SLAB_ID); #endif + #ifdef RANDOMIZE_ALLOCATED_MEMORY /* fill the allocated space with junk */ - randomize_mem((char *) SlabChunkGetPointer(chunk), size); + randomize_mem((char *) pointer, size); #endif Assert(slab->nblocks * slab->blockSize == context->mem_allocated); - return SlabChunkGetPointer(chunk); + return pointer; } /* * SlabFree * Frees allocated memory; memory is removed from the slab. */ -static void -SlabFree(MemoryContext context, void *pointer) +void +SlabFree(void *pointer) { int idx; - SlabContext *slab = castNode(SlabContext, context); - SlabChunk *chunk = SlabPointerGetChunk(pointer); - SlabBlock *block = chunk->block; + SlabContext *slab; + SlabBlock *block; + Size chunksize; + Size blockoffset; +#ifdef MEMORY_CONTEXT_CHECKING + Size reqsize; + + GenericChunkHeaderDecode(pointer, &chunksize, &blockoffset, &reqsize); +#else + GenericChunkHeaderDecode(pointer, &chunksize, &blockoffset); +#endif + + block = (SlabBlock *) ((char *) pointer - blockoffset); + slab = block->slab; #ifdef MEMORY_CONTEXT_CHECKING /* Test for someone scribbling on unused space in chunk */ - if (slab->chunkSize < (slab->fullChunkSize - sizeof(SlabChunk))) + if (slab->chunkSize < (slab->fullChunkSize - SMALL_CHUNK_SIZE)) if (!sentinel_ok(pointer, slab->chunkSize)) elog(WARNING, "detected write past chunk end in %s %p", - slab->header.name, chunk); + slab->header.name, pointer); #endif /* compute index of the chunk with respect to block start */ - idx = SlabChunkIndex(slab, block, chunk); + idx = SlabPointerGetIdx(slab, block, pointer); /* add chunk to freelist, and update block nfree count */ *(int32 *) pointer = block->firstFreeChunk; @@ -527,6 +473,10 @@ SlabFree(MemoryContext context, void *pointer) slab->chunkSize - sizeof(int32)); #endif +#ifdef MEMORY_CONTEXT_CHECKING + GenericChunkHeaderEncode(pointer, block, chunksize, 0, MCTX_SLAB_ID); +#endif + /* remove the block from a freelist */ dlist_delete(&block->node); @@ -560,13 +510,13 @@ SlabFree(MemoryContext context, void *pointer) { free(block); slab->nblocks--; - context->mem_allocated -= slab->blockSize; + slab->header.mem_allocated -= slab->blockSize; } else dlist_push_head(&slab->freelist[block->nfree], &block->node); Assert(slab->nblocks >= 0); - Assert(slab->nblocks * slab->blockSize == context->mem_allocated); + Assert(slab->nblocks * slab->blockSize == slab->header.mem_allocated); } /* @@ -582,10 +532,23 @@ SlabFree(MemoryContext context, void *pointer) * rather pointless - Slab is meant for chunks of constant size, and moreover * realloc is usually used to enlarge the chunk. */ -static void * -SlabRealloc(MemoryContext context, void *pointer, Size size) +void * +SlabRealloc(void *pointer, Size size) { - SlabContext *slab = castNode(SlabContext, context); + SlabContext *slab; + SlabBlock *block; + Size chunksize; + Size blockoffset; +#ifdef MEMORY_CONTEXT_CHECKING + Size reqsize; + + GenericChunkHeaderDecode(pointer, &chunksize, &blockoffset, &reqsize); +#else + GenericChunkHeaderDecode(pointer, &chunksize, &blockoffset); +#endif + + block = (SlabBlock *) ((char *) pointer - blockoffset); + slab = block->slab; Assert(slab); @@ -597,15 +560,54 @@ SlabRealloc(MemoryContext context, void *pointer, Size size) return NULL; /* keep compiler quiet */ } +/* + * SlabGetChunkContext + */ +MemoryContext +SlabGetChunkContext(void *pointer) +{ + SlabContext *slab; + SlabBlock *block; + Size chunksize; + Size blockoffset; +#ifdef MEMORY_CONTEXT_CHECKING + Size reqsize; + + GenericChunkHeaderDecode(pointer, &chunksize, &blockoffset, &reqsize); +#else + GenericChunkHeaderDecode(pointer, &chunksize, &blockoffset); +#endif + + block = (SlabBlock *) ((char *) pointer - blockoffset); + slab = block->slab; + + Assert(slab); + + return &slab->header; +} + /* * SlabGetChunkSpace * Given a currently-allocated chunk, determine the total space * it occupies (including all memory-allocation overhead). */ -static Size -SlabGetChunkSpace(MemoryContext context, void *pointer) +Size +SlabGetChunkSpace(void *pointer) { - SlabContext *slab = castNode(SlabContext, context); + SlabContext *slab; + SlabBlock *block; + Size chunksize; + Size blockoffset; +#ifdef MEMORY_CONTEXT_CHECKING + Size reqsize; + + GenericChunkHeaderDecode(pointer, &chunksize, &blockoffset, &reqsize); +#else + GenericChunkHeaderDecode(pointer, &chunksize, &blockoffset); +#endif + + block = (SlabBlock *) ((char *) pointer - blockoffset); + slab = block->slab; Assert(slab); @@ -616,7 +618,7 @@ SlabGetChunkSpace(MemoryContext context, void *pointer) * SlabIsEmpty * Is an Slab empty of any allocated space? */ -static bool +bool SlabIsEmpty(MemoryContext context) { SlabContext *slab = castNode(SlabContext, context); @@ -635,7 +637,7 @@ SlabIsEmpty(MemoryContext context) * totals: if not NULL, add stats about this context into *totals. * print_to_stderr: print stats to stderr if true, elog otherwise. */ -static void +void SlabStats(MemoryContext context, MemoryStatsPrintFunc printfunc, void *passthru, MemoryContextCounters *totals, @@ -697,7 +699,7 @@ SlabStats(MemoryContext context, * find yourself in an infinite loop when trouble occurs, because this * routine will be entered again when elog cleanup tries to release memory! */ -static void +void SlabCheck(MemoryContext context) { int i; @@ -742,16 +744,16 @@ SlabCheck(MemoryContext context) nfree = 0; while (idx < slab->chunksPerBlock) { - SlabChunk *chunk; + void *pointer; /* count the chunk as free, add it to the bitmap */ nfree++; slab->freechunks[idx] = true; /* read index of the next free chunk */ - chunk = SlabBlockGetChunk(slab, block, idx); - VALGRIND_MAKE_MEM_DEFINED(SlabChunkGetPointer(chunk), sizeof(int32)); - idx = *(int32 *) SlabChunkGetPointer(chunk); + pointer = SlabBlockGetPointer(slab, block, idx); + VALGRIND_MAKE_MEM_DEFINED(pointer, sizeof(int32)); + idx = *(int32 *) pointer; } for (j = 0; j < slab->chunksPerBlock; j++) @@ -759,22 +761,29 @@ SlabCheck(MemoryContext context) /* non-zero bit in the bitmap means chunk the chunk is used */ if (!slab->freechunks[j]) { - SlabChunk *chunk = SlabBlockGetChunk(slab, block, j); + void *pointer = SlabBlockGetPointer(slab, block, j); + SlabBlock *sblock; + Size chunksize; + Size blockoffset; + Size reqSize; + + GenericChunkHeaderDecode(pointer, &chunksize, &blockoffset, &reqSize); + sblock = (SlabBlock *) ((char *) pointer) - blockoffset; - /* chunks have both block and slab pointers, so check both */ - if (chunk->block != block) - elog(WARNING, "problem in slab %s: bogus block link in block %p, chunk %p", - name, block, chunk); + /* check the blockoffset correctly points back to the block */ + if (sblock != block) + elog(WARNING, "problem in slab %s: bogus block link in block %p, pointer %p", + name, block, pointer); - if (chunk->slab != slab) - elog(WARNING, "problem in slab %s: bogus slab link in block %p, chunk %p", - name, block, chunk); + if (sblock->slab != slab) + elog(WARNING, "problem in slab %s: bogus slab link in block %p, pointer %p", + name, block, pointer); /* there might be sentinel (thanks to alignment) */ - if (slab->chunkSize < (slab->fullChunkSize - sizeof(SlabChunk))) - if (!sentinel_ok(chunk, slab->chunkSize)) - elog(WARNING, "problem in slab %s: detected write past chunk end in block %p, chunk %p", - name, block, chunk); + if (slab->chunkSize < slab->fullChunkSize) + if (!sentinel_ok(pointer, slab->chunkSize)) + elog(WARNING, "problem in slab %s: detected write past chunk end in block %p, pointer %p", + name, block, pointer); } } diff --git a/src/include/nodes/memnodes.h b/src/include/nodes/memnodes.h index bbbe151e39..925b42b768 100644 --- a/src/include/nodes/memnodes.h +++ b/src/include/nodes/memnodes.h @@ -59,11 +59,12 @@ typedef struct MemoryContextMethods { void *(*alloc) (MemoryContext context, Size size); /* call this free_p in case someone #define's free() */ - void (*free_p) (MemoryContext context, void *pointer); - void *(*realloc) (MemoryContext context, void *pointer, Size size); + void (*free_p) (void *pointer); + void *(*realloc) (void *pointer, Size size); void (*reset) (MemoryContext context); void (*delete_context) (MemoryContext context); - Size (*get_chunk_space) (MemoryContext context, void *pointer); + MemoryContext (*get_chunk_context) (void *pointer); + Size (*get_chunk_space) (void *pointer); bool (*is_empty) (MemoryContext context); void (*stats) (MemoryContext context, MemoryStatsPrintFunc printfunc, void *passthru, diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h index 495d1af201..244f7bf4ff 100644 --- a/src/include/utils/memutils.h +++ b/src/include/utils/memutils.h @@ -79,6 +79,7 @@ extern void MemoryContextDeleteChildren(MemoryContext context); extern void MemoryContextSetIdentifier(MemoryContext context, const char *id); extern void MemoryContextSetParent(MemoryContext context, MemoryContext new_parent); +extern MemoryContext GetMemoryChunkContext(void *pointer); extern Size GetMemoryChunkSpace(void *pointer); extern MemoryContext MemoryContextGetParent(MemoryContext context); extern bool MemoryContextIsEmpty(MemoryContext context); @@ -98,53 +99,6 @@ extern bool MemoryContextContains(MemoryContext context, void *pointer); #define MemoryContextCopyAndSetIdentifier(cxt, id) \ MemoryContextSetIdentifier(cxt, MemoryContextStrdup(cxt, id)) -/* - * GetMemoryChunkContext - * Given a currently-allocated chunk, determine the context - * it belongs to. - * - * All chunks allocated by any memory context manager are required to be - * preceded by the corresponding MemoryContext stored, without padding, in the - * preceding sizeof(void*) bytes. A currently-allocated chunk must contain a - * backpointer to its owning context. The backpointer is used by pfree() and - * repalloc() to find the context to call. - */ -#ifndef FRONTEND -static inline MemoryContext -GetMemoryChunkContext(void *pointer) -{ - MemoryContext context; - - /* - * Try to detect bogus pointers handed to us, poorly though we can. - * Presumably, a pointer that isn't MAXALIGNED isn't pointing at an - * allocated chunk. - */ - Assert(pointer != NULL); - Assert(pointer == (void *) MAXALIGN(pointer)); - - /* - * OK, it's probably safe to look at the context. - */ - context = *(MemoryContext *) (((char *) pointer) - sizeof(void *)); - - AssertArg(MemoryContextIsValid(context)); - - return context; -} -#endif - -/* - * This routine handles the context-type-independent part of memory - * context creation. It's intended to be called from context-type- - * specific creation routines, and noplace else. - */ -extern void MemoryContextCreate(MemoryContext node, - NodeTag tag, - const MemoryContextMethods *methods, - MemoryContext parent, - const char *name); - extern void HandleLogMemoryContextInterrupt(void); extern void ProcessLogMemoryContextInterrupt(void); diff --git a/src/include/utils/memutils_generichdr.h b/src/include/utils/memutils_generichdr.h new file mode 100644 index 0000000000..82d06715ad --- /dev/null +++ b/src/include/utils/memutils_generichdr.h @@ -0,0 +1,265 @@ +/*------------------------------------------------------------------------- + * + * memutils_generichdr.h + * Contains various inlined functions which can be used by implementations + * of MemoryContexts to use as a memory chunk header. Implementations may + * choose to implement their own header. The only requirement is that the + * 3-bits directly prior to the pointer must be set to the + * MemoryContextMethodID of the given context. + * + * This generic chunk header allows the encoding of various details about the + * memory chunk. Generally, these headers are 8 bytes in length, however for + * larger allocations 24 bytes are required. We refer to the former as + * "small" headers and the latter as "large" headers. + * + * Both small and large headers have a 8-byte portion directly prior to the + * memory pointer for the chunk. We encode 4 separate pieces of information + * into these 8 bytes. These are as follows in order of least significant bit + * first: + * + * 1. 3-bits to indicate the MemoryContextMethodID + * 2. 1-bit to indicate if the chunk is small or large + * 3. 30-bits to indicate the size of the chunk + * 4. 30-bits to indicate the number of bytes that must be subtracted from the + * pointer to obtain the address of the block that the pointer is stored on + * + * Memory allocations where the size of the chunk or the block offset is a + * value that consumes more than 30-bits (1GB) must use a large chunk. + * + * Large chunks, as mentioned above, also contain a 8-byte header, however + * the 30-bit portions #3 and #4 above are unused. Instead the chunk size and + * block offset are stored directly prior to the 8-byte chunk header in the + * following form: + * + * <8 byte header> + * + * Which is 24 bytes when sizeof(Size) is 8 and 16 bytes when sizeof(Size) is + * 4. + * + * When MEMORY_CONTEXT_CHECKING is defined, both small and large headers store + * an additional Size variable which directly prefixes the existing chunk. + * This additional Size field is used to store the requested size of the + * allocation. This can be used by the MemoryContexts implementation to + * perform additional checks to help ensure the code is correct. For example, + * setting sentinal bytes to help ensure that no code overruns the memory + * allocation. + * + * Interface: + * GenericChunkHeaderSize: + * Used to predetermine the size of the chunk. + * + * GenericChunkHeaderDecode: + * Given a pointer to the memory, determine the size of the chunk and + * the number of bytes to subtract from the pointer to obtain the pointer + * to the block. The requested_size is also decoded when + * MEMORY_CONTEXT_CHECKING is defined. + * + * GenericChunkHeaderEncode: + * Given a pointer to the memory and a pointer to the block, determine + * the bytes offset for the block and encode that and the + * MemoryContextMethodID into the chunk. Also encode the chunk size and + * optionally, the requested_size when MEMORY_CONTEXT_CHECKING is + * defined. + * + * Also exports: + * SMALL_CHUNK_LIMIT + * SMALL_CHUNK_SIZE + * LARGE_CHUNK_SIZE + * + * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * src/include/utils/memutils_generichdr.h + * + *------------------------------------------------------------------------- + */ + +#ifndef MEMUTILS_GENERICHDR_H +#define MEMUTILS_GENERICHDR_H + +#include "utils/memutils_internal.h" + +/* + * The threshold, above which a chunk becomes large. The maximum value that + * can be stored in 30-bits. + */ +#define SMALL_CHUNK_LIMIT 0x3FFFFFFF + +/* + * Bit mask with 30 of the least significant bits on. Generally the same as + * SMALL_CHUNK_LIMIT, but kept separate so that SMALL_CHUNK_LIMIT can be + * lowered to make testing large chunks easier. + */ +#define SMALL_CHUNK_SIZE_MASK 0x3FFFFFFF + +/* + * The value to AND onto the 8-byte header to determine if it's a large chunk. + */ +#define LARGE_CHUNK_MASK (1 << 3) + +#ifndef MEMORY_CONTEXT_CHECKING +#define SMALL_CHUNK_SIZE sizeof(uint64) +#define LARGE_CHUNK_SIZE (sizeof(uint64) + (sizeof(Size) * 2)) +#else +#define SMALL_CHUNK_SIZE (sizeof(uint64) + sizeof(Size)) +#define LARGE_CHUNK_SIZE (sizeof(uint64) + (sizeof(Size) * 3)) +#endif + +#define LargeChunkGetChunkSizePtr(p) (Size *) (((char *) (p)) - sizeof(uint64) - sizeof(Size)) +#define LargeChunkGetBlockOffsetPtr(p) (Size *) (((char *) (p)) - sizeof(uint64) - (sizeof(Size) * 2)) + +#ifdef MEMORY_CONTEXT_CHECKING +#define LargeChunkGetRequestedSizePtr(p) (Size *) (((char *) (p)) - sizeof(uint64) - (sizeof(Size) * 3)) +#endif + +/* Get a pointer to the uint64 header for either a small or large chunk */ +#define ChunkGetHeaderPtr(p) ((uint64 *) (((char *) p) - sizeof(uint64))) + +/* + * From a small or large chunk's header uint64 value, get the context method id + */ +#define ChunkHdrGetMethodId(h) ((h) & 7) + +#define SmallChunkHdrGetChunkSize(h) (((h) >> 4) & SMALL_CHUNK_SIZE_MASK) +#define SmallChunkHdrGetBlockOffset(h) (((h) >> 34) & SMALL_CHUNK_SIZE_MASK) + +#ifdef MEMORY_CONTEXT_CHECKING +#define SmallChunkGetRequestedSizePtr(p) (Size *) (((char *) (p)) - sizeof(uint64) - sizeof(Size)) +#endif + + /* + * Determine if offset or size justify requiring a large chunk. Since + * SMALL_CHUNK_LIMIT is a power of 2, we can bitmask OR offset and size to + * get a slightly more efficient way of checking if either is larger than + * SMALL_CHUNK_LIMIT. + */ +#define LargeChunkRequired(offset, size) (((offset) | (size)) > SMALL_CHUNK_LIMIT) + + /* From a given chunk header, return true if it's a large chunk */ +#define IsLargeChunk(h) ((h) & LARGE_CHUNK_MASK) + +/* + * GenericChunkHeaderSize + * Return the size of chunk header required to store details about a + * chunk with the given blockoffset and chunkisize. + */ +static inline Size +GenericChunkHeaderSize(Size blockoffset, Size chunksize) +{ + return LargeChunkRequired(blockoffset, chunksize) ? LARGE_CHUNK_SIZE : SMALL_CHUNK_SIZE; +} + +/* + * GenericChunkHeaderDecode + * Decode the generic chunk header preceeding 'pointer' and populate + * 'chunksize' and 'blockoffset'. Return the size of the generic header. + */ +static inline Size +#ifdef MEMORY_CONTEXT_CHECKING +GenericChunkHeaderDecode(void *pointer, Size *chunksize, Size *blockoffset, + Size *requested_size) +#else +GenericChunkHeaderDecode(void *pointer, Size *chunksize, Size *blockoffset) +#endif +{ + uint64 header = (uint64) *ChunkGetHeaderPtr(pointer); + + if (unlikely(IsLargeChunk(header))) + { + *chunksize = (Size) *LargeChunkGetChunkSizePtr(pointer); + *blockoffset = (Size) *LargeChunkGetBlockOffsetPtr(pointer); + +#ifdef MEMORY_CONTEXT_CHECKING + *requested_size = (Size) *LargeChunkGetRequestedSizePtr(pointer); +#endif + return LARGE_CHUNK_SIZE; + } + else + { + *chunksize = SmallChunkHdrGetChunkSize(header); + *blockoffset = SmallChunkHdrGetBlockOffset(header); +#ifdef MEMORY_CONTEXT_CHECKING + *requested_size = (Size) *SmallChunkGetRequestedSizePtr(pointer); +#endif + + return SMALL_CHUNK_SIZE; + } +} + +static inline void +#ifdef MEMORY_CONTEXT_CHECKING +GenericChunkHeaderEncode(void *pointer, void *block, Size chunksize, + Size requested_size, MemoryContextMethodID methodid) +#else +GenericChunkHeaderEncode(void *pointer, void *block, Size chunksize, + MemoryContextMethodID methodid) +#endif +{ + uint64 *p_header = ChunkGetHeaderPtr(pointer); + Size blockoffset = (char *) pointer - (char *) block; + + /* Check if we need a large chunk */ + if (unlikely(LargeChunkRequired(blockoffset, chunksize))) + { + Size *p_chunksize = LargeChunkGetChunkSizePtr(pointer); + Size *p_blockoffset = LargeChunkGetBlockOffsetPtr(pointer); +#ifdef MEMORY_CONTEXT_CHECKING + Size *p_requestedsize = LargeChunkGetRequestedSizePtr(pointer); +#endif + + /* Encode the 8-byte header */ + *p_header = LARGE_CHUNK_MASK | methodid; + + /* Set the chunk size and block offset */ + *p_chunksize = chunksize; + + *p_blockoffset = blockoffset; + +#ifdef MEMORY_CONTEXT_CHECKING + *p_requestedsize = requested_size; +#endif + } + else + { + *p_header = (blockoffset << 34) | (chunksize << 4) | methodid; +#ifdef MEMORY_CONTEXT_CHECKING + *SmallChunkGetRequestedSizePtr(pointer) = requested_size; +#endif + } + +#ifdef USE_ASSERT_CHECKING + /* verify that we decode the values the same as we've just encoded them */ + { + Size chsz, blkoff; +#ifdef MEMORY_CONTEXT_CHECKING + Size reqsz; + + GenericChunkHeaderDecode(pointer, &chsz, &blkoff, &reqsz); + + Assert(reqsz == requested_size); +#else + GenericChunkHeaderDecode(pointer, &chsz, &blkoff); +#endif + Assert(chsz == chunksize); + Assert(blkoff == blockoffset); + Assert(blkoff > 0); + Assert(GetMemoryChunkMethodID(pointer) == methodid); + } +#endif + +} + +/* cleanup all internal definitions */ +#undef LARGE_CHUNK_MASK +#undef LargeChunkGetChunkSizePtr +#undef LargeChunkGetBlockOffsetPtr +#undef LargeChunkGetRequestedSizePtr +#undef ChunkGetHeaderPtr +#undef ChunkHdrGetMethodId +#undef SmallChunkHdrGetChunkSize +#undef SmallChunkHdrGetBlockOffset +#undef SmallChunkGetRequestedSizePtr +#undef LargeChunkRequired +#undef IsLargeChunk + +#endif /* MEMUTILS_GENERICHDR_H */ diff --git a/src/include/utils/memutils_internal.h b/src/include/utils/memutils_internal.h new file mode 100644 index 0000000000..ab94115ee5 --- /dev/null +++ b/src/include/utils/memutils_internal.h @@ -0,0 +1,117 @@ +/*------------------------------------------------------------------------- + * + * memutils_internal.h + * This file contains declarations for memory allocation utility + * functions for internal use. + * + * + * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group + * Portions Copyright (c) 1994, Regents of the University of California + * + * src/include/utils/memutils_internal.h + * + *------------------------------------------------------------------------- + */ + +#ifndef MEMUTILS_INTERNAL_H +#define MEMUTILS_INTERNAL_H + +#include "utils/memutils.h" + +extern void *AllocSetAlloc(MemoryContext context, Size size); +extern void AllocSetFree(void *pointer); +extern void *AllocSetRealloc(void *pointer, Size size); +extern void AllocSetReset(MemoryContext context); +extern void AllocSetDelete(MemoryContext context); +extern MemoryContext AllocSetGetChunkContext(void *pointer); +extern Size AllocSetGetChunkSpace(void *pointer); +extern bool AllocSetIsEmpty(MemoryContext context); +extern void AllocSetStats(MemoryContext context, + MemoryStatsPrintFunc printfunc, void *passthru, + MemoryContextCounters *totals, + bool print_to_stderr); + +#ifdef MEMORY_CONTEXT_CHECKING +extern void AllocSetCheck(MemoryContext context); +#endif + + +extern void *GenerationAlloc(MemoryContext context, Size size); +extern void GenerationFree(void *pointer); +extern void *GenerationRealloc(void *pointer, Size size); +extern void GenerationReset(MemoryContext context); +extern void GenerationDelete(MemoryContext context); +extern MemoryContext GenerationGetChunkContext(void *pointer); +extern Size GenerationGetChunkSpace(void *pointer); +extern bool GenerationIsEmpty(MemoryContext context); +extern void GenerationStats(MemoryContext context, + MemoryStatsPrintFunc printfunc, void *passthru, + MemoryContextCounters *totals, + bool print_to_stderr); + +#ifdef MEMORY_CONTEXT_CHECKING +extern void GenerationCheck(MemoryContext context); +#endif + + +/* + * These functions implement the MemoryContext API for Slab contexts. + */ +extern void *SlabAlloc(MemoryContext context, Size size); +extern void SlabFree(void *pointer); +extern void *SlabRealloc(void *pointer, Size size); +extern void SlabReset(MemoryContext context); +extern void SlabDelete(MemoryContext context); +extern MemoryContext SlabGetChunkContext(void *pointer); +extern Size SlabGetChunkSpace(void *pointer); +extern bool SlabIsEmpty(MemoryContext context); +extern void SlabStats(MemoryContext context, + MemoryStatsPrintFunc printfunc, void *passthru, + MemoryContextCounters *totals, + bool print_to_stderr); +#ifdef MEMORY_CONTEXT_CHECKING +extern void SlabCheck(MemoryContext context); +#endif + +/* + * MemoryContextMethodID + * A unique identifier for each MemoryContext implementation which + * indicates the index into the mcxt_methods[] array. See mcxt.c. + */ +typedef enum MemoryContextMethodID +{ + MCTX_ASET_ID = 0, + MCTX_GENERATION_ID, + MCTX_SLAB_ID, +} MemoryContextMethodID; + +/* + * This routine handles the context-type-independent part of memory + * context creation. It's intended to be called from context-type- + * specific creation routines, and noplace else. + */ +extern void MemoryContextCreate(MemoryContext node, + NodeTag tag, + MemoryContextMethodID method_id, + MemoryContext parent, + const char *name); + +static inline MemoryContextMethodID +GetMemoryChunkMethodID(void *pointer) +{ + uint64 header; + + /* + * Try to detect bogus pointers handed to us, poorly though we can. + * Presumably, a pointer that isn't MAXALIGNED isn't pointing at an + * allocated chunk. + */ + Assert(pointer != NULL); + Assert(pointer == (void *) MAXALIGN(pointer)); + + header = *((uint64 *) ((char *) pointer - sizeof(uint64))); + + return header & 7; +} + +#endif /* MEMUTILS_INTERNAL_H */ -- 2.35.1.windows.2