MapViewOfFile...

[Emmanuel Derriey]

Hi !
I have a problem with the MapViewOfFile API.
I use this API to store data in files in my own database system management ;
when I need to store new data, is the file is NOT large enought, I close the
file, reopen and grow it (CreateFileMapping) and re-call MapViewOfFile (last
argument is 0, so I map entire file) ; sometimes, it returns NULL and
GetLastError returns 8 (not enought space). I test it with differents
computers, with 512 Mb RAM to 2Gb RAM, with large swap files, but the
problem seems not to be linked to memory... More over, the memory occupied
by my application is always less that physical memory...
It seems to happen when file is about 250 Gb, but it's not systematic...
More over, it seems to have differents behaviours between Win2000 and WinXP.
Is this a problem due to memory fragmentation ?
Thanks for any help...

Emmanuel Derriey

[Jochen Kalmbach]

Post by Emmanuel Derriey
I have a problem with the MapViewOfFile API.
I use this API to store data in files in my own database system
management ; when I need to store new data, is the file is NOT large
enought, I close the file, reopen and grow it (CreateFileMapping) and
re-call MapViewOfFile (last argument is 0, so I map entire file) ;
sometimes, it returns NULL and GetLastError returns 8 (not enought
space).

You already got the answer, didnŽt you ?

The problem is that if you want to map the whole file, there must be an
continous range of virtual address to map the file. For many reasons
there is no virtual address range with the desired length in your apps
address space.

Maybe the low fragmentation heap can help you:

Low-fragmentation Heap
http://msdn.microsoft.com/library/en-
us/memory/base/low_fragmentation_heap.asp

The Windows XP Low Fragmentation Heap Algorithm Feature Is Available for
Windows 2000
http://support.microsoft.com/?kbid=816542


You also should try to split the file mapping, so you do not need to map
the whole file. This can be easily done by a wrapper class for the file
mapping. Access the file mapping with this class. Internally you can map
the needed portion of the file if it is not yet mapped.

[Ivan Brugiolo [MSFT]]

One more comment about the Heap Fragmentation and the address space
fragmentation.
They are different and somewhat unrelated.
A default heap will have regions of virtual address in the sizes of 1M, 1M,
2M, 4M, 8M , etc.
Now, each region can have reserved or committen range according to the heap
usage,
and the committen memory with the same Virtual Address region can be
fragmented
in the sense that a particular order of alloc-free on the LookAside Lists
and the Free-Lists
can cause heap fragmentation, that may not cause address space
fragmentation.
--
This posting is provided "AS IS" with no warranties, and confers no rights.
Use of any included script samples are subject to the terms specified at
http://www.microsoft.com/info/cpyright.htm

Post by Jochen Kalmbach

Post by Emmanuel Derriey
I have a problem with the MapViewOfFile API.
I use this API to store data in files in my own database system
management ; when I need to store new data, is the file is NOT large
enought, I close the file, reopen and grow it (CreateFileMapping) and
re-call MapViewOfFile (last argument is 0, so I map entire file) ;
sometimes, it returns NULL and GetLastError returns 8 (not enought
space).

You already got the answer, didnŽt you ?
The problem is that if you want to map the whole file, there must be an
continous range of virtual address to map the file. For many reasons
there is no virtual address range with the desired length in your apps
address space.
Low-fragmentation Heap
http://msdn.microsoft.com/library/en-
us/memory/base/low_fragmentation_heap.asp
The Windows XP Low Fragmentation Heap Algorithm Feature Is Available for
Windows 2000
http://support.microsoft.com/?kbid=816542
You also should try to split the file mapping, so you do not need to map
the whole file. This can be easily done by a wrapper class for the file
mapping. Access the file mapping with this class. Internally you can map
the needed portion of the file if it is not yet mapped.
--
Greetings
Jochen
Do you need a memory-leak finder ?
http://www.codeproject.com/useritems/leakfinder.asp

[Jochen Kalmbach]

Post by Ivan Brugiolo [MSFT]
One more comment about the Heap Fragmentation and the address space
fragmentation.
They are different and somewhat unrelated.

Yes, you are right.... but maybe if an app uses the low-fragmentation heap
it will use less virtual addresses than without it...

Is there a way to find the used addresses in an appsŽ address space ? or
the larged range of free virtual addresses !?
Maybe VirtualQueryEx !?

[Jochen Kalmbach]

The address space layout of an application can be query-ed with
VirtualQuery(Ex)

Ok, the largest free region can be queried with the following:


<code>
SIZE_T GetLargestFreeMemRegion(LPVOID *lpBaseAddr)
{
SYSTEM_INFO systemInfo;
GetSystemInfo(&systemInfo);

DWORD p = 0;
MEMORY_BASIC_INFORMATION mbi;

SIZE_T largestSize = 0;
while(p < 0x80000000)
{
SIZE_T dwRet = VirtualQuery((LPCVOID) p, &mbi, sizeof(mbi));
if (dwRet > 0)
{
if (mbi.State == MEM_FREE)
{
if (largestSize < mbi.RegionSize)
{
largestSize = mbi.RegionSize;
if (lpBaseAddr != NULL)
*lpBaseAddr = mbi.BaseAddress;
}
}
p += mbi.RegionSize;
}
else
p += systemInfo.dwPageSize;
}
return largestSize;
}


int _tmain(int argc, _TCHAR* argv[])
{
LPVOID baseAddr;
SIZE_T ls = GetLargestFreeMemRegion(&baseAddr);
printf("Largest Free Region: Size: 0x%8.8X at 0x%8.8X\n", ls,
baseAddr);
return 0;
}
</code>

[Patrick Philippot]

Post by Emmanuel Derriey
I have a problem with the MapViewOfFile API.
I use this API to store data in files in my own database system
management ; when I need to store new data, is the file is NOT large
enought, I close the file, reopen and grow it (CreateFileMapping) and
re-call MapViewOfFile (last argument is 0, so I map entire file) ;
sometimes, it returns NULL and GetLastError returns 8 (not enought
space).

Could you please give more details (possibly a code excerpt) about the
exact sequence of operations that take place when you decide to destroy
the current file mapping and create a larger one?

Also, look at the documentation of CreateFileMapping:

dwMaximumSizeLow
[in] Low-order DWORD of the maximum size of the file mapping object. If
this parameter and dwMaximumSizeHigh are zero, ***the maximum size of
the file mapping object is equal to the current size of the file
identified by hFile***.

So, if you need to have a MMF larger than the mapped file, you must
explicitly set the size of the MMF (GetFileSize + extra bytes).

[Emmanuel Derriey]

Ok !
Sorry, I thought it could interest .net developers...

By the way, please ***do not** cross-post. This question has nothing to
do with the .Net technology. So you should have posted to
microsoft.public.win32.programmer.kernel and possibly to
comp.os.ms-windows.programmer.win32. Further posts please only to these
newsgroups.
--
Patrick Philippot - Microsoft MVP [.Net]
MainSoft Consulting Services
www.mainsoft.xx
(replace .xx with .fr when replying by e-mail)