Memory mapped file pages getting cleared when memory overcommitted

Discussion:

(too old to reply)

Michael Vogt

2007-01-26 17:20:02 UTC

I have a win32 program running on XP Pro that is using a memory mapped file
to create shared memory. I find that some or all pages of that memory are
being cleared in memory overcommit situations.

The program does not actually need the persistence of a file, so the
CreateFile() is called with flag FILE_FLAG_DELETE_ON_CLOSE. After doing the
CreateFileMapping()and MapViewOfFile() the program does a CloseHandle() on
the file handle and on the handle to the file mapping object but still keeps
"open" the pointer returned from MapViewOfFile().

Here is the relevant initialization code for the memory:

hFilePtr = CreateFile( TCPDRV32FILE,
GENERIC_READ | GENERIC_WRITE,
FILE_SHARE_READ | FILE_SHARE_WRITE,
NULL,
CREATE_ALWAYS,
FILE_ATTRIBUTE_HIDDEN |
FILE_ATTRIBUTE_TEMPORARY |
FILE_FLAG_DELETE_ON_CLOSE |
FILE_ATTRIBUTE_NORMAL,
NULL );

hFileMap = CreateFileMapping( (HANDLE)hFilePtr,
NULL,
PAGE_READWRITE,
0,
size,
TCPDRV32SHR );

lpMapAddr = MapViewOfFile( hFileMap,
FILE_MAP_ALL_ACCESS,
0, 0,
0 );

CloseHandle( hFileMap );
CloseHandle( hFilePtr );

In the relevant scenarios, no other application process opens the mapped
memory.

The memory is only getting lost/cleared when I do something like open 20 IE
sessions.

The memory clearing can be prevented by any of the following:
- Don't set FILE_FLAG_DELETE_ON_CLOSE
- don't close the file handle
- frequently read from the memory.

I need to know if this behaviour is expected because I need to rule out that
the program is doing something else wrong.

The reason I suspect something else is going on is because in some
application scenarios, the problem does not occur. That is, there are certain
application scenarios that experience the problem and others that do not. In
at least some of those cases the memory is not being read any more frequently
than a failing case.

See this post in the WinDbg newsgroup for some other background:
http://msdn.microsoft.com/newsgroups/default.aspx?dg=microsoft.public.windbg&tid=89af28e3-2d38-48c5-af7d-e17125ea88fb&m=1&p=1

Thanks for any help or insight.

Michael Vogt

Alexander Grigoriev

2007-01-27 04:29:41 UTC

Permalink

I guess, that the underlying backing file becomes deleted and inaccessible
for page-out. Thus, any pageout discards the page. I don't like this
behavior, but it could be by design. If it is, it should better be
documented.

Post by Michael Vogt
I have a win32 program running on XP Pro that is using a memory mapped file
to create shared memory. I find that some or all pages of that memory are
being cleared in memory overcommit situations.
The program does not actually need the persistence of a file, so the
CreateFile() is called with flag FILE_FLAG_DELETE_ON_CLOSE. After doing the
CreateFileMapping()and MapViewOfFile() the program does a CloseHandle() on
the file handle and on the handle to the file mapping object but still keeps
"open" the pointer returned from MapViewOfFile().
hFilePtr = CreateFile( TCPDRV32FILE,
GENERIC_READ | GENERIC_WRITE,
FILE_SHARE_READ | FILE_SHARE_WRITE,
NULL,
CREATE_ALWAYS,
FILE_ATTRIBUTE_HIDDEN |
FILE_ATTRIBUTE_TEMPORARY |
FILE_FLAG_DELETE_ON_CLOSE |
FILE_ATTRIBUTE_NORMAL,
NULL );
hFileMap = CreateFileMapping( (HANDLE)hFilePtr,
NULL,
PAGE_READWRITE,
0,
size,
TCPDRV32SHR );
lpMapAddr = MapViewOfFile( hFileMap,
FILE_MAP_ALL_ACCESS,
0, 0,
0 );
CloseHandle( hFileMap );
CloseHandle( hFilePtr );
In the relevant scenarios, no other application process opens the mapped
memory.
The memory is only getting lost/cleared when I do something like open 20 IE
sessions.
- Don't set FILE_FLAG_DELETE_ON_CLOSE
- don't close the file handle
- frequently read from the memory.
I need to know if this behaviour is expected because I need to rule out that
the program is doing something else wrong.
The reason I suspect something else is going on is because in some
application scenarios, the problem does not occur. That is, there are certain
application scenarios that experience the problem and others that do not. In
at least some of those cases the memory is not being read any more frequently
than a failing case.
http://msdn.microsoft.com/newsgroups/default.aspx?dg=microsoft.public.windbg&tid=89af28e3-2d38-48c5-af7d-e17125ea88fb&m=1&p=1
Thanks for any help or insight.
Michael Vogt

Michael Vogt

2007-01-27 19:46:00 UTC

Permalink

I don't think the file should even be really closed. Even though the
program dose a close of the file handle and the mapping object, it never did
the UnmapViewOfFile. Should not the system keep the file open because of
that.

Post by Alexander Grigoriev
I guess, that the underlying backing file becomes deleted and inaccessible
for page-out. Thus, any pageout discards the page. I don't like this
behavior, but it could be by design. If it is, it should better be
documented.

Alexander Grigoriev

2007-01-29 02:03:03 UTC

Permalink

File mapping object keeps a reference to the FILE_OBJECT, not a handle. The
filesystem gets IRP_MJ_CLEANUP when the last handle to the file is closed.
This is the last chance to delete the file, even though the file mapping
objects still keep a reference to FILE_OBJECT. When the last reference to
FILE_OBJECT is released (file mapping is deleted), the filesystem gets
IRP_MJ_CLOSE, but it cannot delete files at this time because of IRQL
limitations.

I wonder if MS guys can confirm this behavior.

Post by Michael Vogt
I don't think the file should even be really closed. Even though the
program dose a close of the file handle and the mapping object, it never did
the UnmapViewOfFile. Should not the system keep the file open because of
that.

Post by Alexander Grigoriev
I guess, that the underlying backing file becomes deleted and
inaccessible
for page-out. Thus, any pageout discards the page. I don't like this
behavior, but it could be by design. If it is, it should better be
documented.

Post by Michael Vogt
I have a win32 program running on XP Pro that is using a memory mapped file
to create shared memory. I find that some or all pages of that memory are
being cleared in memory overcommit situations.
The program does not actually need the persistence of a file, so the
CreateFile() is called with flag FILE_FLAG_DELETE_ON_CLOSE. After
doing
the
CreateFileMapping()and MapViewOfFile() the program does a CloseHandle() on
the file handle and on the handle to the file mapping object but still keeps
"open" the pointer returned from MapViewOfFile().
hFilePtr = CreateFile( TCPDRV32FILE,
GENERIC_READ | GENERIC_WRITE,
FILE_SHARE_READ | FILE_SHARE_WRITE,
NULL,
CREATE_ALWAYS,
FILE_ATTRIBUTE_HIDDEN |
FILE_ATTRIBUTE_TEMPORARY |
FILE_FLAG_DELETE_ON_CLOSE |
FILE_ATTRIBUTE_NORMAL,
NULL );
hFileMap = CreateFileMapping( (HANDLE)hFilePtr,
NULL,
PAGE_READWRITE,
0,
size,
TCPDRV32SHR );
lpMapAddr = MapViewOfFile( hFileMap,
FILE_MAP_ALL_ACCESS,
0, 0,
0 );
CloseHandle( hFileMap );
CloseHandle( hFilePtr );
In the relevant scenarios, no other application process opens the mapped
memory.
The memory is only getting lost/cleared when I do something like open
20
IE
sessions.
- Don't set FILE_FLAG_DELETE_ON_CLOSE
- don't close the file handle
- frequently read from the memory.
I need to know if this behaviour is expected because I need to rule out that
the program is doing something else wrong.
The reason I suspect something else is going on is because in some
application scenarios, the problem does not occur. That is, there are certain
application scenarios that experience the problem and others that do
not.
In
at least some of those cases the memory is not being read any more frequently
than a failing case.
http://msdn.microsoft.com/newsgroups/default.aspx?dg=microsoft.public.windbg&tid=89af28e3-2d38-48c5-af7d-e17125ea88fb&m=1&p=1
Thanks for any help or insight.
Michael Vogt

Pavel Lebedinsky [MSFT]

2007-02-01 12:49:48 UTC

Permalink

As Alexander said, the file gets deleted when you close the file
handle. After this, if any pages from the view are trimmed and
repurposed and then accessed again, the memory manager will
try to read them from the file (which is now in the "deleted"
state). What happens next depends on the filesystem. NTFS
returns an error code (STATUS_END_OF_FILE) which
causes the memory manager to satisfy the page fault with
zeroed pages. As far as I can tell, this behavior is not documented
so a future version of NTFS (or a different filesystem) could
return a different error, which would result in an inpage
exception instead.

The best solution here would be to use a pagefile-backed
file mapping. Then you wouldn't have to worry about deleting
the file. Keeping the file handle open will also work.
--
This posting is provided "AS IS" with no warranties, and confers no
rights.