WinXP x64 - WriteFile() returns "Insufficient system resources exist to complete the requested service."

[Michael Russell]

I am not sure if this is the right place for this question or not. If not,
kindly direct me to the proper group.
I am having a problem with WriteFile() returning an error. I have whittled
this problem down to a short test program in C which fails under WinXP x64,
but works fine on WinXP x86. I see the problem whether I compile for a win32
or a win64 target. The quick summary is that if I use a size of 33525760 I
can use WriteFile() (after a CreateFile() with the FILE_FLAG_NO_BUFFERING
flag) just fine. If, however I use a file size of 33526272 or larger, then I get:
Insufficient system resources exist to complete the requested service.

I am running on a Dell Precision Workstation 670 w/ dual Xeon processors,
2GB RAM and using WinXP x64 SP1.
I have tried increasing the pagefile.sys size since someone suggested that,
but it did not help.

Here's the sample code:

--- CUT HERE - start of cf-test.cpp ---
#include <stdio.h>
#include <io.h>
#include <fcntl.h>
#include <Windows.h>

// Forward Declaration
void PrintError (void);

int main(int argc, char *argv[])
{
HANDLE hHandle = NULL;
char caFilename[512];
bool bRet = false;
unsigned char *ucpData = NULL;
// This size will work
// long lChunkSizeBytes = 33525760;
// This size will NOT work
long lChunkSizeBytes = 33526272;
unsigned long ulNumBytesWritten = 0;
printf("argc = %d\n", argc);

if (argc > 1)
sscanf(argv[1], "%ld", &lChunkSizeBytes);

strcpy(caFilename, "test.dat");

// Allocate Memory
printf("Allocating memory (%ld bytes)...\n", lChunkSizeBytes);
ucpData = (unsigned char *) VirtualAlloc(NULL,
lChunkSizeBytes,
MEM_COMMIT,
PAGE_READWRITE);
printf("done.\n");

// Open File
printf("Opening file '%s'...\n", caFilename);
hHandle = CreateFile(caFilename,
GENERIC_WRITE,
FILE_SHARE_READ | FILE_SHARE_WRITE,
NULL,
OPEN_ALWAYS,
FILE_FLAG_NO_BUFFERING,
NULL);

if (hHandle == INVALID_HANDLE_VALUE)
{
fprintf(stderr, "Error Opening File\n");
PrintError();
//MemoryFree(&hMem, &ucpData);
exit(-1);
}
printf("done, handle = 0x%x.\n", hHandle);

// Write File
printf("Writing file...\n");
if (!WriteFile(hHandle, ucpData, lChunkSizeBytes, &ulNumBytesWritten,
NULL))
{
fprintf(stderr, "Error Writing File\n");
PrintError();
//MemoryFree(&hMem, &ucpData);
exit(-1);
}
printf("done.\n");

// Close File
printf("Closing file...\n");
if (!CloseHandle(hHandle))
{
fprintf(stderr, "Error Closing File\n");
PrintError();
//MemoryFree(&hMem, &ucpData);
exit(-1);
}
printf("done.\n");

// Free Memory
printf("Releasing Memory...\n");
if (!VirtualFree(ucpData, 0, MEM_RELEASE))
{
fprintf(stderr, "Error Freeing Memory\n");
PrintError();
//MemoryFree(&hMem, &ucpData);
exit(-1);
}
printf("done.\n");
}

void PrintError (void)
{
LPVOID lpMsgBuf;

FormatMessage(
FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_FROM_SYSTEM,
NULL,
GetLastError(),
MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT), // Default language
(LPTSTR) &lpMsgBuf,
0,
NULL
);

// Display the error number & string
fprintf(stderr, " GetLastError(0x%x) = %s\n", GetLastError(),
(LPCTSTR) lpMsgBuf);

// Free the buffer.
LocalFree(lpMsgBuf);
return;
}
--- CUT HERE - end of cf-test.cpp ---

Compilation for x64 is done with the Platform SDK for Windows Server 2003 SP1
and selecting the Build Environment for XP x64 Retail. Then "cl cf-test.cpp
bufferoverflowU.lib". Thanks very much for any assistance.

-Michael Russell
***@mathtech.com

[Skywing]

A read size like 33526272 will never work with a file opened with
FILE_FLAG_NO_BUFFERING. Non-cached/paging IO have special requirements on
the size and offsets for read or write operations.

These requirements are outlined in the documentation for CreateFile().
Quoting them...:

"The system opens a file with no system caching. This flag does not affect
hard disk caching. When combined with FILE_FLAG_OVERLAPPED, the flag gives
maximum asynchronous performance, because the I/O does not rely on the
synchronous operations of the memory manager. However, some I/O operations
take more time, because data is not being held in the cache. Also, the file
metadata may still be cached. To flush the metadata to disk, use the
FlushFileBuffers function.
An application must meet certain requirements when working with files that
are opened with FILE_FLAG_NO_BUFFERING:


File access must begin at byte offsets within a file that are integer
multiples of the volume sector size.
File access must be for numbers of bytes that are integer multiples of the
volume sector size. For example, if the sector size is 512 bytes, an
application can request reads and writes of 512, 1024, or 2048 bytes, but
not of 335, 981, or 7171 bytes.
Buffer addresses for read and write operations should be sector aligned,
which means aligned on addresses in memory that are integer multiples of the
volume sector size. Depending on the disk, this requirement may not be
enforced.
One way to align buffers on integer multiples of the volume sector size is
to use VirtualAlloc to allocate the buffers. It allocates memory that is
aligned on addresses that are integer multiples of the operating system's
memory page size. Because both memory page and volume sector sizes are
powers of 2, this memory is also aligned on addresses that are integer
multiples of a volume sector size.

An application can determine a volume sector size by calling the
GetDiskFreeSpace function."

Post by Michael Russell
I am not sure if this is the right place for this question or not. If not,
kindly direct me to the proper group.
I am having a problem with WriteFile() returning an error. I have
whittled this problem down to a short test program in C which fails under
WinXP x64, but works fine on WinXP x86. I see the problem whether I
compile for a win32 or a win64 target. The quick summary is that if I use
a size of 33525760 I can use WriteFile() (after a CreateFile() with the
FILE_FLAG_NO_BUFFERING flag) just fine. If, however I use a file size of
Insufficient system resources exist to complete the requested service.
I am running on a Dell Precision Workstation 670 w/ dual Xeon
processors, 2GB RAM and using WinXP x64 SP1.
I have tried increasing the pagefile.sys size since someone suggested
that, but it did not help.
--- CUT HERE - start of cf-test.cpp ---
#include <stdio.h>
#include <io.h>
#include <fcntl.h>
#include <Windows.h>
// Forward Declaration
void PrintError (void);
int main(int argc, char *argv[])
{
HANDLE hHandle = NULL;
char caFilename[512];
bool bRet = false;
unsigned char *ucpData = NULL;
// This size will work
// long lChunkSizeBytes = 33525760;
// This size will NOT work
long lChunkSizeBytes = 33526272;
unsigned long ulNumBytesWritten = 0;
printf("argc = %d\n", argc);
if (argc > 1)
sscanf(argv[1], "%ld", &lChunkSizeBytes);
strcpy(caFilename, "test.dat");
// Allocate Memory
printf("Allocating memory (%ld bytes)...\n", lChunkSizeBytes);
ucpData = (unsigned char *) VirtualAlloc(NULL,
lChunkSizeBytes,
MEM_COMMIT,
PAGE_READWRITE);
printf("done.\n");
// Open File
printf("Opening file '%s'...\n", caFilename);
hHandle = CreateFile(caFilename,
GENERIC_WRITE,
FILE_SHARE_READ | FILE_SHARE_WRITE,
NULL,
OPEN_ALWAYS,
FILE_FLAG_NO_BUFFERING,
NULL);
if (hHandle == INVALID_HANDLE_VALUE)
{
fprintf(stderr, "Error Opening File\n");
PrintError();
//MemoryFree(&hMem, &ucpData);
exit(-1);
}
printf("done, handle = 0x%x.\n", hHandle);
// Write File
printf("Writing file...\n");
if (!WriteFile(hHandle, ucpData, lChunkSizeBytes,
&ulNumBytesWritten,
NULL))
{
fprintf(stderr, "Error Writing File\n");
PrintError();
//MemoryFree(&hMem, &ucpData);
exit(-1);
}
printf("done.\n");
// Close File
printf("Closing file...\n");
if (!CloseHandle(hHandle))
{
fprintf(stderr, "Error Closing File\n");
PrintError();
//MemoryFree(&hMem, &ucpData);
exit(-1);
}
printf("done.\n");
// Free Memory
printf("Releasing Memory...\n");
if (!VirtualFree(ucpData, 0, MEM_RELEASE))
{
fprintf(stderr, "Error Freeing Memory\n");
PrintError();
//MemoryFree(&hMem, &ucpData);
exit(-1);
}
printf("done.\n");
}
void PrintError (void)
{
LPVOID lpMsgBuf;
FormatMessage(
FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_FROM_SYSTEM,
NULL,
GetLastError(),
MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT), // Default language
(LPTSTR) &lpMsgBuf,
0,
NULL
);
// Display the error number & string
fprintf(stderr, " GetLastError(0x%x) = %s\n", GetLastError(),
(LPCTSTR) lpMsgBuf);
// Free the buffer.
LocalFree(lpMsgBuf);
return;
}
--- CUT HERE - end of cf-test.cpp ---
Compilation for x64 is done with the Platform SDK for Windows Server 2003
SP1 and selecting the Build Environment for XP x64 Retail. Then "cl
cf-test.cpp bufferoverflowU.lib". Thanks very much for any assistance.
-Michael Russell

[Slava M. Usov]

Post by Michael Russell
I am not sure if this is the right place for this question or not. If not,
kindly direct me to the proper group.

This is the right place.

Post by Michael Russell
I am having a problem with WriteFile() returning an error. I have
whittled this problem down to a short test program in C which fails under
WinXP x64, but works fine on WinXP x86. I see the problem whether I
compile for a win32 or a win64 target. The quick summary is that if I use
a size of 33525760 I can use WriteFile() (after a CreateFile() with the
FILE_FLAG_NO_BUFFERING flag) just fine. If, however I use a file size of
Insufficient system resources exist to complete the requested service.

Apart from the alignment issue, you should understand that the OS does not
guarantee that it will be able to handle arbitrarily large buffers. 30M and
above are large buffers, and they are also unreasonably large for file IO.

S

[Michael Russell]

I believe this is a perfectly valid size - 33526272 is evenly divisible by the
volume's sector size of 512. And, as MS specifies, I am using VirtualAlloc()
to make sure the memory alignment is correct.
Do you have a reference which specifies what a "large buffer" is defined
as? Keep in mind our application works fine under Win32 up to at least 50M
and we are really surprised that with the additional address space of Win64 we
are running into this problem at 33M.
Thanks.

-Mike

Post by Slava M. Usov

Post by Michael Russell
I am not sure if this is the right place for this question or not. If not,
kindly direct me to the proper group.

This is the right place.

Post by Michael Russell
I am having a problem with WriteFile() returning an error. I have
whittled this problem down to a short test program in C which fails under
WinXP x64, but works fine on WinXP x86. I see the problem whether I
compile for a win32 or a win64 target. The quick summary is that if I use
a size of 33525760 I can use WriteFile() (after a CreateFile() with the
FILE_FLAG_NO_BUFFERING flag) just fine. If, however I use a file size of
Insufficient system resources exist to complete the requested service.

Apart from the alignment issue, you should understand that the OS does not
guarantee that it will be able to handle arbitrarily large buffers. 30M and
above are large buffers, and they are also unreasonably large for file IO.
S

[Slava M. Usov]

Post by Michael Russell
I believe this is a perfectly valid size - 33526272 is evenly divisible by
the volume's sector size of 512. And, as MS specifies, I am using
VirtualAlloc() to make sure the memory alignment is correct.

This is true, but it was not clear from the original message, which said
"33526272 or larger".

Post by Michael Russell
Do you have a reference which specifies what a "large buffer" is defined
as?

There is no such reference. Anything bigger than a couple of megs is a large
buffer. Buffers bigger than the combined hardware + drivers maximum transfer
size do not improve IO performance. There is hardware that may support 30M+
transfer sizes, but usually the OS driver stack makes this impossible and
you have tens or hundreds of kilobytes at most. I do not know what hardware
+ drivers you or your customers have, though.

Post by Michael Russell
Keep in mind our application works fine under Win32 up to at least 50M and
we are really surprised that with the additional address space of Win64 we
are running into this problem at 33M.

That is actually expected. x64 is basically x86 but it more than doubles the
overhead required for memory bookkeeping. You must have sizable physical
memory to enjoy "the additional address space".

S

[Alexander Grigoriev]

Since the MDL size field is still 16 bit, while its items are twice as
bigger, I would not be surprised if the max memory for one MDL is now a bit
shy of 32 M.

Post by Slava M. Usov

Post by Michael Russell
I believe this is a perfectly valid size - 33526272 is evenly divisible by
the volume's sector size of 512. And, as MS specifies, I am using
VirtualAlloc() to make sure the memory alignment is correct.

This is true, but it was not clear from the original message, which said
"33526272 or larger".

Post by Michael Russell
Do you have a reference which specifies what a "large buffer" is defined
as?

There is no such reference. Anything bigger than a couple of megs is a large
buffer. Buffers bigger than the combined hardware + drivers maximum transfer
size do not improve IO performance. There is hardware that may support 30M+
transfer sizes, but usually the OS driver stack makes this impossible and
you have tens or hundreds of kilobytes at most. I do not know what hardware
+ drivers you or your customers have, though.

Post by Michael Russell
Keep in mind our application works fine under Win32 up to at least 50M and
we are really surprised that with the additional address space of Win64 we
are running into this problem at 33M.

That is actually expected. x64 is basically x86 but it more than doubles the
overhead required for memory bookkeeping. You must have sizable physical
memory to enjoy "the additional address space".
S

[Slava M. Usov]

Post by Alexander Grigoriev
Since the MDL size field is still 16 bit, while its items are twice as
bigger, I would not be surprised if the max memory for one MDL is now a
bit shy of 32 M.

Good find. 16 bit MDL size was silly on 32 bit platforms, and it's
cardinally nonsensical on 64 bit platforms. Sigh.

S

[Pavel A.]

Post by Alexander Grigoriev
Since the MDL size field is still 16 bit, while its items are twice as
bigger, I would not be surprised if the max memory for one MDL is now a bit
shy of 32 M.

ByteCount field of MDL is ULONG that means 4 GB.
So the max. size depends only on internal implementation behing the
opaque MDL struct, it's not a design problem.

Regards,
--PA

[Slava M. Usov]

Post by Pavel A.

Post by Alexander Grigoriev
Since the MDL size field is still 16 bit, while its items are twice as
bigger, I would not be surprised if the max memory for one MDL is now a
bit shy of 32 M.

ByteCount field of MDL is ULONG that means 4 GB.
So the max. size depends only on internal implementation behing the
opaque MDL struct, it's not a design problem.

MDLs are opaque so you're stuck with the API limitations. Now look up the
DDK article on IoAllocateMdl() and read it carefully.

S