FlushFileBuffers in non-privileged user mode

[Hirakawa]

Good day. I am currently working on an encryption software that need
to flush file cache before the software end.

While searching around for a solution, I got the source code of
SysInternals Sync and I learn that FlushFileBuffers is the right
function that can flush the file cache. Unfortunately, Sync does not
work in non-privileged user mode.

During my searching, I found another possible solution in this usenet
group which can enable CreateFile return a valid handle to let
FlushFileBuffers working in non-privileged user mode.

According to the message, FILE_FLAG_BACKUP_SEMANTICS is the key that
makes the CreateFile return a valid handle for FlushFileBuffers to
flush file cache in user mode but no matter how hard I tried, it
doesn't work at all.

Is there any other method I can try to make FlushFileBuffers works in
non-privileged user mode?

[J de Boyne Pollard]

H> Good day. I am currently working on an encryption software
H> that need to flush file cache before the software end.

Why do you think that it needs to do that?

H> Is there any other method I can try to make FlushFileBuffers
H> works in non-privileged user mode?

<URL:http://homepages.tesco.net./~J.deBoynePollard/FGA/put-down-the-
chocolate-covered-banana.html>

[Hirakawa]

Post by J de Boyne Pollard
H> Good day. I am currently working on an encryption software
H> that need to flush file cache before the software end.
Why do you think that it needs to do that?
H> Is there any other method I can try to make FlushFileBuffers
H> works in non-privileged user mode?
<URL:http://homepages.tesco.net./~J.deBoynePollard/FGA/put-down-the-
chocolate-covered-banana.html>

Hi, thank you for reply.

In Windows 2000 and XP user mode, if the working device that I am
using is a removable disk (either Flash Drive or HDD), I will have the
problem of data corruption or small file (<4k) lost. Therefore I need
to make sure that everything is flush back before user plug out or
safely remove the device.

[Uwe Sieber]

Post by Hirakawa

Post by J de Boyne Pollard
H> Good day. I am currently working on an encryption software
H> that need to flush file cache before the software end.
Why do you think that it needs to do that?
H> Is there any other method I can try to make FlushFileBuffers
H> works in non-privileged user mode?
<URL:http://homepages.tesco.net./~J.deBoynePollard/FGA/put-down-the-
chocolate-covered-banana.html>

Hi, thank you for reply.
In Windows 2000 and XP user mode, if the working device that I am
using is a removable disk (either Flash Drive or HDD), I will have the
problem of data corruption or small file (<4k) lost. Therefore I need
to make sure that everything is flush back before user plug out or
safely remove the device.

'Safely remove' is good enough, it flushes for you.

To prevent caching problems at all just use FILE_FLAG_NO_BUFFERING
or FILE_FLAG_WRITE_THROUGH.


Uwe

[Hirakawa]

Post by Uwe Sieber

Post by Hirakawa

Post by J de Boyne Pollard
H> Good day. I am currently working on an encryption software
H> that need to flush file cache before the software end.
Why do you think that it needs to do that?
H> Is there any other method I can try to make FlushFileBuffers
H> works in non-privileged user mode?
<URL:http://homepages.tesco.net./~J.deBoynePollard/FGA/put-down-the-
chocolate-covered-banana.html>

Hi, thank you for reply.
In Windows 2000 and XP user mode, if the working device that I am
using is a removable disk (either Flash Drive or HDD), I will have the
problem of data corruption or small file (<4k) lost. Therefore I need
to make sure that everything is flush back before user plug out or
safely remove the device.

'Safely remove' is good enough, it flushes for you.
To prevent caching problems at all just use FILE_FLAG_NO_BUFFERING
or FILE_FLAG_WRITE_THROUGH.
Uwe

Hi Uwe, I visited your website before and you have a great eject
device software. Since you're here, can I ask you a question about
safely remove composite usb mass storage device? (I hope you don't
mind)

I got some devices with me right now which cannot be safely remove by
Win2K and WinVista. Those devices are composite device and I think I
might have problem to flush the cache under user mode. May I know how
your eject software flush the file cache under user mode?

Back to my problem, some of my users report to me that under Windows
2000 user mode, they have data corruption problem even though OS
return safely remove successful message. It seems that due to FAT32,
the write back of cache is slower than FAT. That's the reason why I am
seeking for a solution to flush file buffers in non privileged mode.

Currently I am using the following line to get the handle and send to
flushfilebuffers in Administrator Mode:

CreateFile(szVolumeName, GENERIC_READ | GENERIC_WRITE, FILE_SHARE_READ
| FILE_SHARE_WRITE, 0, OPEN_EXISTING, FILE_FLAG_WRITE_THROUGH, 0);

[Uwe Sieber]

Post by Hirakawa

Post by Uwe Sieber

Post by Hirakawa

Post by J de Boyne Pollard
H> Good day. I am currently working on an encryption software
H> that need to flush file cache before the software end.
Why do you think that it needs to do that?
H> Is there any other method I can try to make FlushFileBuffers
H> works in non-privileged user mode?
<URL:http://homepages.tesco.net./~J.deBoynePollard/FGA/put-down-the-
chocolate-covered-banana.html>

Hi, thank you for reply.
In Windows 2000 and XP user mode, if the working device that I am
using is a removable disk (either Flash Drive or HDD), I will have the
problem of data corruption or small file (<4k) lost. Therefore I need
to make sure that everything is flush back before user plug out or
safely remove the device.

'Safely remove' is good enough, it flushes for you.
To prevent caching problems at all just use FILE_FLAG_NO_BUFFERING
or FILE_FLAG_WRITE_THROUGH.
Uwe

Hi Uwe, I visited your website before and you have a great eject
device software. Since you're here, can I ask you a question about
safely remove composite usb mass storage device? (I hope you don't
mind)
I got some devices with me right now which cannot be safely remove by
Win2K and WinVista. Those devices are composite device and I think I
might have problem to flush the cache under user mode. May I know how
your eject software flush the file cache under user mode?

The EjectMediaTool tries to flush but if it fail then it
just do not flush.
IOCTL_STORAGE_EJECT_MEDIA is called only if FSCTL_LOCK_VOLUME
FSCTL_DISMOUNT_VOLUME succeed before.
I tried to get corrupt data by writing a large file in 3,5KB
blocks to fill up the write cache. FSCTL_LOCK_VOLUME always
failed until all data where written. So, this seems to be
good enough.

Post by Hirakawa
Back to my problem, some of my users report to me that under Windows
2000 user mode, they have data corruption problem even though OS
return safely remove successful message. It seems that due to FAT32,
the write back of cache is slower than FAT.

Interesting. I've not tried this so far.

Post by Hirakawa
That's the reason why I am
seeking for a solution to flush file buffers in non privileged mode.
Currently I am using the following line to get the handle and send to
CreateFile(szVolumeName, GENERIC_READ | GENERIC_WRITE, FILE_SHARE_READ
| FILE_SHARE_WRITE, 0, OPEN_EXISTING, FILE_FLAG_WRITE_THROUGH, 0);

The FILE_FLAG should not make a difference for FlushFileBuffers.
It's correct that FlushFileBuffers needs write access. Seems
EjectMedia uses a read handle, so it never flushed...

For FSCTL_LOCK_VOLUME you need at least a read handle. For
restricted uses even this fails. This can be enabled by a
policy:
http://www.uwe-sieber.de/usbstick_e.html#removal_fails
The policy is supposed to enable the ejection of drives with
DriveType DRIVE_REMOVABLE, but in fact it enables read access
to such drive's volume.


Uwe

[J de Boyne Pollard]

<URL:http://homepages.tesco.net./~J.deBoynePollard/FGA/put-down-the-
chocolate-covered-banana.html>

H> Back to my problem, some of my users report to me that under
H> Windows 2000 user mode, they have data corruption problem
H> even though OS return safely remove successful message.

You've checked firsthand that these users are not performing surprise
removal rather than safe removal? It is, after all, not unheard-of
for users to incorrectly describe what they are doing. ("Windows is
telling me after I have removed it that the device requires safe
removal. So I must have removed it safely.")

This is a classic symptom of surprise removal. As <URL:http://
microsoft.com./whdc/device/storage/usbfaq.mspx#EUDAC> says, "Customer
feedback showed that consumers were unplugging USB storage devices
from systems, without going to the 'safely remove hardware' tab in the
system tray on Windows 2000 systems. Such random removal sometimes
left storage devices in a corrupted state.".

Read <URL:http://microsoft.com./whdc/system/pnppwr/hotadd/XPrem-
devs.mspx>, too.

H> It seems that due to FAT32, the write back of cache is slower
H> than FAT.

That's nonsense. FAT32 _is_ FAT.

H> That's the reason why I am seeking for a solution to flush
H> file buffers in non privileged mode.

Put down that chocolate-covered banana. What you should be doing is
(a) confirming with your own eyes that this is most definitely _not_
surprise removal, and (b) looking for reasons why safe removal of a
device would result in filesystem corruption.

[Sten Westerback (MVP SDK 2005-6 :)]

Post by Hirakawa

Post by J de Boyne Pollard
H> Good day. I am currently working on an encryption software
H> that need to flush file cache before the software end.
Why do you think that it needs to do that?
H> Is there any other method I can try to make FlushFileBuffers
H> works in non-privileged user mode?
<URL:http://homepages.tesco.net./~J.deBoynePollard/FGA/put-down-the-
chocolate-covered-banana.html>

Hi, thank you for reply.
In Windows 2000 and XP user mode, if the working device that I am
using is a removable disk (either Flash Drive or HDD), I will have the
problem of data corruption or small file (<4k) lost. Therefore I need
to make sure that everything is flush back before user plug out or
safely remove the device.

Firstly this is something the "Safely Remove hardware" is there for. Use it!
Secondly, before you close your file you should FlushFileBuffers() before
you close a file.
Thirtdly, i think you could use DeviceIoControl with FSCTL_DISMOUNT_VOLUME
yourself... after asking the user if (s)he really want to disconnect the
device....and you have to convince why they would use your tool to do it
while Windows already have such a tool.

- Sten

[Kerem Gümrükcü]

Post by Hirakawa
Good day. I am currently working on an encryption software that need
to flush file cache before the software end.
While searching around for a solution, I got the source code of
SysInternals Sync and I learn that FlushFileBuffers is the right
function that can flush the file cache. Unfortunately, Sync does not
work in non-privileged user mode.
During my searching, I found another possible solution in this usenet
group which can enable CreateFile return a valid handle to let
FlushFileBuffers working in non-privileged user mode.
According to the message, FILE_FLAG_BACKUP_SEMANTICS is the key that
makes the CreateFile return a valid handle for FlushFileBuffers to
flush file cache in user mode but no matter how hard I tried, it
doesn't work at all.
Is there any other method I can try to make FlushFileBuffers works in
non-privileged user mode?

Hi Hirakawa,

what does GetlastError() say?
See remarks section for more information, since there are things to
take care of!

Regards

Kerem





Beste Grüsse / Best regards / Votre bien devoue

Kerem Gümrükcü
***@arcor.de

Best Quote: "Ain't nobody a badass with a double dose
of rock salt...", Kill Bill Vol.2

Latest Open-Source Projects: http://entwicklung.junetz.de
Sign my guestbook: http://entwicklung.junetz.de/guestbook/

-----------------------
"This reply is provided as is, without warranty express or implied."

[Hirakawa]

Post by Kerem Gümrükcü

Post by Hirakawa
Good day. I am currently working on an encryption software that need
to flush file cache before the software end.
While searching around for a solution, I got the source code of
SysInternals Sync and I learn that FlushFileBuffers is the right
function that can flush the file cache. Unfortunately, Sync does not
work in non-privileged user mode.
During my searching, I found another possible solution in this usenet
group which can enable CreateFile return a valid handle to let
FlushFileBuffers working in non-privileged user mode.
According to the message, FILE_FLAG_BACKUP_SEMANTICS is the key that
makes the CreateFile return a valid handle for FlushFileBuffers to
flush file cache in user mode but no matter how hard I tried, it
doesn't work at all.
Is there any other method I can try to make FlushFileBuffers works in
non-privileged user mode?

Hi Hirakawa,
what does GetlastError() say?
See remarks section for more information, since there are things to
take care of!
Regards
Kerem
Beste Gr�sse / Best regards / Votre bien devoue
Kerem G�mr�kc�
Best Quote: "Ain't nobody a badass with a double dose
of rock salt...", Kill Bill Vol.2
Latest Open-Source Projects:http://entwicklung.junetz.de
Sign my guestbook:http://entwicklung.junetz.de/guestbook/
-----------------------
"This reply is provided as is, without warranty express or implied."

Hi Kerem,

Thank you for your reply. The GetLastError() return a value 5.
According to System Error Codes, it is "Access is Denied".

[Kerem Gümrükcü]

Post by Kerem Gümrükcü

Post by Hirakawa
Good day. I am currently working on an encryption software that need
to flush file cache before the software end.
While searching around for a solution, I got the source code of
SysInternals Sync and I learn that FlushFileBuffers is the right
function that can flush the file cache. Unfortunately, Sync does not
work in non-privileged user mode.
During my searching, I found another possible solution in this usenet
group which can enable CreateFile return a valid handle to let
FlushFileBuffers working in non-privileged user mode.
According to the message, FILE_FLAG_BACKUP_SEMANTICS is the key that
makes the CreateFile return a valid handle for FlushFileBuffers to
flush file cache in user mode but no matter how hard I tried, it
doesn't work at all.
Is there any other method I can try to make FlushFileBuffers works in
non-privileged user mode?

Hi Hirakawa,
what does GetlastError() say?
See remarks section for more information, since there are things to
take care of!
Regards
Kerem
Beste Grüsse / Best regards / Votre bien devoue
Kerem Gümrükcü
Best Quote: "Ain't nobody a badass with a double dose
of rock salt...", Kill Bill Vol.2
Latest Open-Source Projects:http://entwicklung.junetz.de
Sign my guestbook:http://entwicklung.junetz.de/guestbook/
-----------------------
"This reply is provided as is, without warranty express or implied."
Hi Kerem,
Thank you for your reply. The GetLastError() return a value 5.
According to System Error Codes, it is "Access is Denied".

Hi Hirakawa,

your user context in which you execute your applications
does not have a specific access right to execute this function
or another part of your code. You should run your application
and then check with i.e. process monitor from sysinternals
which resource does not allow you the file buffer flushing. Did
you check whether your returned handle is valid or not? Did you
run your application with Administrator Account and then with
another less priviliged user?

It would be great if you could some code so that we can see
what exactly you are doing BEFORE you call the function
and please tell in which user context you try to flush your
buffers.

It seems to be a classic missing acces right or missing
privilige problem,...

See the remarks section of the function for more information!
[FlushFileBuffers]
http://msdn2.microsoft.com/en-us/library/aa364439.aspx

By the way, if you need a tool that can translate error codes
on the fly to error messages, check my app out. It is still
under developent but in cases where you need quick error
code description very usefull!

[Windows Exception Viewer]
http://entwicklung.junetz.de/projects/opensource/WindowsExceptionViewer/WindowsExceptionViewer.zip
You will need the .NET 2.0 Runtime!



Regards

Kerem


Beste Grüsse / Best regards / Votre bien devoue

Kerem Gümrükcü
***@arcor.de

Best Quote: "Ain't nobody a badass with a double dose
of rock salt...", Kill Bill Vol.2

Latest Open-Source Projects: http://entwicklung.junetz.de
Sign my guestbook: http://entwicklung.junetz.de/guestbook/

-----------------------
"This reply is provided as is, without warranty express or implied."