Buffering the File Stream in Delphi / C++ Builder
Part of the very fast FireDAC database component library is
TFDFileStream,a class that allows high performance access to trace files, (TFDMoniFlatFileClientLink), text data file reading (TFDBatchMoveTextReader / TFDBatchMoveTextWrtiter), SQL Script file reading (TFDScript) and data serialization to file (TFDMemTable etc)
. Well, it seems it was too good to keep hidden and with the release of 10.1 Berlin, this has been moved from FireDAC.Stan.Util to System.Classes and has been renamed to TBufferedFileStream.
As TBufferedFileStream descends from TFileStream it is a simple replacement to add it into your applications and gain the speed benefits it brings.
Read more for code example..
To see the speed difference, lets run a test! Firstly, using the old style of Read/Write using TFileStream and secondly compared to TBufferedFileStream.
Setting up the TBufferedFileStream Test.
This test uses a blank VCL forms application, (you could use a FireMonkey one instead); added to the main form is a TMemo (memo1) and 3 x TButton (named btnWrite, btnRead, btnReadBuffered. Additionally added to the uses is System.Diagnostics to provide the TStopWatch for timing the speed of TFileStream and TBufferedFileStream.
Writing the file to test the reading methods is pretty simple. For the first button (btnWrite) here is some implemented code to quickly setup 100,000 rows into a text file. (you could make this bigger still if you wish)
procedure TFormReaderWriter.btnWriteClick(Sender: TObject); var sw: TStreamWriter; I: Integer; begin sw := TStreamWriter.Create('test.txt', False, TEncoding.UTF8); try // write 10K lines sw.WriteLine ('Hello, world'); for I := 1 to 99999 do sw.WriteLine ('Hello ' + I.ToString); finally sw.Free; end; Memo1.Lines.Add ('File written'); end;
WriteLn in the code above adds the text and appends character 13 to the end of every line. (in fact you can go open it in notepad if you want to see a long list of Hello x) The following two tests will pass the file and count the number of times char 13 appears to get a total of the rows found. This is a really crude test that is very incremental in the reading operation.
procedure TFormReaderWriter.btnReadClick(Sender: TObject); var fStr: TFileStream; Total, I: Integer; sw: TStopwatch; ch: Char; begin sw := TStopwatch.StartNew; fStr := TFileStream.Create('test.txt', fmOpenRead); try Total := 0; while fStr.Read (ch, 1) = 1 do begin if ch = #13 then Inc(Total); end; Memo1.Lines.Add ('Lines: ' + Total.ToString); finally fStr.Free; end; sw.Stop; Memo1.Lines.Add ('msec: ' + sw.ElapsedMilliseconds.ToString); end;
Running this code inside my VM the StopWatch returns back in around 1573ms to read the 100,000 lines.
Now I know the above code may be simple to parse, but is not very efficient. (unlike loading a text file into a TStringList). So lets make the code as effective as the highly optimised TStringList (even though the code is not as efficient as it could be) by simply copying the above procedure into the 3rd button OnClick event and changing TFileStream for a TBufferedFileStream.
procedure TFormReaderWriter.btnReadBufferedClick(Sender: TObject); var fStr: TBufferedFileStream; Total, I: Integer; sw: TStopwatch; ch: Char; begin sw := TStopwatch.StartNew; fStr := TBufferedFileStream.Create('test.txt', fmOpenRead); try Total := 0; while fStr.Read (ch, 1) = 1 do begin if ch = #13 then Inc(Total); end; Memo1.Lines.Add ('Lines: ' + Total.ToString); finally fStr.Free; end; sw.Stop; Memo1.Lines.Add ('msec: ' + sw.ElapsedMilliseconds.ToString); end;
Running the updated code dramatically reduces the StopWatch time to just 12ms! Awesome!