Buffering the File Stream in Delphi / C++ Builder
Part of the very fast FireDAC database component library is TFDFileStream,
a class that allows high performance access to trace files, (TFDMoniFlatFileClientLink), text data file reading (TFDBatchMoveTextReader / TFDBatchMoveTextWrtiter), SQL Script file reading (TFDScript) and data serialization to file (TFDMemTable etc).
Well, it seems it was too good to keep hidden and with the release of 10.1 Berlin, this has been moved from FireDAC.Stan.Util to System.Classes and has been renamed to TBufferedFileStream.
TBufferedFileStream is a TFileStream descendant that optimises multiple consecutive small writes or reads. In other words, TBufferedFileStream adds buffering support to TFileStream.
As TBufferedFileStream descends from TFileStream it is a simple replacement to add it into your applications and gain the speed benefits it brings.
Read more for code example..
TBufferedFileStream Example
To see the speed difference, lets run a test! Firstly, using the old style of Read/Write using TFileStream and secondly compared to TBufferedFileStream.
Setting up the TBufferedFileStream Test.
This test uses a blank VCL forms application, (you could use a FireMonkey one instead); added to the main form is a TMemo (memo1) and 3 x TButton (named btnWrite, btnRead, btnReadBuffered. Additionally added to the uses is System.Diagnostics to provide the TStopWatch for timing the speed of TFileStream and TBufferedFileStream.
Writing the file to test the reading methods is pretty simple. For the first button (btnWrite) here is some implemented code to quickly setup 100,000 rows into a text file. (you could make this bigger still if you wish)
procedure TFormReaderWriter.btnWriteClick(Sender: TObject); var sw: TStreamWriter; I: Integer; begin sw := TStreamWriter.Create('test.txt', False, TEncoding.UTF8); try // write 10K lines sw.WriteLine ('Hello, world'); for I := 1 to 99999 do sw.WriteLine ('Hello ' + I.ToString); finally sw.Free; end; Memo1.Lines.Add ('File written'); end;
WriteLn in the code above adds the text and appends character 13 to the end of every line. (in fact you can go open it in notepad if you want to see a long list of Hello x) The following two tests will pass the file and count the number of times char 13 appears to get a total of the rows found. This is a really crude test that is very incremental in the reading operation.
procedure TFormReaderWriter.btnReadClick(Sender: TObject); var fStr: TFileStream; Total, I: Integer; sw: TStopwatch; ch: Char; begin sw := TStopwatch.StartNew; fStr := TFileStream.Create('test.txt', fmOpenRead); try Total := 0; while fStr.Read (ch, 1) = 1 do begin if ch = #13 then Inc(Total); end; Memo1.Lines.Add ('Lines: ' + Total.ToString); finally fStr.Free; end; sw.Stop; Memo1.Lines.Add ('msec: ' + sw.ElapsedMilliseconds.ToString); end;
Running this code inside my VM the StopWatch returns back in around 1573ms to read the 100,000 lines.
Now I know the above code may be simple to parse, but is not very efficient. (unlike loading a text file into a TStringList). So lets make the code as effective as the highly optimised TStringList (even though the code is not as efficient as it could be) by simply copying the above procedure into the 3rd button OnClick event and changing TFileStream for a TBufferedFileStream.
procedure TFormReaderWriter.btnReadBufferedClick(Sender: TObject); var fStr: TBufferedFileStream; Total, I: Integer; sw: TStopwatch; ch: Char; begin sw := TStopwatch.StartNew; fStr := TBufferedFileStream.Create('test.txt', fmOpenRead); try Total := 0; while fStr.Read (ch, 1) = 1 do begin if ch = #13 then Inc(Total); end; Memo1.Lines.Add ('Lines: ' + Total.ToString); finally fStr.Free; end; sw.Stop; Memo1.Lines.Add ('msec: ' + sw.ElapsedMilliseconds.ToString); end;
Running the updated code dramatically reduces the StopWatch time to just 12ms! Awesome!
A buffered file stream is very nice, but the part I don’t get is how how this was ever relevant to database streaming, which typically comes over a network, not from the file system…
Thanks Mason, I’ve edited to re-clarify. The streaming is for local storage and reading of data rather than over the network. – The joy of writing multiple blog posts at the same time. FireDAC does have blob streaming support as well which has just been added to InterBase in 10.1 Berlin http://docwiki.embarcadero.com/RADStudio/Berlin/en/Support_for_Blob_Streaming_in_FireDAC
Great!
I will use this quite a bit.
Thanks for the lesson.
Buffered is a good adition ! Related to the example, if this issue is solved (very simple fix), reading text lines from any stream will be much much faster:
https://quality.embarcadero.com/browse/RSP-14620