![]() ![]() I'm using the buffer I just read to count the number of chars I need to go back, and set the stream back that amount of bytes. With optimized compression, and distributed search, we reach TB/s brute force speed. So searching at GB/s speed does make sense. I already hear some people say: "hey, this is premature optimization! the disk is the bottleneck!".īut in 2020, my 1TB SSD reads at more than 3GB/s. It will be faster than any UTF-8 decoding for sure. by StringFromFile() and TSynMemoryStreamMapped.Create() function FileOpenSequentialRead ( const FileName : string ): Integer begin the FILE_FLAG_SEQUENTIAL_SCAN flag under Windows, as stated // by // - under XP, we observed ERROR_NO_SYSTEM_RESOURCES problems with FileRead() // bigger than 32MB // - under POSIX, calls plain FileOpen(FileName,fmOpenRead or fmShareDenyNone) // - is used e.g. / overloaded function optimized for one pass file reading // - will use e.g. See the corresponding function in SynCommons.pas : It bypasses the OS cache, so make it more efficient in your case. Last trick: under Windows, check the FILE_FLAG_SEQUENTIAL_SCAN option when you open such a huge file. It will work, very efficiently, for any size of input files - even TB. I proposed to read the files in a buffer (typically 2MB-32MB), chunk by chunk, searching for the line feeds in it. ![]() Guess you didn't understand what I wrote. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |