BinaryStream file reading speed

General
1

Hi,

can somebody give me hand, please?

I have to read MPEG TS files and parse the TS packets, but I have some ‘reading speed’ issues.

[code] Dim f As FolderItem
Dim bs As BinaryStream
dim length,dummy as Int64
dim m as memoryblock

f = GetOpenFolderItem("")

If f <> NIL And f.Exists Then

bs = BinaryStream.Open(f, False)

If bs <> Nil Then
  length=f.Length / 188
  
  for dummy=0 to length-1
    m =  bs.Read(188,NIL)
    NewPacket(m)
  next
  
  'Do
  'NewPacket( bs.Read(188))
  'loop until bs.eof
  
  
  bs.Close
  
End If

End If[/code]

this code need 13 seconds (sample file) to process and 21 seconds using the do loop / method. Any change to speed that up as the same file is read and processed in ~4 seconds using Java.

THX

Guido

2

Why not read the whole thing into a single memoryblock in one file operation and then split it into packets in memory?

3

Something else to note… If the program has trouble opening the file, it’s going to throw an IOException, not just leave the BinaryStream Nil.

4

Some minor speed suggestions include:

[code] length = ( f.Length / 188 ) - 1

  for dummy=0 to length
    NewPacket( bs.read( 188 ) )
  next[/code]

I suspect that the main cause of the slowdown is probably within the NewPacket function. I use binary stream in many applications in many places and it’s certainly fast enough.

Personally, my concern with doing this is pushing the application memory usage up, so that it spills over and then ends up swapping. Especially if the OP is reading movie files.

5

the new packet function slows it down but that is minor, the reading part is slow …

6

How big is file?

You can load 500 MB in memory.

Maybe load in chunks of 180000 bytes so you get 1000 packets in one read?

7

variable, from a few MB to several GB

8

why read it in 188 byte chunks if it is MB to GB long ?
thats probably a huge waste of time right there

9

But its reading the whole file anyway, just in lumps.
Isnt the overall memory going to be the same-ish?

10

Reading 4 GB at once the app will crash. I’d recommend reading a couple of MBs and then parse piece by piece.

11

yes, that is properly what I will do.

THX