Continuous data recording

General
1

Hello all,

I need a piece of advise.
I am trying to build a long duration data recording with a continuous data throughput of about 24 Mbits/s.
The duration of the recording can be many days without interruption.
I am using the data coming from 8 UART->USB streams (FTDI devices).
As the baud rate is around 3Mbits/s per UART, I plan to use the d2xx.dll to directly access the unit for data read.
I am saving the data in a memory block to be fast enough.
So far, this part is working.

When the memory block is full, I need to save it to disk.
However, while saving the memory block to disk, I must still be able to record the incoming traffic (that never stops).
For the moment, I put the recording code in a Thread with maximum priority.
Not sure that a thread is the best option, though.
I was thinking about using paging (e.g. 2 memory blocks, one being written to disk while the other one received the serial data).

How shall I organise the code to to be able to save at the same time I record?
Any suggestion on how to structure the real-time behaviour of the app will be very welcome.

Xavier

2

Here is an idea.

  • two MB as you suggested
  • but put the DISK WRITE in the thread instead. and make sure the size of the MB is small enough to be written before the next MB can be filled
  • use an SSD to maximize the disk I/O

3Mbit/Sec is almost 400K bytes per second … wow

3

Thanks Dave,

Where would I put the code that records the data?

Currently, I am in an endless loop that polls the devices to see if there is data to read.
This shall be quite fast as I shall not miss any byte.
And I am recording 8 channels concurrently -> 3.2 Mbytes / s.
Note that I still have the possibility to launch 8 times the same app in order to efficiently use the multicore CPUs.
That would bring back the data throughput to 400 kB/s but the disk access might be significantly slower (unless I use one SSD per channel). No HW limitation in the grabbing PC :wink:

Xavier

4

Quite possibly 16 if you account for hyperthreading on most modern CPU’s

5

Good SSDs can easily handle 3.2 MBpbs, in fact NVMe SSDs can handle up x1000 more. You could try the threading approach, but a multi-process architecture is probably more safe. Per IO input line, try with one process to collect the data, and another one to dump it to disk (with in mem data sharing). One the downside of a multi-process architecture is the monitoring and failover which can be more complicated.

6

I tried having 2 threads but any action on the Window/GUI is holding the threads and I am losing data.
I do not really need to have a responding GUI but I need to be able to stop the recording.
And I need 2 parallel processes, one to grab the data and one to write them to disk.

What controls / mechanisms may I use to minimise the CPU time spent on GUI but still have the possibility to communicate with a thread or another process via IPC?

7

And why not a console app that directly records the data in a BinaryStream to the disk?
And another GUI app that lets you start/stop one console app for each channel used. The console app could be started with a parameter for the channel and send back info via IPC to the GUI app.

8

Hello Alain,

Yes, I have been thinking about that too.
But I have not been able yet to properly open an FTDI device in a console app (while the same code works in the GUI app).
Listing the device also take a hell of a time while it is almost instantaneous in a GUI app.
Maybe a problem of declares that are not well supported in console app.

This “simple little” project appears a lot more complex than anticipated.
I am also afraid that the Win10 erratic scheduling of the USB driver of the FTDI device will cause data losses et such high troughput.

9

You should be able to make a “headless” GUI app. That should at least get the IPC communication ironed out.

10

In the console App, have you checked that you give enough time for background processes in the event loop ?

11