Question: I am using an external library which is creating a large amount (>4GB)of binary data. I want to buffer this data and feed it to a second library while it is still being created. 

My program is a 64 bit process, running on Linux. I cannot make any guarantees about the system which is going to be running it regarding its RAM.

For the first library I implement a virtual function which the library calls, to feed me with the binary data:

```
virtual void put_data(uint8_t* data, size_t s_data);

```

I am free to do whatever I want in the implemented function with this data.

The second library expects similarly this data in chunks of 100 bytes. The function looks like that:

```
void write(const uint8_t* buffer);

```

So what I could do in theory is start the first lib in one thread, the second lib in a second and while the data is coming from the first lib, in a loop feed this data to the second lib if it is available. Now obviously I cannot just forward the buffers because they might have a different size. Is there a convenient way of doing this in C++, or do I have to write a new Class for that?

Answer: 
First, you could write files in a [tmpfs](https://en.wikipedia.org/wiki/Tmpfs) mounted file system. Then, the files are sitting in [virtual memory](https://en.wikipedia.org/wiki/Virtual_memory), and, as long as they fit in RAM, no disk IO happens. BTW Linux has a good [page cache](https://en.wikipedia.org/wiki/Page_cache) so when a process writes some file which is quickly read and consumed, few disk IO happens (even on ordinary file systems). Read also <http://linuxatemyram.com/>

Since you mention running *two* threads, you could set up a [pipe(7)](http://man7.org/linux/man-pages/man7/pipe.7.html) to communicate between the threads (with one thread writing to the pipe, and another one reading from it). Read [Advanced Linux Programming](http://advancedlinuxprogramming.com/) & some [pthreads tutorial](https://computing.llnl.gov/tutorials/pthreads/). Consider also learning [C++11 programming](http://www.stroustrup.com/Programming/), redesigning your program entirely and using the many features (like closures - learn also more about [continuations](https://en.wikipedia.org/wiki/Continuation) and [CPS](https://en.wikipedia.org/wiki/Continuation-passing_style), containers, smart pointers, ...) introduced in C++11.

At last, you might keep all the data in virtual memory, perhaps using [std::stringstream](http://en.cppreference.com/w/cpp/io/basic_stringstream), [open\_memstream(3)](http://man7.org/linux/man-pages/man3/open_memstream.3.html), or just `std::string`

See also [fifo(7)](http://man7.org/linux/man-pages/man7/fifo.7.html), [shm\_overview(7)](http://man7.org/linux/man-pages/man7/shm_overview.7.html), [sem\_overview(7)](http://man7.org/linux/man-pages/man7/sem_overview.7.html). Learn more about [C++11 threads](http://en.cppreference.com/w/cpp/thread) and more generally about [Inter Process Communication techniques](https://en.wikipedia.org/wiki/Inter-process_communication) & [event loops](https://en.wikipedia.org/wiki/Event_loop) (e.g. above [poll(2)](http://man7.org/linux/man-pages/man2/poll.2.html) ...).
If you have access to boost, you could use their shared memory library.

[boost interprocess](http://www.boost.org/doc/libs/1_59_0/doc/html/interprocess/sharedmemorybetweenprocesses.html#interprocess.sharedmemorybetweenprocesses.sharedmemory)