Dear all Pipeline Pilot users,
How often does it happen that you're in the scenario where you write out a small file and read it in again just because the text and SD writers have nice formatting options that come in handy sometimes? Or just because it makes the layout of your protocol easier to understand?
Fact is, reading and writing to disk is slow.
I have an example where I write out a tab delimited text file, read it in again so I can put the whole tab-delimited text in a global variable which I can send over the wire (web service). The same web service uses some temp files to write and read back in again, and that's exactly where the performance hits are.
Imagine that there would be a way to write the same data to memory instead of a file. If you do a lot of repetitive tasks using files, writing them and reading them back in, a "memory" reader and "writer" would spectacularly boost performance, because writing data to memory instead of disk is typically much faster. For example, on my computer reading from memory is 50 times faster than reading from disk.
Of course, you can't write 10 gigs to memory because that would fill up your server's memory, but, it's usually the small 'temp' files using a lot of disk I/O.
Therefore, I would like to ask who'd also be interested in such a feature? Maybe we can try to get it in the product if enough people are interested. Of course, if I'm the only one, there's no reason to implement this and I'll happily let it go :-)
How often does it happen that you're in the scenario where you write out a small file and read it in again just because the text and SD writers have nice formatting options that come in handy sometimes? Or just because it makes the layout of your protocol easier to understand?
Fact is, reading and writing to disk is slow.
I have an example where I write out a tab delimited text file, read it in again so I can put the whole tab-delimited text in a global variable which I can send over the wire (web service). The same web service uses some temp files to write and read back in again, and that's exactly where the performance hits are.
Imagine that there would be a way to write the same data to memory instead of a file. If you do a lot of repetitive tasks using files, writing them and reading them back in, a "memory" reader and "writer" would spectacularly boost performance, because writing data to memory instead of disk is typically much faster. For example, on my computer reading from memory is 50 times faster than reading from disk.
Of course, you can't write 10 gigs to memory because that would fill up your server's memory, but, it's usually the small 'temp' files using a lot of disk I/O.
Therefore, I would like to ask who'd also be interested in such a feature? Maybe we can try to get it in the product if enough people are interested. Of course, if I'm the only one, there's no reason to implement this and I'll happily let it go :-)
