I am wondering if there is a way to process my records for my SD file in parallel. For each record, I am making a query to an external server for information (Pubchem). This query takes ~10 seconds. I would like to spin off multiple requests so that I can have several queries working at once.
So far, the only way that I have found to achieve any level of parallelism is to put my components into a subprotocol and then to set the parallel processing options on the implementation tab. This is not ideal, however, since I can only achieve as many parallel processes as there are cpu's available.
I would like to create a protocol that splits my records into two streams (pipes) and processes the external query in parallel on both pipes. Is there a way to do this?
So far, the only way that I have found to achieve any level of parallelism is to put my components into a subprotocol and then to set the parallel processing options on the implementation tab. This is not ideal, however, since I can only achieve as many parallel processes as there are cpu's available.
I would like to create a protocol that splits my records into two streams (pipes) and processes the external query in parallel on both pipes. Is there a way to do this?
