BDSS with Data Transfer Node

[spficklin]

Alex and I were chatting about how BDSS handles when a DTN is used for moving data. In our case at WSU the DTN will be a separate machine not connected to the cluster. BDSS will be used in the Galaxy workflow as a tool, but in order to use the DTN we thought we might need more than one BDSS... one in the Galaxy workflow and one running on the DTN. The BDSS on Galaxy would communicate with the one on the DTN to request files. When the files are available the BDSS on Galaxy would move them over to the cluster storage to get the files off the DTN. Does this sound like a reasonable solution? If so, it seems BDSS will need to have a way to communicate with another instance of itself.

[nawatts]

There could be a two stage transfer mechanism, that connects to the DTN, runs transfers to there, then transfers from DTN to final destination.

I don't think there's a cause for two different copies of the BDSS client. In that case, there'd be redundant lookups of the data URLs in the metadata repository.

Or possibly BDSS doesn't need to contain this entire process. Instead of running bdss on the cluster, maybe install bdss client on the DTN and on the cluster, run a script that looks something like:

ssh dtn bdss transfer urls.txt && globus-url-copy dtn:~/* ./

Or have the DTN file system mounted on the destination?