Start a new topic

How do I transfer my jobs output back If I do not know beforhand what the name of the output will be?

I modified tutorial pegasus to send my jobs. That tutorial passes the name of the output file to the executable and then transfers back that output if I understand right.


How could I have all the output files transferred to the login node without knowing the name of them?

Pegasus requires full knowledge of files in the workflow. You have to tell it explicitly what files a job produces. If you do not know the name of them, the easiest path forward is to have a wrapper around your job, which ones your executable has run, tars up the output files into a tarfile which pegasus knows about.

 

Thank you!

I have an static executable and call it from a wrapper script. The wrapper is executable in the dax-generator.py. Do I need to add the "real" executable as an executable in the dax generator?
I tried it like this:
(but it fails with exit code 127)

 

# Create a abstract dag
dax = ADAG("wordfreq-workflow")

# Add executables to the DAX-level replica catalog
wordfreq = Executable(name="RGS_wraperEnvSetup", arch="x86_64", installed=False)
wordfreq.addPFN(PFN("file://" + base_dir + "/RGS_wraperEnvSetup", "local"))
wordfreq.addProfile(Profile(Namespace.PEGASUS, "clusters.size", 1))
dax.addExecutable(wordfreq)

exe= Executable(name="exe", arch="x86_64", installed=False)
exe.addPFN(PFN("file://" + base_dir + "/RGS", "local"))
exe.addProfile(Profile(Namespace.PEGASUS, "clusters.size", 1))
dax.addExecutable(exe)

# add jobs, one for each input file
inputs_dir = base_dir + "/inputs"
for in_name in os.listdir(inputs_dir):

    # Add input file to the DAX-level replica catalog
    in_file = File(in_name)
    in_file.addPFN(PFN("file://" + inputs_dir + "/" + in_name, "local"))
    dax.addFile(in_file)

    # Add job
    wordfreq_job = Job(name="RGS_wraperEnvSetup")
#    out_file = File("Root/GMK_4mm_dosimetry_detector_180_1.xyz.eDep")
    wordfreq_job.addArguments('4','180', '1', in_file)
    wordfreq_job.uses(in_file, link=Link.INPUT)
    wordfreq_job.uses(exe, link=Link.INPUT)
#    wordfreq_job.uses(out_file, link=Link.OUTPUT)
    dax.addJob(wordfreq_job)

 

Then I call it this way in the wrapper:

export PHYSLIST=STD
export G4LEDATA=/home/tkorokn/software/geant4/geant4.10.01.p01-install/share/Geant4-10.1.1/data/G4EMLOW6.41
#mkdir Root
exe $1 $2 $3 $4

 

 

Login or Signup to post a comment