KIT Configuration

Monitoring

• monitoring of running mcore jobs: http://web-kit.gridka.de/monitoring/mcore.php
  • efficiency plots show the CPU time over the reserved usage time, i.e., job wall time scaled by the number of assigned cores; for efficient job the ratio is near 1.
• SGE v###

jdl to SGE

• number of cores is evaluated from parameter CpuNumber = NOCORES;

• created parallel environment (PE) with dynamic cores
  • PE configuration

    Executable = "/path/to/my/workload.sh";
    StdOutput = "myjob.out";
    StdError = "myjob.err";
    OutputSandbox = {"myjob.out", "myjob.err"};
    OutputSandboxBaseDestURI = "gsiftp://localhost";
    CpuNumber = 4;
    CERequirements = "other.GlueCEPolicyMaxWallClockTime > 31";
    

• translation to GridEngine handled by {/usr/libexec/}sge_local_submit_attributes.sh
• times in jdl are minutes (GlueScheme ###) and are converted to seconds for Grid

• reservations (draining nodes for multi core jobs) need to be enabled in addition
  • '-R y' flag enabling reservations is not added by 'sge_submit.sh'
  • current workaround is running 'qalter -R y $list_of_mc_jobs' by a cron

CREAM

• jdl parameters for wall/clock times are evaluated
  • CERequirements = "other.GlueCEPolicyMaxWallClockTime > MAXWALLTIME"
  • CERequirements = "other.GlueCEPolicyMaxCPUTime > MAXCPUTIME"
    • C-like syntax for combining requirements CERequirements = "other.GlueCEPolicyMaxWallClockTime > MAXWALLTIME && other.GlueCEPolicyMaxCPUTime > MAXCPUTIME"
  • ">" interpreted as upper limit

• memory and virtual memory limits in principle evaluated
  • SGE v## converts erroneously GlueHostMainMemoryRAMSize to s_vmem instead of (correct) GlueHostMainMemoryVirtualSize
  • memory limits are enforced by 's_rss'/'ulimit' - which has no effect in kernels >=2.6 (Goncalo alluded to this comment)
    • SGE v### supports correct memory limits
  • CERequirements = "other.GlueHostMainMemoryVirtualSize > VMEMLIMIT"
  • CERequirements = "other.GlueHostMainMemoryRAMSize > MEMLIMIT"

Questions

• parameter "Requirement" singular, "CERequirements" is plural (no idea, if "Requirement" is actually used at some point. According to some documentation online it should be "Requirements"; what is the difference between Requirement(s) and CERequirements, what is used? From experiences: CERequirements. What schema version was/is "Requirement(s)"??)

Links

• • jdl definition docs
  • requirement forwarding