SAM Prod. Service Client-side failures -- No new tests sent to a resource

TODO

1. Check if the resource (+site) is correctly registered in the SAM DB as explained on General checklist for site/node problems in the SAM DB.
   • NOTE: Carefully check the SERVICEINSTANCE > =ISMONITORED flag, often we have the problem there!!!
2. Try to see what's going on, using the SAM submission framework manually from the SAM UI(s). NOTE: Due to the submission & publishing cron jobs, these operations might not be allowed due to the locking mechanism on the ~/.same/ directory.
   1. Check the status of the last test(using the samops account):

                   [samops]$ /opt/lcg/same/client/bin/same-exec --status <sensor> nodename=<nodename>
                

      This might be a slow operation. In case if you don't specify nodename filter, think about redirecting the long output (2>&1)
      • If there are any finished jobs, in case of (g)CE, (g)RB, FTS sensors, try to publish the results

                        [samops]$ /opt/lcg/same/client/bin/same-exec --publish <sensor> nodename=<nodename>
                      

      • If there are no pending jobs for the resource, try to submit manually

                       [samops]$ /opt/lcg/same/client/bin/same-exec <sensor> nodename=<nodename>
                    

        • In case of CE sensor, you might get error message:

                            Job already submitted to <nodename>
                         

          If the status check showed that the test on the CE is already in Done state, the job was stuck for some reason.
          1. move the ~/.same/CE/nodes/ directory to somewhere for further analysis and open a Savannah bug with the details
          2. re-submit the job to the site

-- JuditNovak - 01 Dec 2007