How many data to store for the data transfer

Some considerations about the amount of data to be stored about data transfer. From a mail thread between Pablo and Julia (21st Nov 2008)

The proposal is to keep data per hour. Nevertheless, Pablo observed that it can be a huge amount of data. In fact a rough calculation gives:

60 sites -> 3600 channels
3 metrics per channel -> 10.000 metrics/h
1 kbytes per entry -> 10 MB/h
4 VO -> 40 MB/h
24h per day -> 1 GB/day

Julia thinks that it is not too much. Compared to info in the job dashboard (150-200K jobs per day for CMS only) with plenty of metrics which are kept essentially forever for several years already, it is nothing. And in case of site monitoring we can keep per hour data for the latest month only and average older data on the daily basis. So she thinks this is not an issue.

But Pablo replicates that it is not much less than the data of dashboard, because for the jobs in Dashboard they don't replicate the information every hour, so:

200k jobs/day x 5k info per job -> 1GB/day

So, basically, it is the same size (and we don't keep 5k info per job in Dashboard). For Dashboard, he doesn't multiply by 4 VOs, since each VO has its data stored in a separate database. In the CMS job database, querying things that are older than one week is really slow. A query since the beginning of the month took almost 4 minutes.

Julia answers: For jobs we have 60 metrics in the job table , many of them are timestamps which are very heavy , plus plenty of tables related to the job via foreign keys. Plus to it you know how complicated are the job queries, we want such and such status, not just summing up certain numeric metrics. So I do not think this is a fair comparison.

If we see that querying monthly data per hour is too slow, we shorten the period of time where we keep data without archiving. I still think that keeping data per channel might be useful for site admins. If ATLAS complains that data from FZK to CNAF does not work, it would be convenient using the same interface to see whether data from FZK to CNAF for CMS works and uses the full bandwidth , etc.. etc.. If at some point we can construct a link to some other monitoring tool providing all detailed info (FTS), we can always drop duplicating data in the central metrics repository.

-- ElisaLanciotti - 08 Jun 2009