Introduction: Automation

What is a script?

## Hi!, this is a set of commands into a script:  

setup some variables

## This is a comment

source a set of other scripts

## This is another comment

set your input and output files and locations

## Finally

execute your main program of set of applications

## Done

Why to use scripts?

An pictorial view of scripts in an analysis

Scripting Languages and the OS as environment

ATLAS

command lines interface support

A list of useful (but far to be completed) of kernels for scripts:

Language/Kernel	Covered here?	Use in HEP?
bash	yes	yes
Python	yes	yes
JavaScript	not	yes
C++	a bit	yes
PowerShell	not	not?
Google Script	not	not?
...	...	...

Q&A

?? Do you have any question so far?
For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL
( please remove SPAMNOT from the email address )

Anatomy of a first script (bash)

#!/bin/bash          

## The line above helps to let the system know that you are asking for the bash shell

## As you can see, using #'s allows you to introduce comments.
## Comments are EXTREMELY useful for self documentation of the code!
## Even if you never share the script (or you code in general) you will thank yourself after few weeks/months and you come back to check what you actually did :)
## And if you share your code, is a nice way to let your user or college know that you follow one of the Good Practices in Programming and Computer Science in general

## A first sugestion is to add to every script you plan to keep for more that few hours:
## - Date of creation: XX/YY/ZZ
## - Date of last update: xx/yy/zz
## - Contact email
 
## Saying all that  (excuse me) lets create a first script following the traditional "Hello World"
## "echo" is a function that will print in a command-shell -terminal- what you put after the "echo" word:

echo "Hello World"

## End

Now, lets use some LINUX/UNIX terminal commands into a script. Here you can start to see what I mean with: Automation. We are "replacing" your fingers (the human) for a scripts (a robot) that can do exactly the same:

    1#!/bin/bash          
    2
    3## Asking where you are:
    4pwd
    5
    6## Listing the files in the current directory
    7
    8ls
    9
   10## Listing the files in your home directory
   11
   12ls $HOME
   13
   14## Listing the files in the temporary directory
   15
   16ls /tmp/
   17
   18## End
   19

Where as you can see, the lines 4,8,12,16 are the only lines that perform an action in this script.

Linux (UNIX) command refreshing review

LINUX has an almost infinite number of commands and shell apps that can do virtually any contemporary operation in a computer or network of computers. some of them are:

cat

> cat      ## Concatenate and print (display) the content of files

cd

> cd       ## Change Directory

chmod

> chmod    ## Change access permissions

clear

> clear    ## Clear terminal screen

comm

> comm     ## Compare two sorted files line by line

cp

> cp       ## Copy one or more files to another location

cut

> cut      ## Divide a file into several parts

date

> date     ## Display or change the date & time

diff

> diff     ## Display the differences between two files

du

> du       ## Estimate file space usage

echo

> echo     ## Display message on screen •

egrep

> egrep    ## Search file(s) for lines that match an extended expression

exit

> exit     ## Exit the shell

export

> export   ## Set an environment variable

find

> find     ## Search for files that meet a desired criteria

grep

> grep     ## Search file(s) for lines that match a given pattern

gzip

> gzip     ## Compress or decompress named file(s)

history

> history  ## Command History

kill

> kill     ## Kill a process by specifying its PID

make

> make     ## Recompile a group of programs

mkdir

> mkdir    ## Create new folder(s)

more

> more     ## Display output one screen at a time

mv

> mv       ## Move or rename files or directories

nohup

> nohup    ## Run a command immune to hangups

passwd

> passwd   ## Modify a user password

paste

> paste    ## Merge lines of files

ping

> ping     ## Test a network connection

ps

> ps       ## Process status

pwd

> pwd      ## Print Working Directory

rcp

> rcp      ## Copy files between two machines

rm

> rm       ## Remove files

scp

> scp      ## Secure copy (remote file copy)

sed

> sed      ## Stream Editor

sleep

> sleep    ## Delay for a specified time

sort

> sort     ## Sort text files

source

> source   ## Run commands from a file '.'

su

> su       ## Substitute user identity

tail

> tail     ## Output the last part of file

time

> time     ## Measure Program running time

uniq

> uniq     ## Uniquify files

vi

> vi       ## Text Editor
> vim       ## Text Editor

wait

> wait     ## Wait for a process to complete •

wc

> wc       ## Print byte, word, and line counts

wget

> wget     ## Retrieve web pages or files via HTTP, HTTPS or FTP

Another interesting use is the inclusion of special characters as is the case of a blank space:

## to include a black space into a command line, use an "\" and add the needed space, for example: 
>  echo Ciao Mondo
Ciao Mondo

> echo Ciao\ Mundo
Ciao Mondo

> echo Ciao     Mondo
Ciao Mondo

> echo Ciao\ \ \ \ \  Mondo
Ciao     Mondo

Some scripts in action

Generic automatic Email for Account "management"

    1#!/bin/bash          
    2###################################################################################################
    3## Author: Arturo Sanchez (arturos@cern.ch)                                                      ##
    4## Created: 07th May 2014                                                                        ##
    5## Updated: 28th October 2015                                                                    ##
    6##                                                                                               ##
    7## How to use this script:                                                                       ##
    8## > ./new_user.sh Name\ LastName nickname 12345 > email_new_users/email_nickname.txt            ##
    9## Example:                                                                                      ##
   10## > ./new_user.sh <Name\ LastName> <username> <password> > email_new_users/email_<username>.txt ##
   11###################################################################################################
   12
   13
   14echo "Bongiorno,"
   15echo ""
   16echo "The account for the user: $1 into the ATLAS Napoli User Interface have been created."
   17echo ""
   18echo ""
   19echo "Your username is:  $2"
   20echo "Your password is:  $3"
   21echo ""
   22echo ""
   23echo "You can access using the ssh connection from a terminal:"
   24echo ""
   25echo "> ssh -YX -p5022 $2@atlasui.na.infn.it"
   26echo ""
   27echo ""
   28echo "Or for a specific UI use the number 01, 02 or 03 as below:" 
   29echo ""
   30echo "> ssh -YX -p5022 $2@atlasui03.na.infn.it"
   31echo ""
   32echo ""
   33echo ""
   34echo "As soon you enter the password, you will get something as below:"
   35echo ""
   36echo "[$2@atlasui03 ~]$"
   37echo ""
   38echo ""
   39echo "Where you can check where you are:"
   40echo ""
   41echo "[$2@atlasui03 ~]$ pwd"
   42echo "/home/$2"
   43echo ""
   44echo ""
   45echo "Please, change your password as soon as possible, using the command:"
   46echo "> yppasswd"
   47echo ""
   48## echo "> passwd"
   49## echo "IMPORTANT! (Please, use the atlasui03 to do this procedure)"
   50## echo ""
   51echo "You will get something like (to introduce your current pass and the new one):"
   52echo ""
   53## echo "[$2@atlasui03 ~]$ passwd"
   54echo "[$2@atlasui03 ~]$ yppasswd"
   55echo "Changing password for user $2."
   56echo "Changing password for $2."
   57echo "(current) UNIX password:"
   58echo "New password: "
   59echo ""
   60echo ""
   61echo ""
   62echo "The UI has space dedicates to big files that the users could need to store there."
   63echo "The path of such a space is: "
   64echo ""
   65echo "/data/<username>"
   66echo ""
   67echo "So, in your case is:"
   68echo ""
   69echo "/data/$2"
   70echo ""
   71echo ""
   72echo "Your /home/$2 folder has a quota of 10GB."
   73echo ""
   74echo "If you have an estimation of the possible space that you will need in /data/ and/or /home/, we will appreciate that information in order to setup your quota according to your work necessities."
   75echo ""
   76echo "In case of any question/suggestion please let me know,"
   77echo ""
   78echo "Cheers,"
   79echo "Account Managers"
   80echo "------"
   81echo ""

Where if you save this script in a file called: new_user.sh and execute the following line:

>  source new_user.sh Jon\ Doe jondoe 12345

...it will give you as a result the printing below ( printing that you can redirect into a text file for bookkeeping and posterior use, like to really send the email )

Bongiorno,

The account for the user: Jon Doe into the ATLAS Napoli User Interface have been created.


Your username is:  jondoe
Your password is:  12345


You can access using the ssh connection from a terminal:

> ssh -YX -p5022 jondoe@atlasui.na.infn.it


Or for a specific UI use the number 01, 02 or 03 as below:

> ssh -YX -p5022 jondoe@atlasui03.na.infn.it



As soon you enter the password, you will get something as below:

[jondoe@atlasui03 ~]$


Where you can check where you are:

[jondoe@atlasui03 ~]$ pwd
/home/jondoe


Please, change your password as soon as possible, using the command:
> yppasswd

You will get something like (to introduce your current pass and the new one):

[jondoe@atlasui03 ~]$ yppasswd
Changing password for user jondoe.
Changing password for jondoe.
(current) UNIX password:
New password: 



The UI has space dedicates to big files that the users could need to store there.
The path of such a space is: 

/data/<username>

So, in your case is:

/data/jondoe


Your /home/jondoe folder has a quota of 10GB.

If you have an estimation of the possible space that you will need in /data/ and/or /home/, we will appreciate that information in order to setup your quota according to your work necessities.

In case of any question/suggestion please let me know,

Cheers,
Account Managers
------

#!/bin/bash          

## This is a Variable that is filled with the input #1
firstWord=$1

## Other variable defined inside the script
secondWord="World"

echo "Hello $1 $secondWord"

## End

Q&A

Do you have any question so far?
For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL
( please remove SPAMNOT from the email address )

Review of concepts with examples:

Script concepts using bash

“Hello” World

#!/bin/bash          

## This is a comment: we will print Hello World again :)

echo "Hello World"

Variables

Lets use some variables to get a feeling of the different use. As in previous sections, they are some of the many ways to use variables but we consider the next examples a representative sample for the majority of the scripts that you may will developed:

• Define variables coming from the user input:
  • Strings

#!/bin/bash          

## Variables coming from "outside", e.g. included as parameters in the script line execution are taken by order using hte character "$"

first_input=$1 
second_input=$2

## etc...

echo "I am the first input \" $first_input \" and the second input \" second_input\" "

• • Integers and floats

#!/bin/bash          

## this is a variable taken from outside
energy_in_tev=$1

## but we always work in MeV of GeV, so a transformation is needed:

energy_gev=$(($energy_in_tev * 1000))

energy_in_mev=$[energy_ingev * 1000]

echo ''The energy for the humans is $energy

• Define variables inside the scripts

As in programming languages like C++, we can have global and local variables in our scripts:

• • Global variables

#!/bin/bash          

## 

• • Local variables

#!/bin/bash          

                    HELLO=Hello 
                    function hello {
                            local HELLO=World
                            echo $HELLO
                    }
                    echo $HELLO
                    hello
                    echo $HELLO

• • Using variables in operations: printing, assigning, concatenating, arithmetic operations:

#!/bin/bash          

## 

Loops

The for loop is a little bit different from other programming languages. Basically, it let's you iterate over a series of 'words' within a string.
The while executes a piece of code if the control expression is true, and only stops when it is false (or a explicit break is found within the executed code.
The until loop is almost equal to the while loop, except that the code is executed while the control expression evaluates to false.

For sample

#!/bin/bash          

            for i in $( ls ); do
                echo item: $i
            done
     

On the second line, we declare i to be the variable that will take the different values contained in $( ls ).
The third line could be longer if needed, or there could be more lines before the done (4).
'done' (4) indicates that the code that used the value of $i has finished and $i can take a new value.

• • C-like for

It's a for loop more similar to C/perl... for. (perl?)

#!/bin/bash          

            for i in `seq 1 10`;
            do
                    echo $i
            done    
 

• • While sample

#!/bin/bash  

             COUNTER=0
             while [  $COUNTER -lt 10 ]; do
                 echo The counter is $COUNTER
                 let COUNTER=COUNTER+1 
             done

This script 'emulates' the well known (C, Pascal, perl, etc) 'for' structure

• • Until sample

#!/bin/bash  

             COUNTER=20
             until [  $COUNTER -lt 10 ]; do
                 echo COUNTER $COUNTER
                 let COUNTER-=1
             done
 

• Note: Loop in python We will talk about this later as well...

    1#!/bin/sh
    2languages=`enscript --help-highlight | grep 'Name:' | cut -d ' ' -f 2`
    3for l in $languages; do
    4    cat << EOF
    5   * $l
    6EOF
    7done

Conditionals

- using numerical values
- using strings values
- conditions for break, as before in loops
- conditions in “if”
- if, else if, else

A word about syntax:
The base for the 'if' constructions in bash is this:

if [expression];

then

code if 'expression' is true.

fi

Basic conditional example if .. then

#!/bin/bash     
                if [ "foo" = "foo" ]; then
                   echo expression evaluated as true
                fi
  

The code to be executed if the expression within braces is true can be found after the 'then' word and before 'fi' which indicates the end of the conditionally executed code.

• Basic conditional example if .. then ... else

#!/bin/bash     
                if [ "foo" = "foo" ]; then
                   echo expression evaluated as true
                else
                   echo expression evaluated as false
                fi

• Conditionals with variables

#!/bin/bash          
                T1="foo"
                T2="bar"
                if [ "$T1" = "$T2" ]; then
                    echo expression evaluated as true
                else
                    echo expression evaluated as false
                fi
                

Pipes

- echo, grep (i, v, c), cat, comm, tail -n X, head -n Y, wc (l,…) as inputs for redirection to files, sed, wc, grep, sort, uniq,...

#!/bin/bash          

## This is a comment

echo "Hello World"

Inputs and Outputs

#!/bin/bash          

## This is a comment

echo "Hello World"

Functions (Moved to Day 3)

- sleep and wait
- define a function
- call functions
- remember of local variables
- use of the .bash_profile and .bash_login file for define functions and scripts in user system/login

#!/bin/bash          

## This is a comment

Q&A

Do you have any question so far?
For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL
( please remove SPAMNOT from the email address )

Q&A

Do you have any question so far?
For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL
( please remove SPAMNOT from the email address )

ROOT Reminder

Review of concepts

Example macros for script proposes

#!/bin/bash

root -l file.root << EOF
$1->Draw()
c1->SaveAs("$1.eps")
EOF

The ATLAS Ecosystem and Tools Part I

The User Interfaces (lxplus and atlasui)

The GRID

What is a Job?

What I need?

Code

Samples

PROXY

Test of the Tutorial ATLASUI account! (INTERNAL)

Q&A

Do you have any question so far?
For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL
( please remove SPAMNOT from the email address )

The ATLAS Ecosystem and Tools Part II

PRUN

Script of the examples jobs:

#!/bin/bash          

## Example 1
prun --exec " echo %IN> input.txt; source Sroot.sh input.txt mini lep_pt lep_n\ \>\ 1" --outDS user.arturos.test.tutorial.$name.job01.V$version --inDS user.arturos.MC.outreach.v2 --nFilesPerJob 1 --outputs output.root --rootVer=6.02/12 --cmtConfig=x86_64-slc6-gcc48-opt

## Example 2
prun --exec " echo %IN> input.txt; source ScriptMultipleDraw.sh input.txt mini lep_pt lep_n\ \>\ 0" --outDS user.arturos.test.tutorial.$name.job02.V$version --inDS user.arturos.MC.outreach.v2 --nFilesPerJob 1 --outputs output.root --rootVer=6.02/12 --cmtConfig=x86_64-slc6-gcc48-opt

echo "Done!"
## End

BigPanda

BigPanda Monitor for our jobs

RUCIO ( dq2 ? depreciate but still useful )

## Example 1
rucio list-dids user.arturos.*.outreach*


## Example 2
rucio list-files user.arturos.MC.outreach.v2

AMI

PBOOK

CERNBOX

JIRA

SVN (Vs) GitLab

Latest ATLAS “Style” guidelines

The ATLASUI

What is inside

Properties

Services

Q&A

Do you have any question so far?
For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL
( please remove SPAMNOT from the email address )

The LXPLUS <---> ATLASUI integration

Second set of examples in Python Vs BASH

One to One equivalent examples for generic use

Online resources and references

Q&A

Do you have any question so far?
For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL
( please remove SPAMNOT from the email address )

Local developed Tools

Existing scripts

Websites and Online

Tier2

Practical Exercises with each the mentioned Tools

Initial Examples

HTML fast view

Old HTML

Scripts

LaTeX

Show example Hide example

\documentclass[a4paper]{article}

\usepackage[english]{babel}
\usepackage[utf8x]{inputenc}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage[colorinlistoftodos]{todonotes}

\title{Your Paper}
\author{You}

\begin{document}
\maketitle

\begin{abstract}
Your abstract.
\end{abstract}

\section{Introduction}

Your introduction goes here! Some examples of commonly used commands and features are listed below, to help you get started. If you have a question, please use the help menu (``?'') on the top bar to search for help or ask us a question. 

\section{Some examples to get started}

\subsection{How to add Comments}

Comments can be added to your project by clicking on the comment icon in the toolbar above. % * <john.hammersley@gmail.com> 2014-09-03T09:54:16.211Z:
%
% Here's an example comment!
%
To reply to a comment, simply click the reply button in the lower right corner of the comment, and you can close them when you're done.

\subsection{How to include Figures}

First you have to upload the image file from your computer using the upload link the project menu. Then use the includegraphics command to include it in your document. Use the figure environment and the caption command to add a number and a caption to your figure. See the code for Figure \ref{fig:frog} in this section for an example.

\begin{figure}
\centering
\includegraphics[width=0.3\textwidth]{frog.jpg}
\caption{\label{fig:frog}This frog was uploaded via the project menu.}
\end{figure}

\subsection{How to add Tables}

Use the table and tabular commands for basic tables --- see Table~\ref{tab:widgets}, for example. 

\begin{table}
\centering
\begin{tabular}{l|r}
Item & Quantity \\\hline
Widgets & 42 \\
Gadgets & 13
\end{tabular}
\caption{\label{tab:widgets}An example table.}
\end{table}

\subsection{How to write Mathematics}

\LaTeX{} is great at typesetting mathematics. Let $X_1, X_2, \ldots, X_n$ be a sequence of independent and identically distributed random variables with $\text{E}[X_i] = \mu$ and $\text{Var}[X_i] = \sigma^2 < \infty$, and let
$$S_n = \frac{X_1 + X_2 + \cdots + X_n}{n}
      = \frac{1}{n}\sum_{i}^{n} X_i$$
denote their mean. Then as $n$ approaches infinity, the random variables $\sqrt{n}(S_n - \mu)$ converge in distribution to a normal $\mathcal{N}(0, \sigma^2)$.


\subsection{How to create Sections and Subsections}

Use section and subsections to organize your document. Simply use the section and subsection buttons in the toolbar to create them, and we'll handle all the formatting and numbering automatically.

\subsection{How to add Lists}

You can make lists with automatic numbering \dots

\begin{enumerate}
\item Like this,
\item and like this.
\end{enumerate}
\dots or bullet points \dots
\begin{itemize}
\item Like this,
\item and like this.
\end{itemize}

We hope you find Overleaf useful, and please let us know if you have any feedback using the help menu above.

\end{document}

Intermediate Examples

Advance Scenarios and Scripts in ATLAS analysis

Q&A

Do you have any question so far?
For offline effects please send your questions to arturos@cernNOSPAMPLEASE.ch including in the subject: SCRIPT TUTORIAL
( please remove SPAMNOT from the email address )