SIR application not displaying correctly (missing static resources)

Description

The static content of the SIR application was not displayed correcly. This problem was intermittent. End users complained they cannot see icons and some HTML components were not displayed correcly due to missing CSS

Impact

• As an impact, end users had to refresh the page until the static resources are displayed.

Time line of the incident

• 25-Mar-13 19:00 - Artur Wiecek switched front-end from iAS to OHS
• 28-Mar-13 - Problem was reported by end users, AIS reported the problem to us
• 02-Apr-13 11:42 - Giovanni Chierico created an JIRA
• 03-Apr-13 12:00 - root cause of the problem was identified
• 03-Apr-13 18:00 - Web configuration updated and problem fixed.

Analysis

• The problem was due to a wrong Web configuration which should have been removed when migrating from Apache/iAS to Apache/OHS.
• The problem was intermittent because the Loab Balanced DNS was pointing to both OLD and new Apache servers. When the user was redirected to the old one, the static content was served. Otherwise it was not.

Follow up

• This issue could have been avoided by a better testing after the same change was made on DEV systems.
• The huge amount of complains from end users also could have been avoided if a better testing after production change was made (no problem reported for 2 days). We did schedule this change 4 days before easter long week end, but it was reported 1 day before the week end, when lot of people were already absent.

-- MarcinBlaszczyk - 20-Sep-2010