WAOL Blog

WAOL ANGEL Performance Update

January 19, 2010 at 11:34 am Leave a comment

Audience: WAOL ANGEL Administrators, Faculty and Staff

As many of you know, WashingtonOnline (WAOL) ANGEL has experienced intermittent performance slowdowns since January 7. This note is an update of current system status, next steps, and a request to complete a very short survey.

What’s New:
Current Status (January 15-17):

• 7.4 patch was successfully applied on Friday, Jan 15
• It has been reported that WAOL ANGEL is running normally from Longview, Olympia and Spokane since the 7.4 patch was applied.
•  There have been no emails re: WAOL ANGEL performance sent system list serves or to SBCTC staff since Friday, Jan 15.
• The last update to the priority ANGEL ticket (# 596-8312283) was on Friday, Jan 15  (The ticket was likely entered after an ANGEL DBA brought down system for 15 min. while testing performance    problem. ANGEL caught it immediately, shut down the test, and the system returned to normal).
• While we hope the 7.4 patch fixed the performance problem, SBCTC and ANGEL are diligently moving forward to diagnose the cause of the problem.

SBCTC / ANGEL Meeting

Most recently SBCTC technical and program staff and ANGEL teams had a productive meeting on Friday, January 15 to discuss performance and recent changes to the WAOL ANGEL hosting environment. We’ve also been in contact with Angel senior technical staff over the weekend.

Outcomes from the meeting included:

• a thorough review of the performance timeline
• a work plan detailing next steps (see “next steps” below)
• leads for SBCTC and ANGEL:
  o Frank Leeds (Service Management Director) is SBCTC’s team lead and will oversee and coordinate all technical aspects of diagnosing and testing performance.
  o Cable Green (eLearning Director) is SBCTC’s lead on system communication.
  o Jason O’Brien (Director, ANGEL Managed Hosting) is lead for ANGEL / Blackboard.

Next Steps

1. URGENT: College ANGEL Domain Administrators: please complete this short survey ASAP.
      a. https://www.surveymonkey.com/s/ZFPY66P
      b. We will use your responses in our next SBCTC / ANGEL meeting (Jan 20th) … and use the data to update the  priority ANGEL ticket.

2. SBCTC collects trouble tickets from Presidium, ANGEL and from survey (see above).  Review knowledge base info on proxy from ANGEL.  ANGEL provides details on which (if any) of its ASP clients are having performance issues with 7.4 and what solutions have been implemented.

3. Determine if patterns arise with respect to sites and performance types by college and activity in ANGEL.

4. Send out a technology survey to college CIOs to determine what network infrastructure is in place at their campus related to proxy, web application firewalls, and traffic management.

5. Collect and collate survey data – look for patterns, trends that help direct troubleshooting.

6. SBCTC sets up a stress test environment for ANGEL to test with current TCP persistence environment at ANGEL.

7. SBCTC runs stress test against ANGEL in TCP persistence mode.

8. ANGEL sets up a separate instance of IP persistence load balancing for WAOL ANGEL environment.

9. SBCTC changes hostname so that we point to IP persistence based front end for our ANGEL instance.

10. SBCTC runs stress test against ANGEL in IP persistence mode.

11. Review and compare the data from here with (9) to see if obvious benefits are gained from IP persistence or if side effects are too great.

12. If 9 and 11 are inconclusive then we will have to regroup – likely go towards creating an independent path from SBCTC Bellevue to ANGEL without proxy, removing IP persistence, and rerunning stress tests.

13. Document findings: test results, conclusions, standard operating procedure for ANGEL and Washington CTC system integration.
Frequent updates will continue to be posted both to the ELC and ITC lists and to the WAOL blog.

College domain administrators: a reminder to please collect and report (through the ANGEL ticketing system – ticket # 596-8312283) detailed information on any performance problems at your college. Detailed information about your users’ experiences is critical information in solving this issue.

Thank you for your patience and cooperation.

Sincerely,

Cable
Cable Green, PhD
eLearning Director
SBCTC

Entry filed under: Angel, Announcements, ELC, Faculty, Staff. Tags: .