Students Staff

March 14, 2019

[Service Alert] [Resolved] Issues affecting FASER’s performance

Filed under: [Resolved], [Service Alert] — Tags: — Benjamin Steeples @ 1:41 pm

Update 26 April:

We now believe that the majority of FASER’s performance and online annotation issues are resolved. Further improvements are planned and will be implemented over the summer.

Update 18 April:

We have taken several steps to reduce the load placed on the Annotaion service, which has improved its performance and reliability. We will continue to monitor things.

Update 03 April:

We’ve released version 2.2.10 of FASER which aims to improve the stability of FASER and reduce its impact on the Annotation service. FASER may be somewhat slower to begin with while it recalculates some common items, but users should notice a more stable online marking experience.

Additional changes are in development, and we are continuing to monitor the situation

Furthermore to the updates, we will be doing the following:

  1. Tuning: This week we will tune the database and underlying server and hardware, this should show improvements and is reversible and non-destructive.
  2. Preloading: After this we will be using the list of expected feedback required and “pre”-preparing these so they are ready for use. Others will queue.
  3. Database shrinking: Next week we will be removing records from the system so that any size issues are removed, this is non reversible and must be tested.
  4. Supplier bug report: We are in contact with the supplier to ask them to address the overly greedy software. Note that we cannot roll back to a previous version after upgrading this software.

Update 29 March:

The service is currently degraded as we are seeing high usage with the annotation component in FASER, which is causing latency with the service. Students should be unaffected, but staff may encounter problems when attempting to annotate submissions.

We have restarted the annotation component, which should reduce the latency and return to normal service.

 

Update 28 March:

We have identified some additional un-used features of the annotation service that are still active, and are having an adverse effect on service performance. These features have been disabled and users should notice an improvement during peak usage times.

Additional update 15:00:

To try to reduce the load on the annotation service, we have changed how submissions are uploaded. After being checked files are no longer automatically uploaded to the annotation service unless submissions are being marked online for an assignment. This means that when you start the first few files will be slightly slower to load while FASER uploads them, but the rest will be uploaded in the background ready.

 

Update 27 March:

We have identified an issue with the annotation service that may affect performance under heavy loads. A patch has been applied, which should improve page load times.

We will continue to monitor the situation

 

Update 22 March:

We are seeing a significant number of deadlines and submissions today, which is having an impact on FASER and its annotation component. Students should be largely unaffected by this, but staff may encounter problems when attempting to annotate submissions.

 

Update 19 March:

This morning, FASER’s annotate component was migrated to faster hardware (SSD storage). We are still monitoring the impact of this, but initial tests are favourable.

 

Original message:

We apologise for the recent performance issues with FASER. Over the last few days, we have been monitoring the service and slowly narrowing down the root cause of the problems that people have been encountering. We are still working on this, but one of the issues identified relates to an update to a third-party component that we rely on to provide online annotation and document checking.

We have implemented various emergency fixes to lessen the impact of these performance problems, and are working to identify other root causes and develop fixes. This is being treated as a priority.

We will continue to keep this blog post updated with developments every few days, but we thank colleagues and students for their continued patience whilst we fully resolve the matter.

Leave a Reply

The University of Essex will moderate comments and there will be a delay before any posts appear.