Over the last week, we’ve encountered some problems in our monitor and processor caused by a large number of pending jobs:
- main monitor thread takes > 1 hr to complete a full scan of pending jobs on disk
- priority job processing depends on this thread
In order to fix these delays, we have made priority job monitoring a separate thread from the main queue thread, but we are working to reduce the delay back to 1-2 minutes (which is what we’re used to!).
Currently, we are blocked by issues with this new method related to filesystem scanning. This is blocking the archiving and data re-import mentioned in our last set up updates. See the related code.
We are working to fix this asap, and will provide updates this evening.