Jobs are stucking

Introduction

In open edx all of your jobs are set in a queue by Xqueue. So if you are having any trouble regarding creating or executing any job like generating a certificate etc. It is mainly cause by Xqueue.

The below are example error you could see in your logs due to Xqueue.

Traceback (most recent call last):
File "/edx/app/certs/certificates/certificate_agent.py", line 183, in <module>
main()
File "/edx/app/certs/certificates/certificate_agent.py", line 54, in main
settings.QUEUE_USER, settings.QUEUE_PASS)
File "/edx/app/certs/certificates/openedx_certificates/queue_xqueue.py", line 28, in _init_
self._login()
File "/edx/app/certs/certificates/openedx_certificates/queue_xqueue.py", line 41, in _login
response = json.loads(request.text)
File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

==> /edx/var/log/supervisor/certs-stdout.log <==
2015-12-25 05:11:48,004 CRITICAL 20454 [certificates: _main_] certificate_agent.py:112 - Unable to parse queue submission ('course_name') : {u'xqueue_files': u'{}', u'xqueue_header': u'{"submission_id": 15, "submission_key": "8f20917442402eacf8cae00ca3787e30"}', u'xqueue_body': u'{"username": "9722e1cec00249debadca60eaa8bb983", "template_pdf": "certificate-template-edX-DemoX.pdf", "name": "John Do
u00eb", "course_id": "course-v1:edX+DemoX+Demo_Course", "example_certificate": true, "action": "create"}'}

So hard coded course_name in /lms/djangoapps/certificates/queue.py:423 to 'Example Course Name' and the example certificate is generating successfully now (logs below) but still the view is not updating

==> /edx/var/log/supervisor/certs-stdout.log <==
2015-12-25 06:25:45,308 INFO 20454 [certificates: _main_] certificate_agent.py:125 - Generating certificate for 113a385c02844a8cb9328d8482dba62c (John Doë), in course-v1:edX+DemoX+Demo_Course, with grade None
2015-12-25 06:25:46,015 INFO 20454 [certificates.gen_cert] gen_cert.py:348 - published /tmp/certificates/tsdIIj/downloads/3fc16c5ca5d344589f5647a7db365e40/Certificate.pdf to /edx/var/certs/www-data/downloads/3fc16c5ca5d344589f5647a7db365e40/Certificate.pdf
2015-12-25 06:25:46,016 INFO 20454 [certificates.gen_cert] gen_cert.py:348 - published /tmp/certificates/tsdIIj/cert/5bd92b7d0f14438e8fe1782f14429c05/valid.html to /edx/var/certs/www-data/cert/5bd92b7d0f14438e8fe1782f14429c05/valid.html
2015-12-25 06:25:46,017 INFO 20454 [certificates.gen_cert] gen_cert.py:348 - published /tmp/certificates/tsdIIj/cert/5bd92b7d0f14438e8fe1782f14429c05/Certificate.pdf.sig to /edx/var/certs/www-data/cert/5bd92b7d0f14438e8fe1782f14429c05/Certificate.pdf.sig
2015-12-25 06:25:46,018 INFO 20454 [certificates.gen_cert] gen_cert.py:348 - published /tmp/certificates/tsdIIj/cert/5bd92b7d0f14438e8fe1782f14429c05/verify.html to /edx/var/certs/www-data/cert/5bd92b7d0f14438e8fe1782f14429c05/verify.html
2015-12-25 06:25:46,019 INFO 20454 [certificates: _main_] certificate_agent.py:191 - Posting result to the LMS: {'xqueue_header': '{"submission_id": 21, "submission_key": "f667fbd6c5a1533baf809cd84ea9d71b"}', 'xqueue_body': '{"username": "113a385c02844a8cb9328d8482dba62c", "url": "http://54.183.110.237:18010/downloads/3fc16c5ca5d344589f5647a7db365e40/Certificate.pdf", "verify_uuid": "5bd92b7d0f14438e8fe1782f14429c05", "download_uuid": "3fc16c5ca5d344589f5647a7db365e40", "action": "create", "course_id": "course-v1:edX+DemoX+Demo_Course"}'}

Solution

The simple solution if your jobs are stucking is just to start the Xqueue. Xqueue can be restarted just by running the below command.

sudo /edx/bin/supervisorctl restart xqueue