Problems with RabbitMQ

Introduction

RabbitMQ is highly stable subsystem that generally works reliably without any administrative oversight whatsoever. If encountered problems with either subsystem it is almost always following a software upgrade, a database restore, or a server migration. Furthermore, the culprit is almost always Celery.

Diagnosing

The below are some symptom you could face in your edx environment which are regarding RabbitMQ.

• The new user registration screen appears to die, and becomes unresponsive after clicking the signup command button. The new user data is never saved into the system and the new user never receives an activation email.

• The screen appears to freeze or die after you click the password reset button.New users do not receive their new user activation email.

• The drag & drop function appears to work, however the changed value is not recognized. Additionally you cannot save results.

• The screen appears to die after submitting a response to an exercise or quiz problem.

• The screen appears to die, and the document is never uploaded. The system provides neither a success nor a failure message.

If you’re experiencing any of these symptoms then you’ll next want to review the Open edX application logs for both the LMS and CMS to look for errors.

cat /edx/var/log/lms/edx.log -n 50
cat /edx/var/log/cms/edx.log -n 50

Change user permissions

If the source of your Celery problem is permissions then this will eliminate the problem.

sudo rabbitmqctl set_permissions -p / celery ".*" ".*" ".*"
sudo rabbitmqctl set_permissions -p / admin ".*" ".*" ".*"
sudo rabbitmqctl set_permissions -p / edx ".*" ".*" ".*"

After completing the above step restart Rabbitmq

sudo service rabbitmq-server restart

Reset the celery password

In order to reset the password for rabbitmq user we have to find the default password first which is stated in my-passwords.yml file (located at /home/ubuntu). Once you successfully find the default password for all the user use the below command to reset the password.

sudo rabbitmqctl change_password celery YourPasswordForTheCeleryUser
sudo rabbitmqctl change_password edx YourPasswordForTheEdxUser
sudo rabbitmqctl change_password admin YourPasswordForTheAdminUser

After completing the above step restart Rabbitmq

sudo service rabbitmq-server restart

Delete and recreate user

In order to delete RabbitMq user including User, EDX or Celery use the below command.

sudo rabbitmqctl delete_user celery 
sudo rabbitmqctl delete_user admin 
sudo rabbitmqctl delete_user edx 

After deleting the user recreate the user by using following commands (also require default password)

sudo rabbitmqctl add_user celery YourPasswordForTheCeleryUser 
sudo rabbitmqctl add_user admin YourPasswordForTheAdminUser 
sudo rabbitmqctl add_user edx YourPasswordForTheEdxUser

Once you have successfully created the user now change the permission.

After changing the permission change the user tag

sudo rabbitmqctl set_user_tags celery administrator
sudo rabbitmqctl set_user_tags admin administrator
sudo rabbitmqctl set_user_tags edx administrator

Test User

To test if all the users are working in rabbitmq use the below commands.

sudo rabbitmqctl authenticate_user celery YourPasswordForTheCeleryUser
sudo rabbitmqctl authenticate_user admin YourPasswordForTheAdminUser
sudo rabbitmqctl authenticate_user edx YourPasswordForTheEdxUser

Restart the platform

After completing we need to restart the open edx platform including all services.

sudo /edx/bin/supervisorctl restart all