Scaling Mongo

Why move Mongo?

MongoDB contains:

• All course content, including text, settings, and static assets.
• All discussion data

When we scale up Open EdX and add multiple application servers, we must migrate Mongo to a shared location so that all application servers reference the same Mongo content.

Choosing a Mongo Provider

AWS and Azure do not have a hosted Mongo service, that we are aware of. Therefore, our most common provider for a managed mongo service is MongoDBAtlas. MongoDBAtlas is a managed service that provided self-healing clusters, automated backups, on-demand scaling, etc for AWS, Azure, and Google Cloud.

Moving MongoDB to remote

Note: Detailed Mongo Migration steps for AWS in 'Migrate Mongo' section.

1. To move MongoDB to a remote instance, you will need a Linux machine preferably Ubuntu 16.04 LTS. The Linux machine can be taken from AWS EC2 or Azure VM.

2. Before installing MongoDB, open inbound port 27017 and whitelist the Open Edx instance IP on Linux machine Network Security Group. The MongoDB runs on 27017 port and this port should be left open, so that an Open Edx instance can communicate in a secure environment.

3. Once the Linux machine has been configured, you should have SSH access to the machine in the form of a .pem key along with the username. If you are not able to connect via SSH, then look for the inbound port and add 22 port in it. The SSH uses port 22 for secure communication and it should only be filled with the IP of the servers that you want to give access to MongoDB.

4. Now check the local MongoDB version using command mongo --version (for eg: 3.2.16) and connect the Linux machine to install the same version on it.

5. Once connected to the Linux machine, run the following commands to install MongoDB.

   wget -qO - https://www.mongodb.org/static/pgp/server-3.2.asc | sudo apt-key add -
   
   echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list
   
   sudo apt-get update
   
   sudo apt-get install -y mongodb-org
   

   This will take some time and after the process you can check your MongoDB status. It will show you status as active (running).

   sudo service mongod status

6. After successful installation of MongoDB on a Linux machine, it needs to run publicly. For this, change the bindIp from 127.0.0.1 to 0.0.0.0 in the file /etc/mongod.conf. Otherwise, Open Edx would not be able to communicate to MongoDB.

7. Create a dump of the local MongoDB which includes edxapp, cs_comments_service, and admin databases. The edxapp and admin are conventional database for storing user related information. While cs_comments_service is responsible to store voting and nested comments. You can use mongo dump command (i.e mongodump) and use relevant users for creating a dump (i.e edxapp for edxapp, cs_comments_service for cs_comments_service and admin for admin). The password will be found in my-password.yml (i.e EDXAPP_MONGO_PASSWORD, FORUM_MONGO_PASSWORD and MONGO_ADMIN_PASSWORD)

8. Now, move the dump using scp to the Linux machine in which MongoDB is installed and restore the dump using mongorestore.

9. After that, the final step is to point the Open Edx instance to the MongoDb instance. For this you need to change the configuration of LMS, CMS and Forum.

10. For LMS and CMS, open lms.auth.json and cms.auth.json and search for 27017 and replace all the occurences for "host": "localhost" to "host": "Your-Mongo-Server-IP". For instance:

    "DOC_STORE_CONFIG": {
        "collection": "modulestore", 
        "connectTimeoutMS": 2000, 
        "db": "edxapp", 
        "host": "{{Your-Mongo-Server-IP}}", 
        "password": "gRhR9psJvrQInTsFPxKvOFYWBS8CwOk7wTj", 
        "port": 27017, 
        "read_preference": "SECONDARY_PREFERRED", 
        "replicaSet": "", 
        "socketTimeoutMS": 3000, 
        "ssl": false, 
        "user": "edxapp"
    }
    

11. For forum, activate forum user and virtual environment. Now, open cs_comments_service/config/mongoid.yml.

    Replace this:

    development:
    clients:
        default:
        <<: *default_uri
        <<: *default_client
    

    With:

    development:
    clients:
        default:
        <<: *default_client
        database: cs_comments_service
        hosts:
            - 172.31.45.195:27017