koa-latest

Migrate EdX (Single-Server)

In this tutorial you'll be migrating from an existing (Source) Open EdX instance to a new (Target) instance. You'll re-install Open EdX on the new machine, then match configurations, then migrate the databases.

REQUIREMENTS FOR A NEW (TARGET) MACHINE:

  • Ubuntu 16.04
  • minimum 8 GB RAM
  • est. 100GB disk space
  • Open Ports:
    • 80 (http)
    • 443 (https)
    • 18010 (studio)
    • 22 (ssh)

Phase 1: Install Open EdX on your Target

  1. Confirm your new machine meets the requirements above.

  2. SSH into the new machine and Update Ubuntu Server.

sudo apt-get update -y &&
sudo apt-get upgrade -y &&
sudo reboot
  1. Set the OPENEDX_RELEASE variable. You can select this variable based on your current Open edx instance.
export OPENEDX_RELEASE=the-tag/you-want-to-install.

OpenEdx Releases: https://edx.readthedocs.io/projects/edx-developer-docs/en/latest/named_releases.html For example:

export OPENEDX_RELEASE=open-release/ironwood.master  (git tag)
  1. Create a file in the current directory named config.yml and set required variables for LMS and CMS.

If you have a domain/subdomain for LMS and CMS, then you can define it instead of using IP's. If you are planning to use domain/subdomain, then please make sure that you have created 'A Record' (a type of DNS record) which will point you domain/subdomain on your newly created server's IP. Also keep in mind that studio/CMS must be a subdomain of the LMS.

Create your file:

nano config.yml

Enter the following. Replace the IP Addresses with your addresses for LMS and CMS

# The host names of LMS and Studio. Don't include the "https://" part:
EDXAPP_LMS_BASE: "54.183.226.25"
EDXAPP_CMS_BASE: "54.183.226.25:18010"
  1. Bootstrap Ansible Script.
wget https://raw.githubusercontent.com/edx/configuration/$OPENEDX_RELEASE/util/install/ansible-bootstrap.sh -O - | sudo bash
  1. Generate Random Password.
wget https://raw.githubusercontent.com/edx/configuration/$OPENEDX_RELEASE/util/install/generate-passwords.sh -O - | bash

After running the command, two files (i.e my-password.yml & password-template.yml) will be created.

  1. Create an install.sh script
nano install.sh

Add the following to your install script

wget https://raw.githubusercontent.com/edx/configuration/$OPENEDX_RELEASE/util/install/native.sh -O - | bash

Make script executable

chmod +x install.sh

Run the script with nohup

nohup ./install.sh &

Check the logs using nohup

tail -f nohup.out

Phase 2: Move Configurations

  1. Connect/SSH into Source Edx and Target Edx into separate terminals. Copy the important keys of lms.env.json & cms.env.json from source edx to target edx ( switch to edxapp user).

To edit the lms.env.json and cms.env.json files

sudo vi /edx/app/edxapp/lms.env.json
sudo vi /edx/app/edxapp/cms.env.json

Modify the following variables in lms.env.json and cms.env.json to match your source machine.

      PLATFORM_NAME

      CMS_BASE

      ALL RELEVANT EMAILS (BUGS_EMAIL, BULK_EMAIL_DEFAULT_FROM_EMAIL, SERVER_EMAIL, CONTACT_EMAIL, DEFAULT_FEEDBACK_EMAIL, DEFAULT_FROM_EMAIL)

      EMAIL_HOST, EMAIL_PORT, EMAIL_USE_TLS, & EMAIL_BACKEND

      LANGUAGE_CODE

      LOGIN_REDIRECT_WHITELIST

      everything from FEATURES

      SESSION_COOKIE_DOMAIN

      BASE_COOKIE_DOMAIN

      SITE_NAME

      ENABLE_COMPREHENSIVE_THEMING

      COMPREHENSIVE_THEME_DIRS

      TIME_ZONE

Note: The env.json files contain the majority of configurations for your Open EdX instance. The list above is some common configuration items, but every installation will be different. It is advisable to run a comparison between the source and the target machines and update everything that needs it.

Make sure to set the same value of SESSION_COOKIE_DOMAIN and BASE_COOKIE_DOMAIN in both lms.env.json and cms.env.json. To check everything is fine, login/register on your new machine/target Edx.

If you have a theming customization then you need to set/copy ENABLE_COMPREHENSIVE_THEMING and COMPREHENSIVE_THEME_DIRS in both lms.env.json and cms.env.json.

Phase 3: Migrate Databases

  1. Find and save the database credentials you'll use for migration.

For migrating databases, you need to know database user and password keys defined in my-password.yml file for both target Edx and source Edx. Using these keys, you will be able to issue migration/dump command in the following steps. For this, go to your my-password.yml file, this could be of location: '/home/ubuntu/my-password.yml' and extract following information and retrieve following important keys (switch to ubuntu user):

Target Edx:

      MONGO_ADMIN_PASSWORD = 'C7bOql12RjuR5bsk0pnW4ua25qC3O8yHpZQ'

      EDXAPP_MONGO_PASSWORD = 'DATAYo6MYwhFhWQtIi2hyZh0wHPiA1WZDIy'

      EDXAPP_MONGO_USER = 'edxapp'

      FORUM_MONGO_USER = "cs_comments_service"

      FORUM_MONGO_PASSWORD = 'VsIKkOxkBDxid4WfKvojoRl6vZfFke7LMxe'

Source Edx:

      MONGO_ADMIN_PASSWORD = '77mNEsys9KztHSIA5qEdfBIeF67HHXr8rNQ'

      EDXAPP_MONGO_PASSWORD = 'gRhR9psJvrQInTsFPxKvOFYWBS8CwOk7wTj'

      EDXAPP_MONGO_USER = 'edxapp'

      FORUM_MONGO_USER = "cs_comments_service"

      FORUM_MONGO_PASSWORD = 'DZzLtmBn5Wg4DUXPOX5YuQ9dCXFHkQIKPqG'

  1. Backup your MySQL database on the target machine

Now we'll Back up the databases on your Target Machine so that if anything goes wrong you can restore it (switch to ubuntu user). This task is optional.

Dump the SQL tables and structure into a file called mysql-fresh.sql

sudo mysqldump --add-drop-database --no-data --databases analytics-api dashboard discovery ecommerce edxapp edxapp_csmh reports xqueue > mysql-fresh.sql

Append the content of those SQL databases into the same file

sudo mysqldump --no-create-info --databases analytics-api dashboard discovery ecommerce edxapp edxapp_csmh reports xqueue >> mysql-fresh.sql
  1. Backup your MongoDB database on your target machine

Stop the MongoDB server first, so that the data would not get updated by the online users accessing the Target Edx.

sudo service mongod stop

Dump the edxapp mongo collection into a folder

mongodump --username [EDXAPP_MONGO_USER] --password [EDXAPP_MONGO_PASSWORD] --db edxapp --out mongo-fresh-edxapp

Dump the cs_comments_service mongo collection into a folder

mongodump --username [FORUM_MONGO_USER] --password [FORUM_MONGO_PASSWORD] --db cs_comments_service --out mongo-fresh-cs_comments_service

Dump the admin mongo collection into a folder

mongodump --username admin --password [MONGO_ADMIN_PASSWORD] --db admin --out mongo-fresh-admin

Now restart the MongoDB server.

sudo service mongod start
  1. Back up the MySQL databases on your Source Machine so that we can move the dump of the database from Source Edx to Target Edx (switch to ubuntu user).

Dump the SQL tables and structure into a file called mysql-old.sql

sudo mysqldump --add-drop-database --no-data --databases analytics-api dashboard discovery ecommerce edxapp edxapp_csmh reports xqueue > mysql-old.sql

Append the content of those SQL databases into the same file

sudo mysqldump --no-create-info --databases analytics-api dashboard discovery ecommerce edxapp edxapp_csmh reports xqueue >> mysql-old.sql
  1. Backup the MongoDB database on your Source Machine so that we can move it to the Target machine.

Stop the MongoDB server first, so that the data would not get updated by the online users accessing the Source Edx.

sudo service mongod stop

Dump the edxapp mongo collection into a folder

mongodump -u admin -p [MONGO_ADMIN_PASSWORD] --host localhost --authenticationDatabase admin --db edxapp --out mongo-old-edxapp

Dump the cs_comments_service mongo collection into a folder

mongodump -u admin -p [MONGO_ADMIN_PASSWORD] --host localhost --authenticationDatabase admin --db cs_comments_service --out mongo-old-cs_comments_service

Note: the slightly modified approach here. We are using the mongo admin username & password to authenticate, as opposed to the db-specific user. I believe both will work. If you are having trouble with one, you can try the other.

Now restart the MongoDB server.

sudo service mongod start
  1. Move the database files from the source Edx to the target Edx (switch to ubuntu user).

The database file size may vary depending on the number of users and courses on source Edx. Mongo dumps are generated as a collection of folders and files. Zip them up into individual .zip files.

cd /home/ubuntu

zip -r mongo-old-edxapp.zip mongo-old-edxapp

zip -r mongo-old-cs_comments_service.zip mongo-old-cs_comments_service

Use scp to transfer your files from source Edx to Target Edx. The commands below will move the appropriate files onto the target instance in the /home/ubuntu directory. Here two things to know i.e IP address of the target Edx and .pem file of target Edx for scp.

Take note of the following items:

      Target EdX Address (IP or URL)

      Source EdX Address (IP or URL)

      Private key file (e.g. key.pem) for the target machine. Move this key file onto your source machine.

Copy data files from Source Edx to Target Edx:

scp -i key.pem mysql-old.sql ubuntu@[TARGET ADDRESS]:/home/ubuntu

scp -i key.pem mongo-old-edxapp.zip ubuntu@[TARGET ADDRESS]:/home/ubuntu

scp -i key.pem mongo-old-cs_comments_service.zip ubuntu@[TARGET ADDRESS]:/home/ubuntu
  1. Unzip the files on the target server (switch to ubuntu user).
cd /home/ubuntu

unzip mongo-old-cs_comments_service.zip

unzip mongo-old-edxapp.zip
  1. Import the databases (i.e MySQL & MongoDB) on the target machine (switch to ubuntu user).
cd /home/ubuntu
sudo mysql < mysql-old.sql

Make sure your folders reside in /home/ubuntu for these commands to work as written:

mongorestore --username [EDXAPP_MONGO_USER] --password [EDXAPP_MONGO_PASSWORD] --drop --db edxapp /home/ubuntu/mongo-old-edxapp/edxapp

mongorestore --username [FORUM_MONGO_USER] --password [FORUM_MONGO_PASSWORD] --drop --db cs_comments_service /home/ubuntu/mongo-old-cs_comments_service/cs_comments_service
  1. The last step is to run database migrations (switch to ubuntu user).
sudo -u edxapp -E /edx/bin/python.edxapp /edx/bin/manage.edxapp lms migrate --settings=production --fake-initial

sudo -u edxapp -E /edx/bin/python.edxapp /edx/bin/manage.edxapp cms migrate --settings=production --fake-initial

  1. Now open your target Edx server IP/domain name on your browser.

    Homepage

← Set or Change a PasswordMigrate the Theme →