-2023-

Setup a Django production server on Ubuntu

using AWS Lightsail, Gunicorn, Nginx, and Git

This is meant to be a one-stop guide for myself if I ever need to restore my Django apps from backup. There is a lot of information about how to do this on the web, but I couldn't quite find a single tutorial that included all parts of my pipeline, and there are some quirks to how things act together that justified creating this custom walkthrough.

I use git to deploy the Django code, use the requirements.txt file to keep track of python dependencies, and keep separate .env files for development and production. I also use PostgreSQL for both my development and production databases.

If you are following this guide to build your production server for the first time, where I indicate to restore the database from a dump file instead create your database and user. If you aren't already using a .env file in your development environment you should set one up first. My .env contains these variables:

Secrets like SECRET_KEY and access tokens
value of DEBUG
database connection info
ALLOWED_HOSTS

My local repository looks like this (abbreviated):

<appname>\
    .git\
    mysite\
        .env
        requirements.txt
        manage.py
        app1\
        app2\
        static\
        media\
        mysite\
            settings.py
            wsgi.py
    venv\

Prerequisites

complete (or at least functioning) Django app
Django already set up with PostgreSQL instead of default MySQL
Django reading settings from .env file with django-environ
git repository
requirements.txt
Amazon Web Services account or another VPS provider like DigitalOcean
An SSH key pair set up with your provider

You don't need a domain. If you don't have one, substitute your VPS's public IP where you see one and use a self-signed certificate instead of Certbot. Something other than django-environ for setting production environment variables would also work. If you don't want to be able to push commits to the server with git you can omit those steps and instead transfer your code to the server with scp (or rewrite it in nano, I guess).

Start

Fire up an AWS Lightsail instance of Ubuntu Server or the cheapest VPS you can find. If you already have your default SSH key set up on amazon, as soon as it's running, all you have to do to log in as default user 'ubuntu' is

ubuntu@<ip_of_instance>

If you are prompted that your ssh config has been locally modified, choose to keep the local version.

sudo apt update
sudo apt upgrade

Do not upgrade to the newest release of Ubuntu.

Install OS level packages we will need

sudo apt install python-dev
sudo apt install supervisor

sudo apt install nginx
sudo apt install python3-certbot-nginx

sudo apt install postgresql

sudo apt install libpq-dev python-dev
sudo apt install python3-pip
sudo pip3 install virtualenv

PostgreSQL

PostgreSQL will have created a user 'postgres' during installation. Switch to it and either set up a new user and database for Django or restore the entire database cluster (which includes the user table) from a dump file.

# Switch to user 'postgres'
sudo -u postgres -i
# Enter PostgreSQL console
psql

# In PostgreSQL console either
# Create a database  and user for your app
CREATE DATABASE <appname>;
CREATE USER <db_user> WITH PASSWORD '<password>';
ALTER ROLE <db_user> SET client_encoding TO 'utf8';
ALTER ROLE <db_user>SET default_transaction_isolation TO 'read committed';
ALTER ROLE <db_user>SET timezone TO 'UTC';
# Or
# Restore from dump file
psql -f <dump_file> template1

Users, Directories, Permissions

Create a user for the app. This user will own the project directory and execute Gunicorn and Django code. Next to it will be a bare git repository owned by 'ubuntu', who will also need write access to the project directory, so create a group 'users' and add user 'ubuntu' to it. I place these under /srv/ and set the app user's home directory there to avoid permission issues that arise when a user owns a directory but not the entire path to that directory.

# Create user app will run as, set up their home directory and set bash as their shell
sudo groupadd --system <appgroup>
sudo useradd --system --gid <appgroup> --shell /bin/bash --home /srv/<appuser> <appuser>

# Create group 'users'
sudo groupadd --system users
# Add ubuntu to group 'users'
sudo usermod -a -G users ubuntu
# Reboot to update group membership
sudo reboot

# Create project directory
sudo mkdir /srv/<appuser>/
# Assign ownership of it to app user 
sudo chown <appuser> /srv/<appuser/
# Add write permission to group users
sudo chmod -R g+w /srv/<appuser>

# Initialize a bare git repository and then create a new hook and edit it
sudo git init --bare /srv/<appuser>.git
sudo nano /srv/<appuser>.git/hooks/post-receive

In the new file:

# /srv/<appuser>.git/hooks/post-receive

git --work-tree=/srv/<appuser> --git-dir=/srv/<appuser>.git checkout -f

then, back in the shell

# Allow post-receive hook to be executed
sudo chmod +x /srv/tfg-site.git/hooks/post-receive

# Change ownership of git repo
sudo chown -R ubuntu /srv/<appuser>.git

# Allow group 'users' to write to working directory so post-receive hook can work
sudo chown -R <appname>:<groupname> /srv/<appuser>
sudo chmod -R g+w /srv/<appuser>

then, back on your local machine

# Add the repository you created on the server as a remote for your local repo
git remote add prod 'ubuntu@<your_server_ip>:/srv/<appname>.git'

Set up virtual environment

# Switch to your app's user then create and activate a virtual environment
sudo su <appuser>
cd /srv/<app>/
virtualenv .
. bin/activate

# Perform as app user with virtual environment activated
pip install -r ~/mysite/requirements.txt
python ~/mysite/manage.py collectstatic


# If you are setting up a project for the first time, also finish setting up Django
python ~mysite/manage.py createsuperuser

Gunicorn

As default user ('ubuntu'), create a bash file and make it executable

# Create a batch file which will start gunicorn
sudo nano /srv/<appname>/bin/gunicorn_start

in that file:

#!/bin/bash

NAME="<appname>"
DJANGODIR=/srv/<appname>/mysite
SOCKFILE=/srv/<appname>/run/gunicorn.sock 
USER=<appname>
GROUP=<groupname>
NUM_WORKERS=3
DJANGO_SETTINGS_MODULE=mysite.settings
DJANGO_WSGI_MODULE=mysite.wsgi

echo "Starting $NAME as `whoami`"

# Activate the virtual environment
cd $DJANGODIR
source ../bin/activate
export DJANGO_SETTINGS_MODULE=$DJANGO_SETTINGS_MODULE
export PYTHONPATH=$DJANGODIR:$PYTHONPATH

# Create the run directory if it doesn't exist
RUNDIR=$(dirname $SOCKFILE)
test -d $RUNDIR || mkdir -p $RUNDIR

# Start Gunicorn
exec ../bin/gunicorn ${DJANGO_WSGI_MODULE}:application \
  --name $NAME \
  --workers $NUM_WORKERS \
  --user=$USER --group=$GROUP \
  --bind=unix:$SOCKFILE \
  --log-level=debug \
  --log-file=-

then

# Make the bash script you just made executable
sudo chmod +x /srv/bin/gunicorn_start

The socket file gunicorn.sock is created automatically.

Supervisor

Create supervisor .conf file to execute gunicorn_start, i.e. :

sudo nano /etc/supervisor/conf.d/<appname>

then, in that file

# /etc/supervisor/conf.d/<appname>.conf
[program:<appname>]
command = /srv/<appname>/bin/gunicorn_start
user = <appname>
stdout_logfile = /srv/<appname>/logs/gunicorn_supervisor.log
redirect_stderr = true
environment=LANG=en_US.UTF-8,LC_ALL=en_US.UTF-8

Create the logs we just told Supervisor to use and then restart to load config. User must own the log files and directory, so switch to that user or change ownership after creation.

# Switch to app user
sudo su <appname>
# Create directory and log file
mkdir /srv/<appname>/logs
touch /srv/<appname>/logs/gunicorn_supervisor.log
# Load the config
sudo supervisorctl reread
sudo supervisorctl update

Nginx

Nginx reads a system-wide config file from /etc/nginx/nginx.conf and then looks for server/app-specific config files in /etc/nginx/sites-enabled . The workflow is to create a config for your app server in sites-available and then create a syslink to it in sites-enabled, allowing you to deactivate the server by deleting the link and retaining the config file.

# Create a config file for your app
sudo nano /etc/nginx/sites-available/<appname>.conf

then, in that file:

# /etc/nginx/sites-available/<appname>.conf

server {
    server_name <yourdomain>.com www.<your_domain>.com;
    root /srv/<appname>/mysite;
    listen 443 ssl;
    listen [::]:443 ssl;

    client_max_body_size 50M;
    ssl_session_cache shared:SSL:10m;
    proxy_connect_timeout 300s;
    proxy_read_timeout 300s;

    access_log /srv/<appname>/logs/nginx-access.log;
    error_log /srv/<appname>/logs/nginx-error.log;

    location = /favicon.ico {access_log off; log_not_found off; }

    location /static {
        alias /srv/<appname>/mysite/static;
        autoindex on;
    }

    location /media {
        alias /srv/<appname>/mysite/media;
        autoindex on;
    }

    location / {
        include proxy_params;
        proxy_pass http://unix:srv/<appname>/run/gunicorn.sock;
    }

    # Error pages
    error_page 500 502 503 504 /500.html;
    location = /500.html {
        root /srv/<appname>/mysite/static/;
    }
}

server {
    listen 80;
    listen [::]:80;
    server_name <your_domain>.com www.<your_domain>.com;
    return 301 https://$server_name@request_uri;
}

then create a symlink to your new config file in sites-enabled and start Nginx

# Make site available by creating link to it in sites-enabled/
sudo ln -s /etc/nginx/sites-available/<appname>.conf /etc/nginx/sites-enabled/<appname>.conf
# Remove the default configuration so it doesn't mask ours
sudo rm /etc/nginx/sites-enabled/default
# Start nginx
sudo service nginx start

Setup SSL Certificate with Certbot

sudo certbot --nginx -d <your_domain>.com -d www.<your_domain>.com
sudo ufw allow ssh
sudo ufw enable
sudo ufw allow 'Nginx Full'

You will also need to create a rule in the AWS Lightsail console to allow HTTPS. Certbot will have added some things to your /etc/nginx/sites-available/.conf file

End

At the end the directory structure on the server looks like this (abbreviated):

# Owned by <appname>, writable by users
\srv\<appname>\
    mysite\
        .env
        requirements.txt
        manage.py
        app1\
        app2\
        static\
        media\
        mysite\
            settings.py
            wsgi.py
    bin\
        activate
        gunicorn_start
    run\
        gunicorn.sock
    logs\
        gunicorn_supervisor.log
        nginx-access.log
        nginx-error.log

# Owned by ubuntu       
\srv\<appname>.git\
    .git\
    hooks\
        post-receive

\etc\nginx\
    sites-available\
        <appname>.conf
    sites-enabled\
        <appname>.conf
    nginx.conf

Troubleshooting

Diagnostics:

# Check if Gunicorn is running out of memory
sudo dmesg | grep gunicorn

# Check memory in general (in Mb)
free -m

Swap File

My server was running out of memory when I uploaded files over some certain dimensions due to some processing with imagekit to create thumbnails. This seems to have fixed it.

# Allocate space
sudo fallocate -l 1G /swapfile
# Reduce permissions for it
sudo chmod 600 /swapfile
# Assign that space as a swap file
sudo mkswap /swapfile
# Activate swap file
sudo swapon /swapfile
# Activate swap file on boot
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

-Published 4 pm Thu, Feb 9 2023-

HOME

Blah

Rand