Nginx as reverse proxy in front of apache

Posted by paul on 2017.07.01

Length: approximately 1500 words

nginx Reverse Proxy and Apache

Here's an asciinema video (7 minutes long) of all the steps I run in this tutorial in 1 go. You can refer to this video to see the actual changes being made. In my tutorial below, I will point out the exact time in the asciinema video.

In this tutorial I will demonstrate how to set up nginx as a Reverse Proxy in front of an Apache server with VirtualHost. I will also explain how to log real source IP in Apache access_log, even when Apache is behind the nginx Reverse Proxy.

Using CentOS 7 on DigitalOcean as the server and Mac as workstation. I will use site.com as the example domain. Since we are using hosts file to browse the test server, you can copy the configs exactly as shown and it should still work.

It'd be helpful if you knew the basics of vim editor. You only need to know how to enter edit mode, escape from edit mode, save the change and quit vim editor. Learning the basics of vim should take less than 5 minutes.

Provision a CentOS 7 server

Provision a server running CentOS 7. You can use DigitalOcean or Linode or AWS. Determine the IP of your new server once your test server is online. Let's assume it is 15.16.17.18 for this tutorial.

Edit HOSTS file on your Mac

On your Mac, run

sudo vim /private/etc/hosts

Enter your Mac password. Enter following in /private/etc/hosts on your Mac.

15.16.17.18 site.com www.site.com

Remember to replace 15.16.17.18 with the actual IP of your test server.

Save /private/etc/hosts.

If you ping "site.com" in Terminal, it should ping 15.16.17.18.

Install Apache, add folders, set permissions

Video 00:00

SSH into the new server. As root on the test CentOS 7 server, run following:

yum -y install httpd

mkdir -p /data/www/site.com/site.com

echo "Hello world" > /data/www/site.com/site.com/index.html

echo "Hello earth" > /data/www/site.com/site.com/earth.html

echo "Hello mars" > /data/www/site.com/site.com/mars.html

chown -R apache: /data/www

find /data/www/ -type d -exec chmod 755 {} \;

find /data/www/ -type f -exec chmod 644 {} \;

systemctl start httpd && systemctl enable httpd

Edit /root/.vimrc

Video 00:48

To get proper formatting when pasting in multiple lines of text in vim, you need to have a file "/root/.vimrc" with a line of config on the CentOS 7 server.

Use vim to open /root/.vimrc, and add following text.

set paste

Create VirtualHost Config file for Apache

Video 01:05

vim /etc/httpd/conf.d/site.com.conf

Copy in following lines of text into site.com.conf.

<VirtualHost *:80>
ServerName site.com
ServerAlias site.com
DocumentRoot /data/www/site.com/site.com

  <Directory /data/www/site.com/site.com>
    Require all granted
  </Directory>
</VirtualHost>

Restart Apache service.

systemctl restart httpd

Update Apache LogFormat to record VirtualHost

Video 01:29

Backup httpd.conf first.

cp /etc/httpd/conf/httpd.conf{,-orig}

In file /etc/httpd/conf/httpd.conf, you will locate following line to edit:
`LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined`

You will add %v right after LogFormat.

vim /etc/httpd/conf/httpd.conf

# BEFORE
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

# AFTER
LogFormat "%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

With v% added, Apache will log the VirtualHost domain being requested by the client. In this case it would be site.com.

Restart Apache service.

systemctl restart httpd

Test browsing site.com

Video 02:09

On your server, tail -f command to monitor the log file.

tail -f /var/log/httpd/access_log

On your Mac, open a Browser window and try visiting site.com. You should see a webpage with following:

Hello world

In the Terminal window connected to the CentOS 7 server, you should see new logs being added to /var/log/httpd/access_log, sililar to below. In this tutorial I am actually browsing the website site.com from another Linux server running elinks browser. Thus the log from your installation may look different.

site.com 138.68.8.53 - - [28/Jun/2017:06:16:14 +0000] "GET / HTTP/1.1" 200 12 "-" "ELinks/0.12pre6 (textmode; Linux; 80x24-2)"

You should see site.com, which is the domain the client requested. And you should see the the real IP of the client being logged.

Hit ctrl+c to exit from 'tail -f'.

Edit main Apache config file before installing nginx

Video 02:21

Let's configure Apache to serve content through port 8080, not 80. This will free up port 80 for nginx to use.

vim /etc/httpd/conf/httpd.conf

Find below line and then comment it out.

BEFORE

Listen 80

AFTER

# Listen 80
Listen 127.0.0.1:8080

Edit Apache VirtualHost config file before installing nginx

Video 03:05

vim /etc/httpd/conf.d/site.com.conf

Replace 1 line as shown below.

BEFORE

<VirtualHost *:80>

AFTER

<VirtualHost 127.0.0.1:8080>

Restart Apache service.

systemctl restart httpd

Now Apache will only listen for incoming traffic on port 8080 of IP 127.0.0.1.

Install nginx rpm

Video 03:31

Install nginx on the CentOS 7 server.

yum -y install epel-release && yum -y install nginx

mkdir /etc/nginx/{sites-available,sites-enabled}

systemctl enable nginx && systemctl start nginx

You will create an nginx Server Block. And these Server Blocks are kept in /etc/nginx/sites-available/, and symlinked to /etc/nginx/sites-enabled/ when it's to be activated.

Update /etc/nginx/nginx.conf

Video 03:57

Add following line inside of http { }

vim /etc/nginx/nginx.conf

include /etc/nginx/sites-enabled/*.conf;

This tutorial is done with nginx version nginx-1.10.2-1. So to add the line in nginx.conf, you'd paste it in as shown below. Line 6 is what we are adding.

        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }

    include /etc/nginx/sites-enabled/*.conf;

    # Settings for a TLS enabled server.

Set up nginx as a Reverse Proxy

Video 04:24

We will create 2 Server Blocks. First one sets nginx to serve content for site.com. The second is to redirect traffic coming to www.site.com onward to site.com. Using `*.proxy.conf` shows that the config file is for a Reverse Proxy. The config file must end with .conf for nginx to load the config.

Run following commands to create the files and create the symlinks.

touch /etc/nginx/sites-available/site.com.proxy.conf
touch /etc/nginx/sites-available/site.com.redirect-no-www.conf

ln -s /etc/nginx/sites-available/site.com.proxy.conf /etc/nginx/sites-enabled/site.com.proxy.conf
ln -s /etc/nginx/sites-available/site.com.redirect-no-www.conf /etc/nginx/sites-enabled/site.com.redirect-no-www.conf

First 2 lines are creating blank files. Last 2 lines are symlinking the files from folder sites-available/ to folder sites-enabled/.

site.com.proxy.conf

Video 04:47

Add following into the first nginx Server Block so that nginx will accept incoming traffic to port 80 and forward it to port 8080 that Apache is listening to.

vim /etc/nginx/sites-available/site.com.proxy.conf

upstream apache8080 {
    # change this if you used a port other than 8080
    server 127.0.0.1:8080 fail_timeout=0;
}


server {
    listen 80;
    listen [::]:80;

    server_name site.com; # replace this with your domain

    location / {
        # Some of these might not be required for other services
        # proxy_pass is the most important
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://apache8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 900s;
    }
}

Note apache8080 in lines 1 and 20 have to match. Line 3 shows the port the upstream server (local apache server in this example) is serving content through. Line 11 shows the Domain nginx will proxy content for.

  • line 1: apache8080
  • line 3: server 127.0.0.1:8080
  • line 11: server_name site.com;
  • line 20: apache8080

Nginx: redirect www to non-www

Video 05:08

vim /etc/nginx/sites-available/site.com.redirect-no-www.conf

The file should look like below.

server {
    server_name www.site.com;
    return 301 $scheme://site.com$request_uri;
}

Test nginx config syntax

Video 5:24

Test the modified nginx config syntax is correct by running the following command:

nginx -t

Restart nginx and apache services

nginx -t && systemctl restart nginx && systemctl restart httpd

Test browsing site.com

Video 05:40

From your Mac, browse site.com with a web browser, and you should see the web page.

Now stop nginx service but keep Apache service running.

systemctl stop nginx

Test browsing site.com and you will notice the site won't load, which is expected. Apache service is running but nginx service is not, thus Apache does not know that a visitor tried to view the website site.com.

Start nginx service up again, and now you can browse site.com successfully.

systemctl start nginx

Let's check the log file of Apache.

tail -f /var/log/httpd/access_log

You may see something like below. Notice 127.0.0.1? Apache should be recording the source IP there and it shouldn't be 127.0.0.1. Let's fix it.

site.com 127.0.0.1 - - [28/Jun/2017:06:21:35 +0000] "GET /earth.html HTTP/1.1" 200 12 "-" "ELinks/0.12pre6 (textmode; Linux; 80x24-2)"

Update Apache LogFormat to record real source IP

Video 06:00

You will edit following line in /etc/httpd/conf/httpd.conf

`LogFormat "%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined`

Replace %h with %a. Note you are updating the second value.

vim /etc/httpd/conf/httpd.conf

# BEFORE
LogFormat "%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

# AFTER
LogFormat "%v %a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

Note v% will add the VirtualHost Domain being requested. In this case it would be site.com. This addition was made earlier in this tutorial.

Parameter a% will set Apache to log the real source IP instead of 127.0.0.1. This real source IP is in the header that nginx forwards to Apache.

Update Apache VirtualHost to log correct client IP

Video 06:28

With Apache Log Formating update done, let's configure the VirtualHost config file to be aware of the source IP in the header that nginx forwards.

Add following in the file, but outside of the Directory config.

vim /etc/httpd/conf.d/site.com.conf

# For getting real source IP for logging
RemoteIPHeader X-Forwarded-For
RemoteIPInternalProxy 127.0.0.1/8

After edit, /etc/httpd/conf.d/site.com.conf should look like below.

<VirtualHost 127.0.0.1:8080>
ServerName site.com
ServerAlias site.com
DocumentRoot /data/www/site.com/site.com

# For getting real source IP for logging
RemoteIPHeader X-Forwarded-For
RemoteIPInternalProxy 127.0.0.1/8

  <Directory /data/www/site.com/site.com>
    Require all granted
  </Directory>
</VirtualHost>

Restart nginx and apache services

nginx -t && systemctl restart nginx && systemctl restart httpd

Test browsing site.com and check log files

Video 06:53

Run following command as root on the server.

tail -f /var/log/httpd/access_log

Above command will keep tailing the log file until you kill it with ctrl-c command.

From your Mac, browse site.com. You should see logs like below being added to /var/log/httpd/access_log. Note real source IP is being logged, instead of 127.0.0.1.

site.com 138.68.8.53 - - [28/Jun/2017:06:23:32 +0000] "GET /mars.html HTTP/1.1" 200 11 "-" "ELinks/0.12pre6 (textmode; Linux; 80x24-2)"
site.com 138.68.8.53 - - [28/Jun/2017:06:23:41 +0000] "GET /earth.html HTTP/1.1" 200 12 "-" "ELinks/0.12pre6 (textmode; Linux; 80x24-2)"

Clean up

Make sure to remove the entry for site.com from /private/etc/hosts on your Mac.