Setting up an AWS instance with Ubuntu 16.04 LTS and virtualmin

: Ubuntu; Ubuntu 16.04 LTS Server; 31 January 2018

Setting up an AWS instance with Ubuntu 16.04 LTS and virtualmin

User Rating: 5 / 5

{jcomments off}

Part 4: Prepare VirtualMin to run websites

Introduction

In this article I set up an AWS EC2 Instance from scratch to run VirtualMin, an open-source virtual server management tool. While Virtumin can be set up on an in-use server, it requires a lot of reconfiguration so it is simpler to set up a new server and proceed from there. At the end of this article we shall have a clean install of VirtualMin ready to accept new websites. This is part 4 where we prepare our Virtualmin setup to run websites.

The article is broken into 4 main steps:

The article is based on various pages by numerous authors who I thank for their invaluable insights, as well as my own experiences. Any errors are most likely mine. I give no guarantee that what you find in these pages will work on your setup - as usual YMMV.

In this series of articles I shall be using an FQDN of myserver.example.com and a user called myuser. The server will reside at the fictitious public ip 12.34.56.78. Make sure you substitute your own FQDN, user and ip where necessary - the example values given will not work.

Move mysql data to external disk

Taken from: https://www.digitalocean.com/community/tutorials/how-to-move-a-mysql-data-directory-to-a-new-location-on-ubuntu-16-04

Databases grow over time, sometimes outgrowing the space on the file system. You can also run into I/O contention when they’re located on the same partition as the rest of the operating system. RAID, network block storage, and other devices can offer redundancy and other desirable features. Whether you’re adding more space, evaluating ways to optimize performance, or looking to take advantage of other storage features, this tutorial will guide you through relocating MySQL’s data directory.

In this example, we’re moving the data to a block storage device mounted at /mnt/mysql.

Step 1 — Moving the MySQL Data Directory

To prepare for moving MySQL’s data directory, let’s verify the current location by starting an interactive MySQL session using the administrative credentials.

sudo mysql -u root -p

When prompted, supply the MySQL root password. Then from the MySQL prompt, select the data directory:

select @@datadir;

+-----------------+
| @@datadir |
+-----------------+
| /var/lib/mysql/ |
+-----------------+
1 row in set (0.00 sec)

This output confirms that MySQL is configured to use the default data directory, /var/lib/mysql/, so that’s the directory we need to move. Once you've confirmed this, type exit to leave the monitor.

exit

To ensure the integrity of the data, we’ll shut down MySQL before we actually make changes to the data directory:

sudo systemctl stop mysql

systemctl doesn't display the outcome of all service management commands, so if you want to be sure you've succeeded, use the following command:

sudo systemctl status mysql

You can be sure it’s shut down if the final line of the output tells you the server is stopped:

sudo systemctl status mysql
. . .
Jul 18 11:24:20 ubuntu-512mb-nyc1-01 systemd[1]: Stopped MySQL Community Server.

Now that the server is shut down, we’ll copy the existing database directory to the new location with rsync. Using the -a flag preserves the permissions and other directory properties, while -v provides verbose output so you can follow the progress.

Note: Be sure there is no trailing slash on the directory, which may be added if you use tab completion. When there’s a trailing slash, rsync will dump the contents of the directory into the mount point instead of transferring it into a containing mysql directory:

sudo rsync -av /var/lib/mysql /mnt/mysql
sudo rsync -av /var/lib/mysql-files /mnt/mysql
sudo rsync -av /var/lib/mysql-keyring /mnt/mysql

Once the rsync is complete, rename the current folder with a .bak extension and keep it until we’ve confirmed the move was successful. By re-naming it, we’ll avoid confusion that could arise from files in both the new and the old location:

sudo mv /var/lib/mysql /var/lib/mysql.bak
sudo mv /var/lib/mysql-files /var/lib/mysql-files.bak
sudo mv /var/lib/mysql-keyring /var/lib/mysql-keyring.bak

Now we’re ready to turn our attention to configuration.

Step 2 — Pointing to the New Data Location

MySQL has several ways to override configuration values. By default, the datadir is set to /var/lib/mysql in the /etc/mysql/mysql.conf.d/mysqld.cnf file. Edit this file to reflect the new data directory:

sudo nano /etc/mysql/mysql.conf.d/mysqld.cnf

Find the line that begins with datadir= and change the path which follows to reflect the new location. In our case, the updated file looks like the output below:

/etc/mysql/mysql.conf.d/mysqld.cnf

. . .
datadir=/mnt/mysql/mysql
. . .

This seems like the right time to bring up MySQL again, but there’s one more thing to configure before we can do that successfully.

Step 3 — Configuring AppArmor Access Control Rules

We’ll need to tell AppArmor to let MySQL write to the new directory by creating an alias between the default directory and the new location. To do this, edit the AppArmor alias file:

sudo nano /etc/apparmor.d/tunables/alias

At the bottom of the file, add the following alias rule:

/etc/apparmor.d/tunables/alias

. . .
alias /var/lib/mysql/ -> /mnt/mysql/mysql/,
. . .

For the changes to take effect, restart AppArmor:

sudo systemctl restart apparmor

Note: If you skipped the AppArmor configuration step, you would run into the following error message:

Job for mysql.service failed because the control process
exited with error code. See "systemctl status mysql.service"
and "journalctl -xe" for details.

The output from both systemctl and journalctl concludes with:

Jul 18 11:03:24 ubuntu-512mb-nyc1-01 systemd[1]:
mysql.service: Main process exited, code=exited, status=1/FAILURE

Since the messages don’t make an explicit connection between AppArmor and the data directory, this error can take some time to figure out.

Step 4 — Restarting MySQL

The next step is to start MySQL, but if you do, you’ll run into another error. This time, instead of an AppArmor issue, the error happens because the script mysql-systemd-start checks for the existence of either a directory, -d, or a symbolic link, -L, that matches two default paths. It fails if they're not found:

/usr/share/mysql/mysql-systemd-start

. . .
if [ ! -d /var/lib/mysql ] && [ ! -L /var/lib/mysql ]; then
echo "MySQL data dir not found at /var/lib/mysql. Please create one."
exit 1
fi
if [ ! -d /var/lib/mysql/mysql ] && [ ! -L /var/lib/mysql/mysql ]; then
echo "MySQL system database not found. Please run mysql_install_db tool."
exit 1
fi
. . .

Since we need these to start the server, we will create the minimal directory structure to pass the script's environment check.

sudo mkdir /var/lib/mysql/mysql -p
sudo ln -s /mnt/mysql/mysql-files /var/lib/mysql-files
sudo ln -s /mnt/mysql/mysql-keyring /var/lib/mysql-keyring

Now we're ready to start MySQL.

sudo systemctl start mysql
sudo systemctl status mysql

To make sure that the new data directory is indeed in use, start the MySQL monitor.

sudo mysql -u root -p

Look at the value for the data directory again:

select @@datadir;
+----------------------------+
| @@datadir |
+----------------------------+
| /mnt/mysql/mysql/ |
+----------------------------+
1 row in set (0.01 sec)

Now that you’ve restarted MySQL and confirmed that it’s using the new location, take the opportunity to ensure that your database is fully functional. Once you’ve verified the integrity of any existing data, you can remove the backup data directory:

sudo rm -Rf /var/lib/mysql.bak

Restart MySQL one final time to be sure that it works as expected:

sudo systemctl restart mysql
sudo systemctl status mysql

Conclusion

In this tutorial, we’ve moved MySQL’s data directory to a new location and updated Ubuntu’s AppArmor ACLs to accommodate the adjustment. Although we were using a Block Storage device, the instructions here should be suitable for redefining the location of the data directory regardless of the underlying technology.

For more on managing MySQL’s data directories, see these sections in the official MySQL documentation:

The Mysql Data Directory
Setting Up Multiple Data Directories

Set up quotas

Go to Webmin → System → Disk Quotas and enable for all disks

Create the main virtual server for the domain

Go to Virtualmin → Create Virtual Server and enter the domain name example.com and your name in Description. Put a strong password in Administration password (note: this is not recoverable – so note it down somewhere). Now open the Enabled Features tab and check Setup SSL website too? And Setup IP-based virtual FTP?, then click on Create Server.

Virtualmin will be unable to find the page after creating the domain’s virtual server since the SSL key has changed. Log out and log in again to fix this.

Make sure that DNS points to example.com and www.example.com to the address of this server.

Go to example.com → Server Configuration → Manage SSL Certificate and open the Let’s Encrypt tab.

Request cert for example.com and www.example.com

Return to example.com→ Server Configuration → Manage SSL Certificate and copy the certificates to all listed programs. ProFTPd will give an error about starting.

In terminal:

sudo systemctl status proftpd.service

sudo systemctl statusproftpd.service
● proftpd.service - LSB: Starts ProFTPD daemon
Loaded: loaded (/etc/init.d/proftpd; bad; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2017-08-29 07:57:10 UTC; 33s ago
Docs: man:systemd-sysv-generator(8)
Process: 6833 ExecStop=/etc/init.d/proftpd stop (code=exited, status=0/SUCCESS)
Process: 7111 ExecStart=/etc/init.d/proftpd start (code=exited, status=1/FAILURE)
Aug 29 07:57:10 server01 systemd[1]: Starting LSB: Starts ProFTPD daemon...
Aug 29 07:57:10 server01 proftpd[7111]: * Starting ftp server proftpd
Aug 29 07:57:10 server01 proftpd[7111]: 2017-08-29 07:57:10,634 server01 proftpd[7121]: fatal: unknown configuration directive 'TLSRSACertificateFile' on line 90 of '/etc/proftpd/proftpd.
Aug 29 07:57:10 server01 proftpd[7111]: ...fail!
Aug 29 07:57:10 server01 systemd[1]: proftpd.service: Control process exited, code=exited status=1
Aug 29 07:57:10 server01 systemd[1]: Failed to start LSB: Starts ProFTPD daemon.
Aug 29 07:57:10 server01 systemd[1]: proftpd.service: Unit entered failed state.
Aug 29 07:57:10 server01 systemd[1]: proftpd.service: Failed with result 'exit-code'.
lines 1-15/15 (END)

sudo nano /etc/proftpd/proftpd.conf

Add the following around the TLS directives:

…
TransferLog /var/log/proftpd/xferlog
SystemLog /var/log/proftpd/proftpd.log
TLSRSACertificateFile /etc/proftpd/proftpd.cert
TLSRSACertificateKeyFile /etc/proftpd/proftpd.key
TLSCACertificateFile /etc/proftpd/proftpd.ca
TLSEngine on
…

And the following under it to prevent ftp overwrite problems:

ServerName gwynsoft.net
User ftp
Group nogroup
UserAlias anonymous ftp
DenyAll
RequireValidShell off
AllowOverwrite on

Then restart proftpd

sudo service proftpd restart

Add an email address to the base domain

Ensure that your DNS provider has an A and an MX entry for mail.example.com.

First we need to set up the format for usernames on virtualmin.

Go to Virtualmin → System Settings → Virtualmin Configuration, find the section ‘Defaults for new domains’ and set ‘Domain name style in username’ to ‘Full domain name’. Now go to Virtualmin → System Settings → Server Templates and select ‘Default Settings’; at the top select ‘Mail for domain’ and, at the bottom, set ‘Format for usernames that include domain’ to ‘username@domain’

Note:

This is not recommended unless transferring existing users from another server where they already use this format. Postfix has trouble with the ‘@’ symbol in usernames.

Set up SSL for email

In /etc/default/saslauthd change OPTIONS="-c -m /var/spool/postfix/var/run/saslauthd" to OPTIONS="-c -r -m /var/spool/postfix/var/run/saslauthd" and restart saslauthd.

sudo nano /etc/default/saslauthd

#OPTIONS="-c -m /var/spool/postfix/var/run/saslauthd"
OPTIONS="-c -r -m /var/spool/postfix/var/run/saslauthd"

sudo service saslauthd restart

Use firewalld to add service smtps in public (default) zone.

sudo reboot

{jcomments on}

Tags: virtualmin, aws, amazon aws, tls, website, mysql, quota, webmin, configuration

Rating:

( 5 Ratings )

GwynSoft

Life's a beach...

Setting up an AWS instance with Ubuntu 16.04 LTS and virtualmin

Part 4: Prepare VirtualMin to run websites

Introduction

Move mysql data to external disk

Step 1 — Moving the MySQL Data Directory

Step 2 — Pointing to the New Data Location

Step 3 — Configuring AppArmor Access Control Rules

Step 4 — Restarting MySQL

Conclusion

Set up quotas

Create the main virtual server for the domain

Add an email address to the base domain

Set up SSL for email