Self-host MongoDB

Create a VM#

On your computer (pick another name if my-mongodb is taken):

ssh exe.dev new --name=my-mongodb
ssh my-mongodb.exe.xyz

Adding MongoDB next to an app? Skip this step and run the rest on the app's VM. All remaining commands run on the VM, as exedev.

Install MongoDB 9.0#

Ubuntu doesn't package MongoDB. MongoDB's own apt repository has Community Edition for each Ubuntu LTS release; add it with MongoDB's signing key, on the 9.0 series:

curl -fsSL https://pgp.mongodb.com/server-9.asc | sudo gpg --dearmor --yes -o /usr/share/keyrings/mongodb-server-9.gpg
echo "deb [ arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-9.gpg ] https://repo.mongodb.org/apt/ubuntu $(. /etc/os-release; echo $VERSION_CODENAME)/mongodb-org/9.0 multiverse" |
  sudo tee /etc/apt/sources.list.d/mongodb-org-9.0.list
sudo apt-get update
apt-cache policy mongodb-org | head -n 3

The candidate should be 9.0.x. Install the server, mongosh, and the database tools (mongodump, mongorestore, and friends):

sudo DEBIAN_FRONTEND=noninteractive apt-get install -y mongodb-org
mongod --version | head -n 1
mongosh --version

Expect db version v9.0.x and a 2.x version of mongosh. The package doesn't start MongoDB yet; the next step configures it first.

Turn on access control and the replica set#

This replaces /etc/mongod.conf with one that keeps MongoDB on localhost, makes it a one-member replica set called rs0, and requires a user and password for everything. A replica set with access control also needs a key file, which its members use to trust each other:

sudo test -e /etc/mongod.keyfile ||
  openssl rand -base64 756 | sudo install -o mongodb -g mongodb -m 400 /dev/stdin /etc/mongod.keyfile
sudo tee /etc/mongod.conf >/dev/null <<'EOF'
# MongoDB settings. Reference:
# https://www.mongodb.com/docs/manual/reference/configuration-options/
storage:
  dbPath: /var/lib/mongodb
  # The WiredTiger cache defaults to half of (RAM - 1 GB). To cap it,
  # say on a VM shared with an app, uncomment these lines and set a size:
  # wiredTiger:
  #   engineConfig:
  #     cacheSizeGB: 1
systemLog:
  destination: file
  logAppend: true
  logRotate: reopen
  path: /var/log/mongodb/mongod.log
net:
  port: 27017
  bindIp: 127.0.0.1
  ipv6: true
processManagement:
  timeZoneInfo: /usr/share/zoneinfo
replication:
  replSetName: rs0
security:
  authorization: enabled
  keyFile: /etc/mongod.keyfile
EOF
sudo systemctl enable mongod
sudo systemctl restart mongod
timeout 60 sh -c 'until mongosh --quiet --eval 1 >/dev/null 2>&1; do sleep 1; done'
systemctl is-active mongod

Expect active. MongoDB now starts at boot. With no users yet, MongoDB lets a connection from the VM itself start the replica set and create the first user, and nothing else (its "localhost exception"). Start the replica set, with the member's address as 127.0.0.1:

mongosh --quiet --eval 'rs.initiate({_id: "rs0", members: [{_id: 0, host: "127.0.0.1:27017"}]})'
timeout 60 sh -c 'until mongosh --quiet --eval "db.hello().isWritablePrimary" | grep -q true; do sleep 1; done'
mongosh --quiet --eval 'db.hello().setName'

Expect { ok: 1 }, then rs0. Now create the admin user with a random password, and save its connection string to ~/mongodb/admin.env. (If that file already exists, the user does too; the script leaves both alone.)

mkdir -p -m 700 ~/mongodb
if [ ! -e ~/mongodb/admin.env ]; then
  pass=$(openssl rand -hex 24)
  mongosh --quiet admin --eval "db.createUser({user: 'admin', pwd: '$pass', roles: ['root']})" >/dev/null
  (umask 077; echo "MONGODB_URI='mongodb://admin:$pass@127.0.0.1:27017/?authSource=admin&directConnection=true'" > ~/mongodb/admin.env)
fi
cat > ~/mongodb/mongosh.sh <<'EOF'
#!/usr/bin/env bash
# mongosh, signed in as MongoDB's admin user. Arguments go to mongosh.
set -euo pipefail
. "$HOME/mongodb/admin.env"
exec mongosh "$MONGODB_URI" "$@"
EOF
chmod +x ~/mongodb/mongosh.sh

Check that a connection without a password is turned away and the admin's works:

mongosh --quiet --eval 'db.adminCommand({listDatabases: 1})' 2>&1 | tail -n 1
~/mongodb/mongosh.sh --quiet --eval 'db.adminCommand({listDatabases: 1}).databases.map(d => d.name).join(" ")'
~/mongodb/mongosh.sh --quiet --eval 'rs.status().members[0].stateStr'
ss -ltn | grep 27017

Expect MongoServerError: Command listDatabases requires authentication, then admin config local, PRIMARY, and MongoDB on 127.0.0.1:27017 only.

Size it for the VM#

MongoDB sizes its WiredTiger cache from the VM's memory when it starts: half of (RAM − 1 GB), at least 256 MB. Resize the VM and the cache follows at the next restart. See what it chose:

~/mongodb/mongosh.sh --quiet --eval 'Math.round(db.serverStatus().wiredTiger.cache["maximum bytes configured"] / 1048576) + " MB"'

On an 8 GB VM, expect about 3450 MB. That suits a VM of its own. On a VM shared with an app, cap it: uncomment the wiredTiger lines in /etc/mongod.conf, set cacheSizeGB (it takes fractions, such as 0.5), and sudo systemctl restart mongod.

MongoDB's packages don't rotate its log, so add that too. logRotate: reopen in the configuration makes MongoDB open a fresh file when logrotate signals it:

sudo tee /etc/logrotate.d/mongod >/dev/null <<'EOF'
/var/log/mongodb/mongod.log {
    weekly
    rotate 8
    compress
    delaycompress
    missingok
    notifempty
    postrotate
        systemctl kill --signal=SIGUSR1 --kill-whom=main mongod.service
    endscript
}
EOF
sudo logrotate --debug /etc/logrotate.d/mongod 2>&1 | grep '^error' || echo ok

Expect ok.

Create a database for your app#

This script creates a user with the same name as the database, gives it readWrite and dbAdmin on that database only, and writes its settings to ~/mongodb/<name>.env. It never touches an existing one:

cat > ~/mongodb/create-db.sh <<'EOF'
#!/usr/bin/env bash
# Usage: create-db.sh NAME
# Create a MongoDB user called NAME that owns the database NAME, and
# write its settings to ~/mongodb/NAME.env (mode 600).
set -euo pipefail
db=${1:?usage: create-db.sh NAME}
[[ $db =~ ^[a-z_][a-z0-9_]{0,31}$ ]] || { echo "Use up to 32 lowercase letters, digits, and _." >&2; exit 1; }
case $db in admin|config|local) echo "$db is MongoDB's own database." >&2; exit 1;; esac
env="$HOME/mongodb/$db.env"
if [ -e "$env" ]; then echo "$env already exists; leaving $db alone."; exit 0; fi
pass=$(openssl rand -hex 24)
"$HOME/mongodb/mongosh.sh" --quiet --eval "db.getSiblingDB('$db').createUser({user: '$db', pwd: '$pass',
  roles: [{role: 'readWrite', db: '$db'}, {role: 'dbAdmin', db: '$db'}]})" >/dev/null
url="mongodb://$db:$pass@127.0.0.1:27017/$db?directConnection=true"
(umask 077; printf 'MONGODB_URI=%s\nDATABASE_URL=%s\nMONGODB_DB=%s\n' "$url" "$url" "$db" > "$env")
echo "$env"
EOF
chmod +x ~/mongodb/create-db.sh
~/mongodb/create-db.sh app

Check that the app's user can sign in, write, use a transaction, and watch a change stream, and can't see other databases:

. ~/mongodb/app.env
mongosh --quiet "$MONGODB_URI" --eval '
  const cs = db.check.watch();
  cs.disableBlockWarnings();
  const s = db.getMongo().startSession();
  s.startTransaction();
  s.getDatabase("app").check.insertOne({note: "Grüße ✓"});
  s.commitTransaction();
  print(cs.next().operationType, db.check.findOne().note);
  db.check.drop();
  try { db.getSiblingDB("admin").system.users.findOne() } catch (e) { print(e.codeName) }'
stat -c '%a %n' ~/mongodb/app.env

Expect insert Grüße ✓, then Unauthorized, then 600.

Set up nightly backups#

This script dumps every database and every user with mongodump, into one compressed archive. --oplog also records the writes that happen while it runs, so a restore lands on one moment in time. Beside the archive it saves the .env files, and it keeps the 14 newest backups:

cat > ~/mongodb/backup.sh <<'EOF'
#!/usr/bin/env bash
# Back up every database and user, and ~/mongodb/*.env, into
# ~/mongodb/backups/<UTC time>/. Keeps the 14 newest backups.
set -euo pipefail
umask 077
dest="$HOME/mongodb/backups"
keep=14
mkdir -p "$dest"
out="$dest/$(date -u +%Y%m%d-%H%M%S)"
while [ -e "$out" ] || [ -e "$out.part" ]; do sleep 1; out="$dest/$(date -u +%Y%m%d-%H%M%S)"; done
trap 'rm -rf "$out.part"' EXIT
mkdir "$out.part"

admin=$(. "$HOME/mongodb/admin.env"; echo "$MONGODB_URI")
mongodump --uri="$admin" --oplog --gzip --archive="$out.part/dump.archive.gz" 2> "$out.part/mongodump.log" ||
  { cat "$out.part/mongodump.log" >&2; exit 1; }
cp -p "$HOME"/mongodb/*.env "$out.part/"

mv "$out.part" "$out"
ls -1d "$dest"/2*/ | head -n -"$keep" | xargs -r rm -rf --
echo "$out"
EOF
chmod +x ~/mongodb/backup.sh

Schedule it, then take a first backup:

sudo tee /etc/systemd/system/mongodb-backup.service >/dev/null <<EOF
[Unit]
Description=Back up MongoDB to ~/mongodb/backups
After=mongod.service

[Service]
Type=oneshot
User=$USER
ExecStart=$HOME/mongodb/backup.sh
EOF
sudo tee /etc/systemd/system/mongodb-backup.timer >/dev/null <<'EOF'
[Unit]
Description=Back up MongoDB every night

[Timer]
OnCalendar=*-*-* 03:30:00
RandomizedDelaySec=30m
Persistent=true

[Install]
WantedBy=timers.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now mongodb-backup.timer
sudo systemctl start mongodb-backup.service
systemctl list-timers mongodb-backup.timer
ls -l ~/mongodb/backups/*/

Expect a next run time, and admin.env, app.env, dump.archive.gz, and mongodump.log. For failures: journalctl -u mongodb-backup.service.

Test a backup#

Put a document with some non-ASCII text in a scratch database, take a backup, and restore it into a scratch MongoDB server on port 27019, exactly as you would on a new VM:

~/mongodb/mongosh.sh --quiet --eval 'db.getSiblingDB("restore_check").t.insertOne({_id: 1, note: "Grüße ✓"})'
latest=$(~/mongodb/backup.sh)
s=/tmp/mongodb-scratch
mkdir -m 700 $s
mongod --dbpath $s --port 27019 --bind_ip 127.0.0.1 --fork --logpath $s/mongod.log >/dev/null
mongorestore --uri=mongodb://127.0.0.1:27019/ --gzip --archive="$latest/dump.archive.gz" --oplogReplay
mongosh --quiet mongodb://127.0.0.1:27019/restore_check --eval 'db.t.findOne().note'
. "$latest/app.env"
mongosh --quiet "${MONGODB_URI/27017/27019}" --eval 'db.runCommand({connectionStatus: 1}).authInfo.authenticatedUsers[0].user'

mongorestore should end with 0 document(s) failed to restore (lines saying it don't know what to do with subdirectory are normal for an archive), then expect Grüße ✓ and app: the restored user still signs in with the password from the backup. Stop and remove the scratch server, and drop the scratch database:

mongod --dbpath $s --shutdown >/dev/null
rm -rf $s
~/mongodb/mongosh.sh --quiet --eval 'db.getSiblingDB("restore_check").dropDatabase(); db.adminCommand({listDatabases: 1}).databases.map(d => d.name).join(" ")'

The list should show only MongoDB's own admin config local (app appears once your app writes to it). Installation done; continue with Connect from your computer.

On the VM, ~/mongodb/backup.sh takes an extra backup and prints its folder. Each folder has dump.archive.gz (every database, user, and role), the .env files, and mongodump.log. mongorestore reads the archive; its lines saying it don't know what to do with subdirectory are normal.

Undo a mistake#

Dropped the wrong collection? Restore a backup next to the live database. On the VM, pick a backup folder from the list:

ls ~/mongodb/backups/
when=20261031-033012
db=app
. ~/mongodb/admin.env
mongorestore --uri="$MONGODB_URI" --gzip --archive=$HOME/mongodb/backups/$when/dump.archive.gz \
  --nsInclude="$db.*" --nsFrom="$db.*" --nsTo="${db}_restored.*"
~/mongodb/mongosh.sh --quiet --eval "db.getSiblingDB('${db}_restored').getCollectionNames()"

Say the lost collection was orders. Copy it back with $out, which replaces any collection of that name in app, then drop the copy:

~/mongodb/mongosh.sh --quiet --eval '
  db.getSiblingDB("app_restored").orders.aggregate([{$out: {db: "app", coll: "orders"}}]);
  db.getSiblingDB("app_restored").dropDatabase()'

($out copies documents but not indexes; recreate those, or let your app do it on start.) To roll the whole database back instead, stop your app, keep a copy of its current state, and restore the backup over it. --drop replaces each collection that's in the backup; users stay as they are:

when=20261031-033012
db=app
. ~/mongodb/admin.env
mongodump --uri="$MONGODB_URI" --db=$db --gzip --archive=$HOME/$db-before-rollback.archive.gz
mongorestore --uri="$MONGODB_URI" --gzip --archive=$HOME/mongodb/backups/$when/dump.archive.gz \
  --nsInclude="$db.*" --drop

Start the app again, and delete ~/app-before-rollback.archive.gz once you're happy.

Restore everything on a new VM#

On a new VM, follow Install by hand up to and including "Size it for the VM", but don't create the app database. Then copy a backup folder over. On your computer:

scp -r mongodb-backups/20261031-033012 my-mongodb.exe.xyz:restore

On the VM, restore everything, users included. First give the new admin user the old password, so that it's the same before and after the restore, and let the old .env files take over:

oldpass=$(sed -nE 's|.*//admin:([^@]*)@.*|\1|p' ~/restore/admin.env)
~/mongodb/mongosh.sh --quiet --eval "db.getSiblingDB('admin').changeUserPassword('admin', '$oldpass')" >/dev/null
cp ~/restore/*.env ~/mongodb/
chmod 600 ~/mongodb/*.env
. ~/mongodb/admin.env
mongorestore --uri="$MONGODB_URI" --gzip --archive=$HOME/restore/dump.archive.gz --oplogReplay
~/mongodb/mongosh.sh --quiet --eval 'db.adminCommand({listDatabases: 1}).databases.map(d => d.name).join(" ")'

mongorestore should end with 0 document(s) failed to restore, and every database should be listed. Continue Install by hand from "Create a database for your app" (it leaves the existing app alone), and rm -r ~/restore when you're done. If the old VM had Docker or Tailscale access, redo its bindIpAll change too.

More often#

Nightly dumps can lose up to a day of changes. To back up hourly and keep two days, on the VM:

sudo sed -i 's/^OnCalendar=.*/OnCalendar=hourly/; s/^RandomizedDelaySec=.*/RandomizedDelaySec=5m/' /etc/systemd/system/mongodb-backup.timer
sed -i 's/^keep=.*/keep=48/' ~/mongodb/backup.sh
sudo systemctl daemon-reload && sudo systemctl restart mongodb-backup.timer
systemctl list-timers mongodb-backup.timer

Each backup is a full dump, so this suits databases up to a few GB. exe.dev already replicates the VM's disk to other machines, which guards against hardware failure, not against a bad deleteMany.

Updates don't install themselves on this VM, so check for them now and then. MongoDB lists each patch release, and the security fixes in it, in the 9.0 release notes.

Minor updates#

On the VM, back up, upgrade just the MongoDB packages, then restart MongoDB onto the new version. Apps see a few seconds of errors:

~/mongodb/backup.sh
sudo apt-get update
sudo DEBIAN_FRONTEND=noninteractive apt-get install --only-upgrade -y -o Dpkg::Options::=--force-confold \
  $(dpkg-query -W -f='${db:Status-Abbrev} ${Package}\n' 'mongodb-*' | awk '$1 == "ii" {print $2}')
sudo systemctl restart mongod
timeout 60 sh -c 'until ~/mongodb/mongosh.sh --quiet --eval 1 >/dev/null 2>&1; do sleep 1; done'
~/mongodb/mongosh.sh --quiet --eval 'db.version()'

--force-confold keeps your /etc/mongod.conf.

Major upgrades#

9.0 is supported until October 31, 2031, so there's no rush. MongoDB upgrades one major release at a time, and each one changes the data files once you raise the "feature compatibility version" (FCV) at the end; until then you can still go back. Read the new release's upgrade notes and check that your drivers support it, then take its repository line and key from MongoDB's install guide. Stop your apps, and on the VM:

~/mongodb/backup.sh
~/mongodb/mongosh.sh --quiet --eval 'db.adminCommand({getParameter: 1, featureCompatibilityVersion: 1}).featureCompatibilityVersion.version'

The FCV must be 9.0 before you start. Replace /etc/apt/sources.list.d/mongodb-org-9.0.list and the key with the new ones, run sudo apt-get update, the same apt-get install line as in Install by hand, and sudo systemctl restart mongod. Check db.version() and your app. When you're sure you'll stay, raise the FCV in ~/mongodb/mongosh.sh, with the new version number:

db.adminCommand({setFeatureCompatibilityVersion: "10.0", confirm: true})

On the VM, MongoDB's status and log usually have the answer. Each log line is a JSON object; "s":"E" marks errors:

systemctl status mongod
sudo tail -n 30 /var/log/mongodb/mongod.log
sudo journalctl -u mongod -n 20

• It won't start, and the journal says BadValue: a setting in /etc/mongod.conf is wrong. YAML wants spaces, not tabs, and security.keyFile is required alongside authorization and replSetName. The journal names the bad setting.

• ECONNREFUSED 127.0.0.1:27017: MongoDB isn't running. Start it with sudo systemctl start mongod and read the log if it stops.

• Authentication failed: the password doesn't match the .env file, or the connection string is missing the database name (the user signs in to its own database, /app). To set a new password in both places, then restart the app:

  pass=$(openssl rand -hex 24)
  ~/mongodb/mongosh.sh --quiet --eval "db.getSiblingDB('app').changeUserPassword('app', '$pass')" >/dev/null
  sed -i "s|//app:[^@]*@|//app:$pass@|" ~/mongodb/app.env
  

• Connections time out with Server selection timed out, often mentioning 127.0.0.1:27017 from another machine or a container: the connection string lacks directConnection=true, so the driver went looking for the replica set's member address. Add it.

• Transaction numbers are only allowed on a replica set member or mongos or The $changeStream stage is only supported on replica sets: this MongoDB wasn't set up as a replica set. Run "Turn on access control and the replica set" from Install by hand.

• not authorized on admin to execute command: the app's user only has rights in its own database. Do admin work with ~/mongodb/mongosh.sh.

• Startup warnings: Using the XFS filesystem is strongly recommended and Failed to read file (for /sys/kernel/mm/transparent_hugepage) are expected. The VM's disk is ext4, which MongoDB supports, and its kernel has no transparent huge pages for MongoDB to tune. Nothing to do.

• It won't start after making the VM smaller with a cacheSizeGB set: lower it in /etc/mongod.conf, or comment it out to go back to the automatic size.

• The tunnel says Address already in use: something on your computer has port 27018. Pick another, such as -L 27020:localhost:27017.

• SSH refuses the tunnel: try the form that always works, ssh -N -L 27018:localhost:27017 vm+my-mongodb@vm.exe.xyz.

• Disk full: MongoDB stops. Check df -h / and sudo du -sh ~/mongodb/backups /var/lib/mongodb. The replica set's operation log takes about 1 GB of that, and stays that size. Lower keep in ~/mongodb/backup.sh after copying old backups off the VM, or grow the disk: on your computer, ssh exe.dev resize my-mongodb --disk=50GB.

• Slow queries: MongoDB logs every operation slower than 100 ms. Find them with sudo grep '"Slow query"' /var/log/mongodb/mongod.log | tail -n 5, and ask MongoDB how it runs one with .explain("executionStats") in mongosh. A COLLSCAN on a big collection usually wants an index.