fixes in response to ekim's comments

pull/73/head
Christopher Usher 5 years ago
parent 1dbe585837
commit 027c2900e2

@ -45,19 +45,6 @@ After making any changes to `docker-compose.jsonnet`, you will need to rerun `ge
By default the `downloader`, `restreamer`, `backfiller`, `cutter`, `thrimshim` and `nginx` services of the wubloader will be run. To change which services are run edit the `enabled` object in `docker-compose.jsonnet`. A complete wubloader set up also requires one and only one `database` service (though having a backup database is a good idea) and one and only one `sheetsync` service. By default the `downloader`, `restreamer`, `backfiller`, `cutter`, `thrimshim` and `nginx` services of the wubloader will be run. To change which services are run edit the `enabled` object in `docker-compose.jsonnet`. A complete wubloader set up also requires one and only one `database` service (though having a backup database is a good idea) and one and only one `sheetsync` service.
### Database setup
When setting up a database node, a number of database specific options can be set.
* `database_path`, the local path to save the database to. If this directory is empty then the database setups scripts will be run to create a new database. Otherwise, the database container will load the database stored in this folder.
* `db_args.user`, `db_args.password`, the username and password for the database user that the rest of the wubloader will connect to.
* `db_super_user`, `super_password`, the username and password for the database superuser that is only accessible from the local machine.
* `db_replication_user`, `db_replication_password`, the username and password for the database user other nodes can connect as to replicate the database. If `db_replication_user` is an empty string, remote replication will be disabled.
* `db_standby`, If true this database node will replicate the database node given by `db_args.host`.
It is recommended that the passwords be changed from the defaults in production.
A database node needs to expose its database on a port. By default this is `5432` but the port exposed to the outside can be changed in the `ports` object.
## Running the wubloader ## Running the wubloader
To start the wubloader, simply run To start the wubloader, simply run
@ -81,7 +68,7 @@ When setting up a database node, a number of database specific options can be se
It is recommended that the passwords be changed from the defaults in production. It is recommended that the passwords be changed from the defaults in production.
A database node needs to expose its database on a port. By default this is `5432` but the port exposed to the outside can be changed in the `ports` object. A database node needs to expose its database on a port. By default this is `5432` but the port exposed to the outside can be changed in the `ports` object.
The `events` table will be automatically populated by the `sheetsync`. The startup script will attempt to populated the `nodes` and `editors` tables from the `nodes.csv` and `editors.csv` files in `segments_path` directory. The expected format for these files is: The `events` table will be automatically populated by the `sheetsync`. If creating a new database, the startup script will attempt to populate the `nodes` and `editors` tables from the `nodes.csv` and `editors.csv` files in `segments_path` directory. The expected format for these files is:
``` ```
nodes.csv nodes.csv
@ -105,3 +92,10 @@ and editors to the database's `editors` table:
`wubloader=> INSERT INTO editors (name, email) VALUES ('example', 'example@gmail.com');` `wubloader=> INSERT INTO editors (name, email) VALUES ('example', 'example@gmail.com');`
### Promoting the standby server
To promote the standby server to primary touch the trigger file in the docker container:
`docker exec wubloader_postgres_1 touch /tmp/touch_to_promote_to_master`
Be careful to prevent the original primary from restarting as another primary.

@ -8,7 +8,7 @@ set -eu
# Pass PUSH=true to also push the resulting images, or PUSH=latest to push them as :latest tag # Pass PUSH=true to also push the resulting images, or PUSH=latest to push them as :latest tag
# The different images we can build # The different images we can build
COMPONENTS=(downloader restreamer backfiller thrimshim cutter sheetsync nginx) COMPONENTS=(downloader restreamer backfiller thrimshim cutter sheetsync nginx postgres)
# Define push if not already defined # Define push if not already defined
PUSH=${PUSH:-} PUSH=${PUSH:-}

@ -502,7 +502,7 @@ def main(dbconnect, youtube_creds_file, name=None, base_dir=".", metrics_port=80
except Exception: except Exception:
delay = common.jitter(10) delay = common.jitter(10)
logging.info('Cannot connect to database. Retrying in {:.0f} s'.format(delay)) logging.info('Cannot connect to database. Retrying in {:.0f} s'.format(delay))
stopping.wait(delay) stop.wait(delay)
youtube_creds = json.load(open(youtube_creds_file)) youtube_creds = json.load(open(youtube_creds_file))
youtube = Youtube( youtube = Youtube(

@ -1,13 +1,16 @@
#! /bin/bash #! /bin/bash
NAME=${1:-postgres} NAME=${1:-postgres}
BUCKET=${2:-wubloader-db}
CONTAINER=$(docker ps --format "{{.Names}}" | grep "$NAME") CONTAINER=$(docker ps --format "{{.Names}}" | grep "$NAME")
if [ -z "$CONTAINER" ] if [ -z "$CONTAINER" ]; then
then echo "Container not found"
echo "Container not found" exit 1
exit 1
fi fi
if [ "$(wc -l <<<"$CONTAINER")" -ne 1 ]; then
echo "Multiple containers found"
exit 1
fi
FILE="wubloader-$(date +%Y-%m-%dT%H:%M:%S).sql" FILE="wubloader-$(date +%Y-%m-%dT%H:%M:%S).sql"
echo "Dumping $CONTAINER to $FILE" echo "Dumping $CONTAINER to $FILE"
docker exec $CONTAINER pg_dump wubloader -U postgres > $FILE docker exec $CONTAINER pg_dump wubloader -U postgres | aws s3 cp - "s3://$BUCKET/$FILE"
aws s3 cp $FILE s3://wubloader-test
rm $FILE

@ -1,25 +1,31 @@
#! /bin/bash #! /bin/bash
if [ -z $1 ] if [ -z "$1" ]; then
then echo "No SQL script file"
echo "No SQL script file" echo "USAGE: $0 SQL_SCRIPT"
exit 1 exit 1
fi fi
NAME=${2:-postgres} NAME=${2:-postgres}
CONTAINER=$(docker ps --format "{{.Names}}" | grep "$NAME") CONTAINER=$(docker ps --format "{{.Names}}" | grep "$NAME")
if [ -z "$CONTAINER" ] if [ -z "$CONTAINER" ]; then
then echo "Container not found"
echo "Container not found" exit 1
exit 1 fi
if [ "$(wc -l <<<"$CONTAINER")" -ne 1 ]; then
echo "Multiple containers found"
exit 1
fi fi
#need to do this in case db dump file has a colon in it #need to do this in case db dump file has a colon in it
cp $1 tmp.sql cp "$1" tmp.sql
docker cp tmp.sql "$CONTAINER:/" docker cp tmp.sql "$CONTAINER:/"
rm tmp.sql rm tmp.sql
docker exec $CONTAINER dropdb wubloader -U postgres --if-exists # this will fail if there are active sessions by users other than 'postgres'
docker exec $CONTAINER createdb wubloader -U postgres # make sure all wubloader components are disconnected
docker exec $CONTAINER psql -d wubloader -f dump.sql -U postgres docker exec "$CONTAINER" dropdb wubloader -U postgres --if-exists
docker exec $CONTAINER rm tmp.sql docker exec "$CONTAINER" createdb wubloader -U postgres
# this assumes that the vst role is in the postgres database
docker exec "$CONTAINER" psql -d wubloader -f tmp.sql -U postgres
docker exec "$CONTAINER" rm tmp.sql

@ -57,6 +57,7 @@
backdoor_port:: 1234, backdoor_port:: 1234,
// Other nodes to always backfill from. You should not include the local node. // Other nodes to always backfill from. You should not include the local node.
// If you are using the database to find peers, you should leave this empty.
peers:: [ peers:: [
], ],
@ -64,7 +65,7 @@
// If database is defined in this config, host and port should be postgres:5432. // If database is defined in this config, host and port should be postgres:5432.
db_args:: { db_args:: {
user: "vst", user: "vst",
password: "dbfh2019", // don't use default in production password: "dbfh2019", // don't use default in production. Must not contain ' or \ as these are not escaped.
host: "postgres", host: "postgres",
port: 5432, port: 5432,
dbname: "wubloader", dbname: "wubloader",
@ -72,9 +73,9 @@
// Other database arguments // Other database arguments
db_super_user:: "postgres", // only accessible from localhost db_super_user:: "postgres", // only accessible from localhost
db_super_password:: "postgres", db_super_password:: "postgres", // Must not contain ' or \ as these are not escaped.
db_replication_user:: "replicate", // if empty, don't allow replication db_replication_user:: "replicate", // if empty, don't allow replication
db_replication_password:: "standby", // don't use default in production db_replication_password:: "standby", // don't use default in production. Must not contain ' or \ as these are not escaped.
db_standby:: false, // set to true to have this database replicate another server db_standby:: false, // set to true to have this database replicate another server
// Path to a JSON file containing google credentials as keys // Path to a JSON file containing google credentials as keys

@ -1,5 +1,5 @@
FROM postgres:latest FROM postgres:latest
COPY postgres/setup.sh /docker-entrypoint-initdb.d/setup.sh COPY postgres/setup.sh /docker-entrypoint-initdb.d/setup.sh
COPY postgres/standby_setup.sh /standby_setup.sh
RUN chmod 0666 /docker-entrypoint-initdb.d/setup.sh RUN chmod 0666 /docker-entrypoint-initdb.d/setup.sh
COPY postgres/standby_setup.sh /standby_setup.sh
RUN chmod 0700 /standby_setup.sh RUN chmod 0700 /standby_setup.sh

@ -2,6 +2,7 @@
set -e set -e
# only allow the $WUBLOADER_USER to connect remotely rather than all users
sed -i "/host all all all/d" "$PGDATA/pg_hba.conf" sed -i "/host all all all/d" "$PGDATA/pg_hba.conf"
echo "host all $WUBLOADER_USER all md5" >> "$PGDATA/pg_hba.conf" echo "host all $WUBLOADER_USER all md5" >> "$PGDATA/pg_hba.conf"
@ -15,6 +16,7 @@ EOSQL
if [ -n "$REPLICATION_USER" ]; then if [ -n "$REPLICATION_USER" ]; then
echo "Creating $REPLICATION_USER" echo "Creating $REPLICATION_USER"
# allow the $REPLICATION user to replicate remotely
echo "host replication $REPLICATION_USER all md5" >> "$PGDATA/pg_hba.conf" echo "host replication $REPLICATION_USER all md5" >> "$PGDATA/pg_hba.conf"
psql -v ON_ERROR_STOP=1 -U postgres <<-EOSQL psql -v ON_ERROR_STOP=1 -U postgres <<-EOSQL
@ -40,7 +42,7 @@ if [ -a /mnt/wubloader/nodes.csv ]; then
url TEXT NOT NULL, url TEXT NOT NULL,
backfill_from BOOLEAN NOT NULL DEFAULT TRUE); backfill_from BOOLEAN NOT NULL DEFAULT TRUE);
COPY nodes FROM '/mnt/wubloader/nodes.csv' DELIMITER ',' CSV HEADER; COPY nodes FROM '/mnt/wubloader/nodes.csv' DELIMITER ',' CSV HEADER;
ALTER TABLE nodes OWNER TO vst; ALTER TABLE nodes OWNER TO $WUBLOADER_USER;
EOF EOF
fi fi
@ -51,7 +53,7 @@ if [ -a /mnt/wubloader/editors.csv ]; then
email TEXT PRIMARY KEY, email TEXT PRIMARY KEY,
name TEXT NOT NULL); name TEXT NOT NULL);
COPY editors FROM '/mnt/wubloader/editors.csv' DELIMITER ',' CSV HEADER; COPY editors FROM '/mnt/wubloader/editors.csv' DELIMITER ',' CSV HEADER;
ALTER TABLE editors OWNER TO vst; ALTER TABLE editors OWNER TO $WUBLOADER_USER;
EOF EOF
fi fi

@ -1,15 +1,17 @@
#! /bin/bash #! /bin/bash
set -e
# if postgres database does not exist in $PGDATA
if [ ! -s "$PGDATA/PG_VERSION" ]; then if [ ! -s "$PGDATA/PG_VERSION" ]; then
# get a binary backup of the database on $MASTER_NODE
pg_basebackup -d "host=$MASTER_NODE password=$REPLICATION_PASSWORD port=5432 user=$REPLICATION_USER" -D ${PGDATA} -vP pg_basebackup -d "host=$MASTER_NODE password=$REPLICATION_PASSWORD port=5432 user=$REPLICATION_USER" -D ${PGDATA} -vP
set -e
cat > ${PGDATA}/recovery.conf <<-EOF cat > ${PGDATA}/recovery.conf <<-EOF
standby_mode = on standby_mode = on
primary_conninfo = 'host=$MASTER_NODE password=$REPLICATION_PASSWORD port=5432 user=$REPLICATION_USER' primary_conninfo = 'host=$MASTER_NODE password=$REPLICATION_PASSWORD port=5432 user=$REPLICATION_USER'
# touch this file to promote this node to master
trigger_file = '/tmp/touch_to_promote_to_master' trigger_file = '/tmp/touch_to_promote_to_master'
EOF EOF
@ -18,4 +20,5 @@ if [ ! -s "$PGDATA/PG_VERSION" ]; then
fi fi
# start postgres
gosu postgres postgres gosu postgres postgres

@ -281,14 +281,13 @@ def main(dbconnect, sheets_creds_file, edit_url, bustime_start, sheet_id, worksh
logging.info("Starting up") logging.info("Starting up")
dbmanager = None dbmanager = None
stopping = gevent.event.Event()
while dbmanager is None: while dbmanager is None:
try: try:
dbmanager = DBManager(dsn=dbconnect) dbmanager = DBManager(dsn=dbconnect)
except Exception: except Exception:
delay = common.jitter(10) delay = common.jitter(10)
logging.info('Cannot connect to database. Retrying in {:.0f} s'.format(delay)) logging.info('Cannot connect to database. Retrying in {:.0f} s'.format(delay))
stopping.wait(delay) stop.wait(delay)
sheets_creds = json.load(open(sheets_creds_file)) sheets_creds = json.load(open(sheets_creds_file))

@ -2,6 +2,7 @@ import datetime
import json import json
import logging import logging
import signal import signal
import sys
import argh import argh
import flask import flask
@ -221,21 +222,27 @@ def main(connection_string, host='0.0.0.0', port=8004, backdoor_port=0):
"""Thrimshim service.""" """Thrimshim service."""
server = WSGIServer((host, port), cors(app)) server = WSGIServer((host, port), cors(app))
app.no_authentication = no_authentication
app.db_manager = None
stopping = gevent.event.Event() stopping = gevent.event.Event()
while app.db_manager is None: def stop():
logging.info("Shutting down")
stopping.set()
# handle when the server is running
if hasattr(server, 'socket'):
server.stop()
# and when not
else:
sys.exit()
gevent.signal(signal.SIGTERM, stop)
app.db_manager = None
while app.db_manager is None and not stopping.is_set():
try: try:
app.db_manager = database.DBManager(dsn=connection_string) app.db_manager = database.DBManager(dsn=connection_string)
except Exception: except Exception:
delay = common.jitter(10) delay = common.jitter(10)
logging.info('Cannot connect to database. Retrying in {:.0f} s'.format(delay)) logging.info('Cannot connect to database. Retrying in {:.0f} s'.format(delay))
stopping.wait(delay) stopping.wait(delay)
def stop():
logging.info("Shutting down")
server.stop()
gevent.signal(signal.SIGTERM, stop)
common.PromLogCountsHandler.install() common.PromLogCountsHandler.install()
common.install_stacksampler() common.install_stacksampler()

Loading…
Cancel
Save