ArchiveBot Redis layout

The ArchiveBot pipelines and backend share a single Redis database. This document describes the keys in that database.

Keys do not follow any namespace-prefixing convention; ArchiveBot assumes it has full control over the database.

Connection

Pipelines connect directly to the Redis database, typically over SSH or spiped. The backend connects the same way. There is no access control from either side.

pipeline:PIPELINE_ID

Type: hash

Keys matching this form describe pipelines. PIPELINE_ID is a hexadecimal number that is generated by a pipeline process on startup. The pipeline process periodically updates its data while it runs.

Hash keys

Key Intended type Meaning
disk_usage Decimal % of the pipeline’s filesystem in use
disk_available Integer Bytes available on the pipeline’s filesystem
fqdn String FQDN of the host running the pipeline
hostname String Short name of the host
id String The pipeline’s ID; always matches the hash key
load_average_1m Decimal Load average over the past minute
load_average_5m Decimal “” “” “” “” over the past 5 minutes
load_average_15m Decimal “” “” “” “” over the past 15 minutes
mem_available Integer Bytes of memory available on the host
mem_usage Decimal % memory in use on the host
nickname String The pipeline nickname
pid Integer The PID of the pipeline process
python String The version of Python running the pipeline
ts UNIX timestamp The last time this pipeline record was updated
version String The pipeline’s version

IDENT (i.e. [a-z0-9]{25,})

Type: hash

These are job records. These are the most common record type in ArchiveBot’s database.

Parts of this record are frequently modified by both the backend and pipeline:

  • whenever a response is recorded
  • whenever job settings are changed

Hash keys

Key Intended type Meaning
bytes_downloaded Integer Bytes downloaded from the target site
concurrency Integer Current number of concurrent downloaders
death_timer Integer Number of liveness checks that have gone without a response
delay_max Integer Maximum delay between two requests on a downloader in ms
delay_min Integer Minimum delay between two requests on a downloader in ms
error_count Integer Number of error (i.e. 4xx, 5xx) responses encountered
fetch_depth String “shallow” for !ao jobs; “inf” for !a jobs
finished_at UNIX ts w/ frac When the job finished; not present if the job is running
heartbeat Integer Set by the pipeline; incremented once per heartbeat
grabber String “phantomjs” for PhantomJS jobs; omitted otherwise
ignore_patterns_set_key String The key storing this job’s ignore patterns
items_downloaded Integer Number of 2xx/3xx responses
items_queued Integer Number of URLs encountered in the job
last_acknowledged_heartbeat Integer Set by the backend; is the last heartbeat received
last_analyzed_log_entry Integer The last log entry index analyzed by the backend [1]
last_broadcasted_log_entry Integer “” “” “” “” “” “” “” “” broadcasted over the firehose [1]
last_trimmed_log_entry Integer “” “” “” “” “” “” “” “” trimmed by the log trimmer [1]
log_key String The key storing this job’s log messages
log_score Integer The current log entry index
next_watermark Integer A threshold for number of queued URLs; currently unused
no_phantomjs_smart_scroll Boolean Whether or not PhantomJS’ smart-scroll should be used
phantomjs_scroll Integer Maximum number of times to scroll the page with PhantomJS
phantomjs_wait Integer Maximum wait time between PhantomJS page interactions
pipeline_id String The pipeline running this job; corresponds to a pipeline:* key
queued_at UNIX ts w/ frac When this job was queued
r1xx Integer Number of 1xx responses
r2xx Integer “” “” “” 2xx responses
r3xx Integer “” “” “” 3xx responses
r4xx Integer “” “” “” 4xx responses
r5xx Integer “” “” “” 5xx responses
runk Integer “” “” “” responses with unknown HTTP status code
recorded_at UNIX ts w/ frac Deprecated. When this job was logged to ArchiveBot’s CouchDB
settings_age Integer Job settings version; incremented for each settings change
slug String WARC/JSON base filename [2]
started_at UNIX ts w/ frac When this job was started by a pipeline
started_by String The user (typically an IRC nick) that submitted the job
started_in String Where the job was started (typically an IRC channel)
suppress_ignore_reports Boolean Whether ignore pattern matches should be reported
ts UNIX ts w/ frac Last update received from a pipeline for this job
url String The URL for this job: either the target or a URL file (for !ao < and !a <)
user_agent String The user-agent to spoof; null if we should use the default agent

[1]: The expected relationship between these values is

last_analyzed_log_entry <= last_broadcasted_log_entry <= last_trimmed_log_entry

[2]: Usually looks like “twitter.com-inf”. The date, time, WARC sequence, extension, etc. are all appended by the pipeline.

IDENT_ignores

Type: set

Ignore patterns for the identified job. Each ignore pattern is a Python regex.

IDENT_log

Type: zset

Log entries generated for a job by the wpull hooks or pipeline stdout capture are sent here. The backend is notified of new entries in this set when the pipeline publishes the job ident on the updates channel.

pipelines

Type: list

Deprecated. This list contains pipeline names, and is still modified by pipelines, but no pipeline listing uses it.

jobs_completed, jobs_aborted, jobs_failed

Type: string

These keys store counts of completed, aborted, and failed jobs, respectively.

A completed job is a job that made it through the entire ArchiveBot pipeline. An aborted job is a job that was terminated using !abort. A failed job is a job that crashed and was reaped using the internal console.

tweets:done, tweets:queue

Type: zset

These are used by ArchiveBot’s Twitter tweeter. They store tweets that were tweeted and tweets in the to-post queue, respectively.

Pubsub channels

updates

Whenever a pipeline has new log entries for a job, it publishes that job’s ident to this channel.

archivebot:job:IDENT

There exists one of these channels per job.

When settings are updated for that job, the new settings age is published via this channel. The job’s settings listener receives the new version. If the new version is greater than the current version, the new settings are read from Redis and applied.