Mike Lang
30c1877b36
Fix bugs with chat_archiver
...
* Order of arguments matters - option can't be between other positional args and *args
* Messed up the count structure
11 months ago
Mike Lang
260293d40d
chat_archiver: Allow passing multiple channels on CLI
11 months ago
Mike Lang
91910c0972
chat_archiver: Fix misconfiguration where MAX_SERVER_LAG < MAX_DELAY
...
This leads to delayed JOIN/PARTs not being put in their proper batch
as it's already been closed. In fact, since each message is re-opening a batch from
more than MAX_SERVER_LAG seconds ago, each message becomes one batch.
11 months ago
Mike Lang
200d2df9ba
chat_archiver: Add code support for archiving multiple channels at once
...
* Join every channel on connect
* Move the "wait for initial ROOMSTATE" logic into the main loop and make it per-channel
* Make batch keys (channel, time) instead of just time
For now the CLI doesn't actually allow you to run chat_archiver in this mode,
it always calls the implementation with a 1-element list of channels.
11 months ago
Mike Lang
b050b71036
chat_archiver: Improve logging and monitoring by using unique client name more
11 months ago
Mike Lang
044dfb8084
Pin argh to avoid stupid breaking changes
11 months ago
Mike Lang
76c9208be5
Move chat_archiver atomic_write() to common for re-use
12 months ago
Mike Lang
30d5ccc483
Fix all old references to github.com/ekimekim/wubloader
1 year ago
Mike Lang
ad4827237f
Fix bug in checking if message has tags
2 years ago
Mike Lang
681da9a76e
Fix a bug where we try to fetch emote "" for messages with no emotes
2 years ago
Mike Lang
3b6ce86c46
chat archiver: Add cli tool for downloading emotes
2 years ago
Mike Lang
e74d655ce5
chat_archiver: Download each seen emote
...
so we have a permanent record, in case they're deleted or changed later
2 years ago
Mike Lang
8e314eea94
Collect metrics for chat_archiver on port 8008
2 years ago
Mike Lang
08257386e2
Add restreamer endpoint for viewing chat messages
2 years ago
Mike Lang
9320251de7
Some extra documentation on chat_archiver
2 years ago
Mike Lang
d8a9b5ddf0
chat_archiver: Always sort json object keys to ensure canonical output
2 years ago
Mike Lang
651658e507
JOINs and PARTs have been observed with up to 30sec difference
...
it turns out to be completely undocumented what the max delay is. so let's assume 45s.
anything > 60s might cause problems due to matching messages being more than 1 batch apart.
2 years ago
Mike Lang
05a989f67d
chat-archiver: fixes
2 years ago
Mike Lang
c1c1c11bce
chat_archiver: Add prometheus metrics
2 years ago
Mike Lang
c25d79e7c2
chat-archiver: Merge all files every minute
2 years ago
Mike Lang
4cfc362f76
chat-archiver: pass in node name
...
instead of using container hostname
2 years ago
Mike Lang
a48beab576
chat-archiver: update girc for py3 support and fixes
2 years ago
Mike Lang
315c9c8297
Integrate chat archiver as a proper component
2 years ago
Mike Lang
f8b3ace148
Backfill chat archives under the "chat" quality
2 years ago
Mike Lang
05ddd39504
chat_archiver: Split files into directories by hour
...
matching how we handle video files
2 years ago
Mike Lang
1d626738bd
chat_archiver: Start a new client on RECONNECT
2 years ago
Mike Lang
96cc212bf0
chat_archiver: fixes, implement merge_all
2 years ago
Mike Lang
d32cbbb7e1
chat-archiver: File merging and other fixes
2 years ago
Mike Lang
0756539b85
chat-archiver: Early work and basic archival
2 years ago