wubloader

Commit Graph

Select branches

Hide Pull Requests

HubbeKing-multiplatform-builds

archive/original-v2.1

bus-post-processing

bus-progress-ui

chrusher-playlist-thumbnails

chrusher-thumbnail-frontend

chrusher/backfiller_error_handler

chrusher/bus_synthesizer

condor-scripts

downloader/follow-game

gunner/thrimbletrimmer-improvements

master

mike/backdoor

mike/backfill-extras

mike/bad-segment-hack

mike/chat-no-end-time

mike/chat-subs

mike/common/split-up

mike/cropping

mike/database/initial

mike/debug-playlists

mike/download-frame-2

mike/downloader/conn-pooling

mike/downloader/handle-404

mike/encode-worker

mike/faster-chat-merge

mike/generate-videos

mike/non-restreamer-stats

mike/playlist-debug

mike/prizebot

mike/render-dashboard

mike/shorts

mike/testing

mlang/suspicious-skew

sheet-updates

suspect-segments

thrimbletrimmer-solid

#1

#10

#100

#101

#102

#103

#107

#109

#11

#113

#114

#115

#116

#118

#119

#12

#120

#128

#129

#13

#132

#133

#134

#136

#137

#138

#139

#14

#140

#142

#143

#144

#146

#147

#148

#149

#15

#150

#151

#152

#153

#154

#155

#156

#157

#158

#159

#160

#163

#17

#173

#174

#175

#176

#177

#178

#18

#180

#181

#182

#183

#184

#185

#186

#187

#188

#189

#19

#190

#191

#192

#193

#194

#195

#196

#197

#198

#199

#2

#20

#200

#201

#202

#203

#204

#205

#206

#207

#21

#210

#211

#212

#213

#22

#222

#223

#224

#225

#227

#229

#23

#230

#231

#233

#234

#235

#236

#237

#239

#24

#240

#242

#243

#244

#245

#246

#247

#248

#249

#25

#250

#253

#254

#255

#256

#257

#258

#26

#260

#261

#262

#263

#264

#265

#266

#267

#268

#269

#27

#270

#28

#288

#289

#29

#290

#291

#292

#294

#295

#297

#298

#299

#3

#30

#300

#302

#304

#305

#306

#307

#308

#309

#31

#310

#311

#312

#313

#314

#315

#315

#316

#317

#318

#319

#32

#320

#321

#322

#323

#324

#327

#328

#329

#33

#330

#331

#332

#334

#335

#336

#337

#34

#340

#342

#343

#344

#347

#349

#35

#350

#352

#353

#354

#355

#356

#357

#36

#360

#361

#362

#363

#367

#369

#37

#370

#373

#374

#375

#38

#388

#388

#389

#39

#390

#391

#392

#396

#397

#4

#40

#400

#401

#407

#408

#409

#41

#410

#413

#414

#415

#416

#417

#418

#419

#42

#423

#425

#426

#427

#428

#43

#432

#433

#434

#435

#436

#437

#438

#439

#44

#440

#441

#441

#443

#444

#447

#448

#449

#45

#452

#453

#454

#455

#456

#456

#457

#459

#46

#460

#461

#462

#463

#464

#464

#465

#465

#466

#467

#468

#47

#471

#472

#476

#478

#48

#480

#481

#483

#483

#484

#487

#487

#488

#49

#495

#495

#5

#50

#51

#519

#52

#520

#522

#523

#523

#524

#525

#526

#526

#53

#54

#540

#540

#55

#56

#57

#58

#59

#6

#60

#61

#62

#63

#64

#65

#66

#67

#68

#69

#70

#71

#72

#73

#74

#75

#76

#77

#78

#79

#87

#89

#93

#94

#96

#97

81aee0ee1e Increase hard timeout for getting segment headers Mike Lang 2019-01-10 10:53:11 -0800
6077f99007 downloader: Stop retrying in SegmentGetter after a long timeout Mike Lang 2019-01-10 11:15:40 -0800
26b60de5e3 Increase hard timeout for getting segment headers Mike Lang 2019-01-10 10:53:11 -0800
787b9002ab restreamer: Use correct name for dateutil Mike Lang 2019-01-09 02:26:15 -0800
3a1e4b0aef restreamer: Fix missing dependency Mike Lang 2019-01-09 02:19:05 -0800
997c1242b2 get_best_segments: Let other things run Mike Lang 2019-01-08 00:49:49 -0800
bf08aa29b8 parse_segment_path: Use datetime.strptime instead of dateutil.parser Mike Lang 2019-01-08 00:24:21 -0800
0445f566d4 get_best_segments: Let other things run Mike Lang 2019-01-08 00:49:49 -0800
ebabda3fb6 parse_segment_path: Use datetime.strptime instead of dateutil.parser Mike Lang 2019-01-08 00:24:21 -0800
bcdb268ce8 Also need to replace locks on the counter float values to prevent deadlocks Mike Lang 2019-01-07 12:39:43 -0800
10cca18922 Fix a deadlock due to signal interactions with prometheus client Mike Lang 2019-01-07 04:12:46 -0800
c9cc8a73a7 generate-flamegraph: Script to create a flamegraph by querying prometheus Mike Lang 2019-01-07 02:29:26 -0800
b75b9a9b00 Add stacksampler to all services Mike Lang 2019-01-07 01:31:30 -0800
b9c2921242 common.stats: Add a stacksampler that records sampled stacks to prometheus Mike Lang 2019-01-07 01:28:10 -0800
8b0ecd6246 Also need to replace locks on the counter float values to prevent deadlocks Mike Lang 2019-01-07 12:39:43 -0800
f11325117c Fix a deadlock due to signal interactions with prometheus client Mike Lang 2019-01-07 04:12:46 -0800
75b9bc195b generate-flamegraph: Script to create a flamegraph by querying prometheus Mike Lang 2019-01-07 02:29:26 -0800
993318b503 Add stacksampler to all services Mike Lang 2019-01-07 01:31:30 -0800
6bb46c582d common.stats: Add a stacksampler that records sampled stacks to prometheus Mike Lang 2019-01-07 01:28:10 -0800
a5213ccb3b downloader: Pool connections when we can Mike Lang 2019-01-08 08:18:41 -0800
5175b099af common: Split segment-related stuff into its own module Mike Lang 2019-01-07 00:15:25 -0800
6f84a23ba6 common: Split stats-related stuff into its own module Mike Lang 2019-01-07 00:15:25 -0800
8fe2fec958 common: convert from module to package Mike Lang 2019-01-07 00:11:13 -0800
96c1566d21

Merge pull request #34 from ekimekim/gunner/restreamer/additional-routes MasterGunner 2019-01-08 19:00:35 -0400
a9569d9e96 Removed unneeded '@has_path_args'. MasterGunner 2019-01-08 18:33:09 -0400
306ac53d08 Added additional routes for listing available streams and variants. MasterGunner 2019-01-08 18:17:46 -0400
25bc29c8c8 downloader: Pool connections when we can mike/downloader/conn-pooling Mike Lang 2019-01-08 08:18:41 -0800
cb929c12b3 common: Split segment-related stuff into its own module mike/common/split-up Mike Lang 2019-01-07 00:15:25 -0800
3edc27cfe6 common: Split stats-related stuff into its own module Mike Lang 2019-01-07 00:15:25 -0800
84b611055b common: convert from module to package Mike Lang 2019-01-07 00:11:13 -0800
901cda4814 Enable backdoor in all services, and add telnet to containers Mike Lang 2019-01-06 22:44:35 -0800
9af7795f34 Add gevent.backdoor as an optional arg to all services Mike Lang 2019-01-06 22:35:30 -0800
47ff92b155 downloader: Fix bug where mark_working wasn't called Mike Lang 2019-01-06 22:15:41 -0800
3042d00516 downloader: Give up on 404 in addition to 403 Mike Lang 2019-01-06 22:02:15 -0800
df73225a5b Enable backdoor in all services, and add telnet to containers mike/backdoor Mike Lang 2019-01-06 22:44:35 -0800
f9aa6ef0e4 Add gevent.backdoor as an optional arg to all services Mike Lang 2019-01-06 22:35:30 -0800
b34caaf208 downloader: Fix bug where mark_working wasn't called mike/downloader/handle-404 Mike Lang 2019-01-06 22:15:41 -0800
80773f130a downloader: Give up on 404 in addition to 403 Mike Lang 2019-01-06 22:02:15 -0800
7f9a1dbe45 downloader: Remove implicit source quality arg Mike Lang 2019-01-06 08:45:43 -0800
89d6b3a6be docker-compose: Add list of peers to backfill from Mike Lang 2019-01-06 06:24:01 -0800
0d627715f3 downloader: Track number of downloaded segments Mike Lang 2019-01-06 05:52:03 -0800
90ccc6d827 backfiller: Track number of successful backfills Mike Lang 2019-01-06 05:43:08 -0800
c59892e148 backfiller: Add ability to set nodes as CLI arg Mike Lang 2019-01-06 05:40:00 -0800
bdcb217d20 docker-compose: Expose metrics ports for other services Mike Lang 2019-01-06 02:14:08 -0800
b4b315b6bc Expose prometheus metrics for backfiller and downloader Mike Lang 2019-01-06 01:55:52 -0800
d90f01b8ce common: Create general function for timing things, and use it to time get_best_segments Mike Lang 2019-01-06 01:50:14 -0800
b0ded641c3 Add a logging handler which counts logs for prometheus stats Mike Lang 2019-01-06 00:14:39 -0800
261c57dc76 downloader: Remove implicit source quality arg mike/non-restreamer-stats Mike Lang 2019-01-06 08:45:43 -0800
00540fac9d docker-compose: Add list of peers to backfill from Mike Lang 2019-01-06 06:24:01 -0800
446befd543 downloader: Track number of downloaded segments Mike Lang 2019-01-06 05:52:03 -0800
4d3fd0815e backfiller: Track number of successful backfills Mike Lang 2019-01-06 05:43:08 -0800
27d7dcf6f4 backfiller: Add ability to set nodes as CLI arg Mike Lang 2019-01-06 05:40:00 -0800
91b85eb99a docker-compose: Expose metrics ports for other services Mike Lang 2019-01-06 02:14:08 -0800
47d8dbc86b Expose prometheus metrics for backfiller and downloader Mike Lang 2019-01-06 01:55:52 -0800
09cf238bb9 common: Create general function for timing things, and use it to time get_best_segments Mike Lang 2019-01-06 01:50:14 -0800
87978db8eb Add a logging handler which counts logs for prometheus stats Mike Lang 2019-01-06 00:14:39 -0800
c9d02b3318 restreamer: Prevent prom client blowing up after two different endpoints are hit Mike Lang 2019-01-05 17:56:27 -0800
7914210ea4 restreamer: Prevent prom client blowing up after two different endpoints are hit Mike Lang 2019-01-05 17:56:27 -0800
30c4bbec1d restreamer: return the actual response from after_request even if untracked Mike Lang 2019-01-05 17:03:34 -0800
96e6904c85 Added monotonic to restreamer setup.py Christopher Usher 2019-01-05 02:04:51 -0800
1a3a511051

Added monotonic to restreamer setup.py Christopher Usher 2019-01-05 02:04:51 -0800
225288980a Added the backfiller to docker-compose Christopher Usher 2019-01-05 00:37:33 -0800
f9f2937bc8 Added the backfiller to docker-compose Christopher Usher 2019-01-05 00:37:33 -0800
3fcd374449 Moved encode_strings to common Christopher Usher 2019-01-04 00:03:30 -0800
93dd216f89 Fixes and suggestions from ekimekim Christopher Usher 2019-01-03 21:25:43 -0800
db1b4e6539 Updated logging to match the other components Christopher Usher 2019-01-03 21:02:48 -0800
bae039977b trying getting the backfiller to actually start Christopher Usher 2019-01-01 20:22:14 -0800
1fcd9b5b36 Adding in stuff to hopefully get this to run Christopher Usher 2019-01-01 19:52:27 -0800
013ad65c68 added a Dockerfile for the backfiller Christopher Usher 2019-01-01 19:30:16 -0800
48d11045d4 Change to backfiller.main to backfill the last 3 hours on start up before doing a full backfill Christopher Usher 2019-01-01 16:26:25 -0800
176633bf7d More messing around with backfill_node to allow finer grained control of order segments are fetched Christopher Usher 2019-01-01 16:12:38 -0800
3a7624b107 added a setup file for the backfiller Christopher Usher 2019-01-01 15:04:25 -0800
ba499fe835 added more logging to backfiller Christopher Usher 2019-01-01 14:47:01 -0800
34372aa7dd

Merge branch 'master' into chrusher/backfiller Christopher Usher 2019-01-04 23:18:47 -0800
7525b7c135 restreamer: Add basic prometheus stats to all endpoints Mike Lang 2019-01-03 21:10:23 -0800
d0e2ce6ae1 restreamer: Add basic prometheus stats to all endpoints Mike Lang 2019-01-03 21:10:23 -0800
747faf6f91 Moved encode_strings to common Christopher Usher 2019-01-04 00:03:30 -0800
7abff26c5d Fixes and suggestions from ekimekim Christopher Usher 2019-01-03 21:25:43 -0800
eeaa4f8d6c Updated logging to match the other components Christopher Usher 2019-01-03 21:02:48 -0800
726b2ec0ff trying getting the backfiller to actually start Christopher Usher 2019-01-01 20:22:14 -0800
22ccd927cf Adding in stuff to hopefully get this to run Christopher Usher 2019-01-01 19:52:27 -0800
fc03f965ca added a Dockerfile for the backfiller Christopher Usher 2019-01-01 19:30:16 -0800
9d69afcfb3 Change to backfiller.main to backfill the last 3 hours on start up before doing a full backfill Christopher Usher 2019-01-01 16:26:25 -0800
54679383df More messing around with backfill_node to allow finer grained control of order segments are fetched Christopher Usher 2019-01-01 16:12:38 -0800
6379d5f71d added a setup file for the backfiller Christopher Usher 2019-01-01 15:04:25 -0800
bcdcc8d634 added more logging to backfiller Christopher Usher 2019-01-01 14:47:01 -0800
f20a779756 Fix some bugs and linter errors introduced by backfiller Mike Lang 2019-01-01 14:27:08 -0800
17972b87aa Allow setting of log level via WUBLOADER_LOG_LEVEL env var Mike Lang 2019-01-03 19:28:45 -0800
c0357680cf downloader: Use caller's logger inside soft_hard_timeout Mike Lang 2019-01-03 19:18:43 -0800
a628676e74 downloader: Log to subloggers instead of the root logger Mike Lang 2019-01-03 19:16:29 -0800
692cdbfa4b Allow setting of log level via WUBLOADER_LOG_LEVEL env var Mike Lang 2019-01-03 19:28:45 -0800
05b33fbe11 downloader: Use caller's logger inside soft_hard_timeout Mike Lang 2019-01-03 19:18:43 -0800
16aed99fdf downloader: Log to subloggers instead of the root logger Mike Lang 2019-01-03 19:16:29 -0800
57e665df2e generate-docker-compose: Clean up the container afterwards Mike Lang 2019-01-03 17:59:39 -0800
c8cc4a68a0 cutter: Fix bugs that meant things wouldn't actually be cut Mike Lang 2019-01-02 07:13:27 -0800
6bf709287a cutter: Introduce an alternate cutting approach that is much faster Mike Lang 2019-01-02 06:36:27 -0800
22df6f0b45 cutter: Fix bugs that meant things wouldn't actually be cut Mike Lang 2019-01-02 07:13:27 -0800
4c49722d95 cutter: Introduce an alternate cutting approach that is much faster Mike Lang 2019-01-02 06:36:27 -0800
6815924097 Fix some bugs and linter errors introduced by backfiller Mike Lang 2019-01-01 14:27:08 -0800
70d1bc466c Fix some bugs and linter errors introduced by backfiller Mike Lang 2019-01-01 14:27:08 -0800