mirror of
https://github.com/ihabunek/twitch-dl
synced 2024-08-30 18:32:25 +00:00
Compare commits
26 Commits
Author | SHA1 | Date | |
---|---|---|---|
838611b834 | |||
cf8d13e80e | |||
6108b15587 | |||
bb2de4af96 | |||
678bf11a8a | |||
69de08e2ba | |||
1cb4598a71 | |||
02e4cdaff6 | |||
c2e9ab9382 | |||
739460bed1 | |||
c92ba5e183 | |||
f4ab045cdc | |||
cdc0495cf1 | |||
c510d5aae4 | |||
2abc56213e | |||
bb16108572 | |||
b982cba566 | |||
f4d442c118 | |||
eecd098f18 | |||
041689bee9 | |||
a245ffb6a4 | |||
ac37f179ef | |||
5b200a2cb7 | |||
bf2a4558f4 | |||
772faa5901 | |||
04ddadef26 |
2
.gitignore
vendored
2
.gitignore
vendored
@ -11,3 +11,5 @@ tmp/
|
||||
/htmlcov
|
||||
/twitch-dl-*.tar.gz
|
||||
/twitch-dl.1.man
|
||||
/bundle
|
||||
/*.pyz
|
||||
|
31
CHANGELOG.md
31
CHANGELOG.md
@ -1,6 +1,37 @@
|
||||
Twitch Downloader change log
|
||||
============================
|
||||
|
||||
1.13.1 (2020-11-23)
|
||||
-------------------
|
||||
|
||||
* Fixed clip download issue (#45)
|
||||
|
||||
1.13.0 (2020-11-10)
|
||||
-------------------
|
||||
|
||||
* Added `clips` command for listing and batch downloading clips (#26)
|
||||
|
||||
1.12.1 (2020-09-29)
|
||||
-------------------
|
||||
|
||||
* Fix bug introduced in previous version which broke joining
|
||||
|
||||
1.12.0 (2020-09-29)
|
||||
-------------------
|
||||
|
||||
* Added `source` as alias for best available quality (#33)
|
||||
* Added `--no-join` option to `download` to skip ffmpeg join (#36)
|
||||
* Added `--overwrite` option to `download` to overwrite target without prompting
|
||||
for confirmation (#37)
|
||||
* Added `--pager` option to `videos`, don't page by default (#30)
|
||||
|
||||
1.11.0 (2020-09-03)
|
||||
-------------------
|
||||
|
||||
* Make downloading more robust, fixes issues with some VODs (#35)
|
||||
* Bundle twitch-dl to a standalone archive, simplifying installation, see
|
||||
installation instructions in README
|
||||
|
||||
1.10.2 (2020-08-11)
|
||||
-------------------
|
||||
|
||||
|
13
Makefile
13
Makefile
@ -6,7 +6,18 @@ dist :
|
||||
|
||||
clean :
|
||||
find . -name "*pyc" | xargs rm -rf $1
|
||||
rm -rf build dist MANIFEST htmlcov deb_dist twitch-dl*.tar.gz twitch-dl.1.man
|
||||
rm -rf build dist bundle MANIFEST htmlcov deb_dist twitch-dl*.tar.gz twitch-dl.1.man
|
||||
|
||||
bundle:
|
||||
mkdir bundle
|
||||
cp twitchdl/__main__.py bundle
|
||||
pip install . --target=bundle
|
||||
rm -rf bundle/*.dist-info
|
||||
find bundle/ -type d -name "__pycache__" -exec rm -rf {} +
|
||||
python -m zipapp \
|
||||
--python "/usr/bin/env python3" \
|
||||
--output twitch-dl.`git describe`.pyz bundle \
|
||||
--compress
|
||||
|
||||
publish :
|
||||
twine upload dist/*.tar.gz dist/*.whl
|
||||
|
129
README.md
129
README.md
@ -1,10 +1,10 @@
|
||||
Twitch Downloader
|
||||
=================
|
||||
|
||||
A simple CLI tool for downloading videos from Twitch.
|
||||
CLI tool for downloading videos from twitch.tv
|
||||
|
||||
Inspired by youtube-dl but improves upon it by using multiple concurrent
|
||||
connections to make the download faster.
|
||||
Inspired by [youtube-dl](https://youtube-dl.org/) but improves upon it by using
|
||||
multiple concurrent connections to make the download faster.
|
||||
|
||||
Resources
|
||||
---------
|
||||
@ -17,11 +17,70 @@ Requirements
|
||||
------------
|
||||
|
||||
* Python 3.5+
|
||||
* [ffmpeg](https://ffmpeg.org/) must be installed and in the path
|
||||
* [ffmpeg](https://ffmpeg.org/download.html), installed and on the system path
|
||||
|
||||
Installation
|
||||
------------
|
||||
|
||||
### Download standalone archive
|
||||
|
||||
Go to the [latest release](https://github.com/ihabunek/twitch-dl/releases/latest)
|
||||
and download the `twitch-dl.<version>.pyz` archive.
|
||||
|
||||
Run the archive by either:
|
||||
|
||||
a) passing it to python:
|
||||
|
||||
```
|
||||
python3 twitch-dl.1.13.0.pyz --help
|
||||
```
|
||||
|
||||
b) making it executable and invoking it directly (linux specific):
|
||||
|
||||
```
|
||||
chmod +x twitch-dl.1.13.0.pyz
|
||||
./twitch-dl.1.13.0.pyz --help
|
||||
```
|
||||
|
||||
Feel free to rename the archive to something more managable, like `twitch-dl`.
|
||||
|
||||
To upgrade to a newer version, repeat the process with the newer release.
|
||||
|
||||
### From PYPI using pipx
|
||||
|
||||
**pipx** is a tool which installs python apps into isolated environments, which
|
||||
prevents all kinds of problems later so it's the suggested way to install
|
||||
twitch-dl from PYPI.
|
||||
|
||||
Install pipx as described in
|
||||
[pipx install docs](https://pipxproject.github.io/pipx/installation/).
|
||||
|
||||
Install twitch-dl:
|
||||
|
||||
```
|
||||
pipx install twitch-dl
|
||||
```
|
||||
|
||||
Check installation worked:
|
||||
|
||||
```
|
||||
twitch-dl --help
|
||||
```
|
||||
|
||||
If twitch-dl executable is not found, check that the pipx binary location (by
|
||||
default `~/.local/bin`) is in your PATH.
|
||||
|
||||
To upgrade twitch-dl to the latest version:
|
||||
|
||||
```
|
||||
pipx install twitch-dl
|
||||
```
|
||||
|
||||
Usage
|
||||
-----
|
||||
|
||||
### Listing videos
|
||||
|
||||
List recent streams for a given channel:
|
||||
|
||||
```
|
||||
@ -55,6 +114,8 @@ Use the `--game` option to specify one or more games to show:
|
||||
twitch-dl videos --game "doom eternal" --game "cave story" bananasaurus_rex
|
||||
```
|
||||
|
||||
### Downloading videos
|
||||
|
||||
Download a stream by ID or URL:
|
||||
|
||||
```
|
||||
@ -68,6 +129,32 @@ Specify video quality to download:
|
||||
twitch-dl download -q 720p 221837124
|
||||
```
|
||||
|
||||
Setting quality to `source` will download the best available quality:
|
||||
|
||||
```
|
||||
twitch-dl download -q source 221837124
|
||||
```
|
||||
|
||||
### Listing clips
|
||||
|
||||
List clips for the given period:
|
||||
|
||||
```
|
||||
twitch-dl clips bananasaurus_rex --period last_week
|
||||
```
|
||||
|
||||
Supported periods are: `last_day`, `last_week`, `last_month`, `all_time`.
|
||||
|
||||
For listing a large number of clips, it's nice to page them:
|
||||
|
||||
```
|
||||
twitch-dl clips bananasaurus_rex --period all_time --limit 10 --pager
|
||||
```
|
||||
|
||||
This will show 10 clips at a time and ask to continue.
|
||||
|
||||
### Downloading clips
|
||||
|
||||
Download a clip by slug or URL:
|
||||
|
||||
```
|
||||
@ -83,6 +170,38 @@ twitch-dl download -q 720 VenomousTameWormHumbleLife
|
||||
|
||||
Note that twitch names for clip qualities have no trailing "p".
|
||||
|
||||
### Batch downloading clips
|
||||
|
||||
It's possible to download all clips for a given period:
|
||||
|
||||
```
|
||||
twitch-dl clips bananasaurus_rex --period last_week --download
|
||||
```
|
||||
|
||||
Clips are downloaded in source quality.
|
||||
|
||||
A note about clips
|
||||
------------------
|
||||
|
||||
Currently it doesn't seem to be possible to get a list of clips ordered by time
|
||||
of creation, only by view count. Clips with the same view count seem to be
|
||||
returned in random order. This can break paging resulting in duplicate clips
|
||||
listed or clips missed.
|
||||
|
||||
When batch downloading a large number of clips (over 100), it's possible that
|
||||
some will be missed.
|
||||
|
||||
Temporary files
|
||||
---------------
|
||||
|
||||
By default, twitch-dl will download VODs to your systems temp dir (e.g. `/tmp/`
|
||||
on Linux). To change the location where the files are downloaded you can set
|
||||
the `TMP` environment variable, e.g.
|
||||
|
||||
```
|
||||
TMP=/my/tmp/path/ twitch-dl download 221837124
|
||||
```
|
||||
|
||||
Man page
|
||||
--------
|
||||
|
||||
@ -97,6 +216,6 @@ make man
|
||||
License
|
||||
-------
|
||||
|
||||
Copyright 2018 Ivan Habunek <ivan@habunek.com>
|
||||
Copyright 2018-2020 Ivan Habunek <ivan@habunek.com>
|
||||
|
||||
Licensed under the GPLv3: http://www.gnu.org/licenses/gpl-3.0.html
|
||||
|
2
setup.py
2
setup.py
@ -11,7 +11,7 @@ makes it faster.
|
||||
|
||||
setup(
|
||||
name='twitch-dl',
|
||||
version='1.10.2',
|
||||
version='1.13.1',
|
||||
description='Twitch downloader',
|
||||
long_description=long_description.strip(),
|
||||
author='Ivan Habunek',
|
||||
|
@ -1,3 +1,3 @@
|
||||
__version__ = "1.10.2"
|
||||
__version__ = "1.13.1"
|
||||
|
||||
CLIENT_ID = "kimne78kx3ncx6brgo4mv6wki5h1ko"
|
||||
|
3
twitchdl/__main__.py
Normal file
3
twitchdl/__main__.py
Normal file
@ -0,0 +1,3 @@
|
||||
from twitchdl.console import main
|
||||
|
||||
main()
|
@ -1,19 +1,18 @@
|
||||
import m3u8
|
||||
import os
|
||||
import pathlib
|
||||
import re
|
||||
import requests
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
|
||||
from os import path
|
||||
from pathlib import Path
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from twitchdl import twitch, utils
|
||||
from twitchdl.download import download_file, download_files
|
||||
from twitchdl.exceptions import ConsoleError
|
||||
from twitchdl.output import print_out, print_video
|
||||
from twitchdl.output import print_out, print_clip, print_video, print_json
|
||||
|
||||
|
||||
def _continue():
|
||||
@ -46,6 +45,63 @@ def _get_game_ids(names):
|
||||
return game_ids
|
||||
|
||||
|
||||
def _clips_json(args):
|
||||
clips = twitch.get_channel_clips(args.channel_name, args.period, args.limit)
|
||||
nodes = list(edge["node"] for edge in clips["edges"])
|
||||
print_json(nodes)
|
||||
|
||||
|
||||
def _clips_download(args):
|
||||
generator = twitch.channel_clips_generator(args.channel_name, args.period, 100)
|
||||
for clips, _ in generator:
|
||||
for clip in clips["edges"]:
|
||||
clip = clip["node"]
|
||||
url = clip["videoQualities"][0]["sourceURL"]
|
||||
target = _clip_target_filename(clip)
|
||||
if path.exists(target):
|
||||
print_out("Already downloaded: <green>{}</green>".format(target))
|
||||
else:
|
||||
print_out("Downloading: <yellow>{}</yellow>".format(target))
|
||||
download_file(url, target)
|
||||
|
||||
|
||||
def clips(args):
|
||||
if args.json:
|
||||
return _clips_json(args)
|
||||
|
||||
if args.download:
|
||||
return _clips_download(args)
|
||||
|
||||
print_out("<dim>Loading clips...</dim>")
|
||||
generator = twitch.channel_clips_generator(args.channel_name, args.period, args.limit)
|
||||
|
||||
first = 1
|
||||
|
||||
for clips, has_more in generator:
|
||||
count = len(clips["edges"]) if "edges" in clips else 0
|
||||
last = first + count - 1
|
||||
|
||||
print_out("-" * 80)
|
||||
print_out("<yellow>Showing clips {}-{} of ??</yellow>".format(first, last))
|
||||
|
||||
for clip in clips["edges"]:
|
||||
print_clip(clip["node"])
|
||||
|
||||
if not args.pager:
|
||||
print_out(
|
||||
"\n<dim>There are more clips. "
|
||||
"Increase the --limit or use --pager to see the rest.</dim>"
|
||||
)
|
||||
break
|
||||
|
||||
if not has_more or not _continue():
|
||||
break
|
||||
|
||||
first += count
|
||||
else:
|
||||
print_out("<yellow>No clips found</yellow>")
|
||||
|
||||
|
||||
def videos(args):
|
||||
game_ids = _get_game_ids(args.game)
|
||||
|
||||
@ -66,6 +122,13 @@ def videos(args):
|
||||
for video in videos["edges"]:
|
||||
print_video(video["node"])
|
||||
|
||||
if not args.pager:
|
||||
print_out(
|
||||
"\n<dim>There are more videos. "
|
||||
"Increase the --limit or use --pager to see the rest.</dim>"
|
||||
)
|
||||
break
|
||||
|
||||
if not has_more or not _continue():
|
||||
break
|
||||
|
||||
@ -84,6 +147,10 @@ def _parse_playlists(playlists_m3u8):
|
||||
|
||||
|
||||
def _get_playlist_by_name(playlists, quality):
|
||||
if quality == "source":
|
||||
_, _, uri = playlists[0]
|
||||
return uri
|
||||
|
||||
for name, _, uri in playlists:
|
||||
if name == quality:
|
||||
return uri
|
||||
@ -103,24 +170,23 @@ def _select_playlist_interactive(playlists):
|
||||
return uri
|
||||
|
||||
|
||||
def _join_vods(directory, file_paths, target):
|
||||
input_path = "{}/files.txt".format(directory)
|
||||
|
||||
with open(input_path, 'w') as f:
|
||||
for path in file_paths:
|
||||
f.write('file {}\n'.format(os.path.basename(path)))
|
||||
|
||||
result = subprocess.run([
|
||||
def _join_vods(playlist_path, target, overwrite):
|
||||
command = [
|
||||
"ffmpeg",
|
||||
"-f", "concat",
|
||||
"-i", input_path,
|
||||
"-i", playlist_path,
|
||||
"-c", "copy",
|
||||
target,
|
||||
"-stats",
|
||||
"-loglevel", "warning",
|
||||
])
|
||||
]
|
||||
|
||||
result.check_returncode()
|
||||
if overwrite:
|
||||
command.append("-y")
|
||||
|
||||
print_out("<dim>{}</dim>".format(" ".join(command)))
|
||||
result = subprocess.run(command)
|
||||
if result.returncode != 0:
|
||||
raise ConsoleError("Joining files failed")
|
||||
|
||||
|
||||
def _video_target_filename(video, format):
|
||||
@ -137,8 +203,27 @@ def _video_target_filename(video, format):
|
||||
return name + "." + format
|
||||
|
||||
|
||||
def _get_files(playlist, start, end):
|
||||
"""Extract files for download from playlist."""
|
||||
def _clip_target_filename(clip):
|
||||
url = clip["videoQualities"][0]["sourceURL"]
|
||||
_, ext = path.splitext(url)
|
||||
ext = ext.lstrip(".")
|
||||
|
||||
match = re.search(r"^(\d{4})-(\d{2})-(\d{2})T", clip["createdAt"])
|
||||
date = "".join(match.groups())
|
||||
|
||||
name = "_".join([
|
||||
date,
|
||||
clip["id"],
|
||||
clip["broadcaster"]["channel"]["name"],
|
||||
utils.slugify(clip["title"]),
|
||||
])
|
||||
|
||||
return "{}.{}".format(name, ext)
|
||||
|
||||
|
||||
def _get_vod_paths(playlist, start, end):
|
||||
"""Extract unique VOD paths for download from playlist."""
|
||||
files = []
|
||||
vod_start = 0
|
||||
for segment in playlist.segments:
|
||||
vod_end = vod_start + segment.duration
|
||||
@ -148,18 +233,20 @@ def _get_files(playlist, start, end):
|
||||
start_condition = not start or vod_end > start
|
||||
end_condition = not end or vod_start < end
|
||||
|
||||
if start_condition and end_condition:
|
||||
yield segment.uri
|
||||
if start_condition and end_condition and segment.uri not in files:
|
||||
files.append(segment.uri)
|
||||
|
||||
vod_start = vod_end
|
||||
|
||||
return files
|
||||
|
||||
|
||||
def _crete_temp_dir(base_uri):
|
||||
"""Create a temp dir to store downloads if it doesn't exist."""
|
||||
path = urlparse(base_uri).path
|
||||
directory = '{}/twitch-dl{}'.format(tempfile.gettempdir(), path)
|
||||
pathlib.Path(directory).mkdir(parents=True, exist_ok=True)
|
||||
return directory
|
||||
path = urlparse(base_uri).path.lstrip("/")
|
||||
temp_dir = Path(tempfile.gettempdir(), "twitch-dl", path)
|
||||
temp_dir.mkdir(parents=True, exist_ok=True)
|
||||
return temp_dir
|
||||
|
||||
|
||||
VIDEO_PATTERNS = [
|
||||
@ -195,6 +282,9 @@ def _get_clip_url(clip, args):
|
||||
|
||||
# Quality given as an argument
|
||||
if args.quality:
|
||||
if args.quality == "source":
|
||||
return qualities[0]["sourceURL"]
|
||||
|
||||
selected_quality = args.quality.rstrip("p") # allow 720p as well as 720
|
||||
for q in qualities:
|
||||
if q["quality"] == selected_quality:
|
||||
@ -232,18 +322,12 @@ def _download_clip(slug, args):
|
||||
url = _get_clip_url(clip, args)
|
||||
print_out("<dim>Selected URL: {}</dim>".format(url))
|
||||
|
||||
url_path = urlparse(url).path
|
||||
extension = Path(url_path).suffix
|
||||
filename = "{}_{}{}".format(
|
||||
clip["broadcaster"]["login"],
|
||||
utils.slugify(clip["title"]),
|
||||
extension
|
||||
)
|
||||
target = _clip_target_filename(clip)
|
||||
|
||||
print_out("Downloading clip...")
|
||||
download_file(url, filename)
|
||||
download_file(url, target)
|
||||
|
||||
print_out("Downloaded: {}".format(filename))
|
||||
print_out("Downloaded: {}".format(target))
|
||||
|
||||
|
||||
def _download_video(video_id, args):
|
||||
@ -272,26 +356,43 @@ def _download_video(video_id, args):
|
||||
|
||||
base_uri = re.sub("/[^/]+$", "/", playlist_uri)
|
||||
target_dir = _crete_temp_dir(base_uri)
|
||||
filenames = list(_get_files(playlist, args.start, args.end))
|
||||
vod_paths = _get_vod_paths(playlist, args.start, args.end)
|
||||
|
||||
# Save playlists for debugging purposes
|
||||
with open(target_dir + "playlists.m3u8", "w") as f:
|
||||
with open(path.join(target_dir, "playlists.m3u8"), "w") as f:
|
||||
f.write(playlists_m3u8)
|
||||
with open(target_dir + "playlist.m3u8", "w") as f:
|
||||
with open(path.join(target_dir, "playlist.m3u8"), "w") as f:
|
||||
f.write(response.text)
|
||||
|
||||
print_out("\nDownloading {} VODs using {} workers to {}".format(
|
||||
len(filenames), args.max_workers, target_dir))
|
||||
file_paths = download_files(base_uri, target_dir, filenames, args.max_workers)
|
||||
len(vod_paths), args.max_workers, target_dir))
|
||||
path_map = download_files(base_uri, target_dir, vod_paths, args.max_workers)
|
||||
|
||||
# Make a modified playlist which references downloaded VODs
|
||||
# Keep only the downloaded segments and skip the rest
|
||||
org_segments = playlist.segments.copy()
|
||||
playlist.segments.clear()
|
||||
for segment in org_segments:
|
||||
if segment.uri in path_map:
|
||||
segment.uri = path_map[segment.uri]
|
||||
playlist.segments.append(segment)
|
||||
|
||||
playlist_path = path.join(target_dir, "playlist_downloaded.m3u8")
|
||||
playlist.dump(playlist_path)
|
||||
|
||||
if args.no_join:
|
||||
print_out("\n\n<dim>Skipping joining files...</dim>")
|
||||
print_out("VODs downloaded to:\n<blue>{}</blue>".format(target_dir))
|
||||
return
|
||||
|
||||
print_out("\n\nJoining files...")
|
||||
target = _video_target_filename(video, args.format)
|
||||
_join_vods(target_dir, file_paths, target)
|
||||
_join_vods(playlist_path, target, args.overwrite)
|
||||
|
||||
if args.keep:
|
||||
print_out("\nTemporary files not deleted: {}".format(target_dir))
|
||||
print_out("\n<dim>Temporary files not deleted: {}</dim>".format(target_dir))
|
||||
else:
|
||||
print_out("\nDeleting temporary files...")
|
||||
print_out("\n<dim>Deleting temporary files...</dim>")
|
||||
shutil.rmtree(target_dir)
|
||||
|
||||
print_out("Downloaded: {}".format(target))
|
||||
print_out("\nDownloaded: <green>{}</green>".format(target))
|
||||
|
@ -77,6 +77,47 @@ COMMANDS = [
|
||||
"choices": ["archive", "highlight", "upload"],
|
||||
"default": "archive",
|
||||
}),
|
||||
(["-p", "--pager"], {
|
||||
"help": "If there are more results than LIMIT, ask to show next page",
|
||||
"action": "store_true",
|
||||
"default": False,
|
||||
}),
|
||||
],
|
||||
),
|
||||
Command(
|
||||
name="clips",
|
||||
description="List clips",
|
||||
arguments=[
|
||||
(["channel_name"], {
|
||||
"help": "channel name",
|
||||
"type": str,
|
||||
}),
|
||||
(["-l", "--limit"], {
|
||||
"help": "Number of videos to fetch (default 10, max 100)",
|
||||
"type": limit,
|
||||
"default": 10,
|
||||
}),
|
||||
(["-P", "--period"], {
|
||||
"help": "Period from which to return clips. (default: 'all_time')",
|
||||
"type": str,
|
||||
"choices": ["last_day", "last_week", "last_month", "all_time"],
|
||||
"default": "all_time",
|
||||
}),
|
||||
(["-j", "--json"], {
|
||||
"help": "Show results as JSON",
|
||||
"action": "store_true",
|
||||
"default": False,
|
||||
}),
|
||||
(["-p", "--pager"], {
|
||||
"help": "If there are more results than LIMIT, ask to show next page",
|
||||
"action": "store_true",
|
||||
"default": False,
|
||||
}),
|
||||
(["-d", "--download"], {
|
||||
"help": "Download all videos in given period (in source quality)",
|
||||
"action": "store_true",
|
||||
"default": False,
|
||||
}),
|
||||
],
|
||||
),
|
||||
Command(
|
||||
@ -115,9 +156,19 @@ COMMANDS = [
|
||||
"default": False,
|
||||
}),
|
||||
(["-q", "--quality"], {
|
||||
"help": "Video quality.",
|
||||
"help": "Video quality, e.g. 720p. Set to 'source' to get best quality.",
|
||||
"type": str,
|
||||
}),
|
||||
(["--no-join"], {
|
||||
"help": "Don't run ffmpeg to join the downloaded vods, implies --keep.",
|
||||
"action": "store_true",
|
||||
"default": False,
|
||||
}),
|
||||
(["--overwrite"], {
|
||||
"help": "Overwrite the target file if it already exists without prompting.",
|
||||
"action": "store_true",
|
||||
"default": False,
|
||||
})
|
||||
],
|
||||
),
|
||||
]
|
||||
|
@ -1,6 +1,7 @@
|
||||
import os
|
||||
import requests
|
||||
|
||||
from collections import OrderedDict
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from datetime import datetime
|
||||
from functools import partial
|
||||
@ -75,13 +76,17 @@ def _print_progress(futures):
|
||||
print_out("\r" + msg.ljust(max_msg_size), end="")
|
||||
|
||||
|
||||
def download_files(base_url, directory, filenames, max_workers):
|
||||
urls = [base_url + f for f in filenames]
|
||||
paths = ["{}{:05d}.vod".format(directory, k) for k, _ in enumerate(filenames)]
|
||||
partials = (partial(download_file, url, path) for url, path in zip(urls, paths))
|
||||
def download_files(base_url, target_dir, vod_paths, max_workers):
|
||||
"""
|
||||
Downloads a list of VODs defined by a common `base_url` and a list of
|
||||
`vod_paths`, returning a dict which maps the paths to the downloaded files.
|
||||
"""
|
||||
urls = [base_url + path for path in vod_paths]
|
||||
targets = [os.path.join(target_dir, "{:05d}.ts".format(k)) for k, _ in enumerate(vod_paths)]
|
||||
partials = (partial(download_file, url, path) for url, path in zip(urls, targets))
|
||||
|
||||
with ThreadPoolExecutor(max_workers=max_workers) as executor:
|
||||
futures = [executor.submit(fn) for fn in partials]
|
||||
_print_progress(futures)
|
||||
|
||||
return paths
|
||||
return OrderedDict(zip(vod_paths, targets))
|
||||
|
@ -1,5 +1,6 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
|
||||
import json
|
||||
import sys
|
||||
import re
|
||||
|
||||
@ -51,6 +52,10 @@ def print_out(*args, **kwargs):
|
||||
print(*args, **kwargs)
|
||||
|
||||
|
||||
def print_json(data):
|
||||
print(json.dumps(data))
|
||||
|
||||
|
||||
def print_err(*args, **kwargs):
|
||||
args = ["<red>{}</red>".format(a) for a in args]
|
||||
args = [colorize(a) if USE_ANSI_COLOR else strip_tags(a) for a in args]
|
||||
@ -62,15 +67,34 @@ def print_video(video):
|
||||
length = utils.format_duration(video["lengthSeconds"])
|
||||
channel = video["creator"]["channel"]["displayName"]
|
||||
playing = (
|
||||
" playing <blue>{}</blue>".format(video["game"]["name"])
|
||||
"playing <blue>{}</blue>".format(video["game"]["name"])
|
||||
if video["game"] else ""
|
||||
)
|
||||
|
||||
# Can't find URL in video object, strange
|
||||
url = "https://twitch.tv/videos/{}".format(video["id"])
|
||||
url = "https://www.twitch.tv/videos/{}".format(video["id"])
|
||||
|
||||
print_out("\n<b>{}</b>".format(video["id"]))
|
||||
print_out("<green>{}</green>".format(video["title"]))
|
||||
print_out("<blue>{}</blue> {}".format(channel, playing))
|
||||
print_out("Published <blue>{}</blue> Length: <blue>{}</blue> ".format(published_at, length))
|
||||
print_out("<i>{}</i>".format(url))
|
||||
|
||||
|
||||
def print_clip(clip):
|
||||
published_at = clip["createdAt"].replace("T", " @ ").replace("Z", "")
|
||||
length = utils.format_duration(clip["durationSeconds"])
|
||||
channel = clip["broadcaster"]["channel"]["displayName"]
|
||||
playing = (
|
||||
"playing <blue>{}</blue>".format(clip["game"]["name"])
|
||||
if clip["game"] else ""
|
||||
)
|
||||
|
||||
print_out("\n<b>{}</b>".format(clip["slug"]))
|
||||
print_out("<green>{}</green>".format(clip["title"]))
|
||||
print_out("<blue>{}</blue> {}".format(channel, playing))
|
||||
print_out(
|
||||
"Published <blue>{}</blue>"
|
||||
" Length: <blue>{}</blue>"
|
||||
" Views: <blue>{}</blue>".format(published_at, length, clip["viewCount"]))
|
||||
print_out("<i>{}</i>".format(clip["url"]))
|
||||
|
@ -74,7 +74,9 @@ def get_clip(slug):
|
||||
query = """
|
||||
{{
|
||||
clip(slug: "{}") {{
|
||||
id
|
||||
title
|
||||
createdAt
|
||||
durationSeconds
|
||||
game {{
|
||||
name
|
||||
@ -82,6 +84,9 @@ def get_clip(slug):
|
||||
broadcaster {{
|
||||
login
|
||||
displayName
|
||||
channel {{
|
||||
name
|
||||
}}
|
||||
}}
|
||||
videoQualities {{
|
||||
frameRate
|
||||
@ -96,6 +101,89 @@ def get_clip(slug):
|
||||
return response["data"]["clip"]
|
||||
|
||||
|
||||
def get_channel_clips(channel_id, period, limit, after=None):
|
||||
"""
|
||||
List channel clips.
|
||||
|
||||
At the time of writing this:
|
||||
* filtering by game name returns an error
|
||||
* sorting by anything but VIEWS_DESC or TRENDING returns an error
|
||||
* sorting by VIEWS_DESC and TRENDING returns the same results
|
||||
* there is no totalCount
|
||||
"""
|
||||
query = """
|
||||
{{
|
||||
user(login: "{channel_id}") {{
|
||||
clips(first: {limit}, after: "{after}", criteria: {{ period: {period}, sort: VIEWS_DESC }}) {{
|
||||
pageInfo {{
|
||||
hasNextPage
|
||||
hasPreviousPage
|
||||
}}
|
||||
edges {{
|
||||
cursor
|
||||
node {{
|
||||
id
|
||||
slug
|
||||
title
|
||||
createdAt
|
||||
viewCount
|
||||
durationSeconds
|
||||
url
|
||||
videoQualities {{
|
||||
frameRate
|
||||
quality
|
||||
sourceURL
|
||||
}}
|
||||
game {{
|
||||
id
|
||||
name
|
||||
}}
|
||||
broadcaster {{
|
||||
channel {{
|
||||
name
|
||||
displayName
|
||||
}}
|
||||
}}
|
||||
}}
|
||||
}}
|
||||
}}
|
||||
}}
|
||||
}}
|
||||
"""
|
||||
|
||||
query = query.format(**{
|
||||
"channel_id": channel_id,
|
||||
"after": after if after else "",
|
||||
"limit": limit,
|
||||
"period": period.upper(),
|
||||
})
|
||||
|
||||
response = gql_query(query)
|
||||
user = response["data"]["user"]
|
||||
if not user:
|
||||
raise ConsoleError("Channel {} not found".format(channel_id))
|
||||
|
||||
return response["data"]["user"]["clips"]
|
||||
|
||||
|
||||
def channel_clips_generator(channel_id, period, limit):
|
||||
cursor = ""
|
||||
while True:
|
||||
clips = get_channel_clips(
|
||||
channel_id, period, limit, after=cursor)
|
||||
|
||||
if not clips["edges"]:
|
||||
break
|
||||
|
||||
has_next = clips["pageInfo"]["hasNextPage"]
|
||||
cursor = clips["edges"][-1]["cursor"] if has_next else None
|
||||
|
||||
yield clips, has_next
|
||||
|
||||
if not cursor:
|
||||
break
|
||||
|
||||
|
||||
def get_channel_videos(channel_id, limit, sort, type="archive", game_ids=[], after=None):
|
||||
query = """
|
||||
{{
|
||||
@ -139,7 +227,7 @@ def get_channel_videos(channel_id, limit, sort, type="archive", game_ids=[], aft
|
||||
query = query.format(**{
|
||||
"channel_id": channel_id,
|
||||
"game_ids": game_ids,
|
||||
"after": after,
|
||||
"after": after if after else "",
|
||||
"limit": limit,
|
||||
"sort": sort.upper(),
|
||||
"type": type.upper(),
|
||||
@ -150,7 +238,7 @@ def get_channel_videos(channel_id, limit, sort, type="archive", game_ids=[], aft
|
||||
|
||||
|
||||
def channel_videos_generator(channel_id, limit, sort, type, game_ids=None):
|
||||
cursor = None
|
||||
cursor = ""
|
||||
while True:
|
||||
videos = get_channel_videos(
|
||||
channel_id, limit, sort, type, game_ids=game_ids, after=cursor)
|
||||
|
Reference in New Issue
Block a user