Compare commits

..

26 Commits

Author SHA1 Message Date
838611b834 Bump version 2020-11-23 16:08:15 +01:00
cf8d13e80e Fix clip download due to missing fields in query
fixes #45
2020-11-23 16:06:09 +01:00
6108b15587 Handle channel not found in clips 2020-11-10 12:40:11 +01:00
bb2de4af96 Bump version 2020-11-10 10:48:21 +01:00
678bf11a8a Fix print_clip when game is not set 2020-11-10 10:44:18 +01:00
69de08e2ba Update readme 2020-11-10 10:44:09 +01:00
1cb4598a71 Add downloading clips 2020-11-10 09:22:07 +01:00
02e4cdaff6 Add listing clips 2020-11-10 08:30:01 +01:00
c2e9ab9382 Bump version 2020-09-29 12:22:24 +02:00
739460bed1 Remove extraneous parameter 2020-09-29 12:21:13 +02:00
c92ba5e183 Bump version 2020-09-29 11:28:35 +02:00
f4ab045cdc Add --pager option, don't page by default
issue #30
2020-09-29 11:28:35 +02:00
cdc0495cf1 Document temp file shenanigans 2020-09-29 11:28:34 +02:00
c510d5aae4 Add --overwrite option
issue #37
2020-09-29 11:28:03 +02:00
2abc56213e Add --no-join option to skip ffmpeg join
issue #36
2020-09-29 11:27:49 +02:00
bb16108572 Add source quality alias
issue #33
2020-09-29 08:26:40 +02:00
b982cba566 Bump version, changelog 2020-09-03 12:56:57 +02:00
f4d442c118 Document installation instructions 2020-09-03 12:56:08 +02:00
eecd098f18 Add www before twitch.tv since that's what twitch uses 2020-09-03 12:55:45 +02:00
041689bee9 Construct paths using path libs
Fixes issues with paths on windows.

issue #35
2020-09-03 12:26:28 +02:00
a245ffb6a4 Fix issue with partial downloads
When using --start or --end, only keep the segments which have been
downloaded, and skip the rest.
2020-09-03 11:59:44 +02:00
ac37f179ef Improve bundle command
* directly save to desired file name
* add version number to file name
* remove __pycache__ folders before bundling
* compress the archive
2020-09-03 11:09:38 +02:00
5b200a2cb7 Add a splash of color 2020-09-03 10:38:29 +02:00
bf2a4558f4 Improve VOD joining logic
Instead of creating a file list, create a modified playlist which
references the downloaded files, and give this as input to ffmpeg. Since
ffmpeg handles M3U8 playlists, this means options such as
`EXT-X-BYTERANGE` are supported.

issue #35
2020-09-03 10:38:29 +02:00
772faa5901 Log ffmpeg command and handle errors better 2020-09-03 10:38:28 +02:00
04ddadef26 Add bundling with zipapp 2020-08-13 13:47:03 +02:00
12 changed files with 494 additions and 59 deletions

2
.gitignore vendored
View File

@ -11,3 +11,5 @@ tmp/
/htmlcov
/twitch-dl-*.tar.gz
/twitch-dl.1.man
/bundle
/*.pyz

View File

@ -1,6 +1,37 @@
Twitch Downloader change log
============================
1.13.1 (2020-11-23)
-------------------
* Fixed clip download issue (#45)
1.13.0 (2020-11-10)
-------------------
* Added `clips` command for listing and batch downloading clips (#26)
1.12.1 (2020-09-29)
-------------------
* Fix bug introduced in previous version which broke joining
1.12.0 (2020-09-29)
-------------------
* Added `source` as alias for best available quality (#33)
* Added `--no-join` option to `download` to skip ffmpeg join (#36)
* Added `--overwrite` option to `download` to overwrite target without prompting
for confirmation (#37)
* Added `--pager` option to `videos`, don't page by default (#30)
1.11.0 (2020-09-03)
-------------------
* Make downloading more robust, fixes issues with some VODs (#35)
* Bundle twitch-dl to a standalone archive, simplifying installation, see
installation instructions in README
1.10.2 (2020-08-11)
-------------------

View File

@ -6,7 +6,18 @@ dist :
clean :
find . -name "*pyc" | xargs rm -rf $1
rm -rf build dist MANIFEST htmlcov deb_dist twitch-dl*.tar.gz twitch-dl.1.man
rm -rf build dist bundle MANIFEST htmlcov deb_dist twitch-dl*.tar.gz twitch-dl.1.man
bundle:
mkdir bundle
cp twitchdl/__main__.py bundle
pip install . --target=bundle
rm -rf bundle/*.dist-info
find bundle/ -type d -name "__pycache__" -exec rm -rf {} +
python -m zipapp \
--python "/usr/bin/env python3" \
--output twitch-dl.`git describe`.pyz bundle \
--compress
publish :
twine upload dist/*.tar.gz dist/*.whl

129
README.md
View File

@ -1,10 +1,10 @@
Twitch Downloader
=================
A simple CLI tool for downloading videos from Twitch.
CLI tool for downloading videos from twitch.tv
Inspired by youtube-dl but improves upon it by using multiple concurrent
connections to make the download faster.
Inspired by [youtube-dl](https://youtube-dl.org/) but improves upon it by using
multiple concurrent connections to make the download faster.
Resources
---------
@ -17,11 +17,70 @@ Requirements
------------
* Python 3.5+
* [ffmpeg](https://ffmpeg.org/) must be installed and in the path
* [ffmpeg](https://ffmpeg.org/download.html), installed and on the system path
Installation
------------
### Download standalone archive
Go to the [latest release](https://github.com/ihabunek/twitch-dl/releases/latest)
and download the `twitch-dl.<version>.pyz` archive.
Run the archive by either:
a) passing it to python:
```
python3 twitch-dl.1.13.0.pyz --help
```
b) making it executable and invoking it directly (linux specific):
```
chmod +x twitch-dl.1.13.0.pyz
./twitch-dl.1.13.0.pyz --help
```
Feel free to rename the archive to something more managable, like `twitch-dl`.
To upgrade to a newer version, repeat the process with the newer release.
### From PYPI using pipx
**pipx** is a tool which installs python apps into isolated environments, which
prevents all kinds of problems later so it's the suggested way to install
twitch-dl from PYPI.
Install pipx as described in
[pipx install docs](https://pipxproject.github.io/pipx/installation/).
Install twitch-dl:
```
pipx install twitch-dl
```
Check installation worked:
```
twitch-dl --help
```
If twitch-dl executable is not found, check that the pipx binary location (by
default `~/.local/bin`) is in your PATH.
To upgrade twitch-dl to the latest version:
```
pipx install twitch-dl
```
Usage
-----
### Listing videos
List recent streams for a given channel:
```
@ -55,6 +114,8 @@ Use the `--game` option to specify one or more games to show:
twitch-dl videos --game "doom eternal" --game "cave story" bananasaurus_rex
```
### Downloading videos
Download a stream by ID or URL:
```
@ -68,6 +129,32 @@ Specify video quality to download:
twitch-dl download -q 720p 221837124
```
Setting quality to `source` will download the best available quality:
```
twitch-dl download -q source 221837124
```
### Listing clips
List clips for the given period:
```
twitch-dl clips bananasaurus_rex --period last_week
```
Supported periods are: `last_day`, `last_week`, `last_month`, `all_time`.
For listing a large number of clips, it's nice to page them:
```
twitch-dl clips bananasaurus_rex --period all_time --limit 10 --pager
```
This will show 10 clips at a time and ask to continue.
### Downloading clips
Download a clip by slug or URL:
```
@ -83,6 +170,38 @@ twitch-dl download -q 720 VenomousTameWormHumbleLife
Note that twitch names for clip qualities have no trailing "p".
### Batch downloading clips
It's possible to download all clips for a given period:
```
twitch-dl clips bananasaurus_rex --period last_week --download
```
Clips are downloaded in source quality.
A note about clips
------------------
Currently it doesn't seem to be possible to get a list of clips ordered by time
of creation, only by view count. Clips with the same view count seem to be
returned in random order. This can break paging resulting in duplicate clips
listed or clips missed.
When batch downloading a large number of clips (over 100), it's possible that
some will be missed.
Temporary files
---------------
By default, twitch-dl will download VODs to your systems temp dir (e.g. `/tmp/`
on Linux). To change the location where the files are downloaded you can set
the `TMP` environment variable, e.g.
```
TMP=/my/tmp/path/ twitch-dl download 221837124
```
Man page
--------
@ -97,6 +216,6 @@ make man
License
-------
Copyright 2018 Ivan Habunek <ivan@habunek.com>
Copyright 2018-2020 Ivan Habunek <ivan@habunek.com>
Licensed under the GPLv3: http://www.gnu.org/licenses/gpl-3.0.html

View File

@ -11,7 +11,7 @@ makes it faster.
setup(
name='twitch-dl',
version='1.10.2',
version='1.13.1',
description='Twitch downloader',
long_description=long_description.strip(),
author='Ivan Habunek',

View File

@ -1,3 +1,3 @@
__version__ = "1.10.2"
__version__ = "1.13.1"
CLIENT_ID = "kimne78kx3ncx6brgo4mv6wki5h1ko"

3
twitchdl/__main__.py Normal file
View File

@ -0,0 +1,3 @@
from twitchdl.console import main
main()

View File

@ -1,19 +1,18 @@
import m3u8
import os
import pathlib
import re
import requests
import shutil
import subprocess
import tempfile
from os import path
from pathlib import Path
from urllib.parse import urlparse
from twitchdl import twitch, utils
from twitchdl.download import download_file, download_files
from twitchdl.exceptions import ConsoleError
from twitchdl.output import print_out, print_video
from twitchdl.output import print_out, print_clip, print_video, print_json
def _continue():
@ -46,6 +45,63 @@ def _get_game_ids(names):
return game_ids
def _clips_json(args):
clips = twitch.get_channel_clips(args.channel_name, args.period, args.limit)
nodes = list(edge["node"] for edge in clips["edges"])
print_json(nodes)
def _clips_download(args):
generator = twitch.channel_clips_generator(args.channel_name, args.period, 100)
for clips, _ in generator:
for clip in clips["edges"]:
clip = clip["node"]
url = clip["videoQualities"][0]["sourceURL"]
target = _clip_target_filename(clip)
if path.exists(target):
print_out("Already downloaded: <green>{}</green>".format(target))
else:
print_out("Downloading: <yellow>{}</yellow>".format(target))
download_file(url, target)
def clips(args):
if args.json:
return _clips_json(args)
if args.download:
return _clips_download(args)
print_out("<dim>Loading clips...</dim>")
generator = twitch.channel_clips_generator(args.channel_name, args.period, args.limit)
first = 1
for clips, has_more in generator:
count = len(clips["edges"]) if "edges" in clips else 0
last = first + count - 1
print_out("-" * 80)
print_out("<yellow>Showing clips {}-{} of ??</yellow>".format(first, last))
for clip in clips["edges"]:
print_clip(clip["node"])
if not args.pager:
print_out(
"\n<dim>There are more clips. "
"Increase the --limit or use --pager to see the rest.</dim>"
)
break
if not has_more or not _continue():
break
first += count
else:
print_out("<yellow>No clips found</yellow>")
def videos(args):
game_ids = _get_game_ids(args.game)
@ -66,6 +122,13 @@ def videos(args):
for video in videos["edges"]:
print_video(video["node"])
if not args.pager:
print_out(
"\n<dim>There are more videos. "
"Increase the --limit or use --pager to see the rest.</dim>"
)
break
if not has_more or not _continue():
break
@ -84,6 +147,10 @@ def _parse_playlists(playlists_m3u8):
def _get_playlist_by_name(playlists, quality):
if quality == "source":
_, _, uri = playlists[0]
return uri
for name, _, uri in playlists:
if name == quality:
return uri
@ -103,24 +170,23 @@ def _select_playlist_interactive(playlists):
return uri
def _join_vods(directory, file_paths, target):
input_path = "{}/files.txt".format(directory)
with open(input_path, 'w') as f:
for path in file_paths:
f.write('file {}\n'.format(os.path.basename(path)))
result = subprocess.run([
def _join_vods(playlist_path, target, overwrite):
command = [
"ffmpeg",
"-f", "concat",
"-i", input_path,
"-i", playlist_path,
"-c", "copy",
target,
"-stats",
"-loglevel", "warning",
])
]
result.check_returncode()
if overwrite:
command.append("-y")
print_out("<dim>{}</dim>".format(" ".join(command)))
result = subprocess.run(command)
if result.returncode != 0:
raise ConsoleError("Joining files failed")
def _video_target_filename(video, format):
@ -137,8 +203,27 @@ def _video_target_filename(video, format):
return name + "." + format
def _get_files(playlist, start, end):
"""Extract files for download from playlist."""
def _clip_target_filename(clip):
url = clip["videoQualities"][0]["sourceURL"]
_, ext = path.splitext(url)
ext = ext.lstrip(".")
match = re.search(r"^(\d{4})-(\d{2})-(\d{2})T", clip["createdAt"])
date = "".join(match.groups())
name = "_".join([
date,
clip["id"],
clip["broadcaster"]["channel"]["name"],
utils.slugify(clip["title"]),
])
return "{}.{}".format(name, ext)
def _get_vod_paths(playlist, start, end):
"""Extract unique VOD paths for download from playlist."""
files = []
vod_start = 0
for segment in playlist.segments:
vod_end = vod_start + segment.duration
@ -148,18 +233,20 @@ def _get_files(playlist, start, end):
start_condition = not start or vod_end > start
end_condition = not end or vod_start < end
if start_condition and end_condition:
yield segment.uri
if start_condition and end_condition and segment.uri not in files:
files.append(segment.uri)
vod_start = vod_end
return files
def _crete_temp_dir(base_uri):
"""Create a temp dir to store downloads if it doesn't exist."""
path = urlparse(base_uri).path
directory = '{}/twitch-dl{}'.format(tempfile.gettempdir(), path)
pathlib.Path(directory).mkdir(parents=True, exist_ok=True)
return directory
path = urlparse(base_uri).path.lstrip("/")
temp_dir = Path(tempfile.gettempdir(), "twitch-dl", path)
temp_dir.mkdir(parents=True, exist_ok=True)
return temp_dir
VIDEO_PATTERNS = [
@ -195,6 +282,9 @@ def _get_clip_url(clip, args):
# Quality given as an argument
if args.quality:
if args.quality == "source":
return qualities[0]["sourceURL"]
selected_quality = args.quality.rstrip("p") # allow 720p as well as 720
for q in qualities:
if q["quality"] == selected_quality:
@ -232,18 +322,12 @@ def _download_clip(slug, args):
url = _get_clip_url(clip, args)
print_out("<dim>Selected URL: {}</dim>".format(url))
url_path = urlparse(url).path
extension = Path(url_path).suffix
filename = "{}_{}{}".format(
clip["broadcaster"]["login"],
utils.slugify(clip["title"]),
extension
)
target = _clip_target_filename(clip)
print_out("Downloading clip...")
download_file(url, filename)
download_file(url, target)
print_out("Downloaded: {}".format(filename))
print_out("Downloaded: {}".format(target))
def _download_video(video_id, args):
@ -272,26 +356,43 @@ def _download_video(video_id, args):
base_uri = re.sub("/[^/]+$", "/", playlist_uri)
target_dir = _crete_temp_dir(base_uri)
filenames = list(_get_files(playlist, args.start, args.end))
vod_paths = _get_vod_paths(playlist, args.start, args.end)
# Save playlists for debugging purposes
with open(target_dir + "playlists.m3u8", "w") as f:
with open(path.join(target_dir, "playlists.m3u8"), "w") as f:
f.write(playlists_m3u8)
with open(target_dir + "playlist.m3u8", "w") as f:
with open(path.join(target_dir, "playlist.m3u8"), "w") as f:
f.write(response.text)
print_out("\nDownloading {} VODs using {} workers to {}".format(
len(filenames), args.max_workers, target_dir))
file_paths = download_files(base_uri, target_dir, filenames, args.max_workers)
len(vod_paths), args.max_workers, target_dir))
path_map = download_files(base_uri, target_dir, vod_paths, args.max_workers)
# Make a modified playlist which references downloaded VODs
# Keep only the downloaded segments and skip the rest
org_segments = playlist.segments.copy()
playlist.segments.clear()
for segment in org_segments:
if segment.uri in path_map:
segment.uri = path_map[segment.uri]
playlist.segments.append(segment)
playlist_path = path.join(target_dir, "playlist_downloaded.m3u8")
playlist.dump(playlist_path)
if args.no_join:
print_out("\n\n<dim>Skipping joining files...</dim>")
print_out("VODs downloaded to:\n<blue>{}</blue>".format(target_dir))
return
print_out("\n\nJoining files...")
target = _video_target_filename(video, args.format)
_join_vods(target_dir, file_paths, target)
_join_vods(playlist_path, target, args.overwrite)
if args.keep:
print_out("\nTemporary files not deleted: {}".format(target_dir))
print_out("\n<dim>Temporary files not deleted: {}</dim>".format(target_dir))
else:
print_out("\nDeleting temporary files...")
print_out("\n<dim>Deleting temporary files...</dim>")
shutil.rmtree(target_dir)
print_out("Downloaded: {}".format(target))
print_out("\nDownloaded: <green>{}</green>".format(target))

View File

@ -77,6 +77,47 @@ COMMANDS = [
"choices": ["archive", "highlight", "upload"],
"default": "archive",
}),
(["-p", "--pager"], {
"help": "If there are more results than LIMIT, ask to show next page",
"action": "store_true",
"default": False,
}),
],
),
Command(
name="clips",
description="List clips",
arguments=[
(["channel_name"], {
"help": "channel name",
"type": str,
}),
(["-l", "--limit"], {
"help": "Number of videos to fetch (default 10, max 100)",
"type": limit,
"default": 10,
}),
(["-P", "--period"], {
"help": "Period from which to return clips. (default: 'all_time')",
"type": str,
"choices": ["last_day", "last_week", "last_month", "all_time"],
"default": "all_time",
}),
(["-j", "--json"], {
"help": "Show results as JSON",
"action": "store_true",
"default": False,
}),
(["-p", "--pager"], {
"help": "If there are more results than LIMIT, ask to show next page",
"action": "store_true",
"default": False,
}),
(["-d", "--download"], {
"help": "Download all videos in given period (in source quality)",
"action": "store_true",
"default": False,
}),
],
),
Command(
@ -115,9 +156,19 @@ COMMANDS = [
"default": False,
}),
(["-q", "--quality"], {
"help": "Video quality.",
"help": "Video quality, e.g. 720p. Set to 'source' to get best quality.",
"type": str,
}),
(["--no-join"], {
"help": "Don't run ffmpeg to join the downloaded vods, implies --keep.",
"action": "store_true",
"default": False,
}),
(["--overwrite"], {
"help": "Overwrite the target file if it already exists without prompting.",
"action": "store_true",
"default": False,
})
],
),
]

View File

@ -1,6 +1,7 @@
import os
import requests
from collections import OrderedDict
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime
from functools import partial
@ -75,13 +76,17 @@ def _print_progress(futures):
print_out("\r" + msg.ljust(max_msg_size), end="")
def download_files(base_url, directory, filenames, max_workers):
urls = [base_url + f for f in filenames]
paths = ["{}{:05d}.vod".format(directory, k) for k, _ in enumerate(filenames)]
partials = (partial(download_file, url, path) for url, path in zip(urls, paths))
def download_files(base_url, target_dir, vod_paths, max_workers):
"""
Downloads a list of VODs defined by a common `base_url` and a list of
`vod_paths`, returning a dict which maps the paths to the downloaded files.
"""
urls = [base_url + path for path in vod_paths]
targets = [os.path.join(target_dir, "{:05d}.ts".format(k)) for k, _ in enumerate(vod_paths)]
partials = (partial(download_file, url, path) for url, path in zip(urls, targets))
with ThreadPoolExecutor(max_workers=max_workers) as executor:
futures = [executor.submit(fn) for fn in partials]
_print_progress(futures)
return paths
return OrderedDict(zip(vod_paths, targets))

View File

@ -1,5 +1,6 @@
# -*- coding: utf-8 -*-
import json
import sys
import re
@ -51,6 +52,10 @@ def print_out(*args, **kwargs):
print(*args, **kwargs)
def print_json(data):
print(json.dumps(data))
def print_err(*args, **kwargs):
args = ["<red>{}</red>".format(a) for a in args]
args = [colorize(a) if USE_ANSI_COLOR else strip_tags(a) for a in args]
@ -62,15 +67,34 @@ def print_video(video):
length = utils.format_duration(video["lengthSeconds"])
channel = video["creator"]["channel"]["displayName"]
playing = (
" playing <blue>{}</blue>".format(video["game"]["name"])
"playing <blue>{}</blue>".format(video["game"]["name"])
if video["game"] else ""
)
# Can't find URL in video object, strange
url = "https://twitch.tv/videos/{}".format(video["id"])
url = "https://www.twitch.tv/videos/{}".format(video["id"])
print_out("\n<b>{}</b>".format(video["id"]))
print_out("<green>{}</green>".format(video["title"]))
print_out("<blue>{}</blue> {}".format(channel, playing))
print_out("Published <blue>{}</blue> Length: <blue>{}</blue> ".format(published_at, length))
print_out("<i>{}</i>".format(url))
def print_clip(clip):
published_at = clip["createdAt"].replace("T", " @ ").replace("Z", "")
length = utils.format_duration(clip["durationSeconds"])
channel = clip["broadcaster"]["channel"]["displayName"]
playing = (
"playing <blue>{}</blue>".format(clip["game"]["name"])
if clip["game"] else ""
)
print_out("\n<b>{}</b>".format(clip["slug"]))
print_out("<green>{}</green>".format(clip["title"]))
print_out("<blue>{}</blue> {}".format(channel, playing))
print_out(
"Published <blue>{}</blue>"
" Length: <blue>{}</blue>"
" Views: <blue>{}</blue>".format(published_at, length, clip["viewCount"]))
print_out("<i>{}</i>".format(clip["url"]))

View File

@ -74,7 +74,9 @@ def get_clip(slug):
query = """
{{
clip(slug: "{}") {{
id
title
createdAt
durationSeconds
game {{
name
@ -82,6 +84,9 @@ def get_clip(slug):
broadcaster {{
login
displayName
channel {{
name
}}
}}
videoQualities {{
frameRate
@ -96,6 +101,89 @@ def get_clip(slug):
return response["data"]["clip"]
def get_channel_clips(channel_id, period, limit, after=None):
"""
List channel clips.
At the time of writing this:
* filtering by game name returns an error
* sorting by anything but VIEWS_DESC or TRENDING returns an error
* sorting by VIEWS_DESC and TRENDING returns the same results
* there is no totalCount
"""
query = """
{{
user(login: "{channel_id}") {{
clips(first: {limit}, after: "{after}", criteria: {{ period: {period}, sort: VIEWS_DESC }}) {{
pageInfo {{
hasNextPage
hasPreviousPage
}}
edges {{
cursor
node {{
id
slug
title
createdAt
viewCount
durationSeconds
url
videoQualities {{
frameRate
quality
sourceURL
}}
game {{
id
name
}}
broadcaster {{
channel {{
name
displayName
}}
}}
}}
}}
}}
}}
}}
"""
query = query.format(**{
"channel_id": channel_id,
"after": after if after else "",
"limit": limit,
"period": period.upper(),
})
response = gql_query(query)
user = response["data"]["user"]
if not user:
raise ConsoleError("Channel {} not found".format(channel_id))
return response["data"]["user"]["clips"]
def channel_clips_generator(channel_id, period, limit):
cursor = ""
while True:
clips = get_channel_clips(
channel_id, period, limit, after=cursor)
if not clips["edges"]:
break
has_next = clips["pageInfo"]["hasNextPage"]
cursor = clips["edges"][-1]["cursor"] if has_next else None
yield clips, has_next
if not cursor:
break
def get_channel_videos(channel_id, limit, sort, type="archive", game_ids=[], after=None):
query = """
{{
@ -139,7 +227,7 @@ def get_channel_videos(channel_id, limit, sort, type="archive", game_ids=[], aft
query = query.format(**{
"channel_id": channel_id,
"game_ids": game_ids,
"after": after,
"after": after if after else "",
"limit": limit,
"sort": sort.upper(),
"type": type.upper(),
@ -150,7 +238,7 @@ def get_channel_videos(channel_id, limit, sort, type="archive", game_ids=[], aft
def channel_videos_generator(channel_id, limit, sort, type, game_ids=None):
cursor = None
cursor = ""
while True:
videos = get_channel_videos(
channel_id, limit, sort, type, game_ids=game_ids, after=cursor)