Re-apply fix lost to a merge conflict

issue #53

.gitignore

@@ -11,3 +11,5 @@ tmp/
 /htmlcov
 /twitch-dl-*.tar.gz
 /twitch-dl.1.man
+/bundle
+/*.pyz

CHANGELOG.md

@@ -1,6 +1,48 @@
 Twitch Downloader change log
 ============================
 
+1.14.0 (2021-01-14)
+-------------------
+
+* Added `info` command for displaying video or clip info (#51)
+* Fixed Twitch regression for getting the access token (#53)
+
+1.13.1 (2020-11-23)
+-------------------
+
+* Fixed clip download issue (#45)
+
+1.13.0 (2020-11-10)
+-------------------
+
+* Added `clips` command for listing and batch downloading clips (#26)
+
+1.12.1 (2020-09-29)
+-------------------
+
+* Fix bug introduced in previous version which broke joining
+
+1.12.0 (2020-09-29)
+-------------------
+
+* Added `source` as alias for best available quality (#33)
+* Added `--no-join` option to `download` to skip ffmpeg join (#36)
+* Added `--overwrite` option to `download` to overwrite target without prompting
+  for confirmation (#37)
+* Added `--pager` option to `videos`, don't page by default (#30)
+
+1.11.0 (2020-09-03)
+-------------------
+
+* Make downloading more robust, fixes issues with some VODs (#35)
+* Bundle twitch-dl to a standalone archive, simplifying installation, see
+  installation instructions in README
+
+1.10.2 (2020-08-11)
+-------------------
+
+* Fix version number displayed by `twitch-dl --version` (#29)
+
 1.10.1 (2020-08-09)
 -------------------
 

Makefile

@@ -6,7 +6,18 @@ dist :
 
 clean :
 	find . -name "*pyc" | xargs rm -rf $1
-	rm -rf build dist MANIFEST htmlcov deb_dist twitch-dl*.tar.gz twitch-dl.1.man
+	rm -rf build dist bundle MANIFEST htmlcov deb_dist twitch-dl*.tar.gz twitch-dl.1.man
+
+bundle:
+	mkdir bundle
+	cp twitchdl/__main__.py bundle
+	pip install . --target=bundle
+	rm -rf bundle/*.dist-info
+	find bundle/ -type d -name "__pycache__" -exec rm -rf {} +
+	python -m zipapp \
+		--python "/usr/bin/env python3" \
+		--output twitch-dl.`git describe`.pyz bundle \
+		--compress
 
 publish :
 	twine upload dist/*.tar.gz dist/*.whl

README.md

@@ -1,10 +1,10 @@
 Twitch Downloader
 =================
 
-A simple CLI tool for downloading videos from Twitch.
+CLI tool for downloading videos from twitch.tv
 
-Inspired by youtube-dl but improves upon it by using multiple concurrent
-connections to make the download faster.
+Inspired by [youtube-dl](https://youtube-dl.org/) but improves upon it by using
+multiple concurrent connections to make the download faster.
 
 Resources
 ---------
@@ -17,45 +17,118 @@ Requirements
 ------------
 
 * Python 3.5+
-* [ffmpeg](https://ffmpeg.org/) must be installed and in the path
+* [ffmpeg](https://ffmpeg.org/download.html), installed and on the system path
+
+Installation
+------------
+
+### Download standalone archive
+
+Go to the [latest release](https://github.com/ihabunek/twitch-dl/releases/latest)
+and download the `twitch-dl.<version>.pyz` archive.
+
+Run the archive by either:
+
+a) passing it to python:
+
+```
+python3 twitch-dl.1.13.0.pyz --help
+```
+
+b) making it executable and invoking it directly (linux specific):
+
+```
+chmod +x twitch-dl.1.13.0.pyz
+./twitch-dl.1.13.0.pyz --help
+```
+
+Feel free to rename the archive to something more managable, like `twitch-dl`.
+
+To upgrade to a newer version, repeat the process with the newer release.
+
+### From PYPI using pipx
+
+**pipx** is a tool which installs python apps into isolated environments, which
+prevents all kinds of problems later so it's the suggested way to install
+twitch-dl from PYPI.
+
+Install pipx as described in
+[pipx install docs](https://pipxproject.github.io/pipx/installation/).
+
+Install twitch-dl:
+
+```
+pipx install twitch-dl
+```
+
+Check installation worked:
+
+```
+twitch-dl --help
+```
+
+If twitch-dl executable is not found, check that the pipx binary location (by
+default `~/.local/bin`) is in your PATH.
+
+To upgrade twitch-dl to the latest version:
+
+```
+pipx install twitch-dl
+```
 
 Usage
 -----
 
+This section does an overview of available features.
+
+To see a list of available commands run:
+
+```
+twitch-dl --help
+```
+
+And to see description and all arguments for a given command run:
+
+```
+twitch-dl <command> --help
+```
+
+### Print clip or video info
+
+Videos can be referenced by URL or ID:
+
+```
+twitch-dl info 863849735
+twitch-dl info https://www.twitch.tv/videos/863849735
+```
+
+Clips by slug or ID:
+
+```
+twitch-dl info BusyBlushingCattleItsBoshyTime
+twitch-dl info https://www.twitch.tv/bananasaurus_rex/clip/BusyBlushingCattleItsBoshyTime
+```
+
+Shows info about the video or clip as well as download URLs for clips and
+playlist URLs for videos.
+
+### Listing videos
+
 List recent streams for a given channel:
 
 ```
 twitch-dl videos bananasaurus_rex
 ```
 
-Yields (trimmed):
-
-```
-Found 33 videos
-
-221837124
-SUPER MARIO ODYSSSEY - Stream #2 / 600,000,000
-Bananasaurus_Rex playing Super Mario Odyssey
-Published 2018-01-24 @ 12:05:25  Length: 3h 40min
-
-221418913
-Dead Space and then SUPER MARIO ODYSSEY PogChamp
-Bananasaurus_Rex playing Dead Space
-Published 2018-01-23 @ 02:40:58  Length: 6h 2min
-
-220783179
-Dead Space | Got my new setup working! rexChamp
-Bananasaurus_Rex playing Dead Space
-Published 2018-01-21 @ 05:47:03  Length: 5h 7min
-```
-
 Use the `--game` option to specify one or more games to show:
 
 ```
 twitch-dl videos --game "doom eternal" --game "cave story" bananasaurus_rex
 ```
 
-Download a stream by ID or URL:
+### Downloading videos
+
+Download a video by ID or URL:
 
 ```
 twitch-dl download 221837124
@@ -68,6 +141,32 @@ Specify video quality to download:
 twitch-dl download -q 720p 221837124
 ```
 
+Setting quality to `source` will download the best available quality:
+
+```
+twitch-dl download -q source 221837124
+```
+
+### Listing clips
+
+List clips for the given period:
+
+```
+twitch-dl clips bananasaurus_rex --period last_week
+```
+
+Supported periods are: `last_day`, `last_week`, `last_month`, `all_time`.
+
+For listing a large number of clips, it's nice to page them:
+
+```
+twitch-dl clips bananasaurus_rex --period all_time --limit 10 --pager
+```
+
+This will show 10 clips at a time and ask to continue.
+
+### Downloading clips
+
 Download a clip by slug or URL:
 
 ```
@@ -83,6 +182,38 @@ twitch-dl download -q 720 VenomousTameWormHumbleLife
 
 Note that twitch names for clip qualities have no trailing "p".
 
+### Batch downloading clips
+
+It's possible to download all clips for a given period:
+
+```
+twitch-dl clips bananasaurus_rex --period last_week --download
+```
+
+Clips are downloaded in source quality.
+
+A note about clips
+------------------
+
+Currently it doesn't seem to be possible to get a list of clips ordered by time
+of creation, only by view count. Clips with the same view count seem to be
+returned in random order. This can break paging resulting in duplicate clips
+listed or clips missed.
+
+When batch downloading a large number of clips (over 100), it's possible that
+some will be missed.
+
+Temporary files
+---------------
+
+By default, twitch-dl will download VODs to your systems temp dir (e.g. `/tmp/`
+on Linux). To change the location where the files are downloaded you can set
+the `TMP` environment variable, e.g.
+
+```
+TMP=/my/tmp/path/ twitch-dl download 221837124
+```
+
 Man page
 --------
 
@@ -97,6 +228,6 @@ make man
 License
 -------
 
-Copyright 2018 Ivan Habunek <ivan@habunek.com>
+Copyright 2018-2020 Ivan Habunek <ivan@habunek.com>
 
 Licensed under the GPLv3: http://www.gnu.org/licenses/gpl-3.0.html

setup.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python
 
-from setuptools import setup
+from setuptools import setup, find_packages
 
 long_description = """
 Quickly download videos from twitch.tv.
@@ -11,7 +11,7 @@ makes it faster.
 
 setup(
     name='twitch-dl',
-    version='1.10.1',
+    version='1.14.0',
     description='Twitch downloader',
     long_description=long_description.strip(),
     author='Ivan Habunek',
@@ -27,7 +27,7 @@ setup(
         'Programming Language :: Python :: 3.6',
         'Programming Language :: Python :: 3.7',
     ],
-    packages=['twitchdl'],
+    packages=find_packages(),
     python_requires='>=3.5',
     install_requires=[
         "m3u8>=0.3.12,<0.4",

twitchdl/__init__.py

@@ -1,3 +1,3 @@
-__version__ = "1.9.0"
+__version__ = "1.14.0"
 
 CLIENT_ID = "kimne78kx3ncx6brgo4mv6wki5h1ko"

twitchdl/__main__.py

@@ -0,0 +1,3 @@
+from twitchdl.console import main
+
+main()

twitchdl/commands/__init__.py

@@ -0,0 +1,11 @@
+from .clips import clips
+from .download import download
+from .info import info
+from .videos import videos
+
+__all__ = [
+    clips,
+    download,
+    info,
+    videos,
+]

twitchdl/commands/clips.py

@@ -0,0 +1,114 @@
+import re
+
+from os import path
+
+from twitchdl import twitch, utils
+from twitchdl.download import download_file
+from twitchdl.exceptions import ConsoleError
+from twitchdl.output import print_out, print_clip, print_json
+
+
+def _continue():
+    print_out(
+        "\nThere are more clips. "
+        "Press <green><b>Enter</green> to continue, "
+        "<yellow><b>Ctrl+C</yellow> to break."
+    )
+
+    try:
+        input()
+    except KeyboardInterrupt:
+        return False
+
+    return True
+
+
+def _get_game_ids(names):
+    if not names:
+        return []
+
+    game_ids = []
+    for name in names:
+        print_out("<dim>Looking up game '{}'...</dim>".format(name))
+        game_id = twitch.get_game_id(name)
+        if not game_id:
+            raise ConsoleError("Game '{}' not found".format(name))
+        game_ids.append(int(game_id))
+
+    return game_ids
+
+
+def _clips_json(args):
+    clips = twitch.get_channel_clips(args.channel_name, args.period, args.limit)
+    nodes = list(edge["node"] for edge in clips["edges"])
+    print_json(nodes)
+
+
+def _clip_target_filename(clip):
+    url = clip["videoQualities"][0]["sourceURL"]
+    _, ext = path.splitext(url)
+    ext = ext.lstrip(".")
+
+    match = re.search(r"^(\d{4})-(\d{2})-(\d{2})T", clip["createdAt"])
+    date = "".join(match.groups())
+
+    name = "_".join([
+        date,
+        clip["id"],
+        clip["broadcaster"]["login"],
+        utils.slugify(clip["title"]),
+    ])
+
+    return "{}.{}".format(name, ext)
+
+
+def _clips_download(args):
+    generator = twitch.channel_clips_generator(args.channel_name, args.period, 100)
+    for clips, _ in generator:
+        for clip in clips["edges"]:
+            clip = clip["node"]
+            url = clip["videoQualities"][0]["sourceURL"]
+            target = _clip_target_filename(clip)
+            if path.exists(target):
+                print_out("Already downloaded: <green>{}</green>".format(target))
+            else:
+                print_out("Downloading: <yellow>{}</yellow>".format(target))
+                download_file(url, target)
+
+
+def clips(args):
+    if args.json:
+        return _clips_json(args)
+
+    if args.download:
+        return _clips_download(args)
+
+    print_out("<dim>Loading clips...</dim>")
+    generator = twitch.channel_clips_generator(args.channel_name, args.period, args.limit)
+
+    first = 1
+
+    for clips, has_more in generator:
+        count = len(clips["edges"]) if "edges" in clips else 0
+        last = first + count - 1
+
+        print_out("-" * 80)
+        print_out("<yellow>Showing clips {}-{} of ??</yellow>".format(first, last))
+
+        for clip in clips["edges"]:
+            print_out()
+            print_clip(clip["node"])
+
+        if not args.pager:
+            print_out(
+                "\n<dim>There are more clips. "
+                "Increase the --limit or use --pager to see the rest.</dim>"
+            )
+            break
+
+        if not has_more or not _continue():
+            break
+
+        first += count
+    else:
+        print_out("<yellow>No clips found</yellow>")

twitchdl/commands/download.py

@@ -1,77 +1,18 @@
 import m3u8
-import os
-import pathlib
 import re
 import requests
 import shutil
 import subprocess
 import tempfile
 
+from os import path
 from pathlib import Path
 from urllib.parse import urlparse
 
 from twitchdl import twitch, utils
 from twitchdl.download import download_file, download_files
 from twitchdl.exceptions import ConsoleError
-from twitchdl.output import print_out, print_video
-
-
-def _continue():
-    print_out(
-        "\nThere are more videos. "
-        "Press <green><b>Enter</green> to continue, "
-        "<yellow><b>Ctrl+C</yellow> to break."
-    )
-
-    try:
-        input()
-    except KeyboardInterrupt:
-        return False
-
-    return True
-
-
-def _get_game_ids(names):
-    if not names:
-        return []
-
-    game_ids = []
-    for name in names:
-        print_out("<dim>Looking up game '{}'...</dim>".format(name))
-        game_id = twitch.get_game_id(name)
-        if not game_id:
-            raise ConsoleError("Game '{}' not found".format(name))
-        game_ids.append(int(game_id))
-
-    return game_ids
-
-
-def videos(args):
-    game_ids = _get_game_ids(args.game)
-
-    print_out("<dim>Loading videos...</dim>")
-    generator = twitch.channel_videos_generator(
-        args.channel_name, args.limit, args.sort, args.type, game_ids=game_ids)
-
-    first = 1
-
-    for videos, has_more in generator:
-        count = len(videos["edges"]) if "edges" in videos else 0
-        total = videos["totalCount"]
-        last = first + count - 1
-
-        print_out("-" * 80)
-        print_out("<yellow>Showing videos {}-{} of {}</yellow>".format(first, last, total))
-
-        for video in videos["edges"]:
-            print_video(video["node"])
-
-        if not has_more or not _continue():
-            break
-
-        first += count
-    else:
-        print_out("<yellow>No videos found</yellow>")
+from twitchdl.output import print_out
 
 
 def _parse_playlists(playlists_m3u8):
@@ -84,6 +25,10 @@ def _parse_playlists(playlists_m3u8):
 
 
 def _get_playlist_by_name(playlists, quality):
+    if quality == "source":
+        _, _, uri = playlists[0]
+        return uri
+
     for name, _, uri in playlists:
         if name == quality:
             return uri
@@ -103,42 +48,60 @@ def _select_playlist_interactive(playlists):
     return uri
 
 
-def _join_vods(directory, file_paths, target):
-    input_path = "{}/files.txt".format(directory)
-
-    with open(input_path, 'w') as f:
-        for path in file_paths:
-            f.write('file {}\n'.format(os.path.basename(path)))
-
-    result = subprocess.run([
+def _join_vods(playlist_path, target, overwrite):
+    command = [
         "ffmpeg",
-        "-f", "concat",
-        "-i", input_path,
+        "-i", playlist_path,
         "-c", "copy",
         target,
         "-stats",
         "-loglevel", "warning",
-    ])
+    ]
 
-    result.check_returncode()
+    if overwrite:
+        command.append("-y")
+
+    print_out("<dim>{}</dim>".format(" ".join(command)))
+    result = subprocess.run(command)
+    if result.returncode != 0:
+        raise ConsoleError("Joining files failed")
 
 
 def _video_target_filename(video, format):
-    match = re.search(r"^(\d{4})-(\d{2})-(\d{2})T", video['published_at'])
+    match = re.search(r"^(\d{4})-(\d{2})-(\d{2})T", video['publishedAt'])
     date = "".join(match.groups())
 
     name = "_".join([
         date,
-        video['_id'][1:],
-        video['channel']['name'],
+        video['id'][1:],
+        video['creator']['login'],
         utils.slugify(video['title']),
     ])
 
     return name + "." + format
 
 
-def _get_files(playlist, start, end):
-    """Extract files for download from playlist."""
+def _clip_target_filename(clip):
+    url = clip["videoQualities"][0]["sourceURL"]
+    _, ext = path.splitext(url)
+    ext = ext.lstrip(".")
+
+    match = re.search(r"^(\d{4})-(\d{2})-(\d{2})T", clip["createdAt"])
+    date = "".join(match.groups())
+
+    name = "_".join([
+        date,
+        clip["id"],
+        clip["broadcaster"]["login"],
+        utils.slugify(clip["title"]),
+    ])
+
+    return "{}.{}".format(name, ext)
+
+
+def _get_vod_paths(playlist, start, end):
+    """Extract unique VOD paths for download from playlist."""
+    files = []
     vod_start = 0
     for segment in playlist.segments:
         vod_end = vod_start + segment.duration
@@ -148,46 +111,32 @@ def _get_files(playlist, start, end):
         start_condition = not start or vod_end > start
         end_condition = not end or vod_start < end
 
-        if start_condition and end_condition:
-            yield segment.uri
+        if start_condition and end_condition and segment.uri not in files:
+            files.append(segment.uri)
 
         vod_start = vod_end
 
+    return files
+
 
 def _crete_temp_dir(base_uri):
     """Create a temp dir to store downloads if it doesn't exist."""
-    path = urlparse(base_uri).path
-    directory = '{}/twitch-dl{}'.format(tempfile.gettempdir(), path)
-    pathlib.Path(directory).mkdir(parents=True, exist_ok=True)
-    return directory
-
-
-VIDEO_PATTERNS = [
-    r"^(?P<id>\d+)?$",
-    r"^https://(www.)?twitch.tv/videos/(?P<id>\d+)(\?.+)?$",
-]
-
-CLIP_PATTERNS = [
-    r"^(?P<slug>[A-Za-z0-9]+)$",
-    r"^https://(www.)?twitch.tv/\w+/clip/(?P<slug>[A-Za-z0-9]+)(\?.+)?$",
-    r"^https://clips.twitch.tv/(?P<slug>[A-Za-z0-9]+)(\?.+)?$",
-]
+    path = urlparse(base_uri).path.lstrip("/")
+    temp_dir = Path(tempfile.gettempdir(), "twitch-dl", path)
+    temp_dir.mkdir(parents=True, exist_ok=True)
+    return temp_dir
 
 
 def download(args):
-    for pattern in VIDEO_PATTERNS:
-        match = re.match(pattern, args.video)
-        if match:
-            video_id = match.group('id')
-            return _download_video(video_id, args)
+    video_id = utils.parse_video_identifier(args.video)
+    if video_id:
+        return _download_video(video_id, args)
 
-    for pattern in CLIP_PATTERNS:
-        match = re.match(pattern, args.video)
-        if match:
-            clip_slug = match.group('slug')
-            return _download_clip(clip_slug, args)
+    clip_slug = utils.parse_clip_identifier(args.video)
+    if clip_slug:
+        return _download_clip(clip_slug, args)
 
-    raise ConsoleError("Invalid video: {}".format(args.video))
+    raise ConsoleError("Invalid input: {}".format(args.video))
 
 
 def _get_clip_url(clip, args):
@@ -195,6 +144,9 @@ def _get_clip_url(clip, args):
 
     # Quality given as an argument
     if args.quality:
+        if args.quality == "source":
+            return qualities[0]["sourceURL"]
+
         selected_quality = args.quality.rstrip("p")  # allow 720p as well as 720
         for q in qualities:
             if q["quality"] == selected_quality:
@@ -232,18 +184,12 @@ def _download_clip(slug, args):
     url = _get_clip_url(clip, args)
     print_out("<dim>Selected URL: {}</dim>".format(url))
 
-    url_path = urlparse(url).path
-    extension = Path(url_path).suffix
-    filename = "{}_{}{}".format(
-        clip["broadcaster"]["login"],
-        utils.slugify(clip["title"]),
-        extension
-    )
+    target = _clip_target_filename(clip)
 
     print_out("Downloading clip...")
-    download_file(url, filename)
+    download_file(url, target)
 
-    print_out("Downloaded: {}".format(filename))
+    print_out("Downloaded: {}".format(target))
 
 
 def _download_video(video_id, args):
@@ -254,7 +200,7 @@ def _download_video(video_id, args):
     video = twitch.get_video(video_id)
 
     print_out("Found: <blue>{}</blue> by <yellow>{}</yellow>".format(
-        video['title'], video['channel']['display_name']))
+        video['title'], video['creator']['displayName']))
 
     print_out("<dim>Fetching access token...</dim>")
     access_token = twitch.get_access_token(video_id)
@@ -272,26 +218,43 @@ def _download_video(video_id, args):
 
     base_uri = re.sub("/[^/]+$", "/", playlist_uri)
     target_dir = _crete_temp_dir(base_uri)
-    filenames = list(_get_files(playlist, args.start, args.end))
+    vod_paths = _get_vod_paths(playlist, args.start, args.end)
 
     # Save playlists for debugging purposes
-    with open(target_dir + "playlists.m3u8", "w") as f:
+    with open(path.join(target_dir, "playlists.m3u8"), "w") as f:
         f.write(playlists_m3u8)
-    with open(target_dir + "playlist.m3u8", "w") as f:
+    with open(path.join(target_dir, "playlist.m3u8"), "w") as f:
         f.write(response.text)
 
     print_out("\nDownloading {} VODs using {} workers to {}".format(
-        len(filenames), args.max_workers, target_dir))
-    file_paths = download_files(base_uri, target_dir, filenames, args.max_workers)
+        len(vod_paths), args.max_workers, target_dir))
+    path_map = download_files(base_uri, target_dir, vod_paths, args.max_workers)
+
+    # Make a modified playlist which references downloaded VODs
+    # Keep only the downloaded segments and skip the rest
+    org_segments = playlist.segments.copy()
+    playlist.segments.clear()
+    for segment in org_segments:
+        if segment.uri in path_map:
+            segment.uri = path_map[segment.uri]
+            playlist.segments.append(segment)
+
+    playlist_path = path.join(target_dir, "playlist_downloaded.m3u8")
+    playlist.dump(playlist_path)
+
+    if args.no_join:
+        print_out("\n\n<dim>Skipping joining files...</dim>")
+        print_out("VODs downloaded to:\n<blue>{}</blue>".format(target_dir))
+        return
 
     print_out("\n\nJoining files...")
     target = _video_target_filename(video, args.format)
-    _join_vods(target_dir, file_paths, target)
+    _join_vods(playlist_path, target, args.overwrite)
 
     if args.keep:
-        print_out("\nTemporary files not deleted: {}".format(target_dir))
+        print_out("\n<dim>Temporary files not deleted: {}</dim>".format(target_dir))
     else:
-        print_out("\nDeleting temporary files...")
+        print_out("\n<dim>Deleting temporary files...</dim>")
         shutil.rmtree(target_dir)
 
-    print_out("Downloaded: {}".format(target))
+    print_out("\nDownloaded: <green>{}</green>".format(target))

twitchdl/commands/info.py

@@ -0,0 +1,78 @@
+import m3u8
+
+from twitchdl import utils, twitch
+from twitchdl.exceptions import ConsoleError
+from twitchdl.output import print_video, print_clip, print_json, print_out, print_log
+
+
+def info(args):
+    video_id = utils.parse_video_identifier(args.identifier)
+    if video_id:
+        print_log("Fetching video...")
+        video = twitch.get_video(video_id)
+
+        print_log("Fetching access token...")
+        access_token = twitch.get_access_token(video_id)
+
+        print_log("Fetching playlists...")
+        playlists = twitch.get_playlists(video_id, access_token)
+
+        if video:
+            if args.json:
+                video_json(video, playlists)
+            else:
+                video_info(video, playlists)
+            return
+
+        raise ConsoleError("Video #{} not found".format(video_id))
+
+    clip_slug = utils.parse_clip_identifier(args.identifier)
+    if clip_slug:
+        print_log("Fetching clip...")
+        clip = twitch.get_clip(clip_slug)
+        if clip:
+            if args.json:
+                print_json(clip)
+            else:
+                clip_info(clip)
+            return
+
+        raise ConsoleError("Clip {} not found".format(clip_slug))
+
+    raise ConsoleError("Invalid input: {}".format(args.video))
+
+
+def video_info(video, playlists):
+    print_out()
+    print_video(video)
+
+    print_out()
+    print_out("Playlists:")
+    for p in m3u8.loads(playlists).playlists:
+        print_out("<b>{}</b> {}".format(p.stream_info.video, p.uri))
+
+
+def video_json(video, playlists):
+    playlists = m3u8.loads(playlists).playlists
+
+    video["playlists"] = [
+        {
+            "bandwidth": p.stream_info.bandwidth,
+            "resolution": p.stream_info.resolution,
+            "codecs": p.stream_info.codecs,
+            "video": p.stream_info.video,
+            "uri": p.uri
+        } for p in playlists
+    ]
+
+    print_json(video)
+
+
+def clip_info(clip):
+    print_out()
+    print_clip(clip)
+    print_out()
+    print_out("Download links:")
+
+    for q in clip["videoQualities"]:
+        print_out("<b>{quality}p{frameRate}</b> {sourceURL}".format(**q))

twitchdl/commands/videos.py

@@ -0,0 +1,69 @@
+from twitchdl import twitch
+from twitchdl.exceptions import ConsoleError
+from twitchdl.output import print_out, print_video
+
+
+def _continue():
+    print_out(
+        "\nThere are more videos. "
+        "Press <green><b>Enter</green> to continue, "
+        "<yellow><b>Ctrl+C</yellow> to break."
+    )
+
+    try:
+        input()
+    except KeyboardInterrupt:
+        return False
+
+    return True
+
+
+def _get_game_ids(names):
+    if not names:
+        return []
+
+    game_ids = []
+    for name in names:
+        print_out("<dim>Looking up game '{}'...</dim>".format(name))
+        game_id = twitch.get_game_id(name)
+        if not game_id:
+            raise ConsoleError("Game '{}' not found".format(name))
+        game_ids.append(int(game_id))
+
+    return game_ids
+
+
+def videos(args):
+    game_ids = _get_game_ids(args.game)
+
+    print_out("<dim>Loading videos...</dim>")
+    generator = twitch.channel_videos_generator(
+        args.channel_name, args.limit, args.sort, args.type, game_ids=game_ids)
+
+    first = 1
+
+    for videos, has_more in generator:
+        count = len(videos["edges"]) if "edges" in videos else 0
+        total = videos["totalCount"]
+        last = first + count - 1
+
+        print_out("-" * 80)
+        print_out("<yellow>Showing videos {}-{} of {}</yellow>".format(first, last, total))
+
+        for video in videos["edges"]:
+            print_out()
+            print_video(video["node"])
+
+        if not args.pager and has_more:
+            print_out(
+                "\n<dim>There are more videos. "
+                "Increase the --limit or use --pager to see the rest.</dim>"
+            )
+            break
+
+        if not has_more or not _continue():
+            break
+
+        first += count
+    else:
+        print_out("<yellow>No videos found</yellow>")

twitchdl/console.py

@@ -77,6 +77,47 @@ COMMANDS = [
                 "choices": ["archive", "highlight", "upload"],
                 "default": "archive",
             }),
+            (["-p", "--pager"], {
+                "help": "If there are more results than LIMIT, ask to show next page",
+                "action": "store_true",
+                "default": False,
+            }),
+        ],
+    ),
+    Command(
+        name="clips",
+        description="List clips",
+        arguments=[
+            (["channel_name"], {
+                "help": "channel name",
+                "type": str,
+            }),
+            (["-l", "--limit"], {
+                "help": "Number of videos to fetch (default 10, max 100)",
+                "type": limit,
+                "default": 10,
+            }),
+            (["-P", "--period"], {
+                "help": "Period from which to return clips. (default: 'all_time')",
+                "type": str,
+                "choices": ["last_day", "last_week", "last_month", "all_time"],
+                "default": "all_time",
+            }),
+            (["-j", "--json"], {
+                "help": "Show results as JSON",
+                "action": "store_true",
+                "default": False,
+            }),
+            (["-p", "--pager"], {
+                "help": "If there are more results than LIMIT, ask to show next page",
+                "action": "store_true",
+                "default": False,
+            }),
+            (["-d", "--download"], {
+                "help": "Download all videos in given period (in source quality)",
+                "action": "store_true",
+                "default": False,
+            }),
         ],
     ),
     Command(
@@ -115,11 +156,36 @@ COMMANDS = [
                 "default": False,
             }),
             (["-q", "--quality"], {
-                "help": "Video quality.",
+                "help": "Video quality, e.g. 720p. Set to 'source' to get best quality.",
                 "type": str,
             }),
+            (["--no-join"], {
+                "help": "Don't run ffmpeg to join the downloaded vods, implies --keep.",
+                "action": "store_true",
+                "default": False,
+            }),
+            (["--overwrite"], {
+                "help": "Overwrite the target file if it already exists without prompting.",
+                "action": "store_true",
+                "default": False,
+            })
         ],
     ),
+    Command(
+        name="info",
+        description="Print information for a given Twitch URL, video ID or clip slug",
+        arguments=[
+            (["identifier"], {
+                "help": "identifier",
+                "type": str,
+            }),
+            (["-j", "--json"], {
+                "help": "Show results as JSON",
+                "action": "store_true",
+                "default": False,
+            }),
+        ],
+    )
 ]
 
 COMMON_ARGUMENTS = [

twitchdl/download.py

@@ -1,6 +1,7 @@
 import os
 import requests
 
+from collections import OrderedDict
 from concurrent.futures import ThreadPoolExecutor, as_completed
 from datetime import datetime
 from functools import partial
@@ -75,13 +76,17 @@ def _print_progress(futures):
         print_out("\r" + msg.ljust(max_msg_size), end="")
 
 
-def download_files(base_url, directory, filenames, max_workers):
-    urls = [base_url + f for f in filenames]
-    paths = ["{}{:05d}.vod".format(directory, k) for k, _ in enumerate(filenames)]
-    partials = (partial(download_file, url, path) for url, path in zip(urls, paths))
+def download_files(base_url, target_dir, vod_paths, max_workers):
+    """
+    Downloads a list of VODs defined by a common `base_url` and a list of
+    `vod_paths`, returning a dict which maps the paths to the downloaded files.
+    """
+    urls = [base_url + path for path in vod_paths]
+    targets = [os.path.join(target_dir, "{:05d}.ts".format(k)) for k, _ in enumerate(vod_paths)]
+    partials = (partial(download_file, url, path) for url, path in zip(urls, targets))
 
     with ThreadPoolExecutor(max_workers=max_workers) as executor:
         futures = [executor.submit(fn) for fn in partials]
         _print_progress(futures)
 
-    return paths
+    return OrderedDict(zip(vod_paths, targets))

twitchdl/output.py

@@ -1,5 +1,6 @@
 # -*- coding: utf-8 -*-
 
+import json
 import sys
 import re
 
@@ -51,26 +52,60 @@ def print_out(*args, **kwargs):
     print(*args, **kwargs)
 
 
+def print_json(data):
+    print(json.dumps(data))
+
+
 def print_err(*args, **kwargs):
     args = ["<red>{}</red>".format(a) for a in args]
     args = [colorize(a) if USE_ANSI_COLOR else strip_tags(a) for a in args]
     print(*args, file=sys.stderr, **kwargs)
 
 
+def print_log(*args, **kwargs):
+    args = ["<dim>{}</dim>".format(a) for a in args]
+    args = [colorize(a) if USE_ANSI_COLOR else strip_tags(a) for a in args]
+    print(*args, file=sys.stderr, **kwargs)
+
+
 def print_video(video):
     published_at = video["publishedAt"].replace("T", " @ ").replace("Z", "")
     length = utils.format_duration(video["lengthSeconds"])
-    channel = video["creator"]["channel"]["displayName"]
+    channel = video["creator"]["displayName"]
     playing = (
-        " playing <blue>{}</blue>".format(video["game"]["name"])
+        "playing <blue>{}</blue>".format(video["game"]["name"])
         if video["game"] else ""
     )
 
     # Can't find URL in video object, strange
-    url = "https://twitch.tv/videos/{}".format(video["id"])
+    url = "https://www.twitch.tv/videos/{}".format(video["id"])
 
-    print_out("\n<b>{}</b>".format(video["id"]))
+    print_out("<b>Video {}</b>".format(video["id"]))
     print_out("<green>{}</green>".format(video["title"]))
     print_out("<blue>{}</blue> {}".format(channel, playing))
     print_out("Published <blue>{}</blue>  Length: <blue>{}</blue> ".format(published_at, length))
     print_out("<i>{}</i>".format(url))
+
+
+def print_clip(clip):
+    published_at = clip["createdAt"].replace("T", " @ ").replace("Z", "")
+    length = utils.format_duration(clip["durationSeconds"])
+    channel = clip["broadcaster"]["displayName"]
+    playing = (
+        "playing <blue>{}</blue>".format(clip["game"]["name"])
+        if clip["game"] else ""
+    )
+
+    print_out("Clip <b>{}</b>".format(clip["slug"]))
+    print_out("<green>{}</green>".format(clip["title"]))
+    print_out("<blue>{}</blue> {}".format(channel, playing))
+    print_out(
+        "Published <blue>{}</blue>"
+        "  Length: <blue>{}</blue>"
+        "  Views: <blue>{}</blue>".format(published_at, length, clip["viewCount"]))
+    print_out("<i>{}</i>".format(clip["url"]))
+
+
+def print_clip_urls(clip):
+    from pprint import pprint
+    pprint(clip)

twitchdl/twitch.py

@@ -61,7 +61,7 @@ def gql_query(query):
     return response
 
 
-def get_video(video_id):
+def get_video_legacy(video_id):
     """
     https://dev.twitch.tv/docs/v5/reference/videos#get-video
     """
@@ -70,24 +70,61 @@ def get_video(video_id):
     return kraken_get(url).json()
 
 
+VIDEO_FIELDS = """
+    id
+    title
+    publishedAt
+    broadcastType
+    lengthSeconds
+    game {
+        name
+    }
+    creator {
+        login
+        displayName
+    }
+"""
+
+
+def get_video(video_id):
+    query = """
+    {{
+        video(id: "{video_id}") {{
+            {fields}
+        }}
+    }}
+    """
+
+    query = query.format(video_id=video_id, fields=VIDEO_FIELDS)
+
+    response = gql_query(query)
+    return response["data"]["video"]
+
+
 def get_clip(slug):
     query = """
     {{
         clip(slug: "{}") {{
+            id
+            slug
             title
+            createdAt
+            viewCount
             durationSeconds
-            game {{
-                name
-            }}
-            broadcaster {{
-                login
-                displayName
-            }}
+            url
             videoQualities {{
                 frameRate
                 quality
                 sourceURL
             }}
+            game {{
+                id
+                name
+            }}
+            broadcaster {{
+                displayName
+                login
+            }}
         }}
     }}
     """
@@ -96,6 +133,87 @@ def get_clip(slug):
     return response["data"]["clip"]
 
 
+def get_channel_clips(channel_id, period, limit, after=None):
+    """
+    List channel clips.
+
+    At the time of writing this:
+    * filtering by game name returns an error
+    * sorting by anything but VIEWS_DESC or TRENDING returns an error
+    * sorting by VIEWS_DESC and TRENDING returns the same results
+    * there is no totalCount
+    """
+    query = """
+    {{
+      user(login: "{channel_id}") {{
+        clips(first: {limit}, after: "{after}", criteria: {{ period: {period}, sort: VIEWS_DESC }}) {{
+          pageInfo {{
+            hasNextPage
+            hasPreviousPage
+          }}
+          edges {{
+            cursor
+            node {{
+              id
+              slug
+              title
+              createdAt
+              viewCount
+              durationSeconds
+              url
+              videoQualities {{
+                frameRate
+                quality
+                sourceURL
+              }}
+              game {{
+                id
+                name
+              }}
+              broadcaster {{
+                displayName
+                login
+              }}
+            }}
+          }}
+        }}
+      }}
+    }}
+    """
+
+    query = query.format(**{
+        "channel_id": channel_id,
+        "after": after if after else "",
+        "limit": limit,
+        "period": period.upper(),
+    })
+
+    response = gql_query(query)
+    user = response["data"]["user"]
+    if not user:
+        raise ConsoleError("Channel {} not found".format(channel_id))
+
+    return response["data"]["user"]["clips"]
+
+
+def channel_clips_generator(channel_id, period, limit):
+    cursor = ""
+    while True:
+        clips = get_channel_clips(
+            channel_id, period, limit, after=cursor)
+
+        if not clips["edges"]:
+            break
+
+        has_next = clips["pageInfo"]["hasNextPage"]
+        cursor = clips["edges"][-1]["cursor"] if has_next else None
+
+        yield clips, has_next
+
+        if not cursor:
+            break
+
+
 def get_channel_videos(channel_id, limit, sort, type="archive", game_ids=[], after=None):
     query = """
     {{
@@ -125,9 +243,8 @@ def get_channel_videos(channel_id, limit, sort, type="archive", game_ids=[], aft
                             name
                         }}
                         creator {{
-                            channel {{
-                                displayName
-                            }}
+                            login
+                            displayName
                         }}
                     }}
                 }}
@@ -139,7 +256,7 @@ def get_channel_videos(channel_id, limit, sort, type="archive", game_ids=[], aft
     query = query.format(**{
         "channel_id": channel_id,
         "game_ids": game_ids,
-        "after": after,
+        "after": after if after else "",
         "limit": limit,
         "sort": sort.upper(),
         "type": type.upper(),
@@ -150,7 +267,7 @@ def get_channel_videos(channel_id, limit, sort, type="archive", game_ids=[], aft
 
 
 def channel_videos_generator(channel_id, limit, sort, type, game_ids=None):
-    cursor = None
+    cursor = ""
     while True:
         videos = get_channel_videos(
             channel_id, limit, sort, type, game_ids=game_ids, after=cursor)
@@ -168,9 +285,26 @@ def channel_videos_generator(channel_id, limit, sort, type, game_ids=None):
 
 
 def get_access_token(video_id):
-    url = "https://api.twitch.tv/api/vods/{}/access_token".format(video_id)
+    query = """
+    {{
+        videoPlaybackAccessToken(
+            id: {video_id},
+            params: {{
+                platform: "web",
+                playerBackend: "mediaplayer",
+                playerType: "site"
+            }}
+        ) {{
+            signature
+            value
+        }}
+    }}
+    """
 
-    return authenticated_get(url).json()
+    query = query.format(video_id=video_id)
+
+    response = gql_query(query)
+    return response["data"]["videoPlaybackAccessToken"]
 
 
 def get_playlists(video_id, access_token):
@@ -180,8 +314,8 @@ def get_playlists(video_id, access_token):
     url = "http://usher.twitch.tv/vod/{}".format(video_id)
 
     response = requests.get(url, params={
-        "nauth": access_token['token'],
-        "nauthsig": access_token['sig'],
+        "nauth": access_token['value'],
+        "nauthsig": access_token['signature'],
         "allow_source": "true",
         "player": "twitchweb",
     })

twitchdl/utils.py

@@ -61,3 +61,31 @@ def slugify(value):
     value = unicodedata.normalize('NFKC', value)
     value = re_pattern.sub('', value).strip().lower()
     return re_spaces.sub('_', value)
+
+
+VIDEO_PATTERNS = [
+    r"^(?P<id>\d+)?$",
+    r"^https://(www.)?twitch.tv/videos/(?P<id>\d+)(\?.+)?$",
+]
+
+CLIP_PATTERNS = [
+    r"^(?P<slug>[A-Za-z0-9]+)$",
+    r"^https://(www.)?twitch.tv/\w+/clip/(?P<slug>[A-Za-z0-9]+)(\?.+)?$",
+    r"^https://clips.twitch.tv/(?P<slug>[A-Za-z0-9]+)(\?.+)?$",
+]
+
+
+def parse_video_identifier(identifier):
+    """Given a video ID or URL returns the video ID, or null if not matched"""
+    for pattern in VIDEO_PATTERNS:
+        match = re.match(pattern, identifier)
+        if match:
+            return match.group("id")
+
+
+def parse_clip_identifier(identifier):
+    """Given a clip slug or URL returns the clip slug, or null if not matched"""
+    for pattern in CLIP_PATTERNS:
+        match = re.match(pattern, identifier)
+        if match:
+            return match.group("slug")