Update changelog

data works, but should be used for form data passed as a dict, while
content takes bytes or str

.github/workflows/test.yml

@@ -0,0 +1,27 @@
+name: Run tests
+
+on: [push, pull_request]
+
+jobs:
+  test:
+    runs-on: ubuntu-22.04
+    strategy:
+      matrix:
+        python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
+
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[test]"
+      - name: Run tests
+        run: |
+          pytest
+      - name: Validate minimum required version
+        run: |
+          vermin --no-tips twitchdl

.gitignore

@@ -13,3 +13,5 @@ tmp/
 /twitch-dl.1.man
 /bundle
 /*.pyz
+/pyrightconfig.json
+/book

.vermin

@@ -0,0 +1,4 @@
+[vermin]
+only_show_violations = yes
+show_tips = no
+targets = 3.8

CHANGELOG.md

@@ -1,18 +1,164 @@
-Twitch Downloader change log
-============================
+twitch-dl changelog
+===================
 
-1.13.0 (2020-11-10)
--------------------
+<!-- Do not edit. This file is automatically generated from changelog.yaml.-->
+
+### [2.2.4 (2024-04-25)](https://github.com/ihabunek/twitch-dl/releases/tag/2.2.4)
+
+* Add m dot url support to video and clip regexes (thanks @localnerve)
+
+### [2.2.3 (2024-04-24)](https://github.com/ihabunek/twitch-dl/releases/tag/2.2.3)
+
+* Respect --dry-run option when downloading videos
+* Add automated tests on github actions
+
+### [2.2.2 (2024-04-23)](https://github.com/ihabunek/twitch-dl/releases/tag/2.2.2)
+
+* Fix more compat issues Python < 3.10 (#152)
+
+### [2.2.1 (2024-04-23)](https://github.com/ihabunek/twitch-dl/releases/tag/2.2.1)
+
+* Fix compat with Python < 3.10 (#152)
+* Fix division by zero in progress calculation when video duration is reported
+  as 0
+
+### [2.2.0 (2024-04-10)](https://github.com/ihabunek/twitch-dl/releases/tag/2.2.0)
+
+* **Requires Python 3.8+**
+* Migrated to Click library for generating the commandline interface
+* Add shell auto completion, see 'Shell completion' in docs.
+* Add setting defaults via environment variables, see 'Environment variables' in
+  docs
+* Add `download --concat` option to avoid using ffmeg for joinig vods and concat
+  them instead. This will produce a `.ts` file by default.
+* Add `download --dry-run` option to skip actual download (thanks @metacoma)
+* Add video description to metadata (#129)
+* Add `clips --compact` option for listing clips in one-per-line mode
+
+### [2.1.4 (2024-01-06)](https://github.com/ihabunek/twitch-dl/releases/tag/2.1.4)
+
+* Fix error caused by twitch requiring https for the usher api (thanks
+  @deanpcmad)
+
+### [2.1.3 (2023-05-07)](https://github.com/ihabunek/twitch-dl/releases/tag/2.1.3)
+
+* Replace client ID with one that works for now (thanks @mwhite34)
+
+### [2.1.2 (2023-04-18)](https://github.com/ihabunek/twitch-dl/releases/tag/2.1.2)
+
+* Fix error caused by twitch changing the Usher domain (thanks @adsa95)
+
+### [2.1.1 (2022-11-20)](https://github.com/ihabunek/twitch-dl/releases/tag/2.1.1)
+
+* Fix Python 3.7 compatibility (#117, thanks @eliduvid)
+* Fix default value for game_ids (#102, thanks @FunnyPocketBook)
+
+### [2.1.0 (2022-11-20)](https://github.com/ihabunek/twitch-dl/releases/tag/2.1.0)
+
+* Add chapter list to `info` command
+* Add `download --chapter` option for downloading a single chapter
+
+### [2.0.1 (2022-09-09)](https://github.com/ihabunek/twitch-dl/releases/tag/2.0.1)
+
+* Fix an issue where a temp vod file would be renamed while still being open,
+  which caused an exception on Windows (#111)
+
+### [2.0.0 (2022-08-18)](https://github.com/ihabunek/twitch-dl/releases/tag/2.0.0)
+
+This release switches from using `requests` to `httpx` for making http requests,
+and from threads to `asyncio` for concurrency. This enables easier
+implementation of new features, but has no breaking changes for the CLI.
+
+* **BREAKING**: Require Python 3.7 or later.
+* Add `--rate-limit` option to `download` for limiting maximum bandwidth when
+  downloading.
+* Add `--compact` option to `download` for displaying one video per line.
+* Allow passing multiple video ids to `download` to download multiple videos
+  successively.
+* Improved progress meter, updates on each chunk downloaded, instead of each VOD
+  downloaded.
+* Improved speed estimate, displays recent speed instead of average speed since
+  the start of download.
+* Decreased default concurrent downloads to 5. This seems to be enough to
+  saturate the download link in most cases. You can override this by setting the
+  `-w` option. Please test and report back if this works for you.
+
+### [1.22.0 (2022-06-25)](https://github.com/ihabunek/twitch-dl/releases/tag/1.22.0)
+
+* Add support for downloading subscriber-only VODs (#48, thanks @cemiu)
+
+### [1.21.0 (2022-02-27)](https://github.com/ihabunek/twitch-dl/releases/tag/1.21.0)
+
+* Add `env` command for printing environment info for attaching to bug reports
+
+### [1.20.0 (2022-02-25)](https://github.com/ihabunek/twitch-dl/releases/tag/1.20.0)
+
+* Add `--json` option to `videos` command (#92, thanks @miff2000)
+* Add `--all` option to `videos` and `clips` commands to list all clips or
+  videos in one go.
+* Modify how `--pager` works, will make multiple requests if needed to show all
+  available items, ignoring `--limit`.
+
+### [1.19.0 (2022-02-05)](https://github.com/ihabunek/twitch-dl/releases/tag/1.19.0)
+
+* Add support for downloading audio only (#10)
+
+### [1.18.1 (2022-02-05)](https://github.com/ihabunek/twitch-dl/releases/tag/1.18.1)
+
+* Fix issues with output formats (#87, #89)
+* Fix issues when downloading clip with no game set (#78)
+* Add option to use clip slug in `--output` format
+
+### [1.18.0 (2022-01-25)](https://github.com/ihabunek/twitch-dl/releases/tag/1.18.0)
+
+* Add `--output` option to `download` command which allows setting output file
+  template (#70)
+* Ask to overwrite before downloading to avoid later prompt
+
+### [1.17.1 (2022-01-19)](https://github.com/ihabunek/twitch-dl/releases/tag/1.17.1)
+
+* Upgrade m3u8 lib to 1.0.0+
+
+### [1.17.0 (2021-12-03)](https://github.com/ihabunek/twitch-dl/releases/tag/1.17.0)
+
+* Fix speed calculation when resuming download (#75, thanks CroquetteTheThe)
+* Add artist and title metadata to resulting video (#80)
+
+### [1.16.1 (2021-07-31)](https://github.com/ihabunek/twitch-dl/releases/tag/1.16.1)
+
+* Fix compat with older versions of python (#71)
+
+### [1.16.0 (2021-06-09)](https://github.com/ihabunek/twitch-dl/releases/tag/1.16.0)
+
+* Fix clips download caused by Twitch changes (#64, thanks to all participants)
+
+### [1.15.0 (2021-02-15)](https://github.com/ihabunek/twitch-dl/releases/tag/1.15.0)
+
+* Add support for new format of clip slug (thanks @Loveangel1337)
+
+### [1.14.1 (2021-01-14)](https://github.com/ihabunek/twitch-dl/releases/tag/1.14.1)
+
+* Handle videos which don't exist more gracefully
+
+### [1.14.0 (2021-01-14)](https://github.com/ihabunek/twitch-dl/releases/tag/1.14.0)
+
+* Added `info` command for displaying video or clip info (#51)
+* Don't show there are more videos when there aren't (#52, thanks @scottyallen)
+* Fixed Twitch regression for getting the access token (#53)
+
+### [1.13.1 (2020-11-23)](https://github.com/ihabunek/twitch-dl/releases/tag/1.13.1)
+
+* Fixed clip download issue (#45)
+
+### [1.13.0 (2020-11-10)](https://github.com/ihabunek/twitch-dl/releases/tag/1.13.0)
 
 * Added `clips` command for listing and batch downloading clips (#26)
 
-1.12.1 (2020-09-29)
--------------------
+### [1.12.1 (2020-09-29)](https://github.com/ihabunek/twitch-dl/releases/tag/1.12.1)
 
 * Fix bug introduced in previous version which broke joining
 
-1.12.0 (2020-09-29)
--------------------
+### [1.12.0 (2020-09-29)](https://github.com/ihabunek/twitch-dl/releases/tag/1.12.0)
 
 * Added `source` as alias for best available quality (#33)
 * Added `--no-join` option to `download` to skip ffmpeg join (#36)
@@ -20,43 +166,37 @@ Twitch Downloader change log
   for confirmation (#37)
 * Added `--pager` option to `videos`, don't page by default (#30)
 
-1.11.0 (2020-09-03)
--------------------
+### [1.11.0 (2020-09-03)](https://github.com/ihabunek/twitch-dl/releases/tag/1.11.0)
 
 * Make downloading more robust, fixes issues with some VODs (#35)
 * Bundle twitch-dl to a standalone archive, simplifying installation, see
   installation instructions in README
 
-1.10.2 (2020-08-11)
--------------------
+### [1.10.2 (2020-08-11)](https://github.com/ihabunek/twitch-dl/releases/tag/1.10.2)
 
 * Fix version number displayed by `twitch-dl --version` (#29)
 
-1.10.1 (2020-08-09)
--------------------
+### [1.10.1 (2020-08-09)](https://github.com/ihabunek/twitch-dl/releases/tag/1.10.1)
 
 * Fix videos incorrectly identified as clips (#28)
-* Make download command work with video URLs lacking "www" before "twitch.tv"
+* Make download command work with video URLs lacking 'www' before 'twitch.tv'
 * Print an error when video or clip is not found instead of an exception trace
 
-1.10.0 (2020-08-07)
--------------------
+### [1.10.0 (2020-08-07)](https://github.com/ihabunek/twitch-dl/releases/tag/1.10.0)
 
 * Add `--quality` option to `download` command, allows specifying the video
   quality to download. In this case, twitch-dl will require no user input. (#22)
 * Fix download of clips which contain numbers in their slug (#24)
 * Fix URL to video displayed by `videos` command (it was missing /videos/)
 
-1.9.0 (2020-06-10)
-------------------
+### [1.9.0 (2020-06-10)](https://github.com/ihabunek/twitch-dl/releases/tag/1.9.0)
 
 * **Breaking**: wrongly named `--max_workers` option changed to `--max-workers`.
   The shorthand option `-w` remains the same.
 * Fix bug where `videos` command would crash if there was no game info (#21)
 * Allow unicode characters in filenames, no longer strips e.g. cyrillic script
 
-1.8.0 (2020-05-17)
-------------------
+### [1.8.0 (2020-05-17)](https://github.com/ihabunek/twitch-dl/releases/tag/1.8.0)
 
 * Fix videos command (#18)
 * **Breaking**: `videos` command no longer takes the `--offset` parameter due to
@@ -64,58 +204,50 @@ Twitch Downloader change log
 * Add paging to `videos` command to replace offset
 * Add `--game` option to `videos` command to filter by game
 
-1.7.0 (2020-04-25)
-------------------
+### [1.7.0 (2020-04-25)](https://github.com/ihabunek/twitch-dl/releases/tag/1.7.0)
 
 * Support for specifying broadcast type when listing videos (#13)
 
-1.6.0 (2020-04-11)
-------------------
+### [1.6.0 (2020-04-11)](https://github.com/ihabunek/twitch-dl/releases/tag/1.6.0)
 
 * Support for downloading clips (#15)
 
-1.5.1 (2020-04-11)
-------------------
+### [1.5.1 (2020-04-11)](https://github.com/ihabunek/twitch-dl/releases/tag/1.5.1)
 
 * Fix VOD naming issue (#12)
 * Nice console output while downloading
 
-1.5.0 (2020-04-10)
-------------------
+### [1.5.0 (2020-04-10)](https://github.com/ihabunek/twitch-dl/releases/tag/1.5.0)
 
 * Fix video downloads after Twitch deprecated access token access
 * Don't print errors when retrying download, only if all fails
 
-1.4.0 (2019-08-23)
-------------------
+### [1.4.0 (2019-08-23)](https://github.com/ihabunek/twitch-dl/releases/tag/1.4.0)
 
 * Fix usage of deprecated v3 API
 * Use m3u8 lib for parsing playlists
 * Add `--keep` option not preserve downloaded VODs
 
-1.3.1 (2019-08-13)
-------------------
+### [1.3.1 (2019-08-13)](https://github.com/ihabunek/twitch-dl/releases/tag/1.3.1)
 
 * No changes, bumped to fix issue with pypi
 
-1.3.0 (2019-08-13)
-------------------
+### [1.3.0 (2019-08-13)](https://github.com/ihabunek/twitch-dl/releases/tag/1.3.0)
 
 * Add `--sort` and `--offset` options to `videos` command, allows paging (#7)
 * Show video URL in `videos` command output
 
-1.2.0 (2019-07-05)
-------------------
+### [1.2.0 (2019-07-05)](https://github.com/ihabunek/twitch-dl/releases/tag/1.2.0)
 
-* Add `--format` option to `download` command for specifying the output format (#6)
+* Add `--format` option to `download` command for specifying the output format
+  (#6)
 * Add `--version` option for printing program version
 
-1.1.0 (2019-06-06)
-------------------
+### [1.1.0 (2019-06-06)](https://github.com/ihabunek/twitch-dl/releases/tag/1.1.0)
 
 * Allow limiting download by start and end time
 
-1.0.0 (2019-04-30)
-------------------
+### [1.0.0 (2019-04-30)](https://github.com/ihabunek/twitch-dl/releases/tag/1.0.0)
 
 * Initial release
+

Makefile

@@ -1,12 +1,13 @@
+.PHONY: docs
+
 default : clean dist
 
-dist :
-	python setup.py sdist --formats=gztar,zip
-	python setup.py bdist_wheel --python-tag=py3
+dist:
+	python -m build
 
 clean :
 	find . -name "*pyc" | xargs rm -rf $1
-	rm -rf build dist bundle MANIFEST htmlcov deb_dist twitch-dl*.tar.gz twitch-dl.1.man
+	rm -rf build dist bundle MANIFEST htmlcov deb_dist twitch-dl.*.pyz twitch-dl.1.man twitch_dl.egg-info
 
 bundle:
 	mkdir bundle
@@ -23,13 +24,24 @@ publish :
 	twine upload dist/*.tar.gz dist/*.whl
 
 coverage:
-	py.test --cov=toot --cov-report html tests/
-
-deb:
-	@python setup.py --command-packages=stdeb.command bdist_deb
+	pytest --cov=twitchdl --cov-report html tests/
 
 man:
 	scdoc < twitch-dl.1.scd > twitch-dl.1.man
 
 test:
 	pytest
+
+changelog:
+	./scripts/generate_changelog > CHANGELOG.md
+
+docs: changelog
+	python scripts/generate_docs
+	mdbook build
+
+docs-serve:
+	python scripts/generate_docs
+	mdbook serve --port 8000
+
+docs-deploy: docs
+	rsync --archive --compress --delete --stats book/ bezdomni:web/twitch-dl

README.md

@@ -9,213 +9,70 @@ multiple concurrent connections to make the download faster.
 Resources
 ---------
 
-* Homepage: https://github.com/ihabunek/twitch-dl
-* Issues: https://github.com/ihabunek/twitch-dl/issues
-* Python package: https://pypi.org/project/twitch-dl/
+* [Documentation](https://twitch-dl.bezdomni.net/)
+* [Source code](https://github.com/ihabunek/twitch-dl)
+* [Issues](https://github.com/ihabunek/twitch-dl/issues)
+* [Python package](https://pypi.org/project/twitch-dl/)
 
 Requirements
 ------------
 
-* Python 3.5+
+* Python 3.8 or later
 * [ffmpeg](https://ffmpeg.org/download.html), installed and on the system path
 
-Installation
-------------
+Quick start
+-----------
 
-### Download standalone archive
+See [installation instructions](https://twitch-dl.bezdomni.net/installation.html)
+to set up twitch-dl.
 
-Go to the [latest release](https://github.com/ihabunek/twitch-dl/releases/latest)
-and download the `twitch-dl.<version>.pyz` archive.
-
-Run the archive by either:
-
-a) passing it to python:
-
-```
-python3 twitch-dl.1.13.0.pyz --help
-```
-
-b) making it executable and invoking it directly (linux specific):
-
-```
-chmod +x twitch-dl.1.13.0.pyz
-./twitch-dl.1.13.0.pyz --help
-```
-
-Feel free to rename the archive to something more managable, like `twitch-dl`.
-
-To upgrade to a newer version, repeat the process with the newer release.
-
-### From PYPI using pipx
-
-**pipx** is a tool which installs python apps into isolated environments, which
-prevents all kinds of problems later so it's the suggested way to install
-twitch-dl from PYPI.
-
-Install pipx as described in
-[pipx install docs](https://pipxproject.github.io/pipx/installation/).
-
-Install twitch-dl:
-
-```
-pipx install twitch-dl
-```
-
-Check installation worked:
-
-```
-twitch-dl --help
-```
-
-If twitch-dl executable is not found, check that the pipx binary location (by
-default `~/.local/bin`) is in your PATH.
-
-To upgrade twitch-dl to the latest version:
-
-```
-pipx install twitch-dl
-```
-
-Usage
------
-
-### Listing videos
-
-List recent streams for a given channel:
+List videos from a channel.
 
 ```
 twitch-dl videos bananasaurus_rex
 ```
 
-Yields (trimmed):
+List clips from a channel.
 
 ```
-Found 33 videos
-
-221837124
-SUPER MARIO ODYSSSEY - Stream #2 / 600,000,000
-Bananasaurus_Rex playing Super Mario Odyssey
-Published 2018-01-24 @ 12:05:25  Length: 3h 40min
-
-221418913
-Dead Space and then SUPER MARIO ODYSSEY PogChamp
-Bananasaurus_Rex playing Dead Space
-Published 2018-01-23 @ 02:40:58  Length: 6h 2min
-
-220783179
-Dead Space | Got my new setup working! rexChamp
-Bananasaurus_Rex playing Dead Space
-Published 2018-01-21 @ 05:47:03  Length: 5h 7min
+twitch-dl clips bananasaurus_rex
 ```
 
-Use the `--game` option to specify one or more games to show:
+Download a video by URL.
 
 ```
-twitch-dl videos --game "doom eternal" --game "cave story" bananasaurus_rex
+twitch-dl download https://www.twitch.tv/videos/1418494769
 ```
 
-### Downloading videos
-
-Download a stream by ID or URL:
+or by ID
 
 ```
-twitch-dl download 221837124
-twitch-dl download https://www.twitch.tv/videos/221837124
+twitch-dl download 1418494769
 ```
 
-Specify video quality to download:
+Download a clip by URL
 
 ```
-twitch-dl download -q 720p 221837124
+twitch-dl download https://www.twitch.tv/bananasaurus_rex/clip/PlacidColdClipsdadDeIlluminati-hL2s_aLE4CHvVN4J
 ```
 
-Setting quality to `source` will download the best available quality:
+or by slug
 
 ```
-twitch-dl download -q source 221837124
+twitch-dl download PlacidColdClipsdadDeIlluminati-hL2s_aLE4CHvVN4J
 ```
 
-### Listing clips
-
-List clips for the given period:
-
-```
-twitch-dl clips bananasaurus_rex --period last_week
-```
-
-Supported periods are: `last_day`, `last_week`, `last_month`, `all_time`.
-
-For listing a large number of clips, it's nice to page them:
-
-```
-twitch-dl clips bananasaurus_rex --period all_time --limit 10 --pager
-```
-
-This will show 10 clips at a time and ask to continue.
-
-### Downloading clips
-
-Download a clip by slug or URL:
-
-```
-twitch-dl download VenomousTameWormHumbleLife
-twitch-dl download https://www.twitch.tv/bananasaurus_rex/clip/VenomousTameWormHumbleLife
-```
-
-Specify clip quality to download:
-
-```
-twitch-dl download -q 720 VenomousTameWormHumbleLife
-```
-
-Note that twitch names for clip qualities have no trailing "p".
-
-### Batch downloading clips
-
-It's possible to download all clips for a given period:
-
-```
-twitch-dl clips bananasaurus_rex --period last_week --download
-```
-
-Clips are downloaded in source quality.
-
-A note about clips
-------------------
-
-Currently it doesn't seem to be possible to get a list of clips ordered by time
-of creation, only by view count. Clips with the same view count seem to be
-returned in random order. This can break paging resulting in duplicate clips
-listed or clips missed.
-
-When batch downloading a large number of clips (over 100), it's possible that
-some will be missed.
-
-Temporary files
----------------
-
-By default, twitch-dl will download VODs to your systems temp dir (e.g. `/tmp/`
-on Linux). To change the location where the files are downloaded you can set
-the `TMP` environment variable, e.g.
-
-```
-TMP=/my/tmp/path/ twitch-dl download 221837124
-```
-
-Man page
---------
-
-Building the man page for twitch-dl requires scdoc.
-
-The source is in ``twitch-dl.1.scd``, and you can build it by running:
-
-```
-make man
-```
+For more info see [the documentation](https://twitch-dl.bezdomni.net/usage.html).
 
 License
 -------
 
-Copyright 2018-2020 Ivan Habunek <ivan@habunek.com>
+Copyright 2018-2022 Ivan Habunek <ivan@habunek.com>
 
 Licensed under the GPLv3: http://www.gnu.org/licenses/gpl-3.0.html
+
+Useful links for dev
+--------------------
+
+* https://supersonichub1.github.io/twitch-graphql-api/index.html
+* https://github.com/SuperSonicHub1/twitch-graphql-api

TODO.md

@@ -0,0 +1,8 @@
+TODO
+====
+
+Some ideas what to do next.
+
+* gracefully handle aborting the download with Ctrl+C, now it prints out an error stack
+* add keyboard control for e.g. pausing a download
+* test how worker count affects download speeds on low and high-bandwidth links (see https://github.com/ihabunek/twitch-dl/issues/104), adjust default worker count

book.css

@@ -0,0 +1,20 @@
+/* Overrides for the docs theme  */
+table { width: 100% }
+table th { text-align: left }
+code { white-space: pre }
+h2, h3 { margin-top: 2.5rem; }
+h4, h5 { margin-top: 2rem; }
+
+td.code {
+    font-family: "Source Code Pro", Consolas, "Ubuntu Mono", Menlo, "DejaVu Sans Mono", monospace, monospace !important;
+    font-size: 0.875em;
+    width: 20%;
+    white-space: nowrap;
+}
+
+/* Override max width */
+.searchbar-outer,
+.searchresults-outer,
+.content main  {
+    max-width: 1024px;
+}

book.toml

@@ -0,0 +1,10 @@
+[book]
+title = "twitch-dl Documentation"
+description = "CLI tool for downloading videos from twitch.tv"
+author = "Ivan Habunek"
+language = "en"
+multilingual = false
+src = "docs"
+
+[output.html]
+additional-css = ["book.css"]

changelog.yaml

@@ -0,0 +1,276 @@
+2.2.4:
+  date: 2024-04-25
+  changes:
+    - "Add m dot url support to video and clip regexes (thanks @localnerve)"
+
+2.2.3:
+  date: 2024-04-24
+  changes:
+    - "Respect --dry-run option when downloading videos"
+    - "Add automated tests on github actions"
+
+2.2.2:
+  date: 2024-04-23
+  changes:
+    - "Fix more compat issues Python < 3.10 (#152)"
+
+2.2.1:
+  date: 2024-04-23
+  changes:
+    - "Fix compat with Python < 3.10 (#152)"
+    - "Fix division by zero in progress calculation when video duration is reported as 0"
+
+2.2.0:
+  date: 2024-04-10
+  changes:
+    - "**Requires Python 3.8+**"
+    - "Migrated to Click library for generating the commandline interface"
+    - "Add shell auto completion, see 'Shell completion' in docs."
+    - "Add setting defaults via environment variables, see 'Environment variables' in docs"
+    - "Add `download --concat` option to avoid using ffmeg for joinig vods and concat them instead. This will produce a `.ts` file by default."
+    - "Add `download --dry-run` option to skip actual download (thanks @metacoma)"
+    - "Add video description to metadata (#129)"
+    - "Add `clips --compact` option for listing clips in one-per-line mode"
+
+2.1.4:
+  date: 2024-01-06
+  changes:
+    - "Fix error caused by twitch requiring https for the usher api (thanks @deanpcmad)"
+
+2.1.3:
+  date: 2023-05-07
+  changes:
+    - "Replace client ID with one that works for now (thanks @mwhite34)"
+
+2.1.2:
+  date: 2023-04-18
+  changes:
+    - "Fix error caused by twitch changing the Usher domain (thanks @adsa95)"
+
+2.1.1:
+  date: 2022-11-20
+  changes:
+    - "Fix Python 3.7 compatibility (#117, thanks @eliduvid)"
+    - "Fix default value for game_ids (#102, thanks @FunnyPocketBook)"
+
+2.1.0:
+  date: 2022-11-20
+  changes:
+    - "Add chapter list to `info` command"
+    - "Add `download --chapter` option for downloading a single chapter"
+
+2.0.1:
+  date: 2022-09-09
+  changes:
+    - "Fix an issue where a temp vod file would be renamed while still being open, which caused an exception on Windows (#111)"
+
+2.0.0:
+  date: 2022-08-18
+  description: |
+    This release switches from using `requests` to `httpx` for making http
+    requests, and from threads to `asyncio` for concurrency. This enables
+    easier implementation of new features, but has no breaking changes for the
+    CLI.
+  changes:
+    - "**BREAKING**: Require Python 3.7 or later."
+    - "Add `--rate-limit` option to `download` for limiting maximum bandwidth when downloading."
+    - "Add `--compact` option to `download` for displaying one video per line."
+    - "Allow passing multiple video ids to `download` to download multiple videos successively."
+    - "Improved progress meter, updates on each chunk downloaded, instead of each VOD downloaded."
+    - "Improved speed estimate, displays recent speed instead of average speed since the start of download."
+    - |
+      Decreased default concurrent downloads to 5. This seems to be enough to
+      saturate the download link in most cases. You can override this by setting
+      the `-w` option. Please test and report back if this works for you.
+
+1.22.0:
+  date: 2022-06-25
+  changes:
+    - "Add support for downloading subscriber-only VODs (#48, thanks @cemiu)"
+
+1.21.0:
+  date: 2022-02-27
+  changes:
+    - "Add `env` command for printing environment info for attaching to bug reports"
+
+1.20.0:
+  date: 2022-02-25
+  changes:
+    - "Add `--json` option to `videos` command (#92, thanks @miff2000)"
+    - "Add `--all` option to `videos` and `clips` commands to list all clips or videos in one go."
+    - "Modify how `--pager` works, will make multiple requests if needed to show all available items, ignoring `--limit`."
+
+1.19.0:
+  date: 2022-02-05
+  changes:
+    - "Add support for downloading audio only (#10)"
+
+1.18.1:
+  date: 2022-02-05
+  changes:
+    - "Fix issues with output formats (#87, #89)"
+    - "Fix issues when downloading clip with no game set (#78)"
+    - "Add option to use clip slug in `--output` format"
+
+1.18.0:
+  date: 2022-01-25
+  changes:
+    - "Add `--output` option to `download` command which allows setting output file template (#70)"
+    - "Ask to overwrite before downloading to avoid later prompt"
+
+1.17.1:
+  date: 2022-01-19
+  changes:
+    - "Upgrade m3u8 lib to 1.0.0+"
+
+1.17.0:
+  date: 2021-12-03
+  changes:
+    - "Fix speed calculation when resuming download (#75, thanks CroquetteTheThe)"
+    - "Add artist and title metadata to resulting video (#80)"
+
+1.16.1:
+  date: 2021-07-31
+  changes:
+    - "Fix compat with older versions of python (#71)"
+
+1.16.0:
+  date: 2021-06-09
+  changes:
+    - "Fix clips download caused by Twitch changes (#64, thanks to all participants)"
+
+1.15.0:
+  date: 2021-02-15
+  changes:
+    - "Add support for new format of clip slug (thanks @Loveangel1337)"
+
+1.14.1:
+  date: 2021-01-14
+  changes:
+    - "Handle videos which don't exist more gracefully"
+
+1.14.0:
+  date: 2021-01-14
+  changes:
+    - "Added `info` command for displaying video or clip info (#51)"
+    - "Don't show there are more videos when there aren't (#52, thanks @scottyallen)"
+    - "Fixed Twitch regression for getting the access token (#53)"
+
+1.13.1:
+  date: 2020-11-23
+  changes:
+    - "Fixed clip download issue (#45)"
+
+1.13.0:
+  date: 2020-11-10
+  changes:
+    - "Added `clips` command for listing and batch downloading clips (#26)"
+
+1.12.1:
+  date: 2020-09-29
+  changes:
+    - "Fix bug introduced in previous version which broke joining"
+
+1.12.0:
+  date: 2020-09-29
+  changes:
+    - "Added `source` as alias for best available quality (#33)"
+    - "Added `--no-join` option to `download` to skip ffmpeg join (#36)"
+    - "Added `--overwrite` option to `download` to overwrite target without prompting for confirmation (#37)"
+    - "Added `--pager` option to `videos`, don't page by default (#30)"
+
+1.11.0:
+  date: 2020-09-03
+  changes:
+    - "Make downloading more robust, fixes issues with some VODs (#35)"
+    - "Bundle twitch-dl to a standalone archive, simplifying installation, see installation instructions in README"
+
+1.10.2:
+  date: 2020-08-11
+  changes:
+    - "Fix version number displayed by `twitch-dl --version` (#29)"
+
+1.10.1:
+  date: 2020-08-09
+  changes:
+    - "Fix videos incorrectly identified as clips (#28)"
+    - "Make download command work with video URLs lacking 'www' before 'twitch.tv'"
+    - "Print an error when video or clip is not found instead of an exception trace"
+
+1.10.0:
+  date: 2020-08-07
+  changes:
+    - "Add `--quality` option to `download` command, allows specifying the video quality to download. In this case, twitch-dl will require no user input. (#22)"
+    - "Fix download of clips which contain numbers in their slug (#24)"
+    - "Fix URL to video displayed by `videos` command (it was missing /videos/)"
+
+1.9.0:
+  date: 2020-06-10
+  changes:
+    - "**Breaking**: wrongly named `--max_workers` option changed to `--max-workers`. The shorthand option `-w` remains the same."
+    - "Fix bug where `videos` command would crash if there was no game info (#21)"
+    - "Allow unicode characters in filenames, no longer strips e.g. cyrillic script"
+
+1.8.0:
+  date: 2020-05-17
+  changes:
+    - "Fix videos command (#18)"
+    - "**Breaking**: `videos` command no longer takes the `--offset` parameter due to API changes"
+    - "Add paging to `videos` command to replace offset"
+    - "Add `--game` option to `videos` command to filter by game"
+
+1.7.0:
+  date: 2020-04-25
+  changes:
+    - "Support for specifying broadcast type when listing videos (#13)"
+
+1.6.0:
+  date: 2020-04-11
+  changes:
+    - "Support for downloading clips (#15)"
+
+1.5.1:
+  date: 2020-04-11
+  changes:
+    - "Fix VOD naming issue (#12)"
+    - "Nice console output while downloading"
+
+1.5.0:
+  date: 2020-04-10
+  changes:
+    - "Fix video downloads after Twitch deprecated access token access"
+    - "Don't print errors when retrying download, only if all fails"
+
+1.4.0:
+  date: 2019-08-23
+  changes:
+    - "Fix usage of deprecated v3 API"
+    - "Use m3u8 lib for parsing playlists"
+    - "Add `--keep` option not preserve downloaded VODs"
+
+1.3.1:
+  date: 2019-08-13
+  changes:
+    - "No changes, bumped to fix issue with pypi"
+
+1.3.0:
+  date: 2019-08-13
+  changes:
+    - "Add `--sort` and `--offset` options to `videos` command, allows paging (#7)"
+    - "Show video URL in `videos` command output"
+
+1.2.0:
+  date: 2019-07-05
+  changes:
+    - "Add `--format` option to `download` command for specifying the output format (#6)"
+    - "Add `--version` option for printing program version"
+
+1.1.0:
+  date: 2019-06-06
+  changes:
+    - "Allow limiting download by start and end time"
+
+1.0.0:
+  date: 2019-04-30
+  changes:
+    - "Initial release"

docs/SUMMARY.md

@@ -0,0 +1,17 @@
+# Summary
+
+[Introduction](introduction.md)
+
+- [Installation](installation.md)
+- [Usage](usage.md)
+    - [twitch-dl download](commands/download.md)
+    - [twitch-dl videos](commands/videos.md)
+    - [twitch-dl clips](commands/clips.md)
+    - [twitch-dl info](commands/info.md)
+    - [twitch-dl env](commands/env.md)
+- [Environemnt variables](environment_variables.md)
+- [Shell completion](shell_completion.md)
+- [Advanced](advanced.md)
+
+[License](license.md)
+[Changelog](changelog.md)

docs/advanced.md

@@ -0,0 +1,11 @@
+# Advanced
+
+## Temporary files
+
+By default, twitch-dl will download VODs to your systems temp dir (e.g. `/tmp/`
+on Linux). To change the location where the files are downloaded you can set
+the `TMP` environment variable, e.g.
+
+```
+TMP=/my/tmp/path/ twitch-dl download 221837124
+```

docs/changelog.md

@@ -0,0 +1,253 @@
+twitch-dl changelog
+===================
+
+<!-- Do not edit. This file is automatically generated from changelog.yaml.-->
+
+### [2.2.4 (2024-04-25)](https://github.com/ihabunek/twitch-dl/releases/tag/2.2.4)
+
+* Add m dot url support to video and clip regexes (thanks @localnerve)
+
+### [2.2.3 (2024-04-24)](https://github.com/ihabunek/twitch-dl/releases/tag/2.2.3)
+
+* Respect --dry-run option when downloading videos
+* Add automated tests on github actions
+
+### [2.2.2 (2024-04-23)](https://github.com/ihabunek/twitch-dl/releases/tag/2.2.2)
+
+* Fix more compat issues Python < 3.10 (#152)
+
+### [2.2.1 (2024-04-23)](https://github.com/ihabunek/twitch-dl/releases/tag/2.2.1)
+
+* Fix compat with Python < 3.10 (#152)
+* Fix division by zero in progress calculation when video duration is reported
+  as 0
+
+### [2.2.0 (2024-04-10)](https://github.com/ihabunek/twitch-dl/releases/tag/2.2.0)
+
+* **Requires Python 3.8+**
+* Migrated to Click library for generating the commandline interface
+* Add shell auto completion, see 'Shell completion' in docs.
+* Add setting defaults via environment variables, see 'Environment variables' in
+  docs
+* Add `download --concat` option to avoid using ffmeg for joinig vods and concat
+  them instead. This will produce a `.ts` file by default.
+* Add `download --dry-run` option to skip actual download (thanks @metacoma)
+* Add video description to metadata (#129)
+* Add `clips --compact` option for listing clips in one-per-line mode
+
+### [2.1.4 (2024-01-06)](https://github.com/ihabunek/twitch-dl/releases/tag/2.1.4)
+
+* Fix error caused by twitch requiring https for the usher api (thanks
+  @deanpcmad)
+
+### [2.1.3 (2023-05-07)](https://github.com/ihabunek/twitch-dl/releases/tag/2.1.3)
+
+* Replace client ID with one that works for now (thanks @mwhite34)
+
+### [2.1.2 (2023-04-18)](https://github.com/ihabunek/twitch-dl/releases/tag/2.1.2)
+
+* Fix error caused by twitch changing the Usher domain (thanks @adsa95)
+
+### [2.1.1 (2022-11-20)](https://github.com/ihabunek/twitch-dl/releases/tag/2.1.1)
+
+* Fix Python 3.7 compatibility (#117, thanks @eliduvid)
+* Fix default value for game_ids (#102, thanks @FunnyPocketBook)
+
+### [2.1.0 (2022-11-20)](https://github.com/ihabunek/twitch-dl/releases/tag/2.1.0)
+
+* Add chapter list to `info` command
+* Add `download --chapter` option for downloading a single chapter
+
+### [2.0.1 (2022-09-09)](https://github.com/ihabunek/twitch-dl/releases/tag/2.0.1)
+
+* Fix an issue where a temp vod file would be renamed while still being open,
+  which caused an exception on Windows (#111)
+
+### [2.0.0 (2022-08-18)](https://github.com/ihabunek/twitch-dl/releases/tag/2.0.0)
+
+This release switches from using `requests` to `httpx` for making http requests,
+and from threads to `asyncio` for concurrency. This enables easier
+implementation of new features, but has no breaking changes for the CLI.
+
+* **BREAKING**: Require Python 3.7 or later.
+* Add `--rate-limit` option to `download` for limiting maximum bandwidth when
+  downloading.
+* Add `--compact` option to `download` for displaying one video per line.
+* Allow passing multiple video ids to `download` to download multiple videos
+  successively.
+* Improved progress meter, updates on each chunk downloaded, instead of each VOD
+  downloaded.
+* Improved speed estimate, displays recent speed instead of average speed since
+  the start of download.
+* Decreased default concurrent downloads to 5. This seems to be enough to
+  saturate the download link in most cases. You can override this by setting the
+  `-w` option. Please test and report back if this works for you.
+
+### [1.22.0 (2022-06-25)](https://github.com/ihabunek/twitch-dl/releases/tag/1.22.0)
+
+* Add support for downloading subscriber-only VODs (#48, thanks @cemiu)
+
+### [1.21.0 (2022-02-27)](https://github.com/ihabunek/twitch-dl/releases/tag/1.21.0)
+
+* Add `env` command for printing environment info for attaching to bug reports
+
+### [1.20.0 (2022-02-25)](https://github.com/ihabunek/twitch-dl/releases/tag/1.20.0)
+
+* Add `--json` option to `videos` command (#92, thanks @miff2000)
+* Add `--all` option to `videos` and `clips` commands to list all clips or
+  videos in one go.
+* Modify how `--pager` works, will make multiple requests if needed to show all
+  available items, ignoring `--limit`.
+
+### [1.19.0 (2022-02-05)](https://github.com/ihabunek/twitch-dl/releases/tag/1.19.0)
+
+* Add support for downloading audio only (#10)
+
+### [1.18.1 (2022-02-05)](https://github.com/ihabunek/twitch-dl/releases/tag/1.18.1)
+
+* Fix issues with output formats (#87, #89)
+* Fix issues when downloading clip with no game set (#78)
+* Add option to use clip slug in `--output` format
+
+### [1.18.0 (2022-01-25)](https://github.com/ihabunek/twitch-dl/releases/tag/1.18.0)
+
+* Add `--output` option to `download` command which allows setting output file
+  template (#70)
+* Ask to overwrite before downloading to avoid later prompt
+
+### [1.17.1 (2022-01-19)](https://github.com/ihabunek/twitch-dl/releases/tag/1.17.1)
+
+* Upgrade m3u8 lib to 1.0.0+
+
+### [1.17.0 (2021-12-03)](https://github.com/ihabunek/twitch-dl/releases/tag/1.17.0)
+
+* Fix speed calculation when resuming download (#75, thanks CroquetteTheThe)
+* Add artist and title metadata to resulting video (#80)
+
+### [1.16.1 (2021-07-31)](https://github.com/ihabunek/twitch-dl/releases/tag/1.16.1)
+
+* Fix compat with older versions of python (#71)
+
+### [1.16.0 (2021-06-09)](https://github.com/ihabunek/twitch-dl/releases/tag/1.16.0)
+
+* Fix clips download caused by Twitch changes (#64, thanks to all participants)
+
+### [1.15.0 (2021-02-15)](https://github.com/ihabunek/twitch-dl/releases/tag/1.15.0)
+
+* Add support for new format of clip slug (thanks @Loveangel1337)
+
+### [1.14.1 (2021-01-14)](https://github.com/ihabunek/twitch-dl/releases/tag/1.14.1)
+
+* Handle videos which don't exist more gracefully
+
+### [1.14.0 (2021-01-14)](https://github.com/ihabunek/twitch-dl/releases/tag/1.14.0)
+
+* Added `info` command for displaying video or clip info (#51)
+* Don't show there are more videos when there aren't (#52, thanks @scottyallen)
+* Fixed Twitch regression for getting the access token (#53)
+
+### [1.13.1 (2020-11-23)](https://github.com/ihabunek/twitch-dl/releases/tag/1.13.1)
+
+* Fixed clip download issue (#45)
+
+### [1.13.0 (2020-11-10)](https://github.com/ihabunek/twitch-dl/releases/tag/1.13.0)
+
+* Added `clips` command for listing and batch downloading clips (#26)
+
+### [1.12.1 (2020-09-29)](https://github.com/ihabunek/twitch-dl/releases/tag/1.12.1)
+
+* Fix bug introduced in previous version which broke joining
+
+### [1.12.0 (2020-09-29)](https://github.com/ihabunek/twitch-dl/releases/tag/1.12.0)
+
+* Added `source` as alias for best available quality (#33)
+* Added `--no-join` option to `download` to skip ffmpeg join (#36)
+* Added `--overwrite` option to `download` to overwrite target without prompting
+  for confirmation (#37)
+* Added `--pager` option to `videos`, don't page by default (#30)
+
+### [1.11.0 (2020-09-03)](https://github.com/ihabunek/twitch-dl/releases/tag/1.11.0)
+
+* Make downloading more robust, fixes issues with some VODs (#35)
+* Bundle twitch-dl to a standalone archive, simplifying installation, see
+  installation instructions in README
+
+### [1.10.2 (2020-08-11)](https://github.com/ihabunek/twitch-dl/releases/tag/1.10.2)
+
+* Fix version number displayed by `twitch-dl --version` (#29)
+
+### [1.10.1 (2020-08-09)](https://github.com/ihabunek/twitch-dl/releases/tag/1.10.1)
+
+* Fix videos incorrectly identified as clips (#28)
+* Make download command work with video URLs lacking 'www' before 'twitch.tv'
+* Print an error when video or clip is not found instead of an exception trace
+
+### [1.10.0 (2020-08-07)](https://github.com/ihabunek/twitch-dl/releases/tag/1.10.0)
+
+* Add `--quality` option to `download` command, allows specifying the video
+  quality to download. In this case, twitch-dl will require no user input. (#22)
+* Fix download of clips which contain numbers in their slug (#24)
+* Fix URL to video displayed by `videos` command (it was missing /videos/)
+
+### [1.9.0 (2020-06-10)](https://github.com/ihabunek/twitch-dl/releases/tag/1.9.0)
+
+* **Breaking**: wrongly named `--max_workers` option changed to `--max-workers`.
+  The shorthand option `-w` remains the same.
+* Fix bug where `videos` command would crash if there was no game info (#21)
+* Allow unicode characters in filenames, no longer strips e.g. cyrillic script
+
+### [1.8.0 (2020-05-17)](https://github.com/ihabunek/twitch-dl/releases/tag/1.8.0)
+
+* Fix videos command (#18)
+* **Breaking**: `videos` command no longer takes the `--offset` parameter due to
+  API changes
+* Add paging to `videos` command to replace offset
+* Add `--game` option to `videos` command to filter by game
+
+### [1.7.0 (2020-04-25)](https://github.com/ihabunek/twitch-dl/releases/tag/1.7.0)
+
+* Support for specifying broadcast type when listing videos (#13)
+
+### [1.6.0 (2020-04-11)](https://github.com/ihabunek/twitch-dl/releases/tag/1.6.0)
+
+* Support for downloading clips (#15)
+
+### [1.5.1 (2020-04-11)](https://github.com/ihabunek/twitch-dl/releases/tag/1.5.1)
+
+* Fix VOD naming issue (#12)
+* Nice console output while downloading
+
+### [1.5.0 (2020-04-10)](https://github.com/ihabunek/twitch-dl/releases/tag/1.5.0)
+
+* Fix video downloads after Twitch deprecated access token access
+* Don't print errors when retrying download, only if all fails
+
+### [1.4.0 (2019-08-23)](https://github.com/ihabunek/twitch-dl/releases/tag/1.4.0)
+
+* Fix usage of deprecated v3 API
+* Use m3u8 lib for parsing playlists
+* Add `--keep` option not preserve downloaded VODs
+
+### [1.3.1 (2019-08-13)](https://github.com/ihabunek/twitch-dl/releases/tag/1.3.1)
+
+* No changes, bumped to fix issue with pypi
+
+### [1.3.0 (2019-08-13)](https://github.com/ihabunek/twitch-dl/releases/tag/1.3.0)
+
+* Add `--sort` and `--offset` options to `videos` command, allows paging (#7)
+* Show video URL in `videos` command output
+
+### [1.2.0 (2019-07-05)](https://github.com/ihabunek/twitch-dl/releases/tag/1.2.0)
+
+* Add `--format` option to `download` command for specifying the output format
+  (#6)
+* Add `--version` option for printing program version
+
+### [1.1.0 (2019-06-06)](https://github.com/ihabunek/twitch-dl/releases/tag/1.1.0)
+
+* Allow limiting download by start and end time
+
+### [1.0.0 (2019-04-30)](https://github.com/ihabunek/twitch-dl/releases/tag/1.0.0)
+
+* Initial release
+

docs/commands/clips.md

@@ -0,0 +1,108 @@
+<!-- ------------------- generated docs start ------------------- -->
+# twitch-dl clips
+
+List or download clips for given CHANNEL_NAME.
+
+### USAGE
+
+```
+twitch-dl clips [OPTIONS] CHANNEL_NAME
+```
+
+### OPTIONS
+
+<table>
+<tbody>
+<tr>
+    <td class="code">-a, --all</td>
+    <td>Fetch all clips, overrides --limit</td>
+</tr>
+
+<tr>
+    <td class="code">-c, --compact</td>
+    <td>Show clips in compact mode, one line per video</td>
+</tr>
+
+<tr>
+    <td class="code">-d, --download</td>
+    <td>Download clips in given period (in source quality)</td>
+</tr>
+
+<tr>
+    <td class="code">-l, --limit INTEGER</td>
+    <td>Number of clips to fetch. Defaults to 40 in compact mode, 10 otherwise.</td>
+</tr>
+
+<tr>
+    <td class="code">-p, --pager INTEGER</td>
+    <td>Number of clips to show per page. Disabled by default.</td>
+</tr>
+
+<tr>
+    <td class="code">-P, --period TEXT</td>
+    <td>Period from which to return clips Possible values: <code>last_day</code>, <code>last_week</code>, <code>last_month</code>, <code>all_time</code>. [default: <code>all_time</code>]</td>
+</tr>
+
+<tr>
+    <td class="code">--json</td>
+    <td>Print data as JSON rather than human readable text</td>
+</tr>
+</tbody>
+</table>
+
+<!-- ------------------- generated docs end ------------------- -->
+
+## Listing clips
+
+By default returns top 10 clips of all time.
+
+```
+twitch-dl clips bananasaurus_rex
+```
+
+Increase the number of clips returned.
+
+```
+twitch-dl clips bananasaurus_rex --limit 50
+```
+
+Return all clips, may require multiple requests, see notes.
+
+```
+twitch-dl clips bananasaurus_rex --all
+```
+
+Return clips from past day/week/month by changing the period:
+
+```
+twitch-dl clips bananasaurus_rex --period past_week
+```
+
+List all clips, 10 clips at the time:
+
+```
+twitch-dl clips bananasaurus_rex --pager
+```
+
+Print clips data in JSON. Useful for scripting.
+
+```
+twitch-dl clips bananasaurus_rex --json
+```
+
+Download all clips of the past week, won't overwrite exisitng ones:
+
+```
+twitch-dl clips bananasaurus_rex --download --period last_week
+```
+
+## Notes
+
+Clips are fetched in batches no larger than 100. When requesting more than 100
+clips, it takes more than one request so it can take a little time. You can see
+individual requests by passing the `--debug` flag.
+
+Currently it doesn't seem to be possible to get a list of clips ordered by time
+of creation, only by view count. Clips with the same view count seem to be
+returned in random order. This can break paging resulting in duplicate clips
+listed or clips missed.

docs/commands/download.md

@@ -0,0 +1,187 @@
+<!-- ------------------- generated docs start ------------------- -->
+# twitch-dl download
+
+Download videos or clips.
+
+    Pass one or more video ID, clip slug or Twitch URL to download.
+    
+
+### USAGE
+
+```
+twitch-dl download [OPTIONS] [IDS]...
+```
+
+### OPTIONS
+
+<table>
+<tbody>
+<tr>
+    <td class="code">-a, --auth-token TEXT</td>
+    <td>Authentication token, passed to Twitch to access subscriber only VODs. Can be copied from the <code>auth_token</code> cookie in any browser logged in on Twitch.</td>
+</tr>
+
+<tr>
+    <td class="code">-c, --chapter INTEGER</td>
+    <td>Download a single chapter of the video. Specify the chapter number or use the flag without a number to display a chapter select prompt.</td>
+</tr>
+
+<tr>
+    <td class="code">--concat</td>
+    <td>Do not use ffmpeg to join files, concat them instead. This will produce a .ts file by default.</td>
+</tr>
+
+<tr>
+    <td class="code">-d, --dry-run</td>
+    <td>Simulate the download provcess without actually downloading any files.</td>
+</tr>
+
+<tr>
+    <td class="code">-e, --end TEXT</td>
+    <td>Download video up to this time (hh:mm or hh:mm:ss)</td>
+</tr>
+
+<tr>
+    <td class="code">-f, --format TEXT</td>
+    <td>Video format to convert into, passed to ffmpeg as the target file extension. Defaults to <code>mkv</code>. If <code>--concat</code> is passed, defaults to <code>ts</code>.</td>
+</tr>
+
+<tr>
+    <td class="code">-k, --keep</td>
+    <td>Don&#x27;t delete downloaded VODs and playlists after merging.</td>
+</tr>
+
+<tr>
+    <td class="code">--no-join</td>
+    <td>Don&#x27;t run ffmpeg to join the downloaded vods, implies --keep.</td>
+</tr>
+
+<tr>
+    <td class="code">--overwrite</td>
+    <td>Overwrite the target file if it already exists without prompting.</td>
+</tr>
+
+<tr>
+    <td class="code">-o, --output TEXT</td>
+    <td>Output file name template. See docs for details. [default: <code>{date}_{id}_{channel_login}_{title_slug}.{format}</code>]</td>
+</tr>
+
+<tr>
+    <td class="code">-q, --quality TEXT</td>
+    <td>Video quality, e.g. <code>720p</code>. Set to <code>source</code> to get best quality.</td>
+</tr>
+
+<tr>
+    <td class="code">-r, --rate-limit TEXT</td>
+    <td>Limit the maximum download speed in bytes per second. Use &#x27;k&#x27; and &#x27;m&#x27; suffixes for kbps and mbps.</td>
+</tr>
+
+<tr>
+    <td class="code">-s, --start TEXT</td>
+    <td>Download video from this time (hh:mm or hh:mm:ss)</td>
+</tr>
+
+<tr>
+    <td class="code">-w, --max-workers INTEGER</td>
+    <td>Number of workers for downloading vods concurrently [default: <code>5</code>]</td>
+</tr>
+</tbody>
+</table>
+
+<!-- ------------------- generated docs end ------------------- -->
+
+### Examples
+
+Download a video by ID or URL:
+
+```
+twitch-dl download 221837124
+twitch-dl download https://www.twitch.tv/videos/221837124
+```
+
+Specify video quality to download to prevent a prompt:
+
+```
+twitch-dl download -q 720p 221837124
+```
+
+Setting quality to `source` will download the best available quality:
+
+```
+twitch-dl download -q source 221837124
+```
+
+Setting quality to `audio_only` will download only audio:
+
+```
+twitch-dl download -q audio_only 221837124
+```
+
+Download multiple videos one after the other:
+
+```
+twitch-dl download 1559928295 1557034274 1555157293 -q source
+```
+
+### Overriding the target file name
+
+The target filename can be defined by passing the `--output` option followed by
+the desired file name, e.g. `--output strim.mkv`.
+
+The filename uses
+[Python format string syntax](https://docs.python.org/3/library/string.html#format-string-syntax)
+and may contain placeholders in curly braces which will be replaced with
+relevant information tied to the downloaded video, e.g. `--output "{date}_{id}.{format}"`.
+
+The supported placeholders are:
+
+| Placeholder       | Description                    | Sample                        |
+| ----------------- | ------------------------------ | ------------------------------ |
+| `{id}`            | Video ID                       | 1255522958                     |
+| `{title}`         | Video title                    | Dark Souls 3 First playthrough |
+| `{title_slug}`    | Slugified video title          | dark_souls_3_first_playthrough |
+| `{datetime}`      | Video date and time            | 2022-01-07T04:00:27Z           |
+| `{date}`          | Video date                     | 2022-01-07                     |
+| `{time}`          | Video time                     | 04:00:27Z                      |
+| `{channel}`       | Channel name                   | KatLink                        |
+| `{channel_login}` | Channel login                  | katlink                        |
+| `{format}`        | File extension, see `--format` | mkv                            |
+| `{game}`          | Game name                      | Dark Souls III                 |
+| `{game_slug}`     | Slugified game name            | dark_souls_iii                 |
+| `{slug}`          | Clip slug (clips only)         | AbrasivePlacidCatDxAbomb       |
+
+A couple of examples:
+
+|    |    |
+| -- | -- |
+| Pattern | `{date}_{id}_{channel_login}_{title_slug}.{format}` *(default)* |
+| Expands to | `2022-01-07_1255522958_katlink_dark_souls_3_first_playthrough.mkv` |
+
+<br />
+
+|    |    |
+| -- | -- |
+| Pattern | `{channel} - {game} - {title}.{format}` |
+| Expands to | `KatLink - Dark Souls III - Dark Souls 3 First playthrough.mkv` |
+
+
+### Downloading subscriber-only VODs
+
+To download sub-only VODs, you need to find your auth token. It can be found
+using your browser, in a cookie named `auth_token`.
+
+1. Open twitch.tv in your browser and make sure you're logged in.
+2. Open developer tools (F12 shortcut in Firefox and Chrome).
+3. Open the `Storage` tab on Firefox, or `Application` tab in Chrome.
+4. Click on `Cookies` → `https://www.twitch.tv/` in the sidebar.
+5. Find the `auth-token` cookie in the list and copy it's value.
+
+![How to find the auth token in dev tools](./auth_token.png)
+
+The auth token will be a 30 character long string of random letters and numbers,
+something like `iduetx4i107rn4b9wrgctf590aiktv`. Then you can pass this to the
+download command:
+
+```
+twitch-dl download 221837124 --auth-token iduetx4i107rn4b9wrgctf590aiktv
+```

docs/commands/env.md

@@ -0,0 +1,31 @@
+<!-- ------------------- generated docs start ------------------- -->
+# twitch-dl env
+
+Print environment information for inclusion in bug reports.
+
+### USAGE
+
+```
+twitch-dl env [OPTIONS]
+```
+
+<!-- ------------------- generated docs end ------------------- -->
+
+Includes the twitch-dl version, platform details and python version. Does not
+include any personal information.
+
+On Linux this may return something like:
+
+```
+twitch-dl 1.21.0
+Platform: Linux-5.10.0-10-amd64-x86_64-with-glibc2.31
+Python 3.9.2 (default, Feb 28 2021, 17:03:44) [GCC 10.2.1 20210110]
+```
+
+On Windows:
+
+```
+twitch-dl 1.21.0
+Platform: Windows-10-10.0.19041-SP0
+Python 3.9.4 (tags/v3.9.4:1f2e308, Apr 6 2021, 13:40:21) [MSC v.1928 64 bit (AMD64)]
+```

docs/commands/info.md

@@ -0,0 +1,44 @@
+<!-- ------------------- generated docs start ------------------- -->
+# twitch-dl info
+
+Print information for a given Twitch URL, video ID or clip slug.
+
+### USAGE
+
+```
+twitch-dl info [OPTIONS] ID
+```
+
+### OPTIONS
+
+<table>
+<tbody>
+<tr>
+    <td class="code">--json</td>
+    <td>Print data as JSON rather than human readable text</td>
+</tr>
+</tbody>
+</table>
+
+<!-- ------------------- generated docs end ------------------- -->
+
+### Examples
+
+
+Print info about a video:
+
+```
+twitch-dl info 863849735
+```
+
+Print info about a clip:
+
+```
+twitch-dl info BusyBlushingCattleItsBoshyTime
+```
+
+Print JSON encoded info:
+
+```
+twitch-dl info BusyBlushingCattleItsBoshyTime --json
+```

docs/commands/videos.md

@@ -0,0 +1,96 @@
+<!-- ------------------- generated docs start ------------------- -->
+# twitch-dl videos
+
+List or download clips for given CHANNEL_NAME.
+
+### USAGE
+
+```
+twitch-dl videos [OPTIONS] CHANNEL_NAME
+```
+
+### OPTIONS
+
+<table>
+<tbody>
+<tr>
+    <td class="code">-a, --all</td>
+    <td>Fetch all clips, overrides --limit</td>
+</tr>
+
+<tr>
+    <td class="code">-c, --compact</td>
+    <td>Show videos in compact mode, one line per video</td>
+</tr>
+
+<tr>
+    <td class="code">-l, --limit INTEGER</td>
+    <td>Number of videos to fetch. Defaults to 40 in compact mode, 10 otherwise.</td>
+</tr>
+
+<tr>
+    <td class="code">-p, --pager INTEGER</td>
+    <td>Number of videos to show per page. Disabled by default.</td>
+</tr>
+
+<tr>
+    <td class="code">-g, --game TEXT</td>
+    <td>Show videos of given game (can be given multiple times)</td>
+</tr>
+
+<tr>
+    <td class="code">-s, --sort TEXT</td>
+    <td>Sorting order of videos Possible values: <code>views</code>, <code>time</code>. [default: <code>time</code>]</td>
+</tr>
+
+<tr>
+    <td class="code">-t, --type TEXT</td>
+    <td>Broadcast type Possible values: <code>archive</code>, <code>highlight</code>, <code>upload</code>. [default: <code>archive</code>]</td>
+</tr>
+
+<tr>
+    <td class="code">--json</td>
+    <td>Print data as JSON rather than human readable text</td>
+</tr>
+</tbody>
+</table>
+
+<!-- ------------------- generated docs end ------------------- -->
+
+### Examples
+
+List recent channel videos (10 by default):
+
+```
+twitch-dl videos bananasaurus_rex
+```
+
+Limit to videos of one or more games:
+
+```
+twitch-dl videos --game "doom eternal" --game "cave story" bananasaurus_rex
+```
+
+List all channel videos at once:
+
+```
+twitch-dl videos bananasaurus_rex --all
+```
+
+List all channel videos in pages of 10:
+
+```
+twitch-dl videos bananasaurus_rex --pager
+```
+
+Page size can be adjusted by passing number of items per page:
+
+```
+twitch-dl videos bananasaurus_rex --pager 5
+```
+
+Returns all videos as a JSON list. Useful for scripting.
+
+```
+twitch-dl videos bananasaurus_rex --json --all
+```

docs/environment_variables.md

@@ -0,0 +1,15 @@
+# Environment variables
+
+> Introduced in twitch-dl 2.2.0
+
+twitch-dl allows setting defaults for parameters via environment variables.
+
+Environment variables should be named `TWITCH_DL_<COMMAND_NAME>_<OPTION_NAME>`.
+
+For example, when invoking `twitch-dl download`, if you always set `--quality
+source` you can set the following environment variable to make this the
+default:
+
+```
+TWITCH_DL_DOWNLOAD_QUALITY="source"
+```

docs/installation.md

@@ -0,0 +1,76 @@
+# Installation
+
+twitch-dl requires **Python 3.8** or later.
+
+## Prerequisite: FFmpeg
+
+FFmpeg is used to join vods into a single video file.
+
+To check if FFmpeg is available, run:
+
+```
+ffmpeg -version
+```
+
+The version number and some info should be printed.
+
+```
+ffmpeg version 4.3.2-0+deb11u2 Copyright (c) 2000-2021 the FFmpeg developers
+built with gcc 10 (Debian 10.2.1-6)
+```
+
+To install FFmpeg see [FFmpeg documentation](https://ffmpeg.org/download.html).
+
+## Option 1: Download standalone archive
+
+Go to the [latest release](https://github.com/ihabunek/twitch-dl/releases/latest)
+and download the `twitch-dl.<version>.pyz` archive.
+
+Run the archive by either:
+
+a) passing it to python:
+
+```
+python3 twitch-dl.1.13.0.pyz --help
+```
+
+b) making it executable and invoking it directly (linux specific):
+
+```
+chmod +x twitch-dl.1.13.0.pyz
+./twitch-dl.1.13.0.pyz --help
+```
+
+Feel free to rename the archive to something more managable, like `twitch-dl`.
+
+To upgrade to a newer version, repeat the process with the newer release.
+
+## Option 2: Install from PYPI using pipx
+
+**pipx** is a tool which installs python apps into isolated environments, which
+prevents all kinds of problems later so it's the suggested way to install
+twitch-dl from PYPI.
+
+Install pipx as described in
+[pipx install docs](https://pipxproject.github.io/pipx/installation/).
+
+Install twitch-dl:
+
+```
+pipx install twitch-dl
+```
+
+Check installation worked:
+
+```
+twitch-dl --version
+```
+
+If twitch-dl executable is not found, check that the pipx binary location (by
+default `~/.local/bin`) is in your PATH.
+
+To upgrade twitch-dl to the latest version:
+
+```
+pipx upgrade twitch-dl
+```

docs/introduction.md

@@ -0,0 +1,58 @@
+twitch-dl
+=========
+
+Command-line tool for downloading videos from twitch.tv
+
+Inspired by [youtube-dl](https://youtube-dl.org/). Adds twitch-specific features
+and uses concurrent connections to make the download faster.
+
+## Resources
+
+* [Documentation](https://twitch-dl.bezdomni.net/)
+* [Source code on Github](https://github.com/ihabunek/twitch-dl)
+* [Issues on Github](https://github.com/ihabunek/twitch-dl/issues)
+* [Python package](https://pypi.org/project/twitch-dl/)
+
+## Quick start
+
+List videos from a channel.
+
+```
+twitch-dl videos bananasaurus_rex
+```
+
+List clips from a channel.
+
+```
+twitch-dl clips bananasaurus_rex
+```
+
+Download a video by URL.
+
+```
+twitch-dl download https://www.twitch.tv/videos/1418494769
+```
+
+or by ID
+
+```
+twitch-dl download 1418494769
+```
+
+Download a clip by URL
+
+```
+twitch-dl download https://www.twitch.tv/bananasaurus_rex/clip/PlacidColdClipsdadDeIlluminati-hL2s_aLE4CHvVN4J
+```
+
+or by slug
+
+```
+twitch-dl download PlacidColdClipsdadDeIlluminati-hL2s_aLE4CHvVN4J
+```
+
+For more info and examples see [usage](usage.html).
+
+## License
+
+twitch-dl is open source and licensed under the [GNU General Public License v3](/license.html).

docs/license.md

@@ -0,0 +1,675 @@
+### GNU GENERAL PUBLIC LICENSE
+
+Version 3, 29 June 2007
+
+Copyright (C) 2007 Free Software Foundation, Inc.
+<https://fsf.org/>
+
+Everyone is permitted to copy and distribute verbatim copies of this
+license document, but changing it is not allowed.
+
+### Preamble
+
+The GNU General Public License is a free, copyleft license for
+software and other kinds of works.
+
+The licenses for most software and other practical works are designed
+to take away your freedom to share and change the works. By contrast,
+the GNU General Public License is intended to guarantee your freedom
+to share and change all versions of a program--to make sure it remains
+free software for all its users. We, the Free Software Foundation, use
+the GNU General Public License for most of our software; it applies
+also to any other work released this way by its authors. You can apply
+it to your programs, too.
+
+When we speak of free software, we are referring to freedom, not
+price. Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+them if you wish), that you receive source code or can get it if you
+want it, that you can change the software or use pieces of it in new
+free programs, and that you know you can do these things.
+
+To protect your rights, we need to prevent others from denying you
+these rights or asking you to surrender the rights. Therefore, you
+have certain responsibilities if you distribute copies of the
+software, or if you modify it: responsibilities to respect the freedom
+of others.
+
+For example, if you distribute copies of such a program, whether
+gratis or for a fee, you must pass on to the recipients the same
+freedoms that you received. You must make sure that they, too, receive
+or can get the source code. And you must show them these terms so they
+know their rights.
+
+Developers that use the GNU GPL protect your rights with two steps:
+(1) assert copyright on the software, and (2) offer you this License
+giving you legal permission to copy, distribute and/or modify it.
+
+For the developers' and authors' protection, the GPL clearly explains
+that there is no warranty for this free software. For both users' and
+authors' sake, the GPL requires that modified versions be marked as
+changed, so that their problems will not be attributed erroneously to
+authors of previous versions.
+
+Some devices are designed to deny users access to install or run
+modified versions of the software inside them, although the
+manufacturer can do so. This is fundamentally incompatible with the
+aim of protecting users' freedom to change the software. The
+systematic pattern of such abuse occurs in the area of products for
+individuals to use, which is precisely where it is most unacceptable.
+Therefore, we have designed this version of the GPL to prohibit the
+practice for those products. If such problems arise substantially in
+other domains, we stand ready to extend this provision to those
+domains in future versions of the GPL, as needed to protect the
+freedom of users.
+
+Finally, every program is threatened constantly by software patents.
+States should not allow patents to restrict development and use of
+software on general-purpose computers, but in those that do, we wish
+to avoid the special danger that patents applied to a free program
+could make it effectively proprietary. To prevent this, the GPL
+assures that patents cannot be used to render the program non-free.
+
+The precise terms and conditions for copying, distribution and
+modification follow.
+
+### TERMS AND CONDITIONS
+
+#### 0. Definitions.
+
+"This License" refers to version 3 of the GNU General Public License.
+
+"Copyright" also means copyright-like laws that apply to other kinds
+of works, such as semiconductor masks.
+
+"The Program" refers to any copyrightable work licensed under this
+License. Each licensee is addressed as "you". "Licensees" and
+"recipients" may be individuals or organizations.
+
+To "modify" a work means to copy from or adapt all or part of the work
+in a fashion requiring copyright permission, other than the making of
+an exact copy. The resulting work is called a "modified version" of
+the earlier work or a work "based on" the earlier work.
+
+A "covered work" means either the unmodified Program or a work based
+on the Program.
+
+To "propagate" a work means to do anything with it that, without
+permission, would make you directly or secondarily liable for
+infringement under applicable copyright law, except executing it on a
+computer or modifying a private copy. Propagation includes copying,
+distribution (with or without modification), making available to the
+public, and in some countries other activities as well.
+
+To "convey" a work means any kind of propagation that enables other
+parties to make or receive copies. Mere interaction with a user
+through a computer network, with no transfer of a copy, is not
+conveying.
+
+An interactive user interface displays "Appropriate Legal Notices" to
+the extent that it includes a convenient and prominently visible
+feature that (1) displays an appropriate copyright notice, and (2)
+tells the user that there is no warranty for the work (except to the
+extent that warranties are provided), that licensees may convey the
+work under this License, and how to view a copy of this License. If
+the interface presents a list of user commands or options, such as a
+menu, a prominent item in the list meets this criterion.
+
+#### 1. Source Code.
+
+The "source code" for a work means the preferred form of the work for
+making modifications to it. "Object code" means any non-source form of
+a work.
+
+A "Standard Interface" means an interface that either is an official
+standard defined by a recognized standards body, or, in the case of
+interfaces specified for a particular programming language, one that
+is widely used among developers working in that language.
+
+The "System Libraries" of an executable work include anything, other
+than the work as a whole, that (a) is included in the normal form of
+packaging a Major Component, but which is not part of that Major
+Component, and (b) serves only to enable use of the work with that
+Major Component, or to implement a Standard Interface for which an
+implementation is available to the public in source code form. A
+"Major Component", in this context, means a major essential component
+(kernel, window system, and so on) of the specific operating system
+(if any) on which the executable work runs, or a compiler used to
+produce the work, or an object code interpreter used to run it.
+
+The "Corresponding Source" for a work in object code form means all
+the source code needed to generate, install, and (for an executable
+work) run the object code and to modify the work, including scripts to
+control those activities. However, it does not include the work's
+System Libraries, or general-purpose tools or generally available free
+programs which are used unmodified in performing those activities but
+which are not part of the work. For example, Corresponding Source
+includes interface definition files associated with source files for
+the work, and the source code for shared libraries and dynamically
+linked subprograms that the work is specifically designed to require,
+such as by intimate data communication or control flow between those
+subprograms and other parts of the work.
+
+The Corresponding Source need not include anything that users can
+regenerate automatically from other parts of the Corresponding Source.
+
+The Corresponding Source for a work in source code form is that same
+work.
+
+#### 2. Basic Permissions.
+
+All rights granted under this License are granted for the term of
+copyright on the Program, and are irrevocable provided the stated
+conditions are met. This License explicitly affirms your unlimited
+permission to run the unmodified Program. The output from running a
+covered work is covered by this License only if the output, given its
+content, constitutes a covered work. This License acknowledges your
+rights of fair use or other equivalent, as provided by copyright law.
+
+You may make, run and propagate covered works that you do not convey,
+without conditions so long as your license otherwise remains in force.
+You may convey covered works to others for the sole purpose of having
+them make modifications exclusively for you, or provide you with
+facilities for running those works, provided that you comply with the
+terms of this License in conveying all material for which you do not
+control copyright. Those thus making or running the covered works for
+you must do so exclusively on your behalf, under your direction and
+control, on terms that prohibit them from making any copies of your
+copyrighted material outside their relationship with you.
+
+Conveying under any other circumstances is permitted solely under the
+conditions stated below. Sublicensing is not allowed; section 10 makes
+it unnecessary.
+
+#### 3. Protecting Users' Legal Rights From Anti-Circumvention Law.
+
+No covered work shall be deemed part of an effective technological
+measure under any applicable law fulfilling obligations under article
+11 of the WIPO copyright treaty adopted on 20 December 1996, or
+similar laws prohibiting or restricting circumvention of such
+measures.
+
+When you convey a covered work, you waive any legal power to forbid
+circumvention of technological measures to the extent such
+circumvention is effected by exercising rights under this License with
+respect to the covered work, and you disclaim any intention to limit
+operation or modification of the work as a means of enforcing, against
+the work's users, your or third parties' legal rights to forbid
+circumvention of technological measures.
+
+#### 4. Conveying Verbatim Copies.
+
+You may convey verbatim copies of the Program's source code as you
+receive it, in any medium, provided that you conspicuously and
+appropriately publish on each copy an appropriate copyright notice;
+keep intact all notices stating that this License and any
+non-permissive terms added in accord with section 7 apply to the code;
+keep intact all notices of the absence of any warranty; and give all
+recipients a copy of this License along with the Program.
+
+You may charge any price or no price for each copy that you convey,
+and you may offer support or warranty protection for a fee.
+
+#### 5. Conveying Modified Source Versions.
+
+You may convey a work based on the Program, or the modifications to
+produce it from the Program, in the form of source code under the
+terms of section 4, provided that you also meet all of these
+conditions:
+
+-   a) The work must carry prominent notices stating that you modified
+    it, and giving a relevant date.
+-   b) The work must carry prominent notices stating that it is
+    released under this License and any conditions added under
+    section 7. This requirement modifies the requirement in section 4
+    to "keep intact all notices".
+-   c) You must license the entire work, as a whole, under this
+    License to anyone who comes into possession of a copy. This
+    License will therefore apply, along with any applicable section 7
+    additional terms, to the whole of the work, and all its parts,
+    regardless of how they are packaged. This License gives no
+    permission to license the work in any other way, but it does not
+    invalidate such permission if you have separately received it.
+-   d) If the work has interactive user interfaces, each must display
+    Appropriate Legal Notices; however, if the Program has interactive
+    interfaces that do not display Appropriate Legal Notices, your
+    work need not make them do so.
+
+A compilation of a covered work with other separate and independent
+works, which are not by their nature extensions of the covered work,
+and which are not combined with it such as to form a larger program,
+in or on a volume of a storage or distribution medium, is called an
+"aggregate" if the compilation and its resulting copyright are not
+used to limit the access or legal rights of the compilation's users
+beyond what the individual works permit. Inclusion of a covered work
+in an aggregate does not cause this License to apply to the other
+parts of the aggregate.
+
+#### 6. Conveying Non-Source Forms.
+
+You may convey a covered work in object code form under the terms of
+sections 4 and 5, provided that you also convey the machine-readable
+Corresponding Source under the terms of this License, in one of these
+ways:
+
+-   a) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by the
+    Corresponding Source fixed on a durable physical medium
+    customarily used for software interchange.
+-   b) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by a
+    written offer, valid for at least three years and valid for as
+    long as you offer spare parts or customer support for that product
+    model, to give anyone who possesses the object code either (1) a
+    copy of the Corresponding Source for all the software in the
+    product that is covered by this License, on a durable physical
+    medium customarily used for software interchange, for a price no
+    more than your reasonable cost of physically performing this
+    conveying of source, or (2) access to copy the Corresponding
+    Source from a network server at no charge.
+-   c) Convey individual copies of the object code with a copy of the
+    written offer to provide the Corresponding Source. This
+    alternative is allowed only occasionally and noncommercially, and
+    only if you received the object code with such an offer, in accord
+    with subsection 6b.
+-   d) Convey the object code by offering access from a designated
+    place (gratis or for a charge), and offer equivalent access to the
+    Corresponding Source in the same way through the same place at no
+    further charge. You need not require recipients to copy the
+    Corresponding Source along with the object code. If the place to
+    copy the object code is a network server, the Corresponding Source
+    may be on a different server (operated by you or a third party)
+    that supports equivalent copying facilities, provided you maintain
+    clear directions next to the object code saying where to find the
+    Corresponding Source. Regardless of what server hosts the
+    Corresponding Source, you remain obligated to ensure that it is
+    available for as long as needed to satisfy these requirements.
+-   e) Convey the object code using peer-to-peer transmission,
+    provided you inform other peers where the object code and
+    Corresponding Source of the work are being offered to the general
+    public at no charge under subsection 6d.
+
+A separable portion of the object code, whose source code is excluded
+from the Corresponding Source as a System Library, need not be
+included in conveying the object code work.
+
+A "User Product" is either (1) a "consumer product", which means any
+tangible personal property which is normally used for personal,
+family, or household purposes, or (2) anything designed or sold for
+incorporation into a dwelling. In determining whether a product is a
+consumer product, doubtful cases shall be resolved in favor of
+coverage. For a particular product received by a particular user,
+"normally used" refers to a typical or common use of that class of
+product, regardless of the status of the particular user or of the way
+in which the particular user actually uses, or expects or is expected
+to use, the product. A product is a consumer product regardless of
+whether the product has substantial commercial, industrial or
+non-consumer uses, unless such uses represent the only significant
+mode of use of the product.
+
+"Installation Information" for a User Product means any methods,
+procedures, authorization keys, or other information required to
+install and execute modified versions of a covered work in that User
+Product from a modified version of its Corresponding Source. The
+information must suffice to ensure that the continued functioning of
+the modified object code is in no case prevented or interfered with
+solely because modification has been made.
+
+If you convey an object code work under this section in, or with, or
+specifically for use in, a User Product, and the conveying occurs as
+part of a transaction in which the right of possession and use of the
+User Product is transferred to the recipient in perpetuity or for a
+fixed term (regardless of how the transaction is characterized), the
+Corresponding Source conveyed under this section must be accompanied
+by the Installation Information. But this requirement does not apply
+if neither you nor any third party retains the ability to install
+modified object code on the User Product (for example, the work has
+been installed in ROM).
+
+The requirement to provide Installation Information does not include a
+requirement to continue to provide support service, warranty, or
+updates for a work that has been modified or installed by the
+recipient, or for the User Product in which it has been modified or
+installed. Access to a network may be denied when the modification
+itself materially and adversely affects the operation of the network
+or violates the rules and protocols for communication across the
+network.
+
+Corresponding Source conveyed, and Installation Information provided,
+in accord with this section must be in a format that is publicly
+documented (and with an implementation available to the public in
+source code form), and must require no special password or key for
+unpacking, reading or copying.
+
+#### 7. Additional Terms.
+
+"Additional permissions" are terms that supplement the terms of this
+License by making exceptions from one or more of its conditions.
+Additional permissions that are applicable to the entire Program shall
+be treated as though they were included in this License, to the extent
+that they are valid under applicable law. If additional permissions
+apply only to part of the Program, that part may be used separately
+under those permissions, but the entire Program remains governed by
+this License without regard to the additional permissions.
+
+When you convey a copy of a covered work, you may at your option
+remove any additional permissions from that copy, or from any part of
+it. (Additional permissions may be written to require their own
+removal in certain cases when you modify the work.) You may place
+additional permissions on material, added by you to a covered work,
+for which you have or can give appropriate copyright permission.
+
+Notwithstanding any other provision of this License, for material you
+add to a covered work, you may (if authorized by the copyright holders
+of that material) supplement the terms of this License with terms:
+
+-   a) Disclaiming warranty or limiting liability differently from the
+    terms of sections 15 and 16 of this License; or
+-   b) Requiring preservation of specified reasonable legal notices or
+    author attributions in that material or in the Appropriate Legal
+    Notices displayed by works containing it; or
+-   c) Prohibiting misrepresentation of the origin of that material,
+    or requiring that modified versions of such material be marked in
+    reasonable ways as different from the original version; or
+-   d) Limiting the use for publicity purposes of names of licensors
+    or authors of the material; or
+-   e) Declining to grant rights under trademark law for use of some
+    trade names, trademarks, or service marks; or
+-   f) Requiring indemnification of licensors and authors of that
+    material by anyone who conveys the material (or modified versions
+    of it) with contractual assumptions of liability to the recipient,
+    for any liability that these contractual assumptions directly
+    impose on those licensors and authors.
+
+All other non-permissive additional terms are considered "further
+restrictions" within the meaning of section 10. If the Program as you
+received it, or any part of it, contains a notice stating that it is
+governed by this License along with a term that is a further
+restriction, you may remove that term. If a license document contains
+a further restriction but permits relicensing or conveying under this
+License, you may add to a covered work material governed by the terms
+of that license document, provided that the further restriction does
+not survive such relicensing or conveying.
+
+If you add terms to a covered work in accord with this section, you
+must place, in the relevant source files, a statement of the
+additional terms that apply to those files, or a notice indicating
+where to find the applicable terms.
+
+Additional terms, permissive or non-permissive, may be stated in the
+form of a separately written license, or stated as exceptions; the
+above requirements apply either way.
+
+#### 8. Termination.
+
+You may not propagate or modify a covered work except as expressly
+provided under this License. Any attempt otherwise to propagate or
+modify it is void, and will automatically terminate your rights under
+this License (including any patent licenses granted under the third
+paragraph of section 11).
+
+However, if you cease all violation of this License, then your license
+from a particular copyright holder is reinstated (a) provisionally,
+unless and until the copyright holder explicitly and finally
+terminates your license, and (b) permanently, if the copyright holder
+fails to notify you of the violation by some reasonable means prior to
+60 days after the cessation.
+
+Moreover, your license from a particular copyright holder is
+reinstated permanently if the copyright holder notifies you of the
+violation by some reasonable means, this is the first time you have
+received notice of violation of this License (for any work) from that
+copyright holder, and you cure the violation prior to 30 days after
+your receipt of the notice.
+
+Termination of your rights under this section does not terminate the
+licenses of parties who have received copies or rights from you under
+this License. If your rights have been terminated and not permanently
+reinstated, you do not qualify to receive new licenses for the same
+material under section 10.
+
+#### 9. Acceptance Not Required for Having Copies.
+
+You are not required to accept this License in order to receive or run
+a copy of the Program. Ancillary propagation of a covered work
+occurring solely as a consequence of using peer-to-peer transmission
+to receive a copy likewise does not require acceptance. However,
+nothing other than this License grants you permission to propagate or
+modify any covered work. These actions infringe copyright if you do
+not accept this License. Therefore, by modifying or propagating a
+covered work, you indicate your acceptance of this License to do so.
+
+#### 10. Automatic Licensing of Downstream Recipients.
+
+Each time you convey a covered work, the recipient automatically
+receives a license from the original licensors, to run, modify and
+propagate that work, subject to this License. You are not responsible
+for enforcing compliance by third parties with this License.
+
+An "entity transaction" is a transaction transferring control of an
+organization, or substantially all assets of one, or subdividing an
+organization, or merging organizations. If propagation of a covered
+work results from an entity transaction, each party to that
+transaction who receives a copy of the work also receives whatever
+licenses to the work the party's predecessor in interest had or could
+give under the previous paragraph, plus a right to possession of the
+Corresponding Source of the work from the predecessor in interest, if
+the predecessor has it or can get it with reasonable efforts.
+
+You may not impose any further restrictions on the exercise of the
+rights granted or affirmed under this License. For example, you may
+not impose a license fee, royalty, or other charge for exercise of
+rights granted under this License, and you may not initiate litigation
+(including a cross-claim or counterclaim in a lawsuit) alleging that
+any patent claim is infringed by making, using, selling, offering for
+sale, or importing the Program or any portion of it.
+
+#### 11. Patents.
+
+A "contributor" is a copyright holder who authorizes use under this
+License of the Program or a work on which the Program is based. The
+work thus licensed is called the contributor's "contributor version".
+
+A contributor's "essential patent claims" are all patent claims owned
+or controlled by the contributor, whether already acquired or
+hereafter acquired, that would be infringed by some manner, permitted
+by this License, of making, using, or selling its contributor version,
+but do not include claims that would be infringed only as a
+consequence of further modification of the contributor version. For
+purposes of this definition, "control" includes the right to grant
+patent sublicenses in a manner consistent with the requirements of
+this License.
+
+Each contributor grants you a non-exclusive, worldwide, royalty-free
+patent license under the contributor's essential patent claims, to
+make, use, sell, offer for sale, import and otherwise run, modify and
+propagate the contents of its contributor version.
+
+In the following three paragraphs, a "patent license" is any express
+agreement or commitment, however denominated, not to enforce a patent
+(such as an express permission to practice a patent or covenant not to
+sue for patent infringement). To "grant" such a patent license to a
+party means to make such an agreement or commitment not to enforce a
+patent against the party.
+
+If you convey a covered work, knowingly relying on a patent license,
+and the Corresponding Source of the work is not available for anyone
+to copy, free of charge and under the terms of this License, through a
+publicly available network server or other readily accessible means,
+then you must either (1) cause the Corresponding Source to be so
+available, or (2) arrange to deprive yourself of the benefit of the
+patent license for this particular work, or (3) arrange, in a manner
+consistent with the requirements of this License, to extend the patent
+license to downstream recipients. "Knowingly relying" means you have
+actual knowledge that, but for the patent license, your conveying the
+covered work in a country, or your recipient's use of the covered work
+in a country, would infringe one or more identifiable patents in that
+country that you have reason to believe are valid.
+
+If, pursuant to or in connection with a single transaction or
+arrangement, you convey, or propagate by procuring conveyance of, a
+covered work, and grant a patent license to some of the parties
+receiving the covered work authorizing them to use, propagate, modify
+or convey a specific copy of the covered work, then the patent license
+you grant is automatically extended to all recipients of the covered
+work and works based on it.
+
+A patent license is "discriminatory" if it does not include within the
+scope of its coverage, prohibits the exercise of, or is conditioned on
+the non-exercise of one or more of the rights that are specifically
+granted under this License. You may not convey a covered work if you
+are a party to an arrangement with a third party that is in the
+business of distributing software, under which you make payment to the
+third party based on the extent of your activity of conveying the
+work, and under which the third party grants, to any of the parties
+who would receive the covered work from you, a discriminatory patent
+license (a) in connection with copies of the covered work conveyed by
+you (or copies made from those copies), or (b) primarily for and in
+connection with specific products or compilations that contain the
+covered work, unless you entered into that arrangement, or that patent
+license was granted, prior to 28 March 2007.
+
+Nothing in this License shall be construed as excluding or limiting
+any implied license or other defenses to infringement that may
+otherwise be available to you under applicable patent law.
+
+#### 12. No Surrender of Others' Freedom.
+
+If conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License. If you cannot convey a
+covered work so as to satisfy simultaneously your obligations under
+this License and any other pertinent obligations, then as a
+consequence you may not convey it at all. For example, if you agree to
+terms that obligate you to collect a royalty for further conveying
+from those to whom you convey the Program, the only way you could
+satisfy both those terms and this License would be to refrain entirely
+from conveying the Program.
+
+#### 13. Use with the GNU Affero General Public License.
+
+Notwithstanding any other provision of this License, you have
+permission to link or combine any covered work with a work licensed
+under version 3 of the GNU Affero General Public License into a single
+combined work, and to convey the resulting work. The terms of this
+License will continue to apply to the part which is the covered work,
+but the special requirements of the GNU Affero General Public License,
+section 13, concerning interaction through a network will apply to the
+combination as such.
+
+#### 14. Revised Versions of this License.
+
+The Free Software Foundation may publish revised and/or new versions
+of the GNU General Public License from time to time. Such new versions
+will be similar in spirit to the present version, but may differ in
+detail to address new problems or concerns.
+
+Each version is given a distinguishing version number. If the Program
+specifies that a certain numbered version of the GNU General Public
+License "or any later version" applies to it, you have the option of
+following the terms and conditions either of that numbered version or
+of any later version published by the Free Software Foundation. If the
+Program does not specify a version number of the GNU General Public
+License, you may choose any version ever published by the Free
+Software Foundation.
+
+If the Program specifies that a proxy can decide which future versions
+of the GNU General Public License can be used, that proxy's public
+statement of acceptance of a version permanently authorizes you to
+choose that version for the Program.
+
+Later license versions may give you additional or different
+permissions. However, no additional obligations are imposed on any
+author or copyright holder as a result of your choosing to follow a
+later version.
+
+#### 15. Disclaimer of Warranty.
+
+THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
+APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
+HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT
+WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND
+PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE PROGRAM PROVE
+DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR
+CORRECTION.
+
+#### 16. Limitation of Liability.
+
+IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR
+CONVEYS THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
+INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES
+ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT
+NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR
+LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM
+TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER
+PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
+
+#### 17. Interpretation of Sections 15 and 16.
+
+If the disclaimer of warranty and limitation of liability provided
+above cannot be given local legal effect according to their terms,
+reviewing courts shall apply local law that most closely approximates
+an absolute waiver of all civil liability in connection with the
+Program, unless a warranty or assumption of liability accompanies a
+copy of the Program in return for a fee.
+
+END OF TERMS AND CONDITIONS
+
+### How to Apply These Terms to Your New Programs
+
+If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these
+terms.
+
+To do so, attach the following notices to the program. It is safest to
+attach them to the start of each source file to most effectively state
+the exclusion of warranty; and each file should have at least the
+"copyright" line and a pointer to where the full notice is found.
+
+        <one line to give the program's name and a brief idea of what it does.>
+        Copyright (C) <year>  <name of author>
+
+        This program is free software: you can redistribute it and/or modify
+        it under the terms of the GNU General Public License as published by
+        the Free Software Foundation, either version 3 of the License, or
+        (at your option) any later version.
+
+        This program is distributed in the hope that it will be useful,
+        but WITHOUT ANY WARRANTY; without even the implied warranty of
+        MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+        GNU General Public License for more details.
+
+        You should have received a copy of the GNU General Public License
+        along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+Also add information on how to contact you by electronic and paper
+mail.
+
+If the program does terminal interaction, make it output a short
+notice like this when it starts in an interactive mode:
+
+        <program>  Copyright (C) <year>  <name of author>
+        This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
+        This is free software, and you are welcome to redistribute it
+        under certain conditions; type `show c' for details.
+
+The hypothetical commands \`show w' and \`show c' should show the
+appropriate parts of the General Public License. Of course, your
+program's commands might be different; for a GUI interface, you would
+use an "about box".
+
+You should also get your employer (if you work as a programmer) or
+school, if any, to sign a "copyright disclaimer" for the program, if
+necessary. For more information on this, and how to apply and follow
+the GNU GPL, see <https://www.gnu.org/licenses/>.
+
+The GNU General Public License does not permit incorporating your
+program into proprietary programs. If your program is a subroutine
+library, you may consider it more useful to permit linking proprietary
+applications with the library. If this is what you want to do, use the
+GNU Lesser General Public License instead of this License. But first,
+please read <https://www.gnu.org/licenses/why-not-lgpl.html>.

docs/shell_completion.md

@@ -0,0 +1,31 @@
+# Shell completion
+
+> Introduced in twitch-dl 2.2.0
+
+twitch-dl uses [Click shell completion](https://click.palletsprojects.com/en/8.1.x/shell-completion/) which works on Bash, Fish and Zsh.
+
+To enable completion, twitch-dl must be [installed](./installation.html) as a command and available by ivoking `twitch-dl`. Then follow the instructions for your shell.
+
+**Bash**
+
+Add to `~/.bashrc`:
+
+```
+eval "$(_TWITCH_DL_COMPLETE=bash_source twitch-dl)"
+```
+
+**Fish**
+
+Add to `~/.config/fish/completions/twitch-dl.fish`:
+
+```
+_TWITCH_DL_COMPLETE=fish_source twitch-dl | source
+```
+
+**Zsh**
+
+Add to `~/.zshrc`:
+
+```
+eval "$(_TWITCH_DL_COMPLETE=zsh_source twitch-dl)"
+```

docs/usage.md

@@ -0,0 +1,24 @@
+# Usage
+
+To see a list of available commands run:
+
+```
+twitch-dl --help
+```
+
+And to see description and all arguments for a given command run:
+
+```
+twitch-dl <command> --help
+```
+
+## Commands
+
+Click on a command to see it's documentation.
+
+|                                      |                                 |
+|------------------------------------- | ------------------------------- |
+| [`videos`](commands/videos.html)     | List videos from a channel.     |
+| [`clips`](commands/clips.html)       | List clips from a channel.      |
+| [`download`](commands/download.html) | Download a video or clip.       |
+| [`info`](commands/info.html)         | Print info for a video or clip. |

pyproject.toml

@@ -0,0 +1,64 @@
+[build-system]
+requires = ["setuptools>=64", "setuptools_scm>=8"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "twitch-dl"
+authors = [{ name="Ivan Habunek", email="ivan@habunek.com" }]
+description = "Quickly download videos from twitch.tv from the comort of your terminal emulator"
+keywords=["twitch", "vod", "video", "download"]
+readme = "README.md"
+license = { file="LICENSE" }
+requires-python = ">=3.8"
+dynamic = ["version"]
+
+classifiers = [
+    "Development Status :: 5 - Production/Stable",
+    "Environment :: Console",
+    "License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
+    "Operating System :: OS Independent",
+]
+
+dependencies = [
+    "click>=8.0.0,<9.0.0",
+    "httpx>=0.17.0,<1.0.0",
+    "m3u8>=1.0.0,<4.0.0",
+]
+
+[tool.setuptools]
+packages = [
+    "twitchdl",
+    "twitchdl.commands",
+]
+
+[tool.setuptools_scm]
+
+[project.optional-dependencies]
+dev = [
+    "build",
+    "pytest",
+    "pyyaml",
+    "setuptools",
+    "twine",
+    "vermin",
+]
+
+test = [
+    "pytest",
+    "vermin",
+]
+
+[project.urls]
+"Homepage" = "https://twitch-dl.bezdomni.net/"
+"Source" = "https://github.com/ihabunek/twitch-dl"
+
+[project.scripts]
+twitch-dl = "twitchdl.cli:cli"
+
+[tool.pyright]
+typeCheckingMode = "strict"
+pythonVersion = "3.8"
+
+[tool.ruff]
+line-length = 100
+target-version = "py38"

requirements-dev.txt

@@ -1,3 +0,0 @@
-pytest
-twine
-wheel

scripts/generate_changelog

@@ -0,0 +1,40 @@
+#!/usr/bin/env python3
+
+"""
+Generates a more user-readable changelog from changelog.yaml.
+"""
+
+import textwrap
+import yaml
+
+with open("changelog.yaml", "r") as f:
+    data = yaml.safe_load(f)
+
+print("twitch-dl changelog")
+print("===================")
+print()
+print("<!-- Do not edit. This file is automatically generated from changelog.yaml.-->")
+print()
+
+for version in data.keys():
+    date = data[version]["date"]
+    changes = data[version]["changes"]
+    print(f"### [{version} ({date})](https://github.com/ihabunek/twitch-dl/releases/tag/{version})")
+    print()
+
+    if "description" in data[version]:
+        description = data[version]["description"].strip()
+        for line in textwrap.wrap(description, 80):
+            print(line)
+        print()
+
+    for c in changes:
+        lines = textwrap.wrap(c, 78)
+        initial = True
+        for line in lines:
+            if initial:
+                print("* " + line)
+                initial = False
+            else:
+                print("  " + line)
+    print()

scripts/generate_docs

@@ -0,0 +1,148 @@
+#!/usr/bin/env python3
+
+"""
+Auto-generates documentation from command defs in console.py.
+"""
+
+import click
+import html
+import os
+import re
+import shutil
+import textwrap
+
+from click import Command
+
+from twitchdl.cli import cli
+
+
+START_MARKER = "<!-- ------------------- generated docs start ------------------- -->"
+END_MARKER = "<!-- ------------------- generated docs end ------------------- -->"
+
+
+def main():
+    update_changelog()
+
+    parent_ctx = click.Context(cli, info_name="twitch-dl")
+    for name, command in cli.commands.items():
+        ctx = click.Context(cli, info_name=name, parent=parent_ctx)
+        update_docs(command, ctx)
+
+
+def update_changelog():
+    print("Updating: docs/changelog.md")
+    root = os.path.realpath(os.path.dirname(os.path.dirname(__file__)))
+    source = os.path.join(root, "CHANGELOG.md")
+    target = os.path.join(root, "docs/changelog.md")
+    shutil.copy(source, target)
+
+
+def update_docs(command: Command, ctx: click.Context):
+    path = os.path.join("docs", "commands", f"{command.name}.md")
+    content = render_command(command, ctx)
+
+    if not os.path.exists(path):
+        print(f"Creating: {path}")
+        write(path, content)
+    else:
+        print(f"Updating: {path}")
+        [_, handwritten] = read(path).split(END_MARKER)
+        content = f"{content.strip()}\n\n{END_MARKER}\n\n{handwritten.strip()}"
+        write(path, content)
+
+
+def render_command(command: Command, ctx: click.Context):
+    content = START_MARKER
+    content += f"\n# twitch-dl {command.name}\n\n"
+
+    if command.help:
+        content += command.help + "\n\n"
+
+    content += render_usage(ctx, command)
+    content += render_options(ctx, command)
+    return content
+
+
+def render_usage(ctx: click.Context, command: Command):
+    content = "### USAGE\n\n"
+    content += "```\n"
+    content += command.get_usage(ctx).replace("Usage: ", "")
+
+    content += "\n```\n\n"
+    return content
+
+
+def render_options(ctx, command: Command):
+    options = list(get_options(command))
+
+    if not options:
+        return ""
+
+    content = "### OPTIONS\n\n"
+
+    content += "<table>\n"
+    content += "<tbody>"
+    for opts, help in options:
+        content += textwrap.dedent(f"""
+        <tr>
+            <td class="code">{escape(opts)}</td>
+            <td>{escape(help)}</td>
+        </tr>
+        """)
+    content += "</tbody>\n"
+    content += "</table>\n\n"
+
+    return content
+
+
+def get_options(command: Command):
+    for option in command.params:
+        if isinstance(option, click.Option):
+            opts = ", ".join(option.opts)
+            opts += option_type(option)
+
+            help = option.help or ""
+            help = re.sub(r"\s+", " ", help)
+            help += choices(option)
+            if option.default:
+                help += f" [default: `{option.default}`]"
+
+            yield opts, help
+
+
+def option_type(option: click.Option):
+    match option.type:
+        case click.types.StringParamType():
+            return " TEXT"
+        case click.types.Choice():
+            return " TEXT"
+        case click.types.IntParamType():
+            return " INTEGER"
+        case _:
+            return ""
+
+def choices(option: click.Option):
+    if isinstance(option.type, click.Choice):
+        choices = ", ".join(f"`{c}`" for c in option.type.choices)
+        return f" Possible values: {choices}."
+    return ""
+
+
+def read(path):
+    with open(path, "r") as f:
+        return f.read()
+
+
+def write(path, content):
+    with open(path, "w") as f:
+        return f.write(content)
+
+
+def escape(text: str):
+    text = html.escape(text)
+    text = re.sub(r"`([\S]+)`", "<code>\\1</code>", text)
+    return text
+
+
+if __name__ == "__main__":
+    main()

scripts/tag_version

@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
+
+"""
+Creates an annotated git tag for a given version number.
+
+The tag will include the version number and changes for given version.
+
+Usage: tag_version [version]
+"""
+
+import subprocess
+import sys
+import textwrap
+from datetime import date
+from os import path
+
+import yaml
+
+path = path.join(path.dirname(path.dirname(path.abspath(__file__))), "changelog.yaml")
+with open(path, "r") as f:
+    changelog = yaml.safe_load(f)
+
+if len(sys.argv) != 2:
+    print("Wrong argument count", file=sys.stderr)
+    sys.exit(1)
+
+version = sys.argv[1]
+
+changelog_item = changelog.get(version)
+if not changelog_item:
+    print(f"Version `{version}` not found in changelog.", file=sys.stderr)
+    sys.exit(1)
+
+release_date = changelog_item["date"]
+changes = changelog_item["changes"]
+description = changelog_item["description"] if "description" in changelog_item else None
+
+if not isinstance(release_date, date):
+    print(f"Release date not set for version `{version}` in the changelog.", file=sys.stderr)
+    sys.exit(1)
+
+commit_message = f"twitch-dl {version}\n\n"
+if description:
+    lines = textwrap.wrap(description.strip(), 72)
+    commit_message += "\n".join(lines) + "\n\n"
+for c in changes:
+    lines = textwrap.wrap(c, 69)
+    initial = True
+    for line in lines:
+        lead = " *" if initial else "  "
+        initial = False
+        commit_message += f"{lead} {line}\n"
+
+proc = subprocess.run(["git", "tag", "-a", version, "-m", commit_message])
+if proc.returncode != 0:
+    sys.exit(1)
+
+print()
+print(commit_message)
+print()
+print(f"Version {version} tagged \\o/")

setup.py

@@ -1,41 +0,0 @@
-#!/usr/bin/env python
-
-from setuptools import setup
-
-long_description = """
-Quickly download videos from twitch.tv.
-
-Works simliarly to youtube-dl but downloads multiple VODs in parallel which
-makes it faster.
-"""
-
-setup(
-    name='twitch-dl',
-    version='1.13.0',
-    description='Twitch downloader',
-    long_description=long_description.strip(),
-    author='Ivan Habunek',
-    author_email='ivan@habunek.com',
-    url='https://github.com/ihabunek/twitch-dl/',
-    keywords='twitch vod video download',
-    license='GPLv3',
-    classifiers=[
-        'Development Status :: 5 - Production/Stable',
-        'License :: OSI Approved :: GNU General Public License v3 (GPLv3)',
-        'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.5',
-        'Programming Language :: Python :: 3.6',
-        'Programming Language :: Python :: 3.7',
-    ],
-    packages=['twitchdl'],
-    python_requires='>=3.5',
-    install_requires=[
-        "m3u8>=0.3.12,<0.4",
-        "requests>=2.13,<3.0",
-    ],
-    entry_points={
-        'console_scripts': [
-            'twitch-dl=twitchdl.console:main',
-        ],
-    }
-)

stdeb.cfg

@@ -1,3 +0,0 @@
-[DEFAULT]
-X-Python3-Version: >= 3.5
-Copyright-File: LICENSE

tests/test_api.py

@@ -0,0 +1,70 @@
+"""
+These tests depend on the channel having some videos and clips published.
+"""
+
+import httpx
+import pytest
+
+from twitchdl import twitch
+from twitchdl.commands.download import get_clip_authenticated_url
+from twitchdl.commands.videos import get_game_ids
+from twitchdl.exceptions import ConsoleError
+from twitchdl.playlists import enumerate_vods, load_m3u8, parse_playlists
+
+TEST_CHANNEL = "bananasaurus_rex"
+
+
+def test_get_videos():
+    videos = twitch.get_channel_videos(TEST_CHANNEL, 3, "time")
+    assert videos["pageInfo"]
+    assert len(videos["edges"]) > 0
+
+    video_id = videos["edges"][0]["node"]["id"]
+    video = twitch.get_video(video_id)
+    assert video is not None
+    assert video["id"] == video_id
+
+    access_token = twitch.get_access_token(video_id)
+    assert "signature" in access_token
+    assert "value" in access_token
+
+    playlists_txt = twitch.get_playlists(video_id, access_token)
+    assert playlists_txt.startswith("#EXTM3U")
+
+    playlists = parse_playlists(playlists_txt)
+    playlist_url = playlists[0].url
+
+    playlist_txt = httpx.get(playlist_url).text
+    assert playlist_txt.startswith("#EXTM3U")
+
+    playlist_m3u8 = load_m3u8(playlist_txt)
+    vods = enumerate_vods(playlist_m3u8)
+    assert vods[0].path == "0.ts"
+
+
+def test_get_clips():
+    """
+    This test depends on the channel having some videos published.
+    """
+    clips = twitch.get_channel_clips(TEST_CHANNEL, "all_time", 3)
+    assert clips["pageInfo"]
+    assert len(clips["edges"]) > 0
+
+    slug = clips["edges"][0]["node"]["slug"]
+    clip = twitch.get_clip(slug)
+    assert clip is not None
+    assert clip["slug"] == slug
+
+    assert get_clip_authenticated_url(slug, "source")
+
+
+def test_get_games():
+    assert get_game_ids([]) == []
+    assert get_game_ids(["Bioshock"]) == ["15866"]
+    assert get_game_ids(["Bioshock", "Portal"]) == ["15866", "6187"]
+
+
+def test_get_games_not_found():
+    with pytest.raises(ConsoleError) as ex:
+        get_game_ids(["the game which does not exist"])
+    assert str(ex.value) == "Game 'the game which does not exist' not found"

tests/test_cli.py

@@ -0,0 +1,156 @@
+import json
+
+import pytest
+from click.testing import CliRunner, Result
+
+from twitchdl import cli
+
+
+@pytest.fixture(scope="session")
+def runner():
+    return CliRunner(mix_stderr=False)
+
+
+def assert_ok(result: Result):
+    if result.exit_code != 0:
+        raise AssertionError(
+            f"Command failed with exit code {result.exit_code}\nStderr: {result.stderr}"
+        )
+
+
+def test_info_video(runner: CliRunner):
+    result = runner.invoke(cli.info, ["2090131595"])
+    assert_ok(result)
+
+    assert "Frost Fatales 2024 Day 1" in result.stdout
+    assert "frozenflygone playing Tomb Raider" in result.stdout
+
+
+def test_info_video_json(runner: CliRunner):
+    result = runner.invoke(cli.info, ["2090131595", "--json"])
+    assert_ok(result)
+
+    video = json.loads(result.stdout)
+    assert video["title"] == "Frost Fatales 2024 Day 1"
+    assert video["game"] == {"id": "2770", "name": "Tomb Raider"}
+    assert video["creator"] == {"login": "frozenflygone", "displayName": "frozenflygone"}
+
+
+def test_info_clip(runner: CliRunner):
+    result = runner.invoke(cli.info, ["PoisedTalentedPuddingChefFrank"])
+    assert_ok(result)
+
+    assert "AGDQ Crashes during Bioshock run" in result.stdout
+    assert "GamesDoneQuick playing BioShock" in result.stdout
+
+
+def test_info_clip_json(runner: CliRunner):
+    result = runner.invoke(cli.info, ["PoisedTalentedPuddingChefFrank", "--json"])
+    assert_ok(result)
+
+    clip = json.loads(result.stdout)
+    assert clip["slug"] == "PoisedTalentedPuddingChefFrank"
+    assert clip["title"] == "AGDQ Crashes during Bioshock run"
+    assert clip["game"] == {"id": "15866", "name": "BioShock"}
+    assert clip["broadcaster"] == {"displayName": "GamesDoneQuick", "login": "gamesdonequick"}
+
+
+def test_info_not_found(runner: CliRunner):
+    result = runner.invoke(cli.info, ["banana"])
+    assert result.exit_code == 1
+    assert "Clip banana not found" in result.stderr
+
+    result = runner.invoke(cli.info, ["12345"])
+    assert result.exit_code == 1
+    assert "Video 12345 not found" in result.stderr
+
+    result = runner.invoke(cli.info, [""])
+    assert result.exit_code == 1
+    assert "Invalid input" in result.stderr
+
+
+def test_download_clip(runner: CliRunner):
+    result = runner.invoke(
+        cli.download,
+        [
+            "PoisedTalentedPuddingChefFrank",
+            "-q",
+            "source",
+            "--dry-run",
+        ],
+    )
+    assert_ok(result)
+    assert (
+        "Found: AGDQ Crashes during Bioshock run by GamesDoneQuick, playing BioShock (30 sec)"
+        in result.stdout
+    )
+    assert (
+        "Target: 2020-01-10_3099545841_gamesdonequick_agdq_crashes_during_bioshock_run.mp4"
+        in result.stdout
+    )
+    assert "Dry run, clip not downloaded." in result.stdout
+
+
+def test_download_video(runner: CliRunner):
+    result = runner.invoke(
+        cli.download,
+        [
+            "2090131595",
+            "-q",
+            "source",
+            "--dry-run",
+        ],
+    )
+    assert_ok(result)
+    assert "Found: Frost Fatales 2024 Day 1 by frozenflygone" in result.stdout
+    assert (
+        "Output: 2024-03-14_2090131595_frozenflygone_frost_fatales_2024_day_1.mkv" in result.stdout
+    )
+    assert "Dry run, video not downloaded." in result.stdout
+
+
+def test_videos(runner: CliRunner):
+    result = runner.invoke(cli.videos, ["gamesdonequick", "--json"])
+    assert_ok(result)
+    videos = json.loads(result.stdout)
+
+    assert videos["count"] == 10
+    assert videos["totalCount"] > 0
+    video = videos["videos"][0]
+
+    result = runner.invoke(cli.videos, "gamesdonequick")
+    assert_ok(result)
+
+    assert f"Video {video['id']}" in result.stdout
+    assert video["title"] in result.stdout
+
+    result = runner.invoke(cli.videos, ["gamesdonequick", "--compact"])
+    assert_ok(result)
+
+    assert video["id"] in result.stdout
+    assert video["title"][:60] in result.stdout
+
+
+def test_videos_channel_not_found(runner: CliRunner):
+    result = runner.invoke(cli.videos, ["doesnotexisthopefully"])
+    assert result.exit_code == 1
+    assert result.stderr.strip() == "Error: Channel doesnotexisthopefully not found"
+
+
+def test_clips(runner: CliRunner):
+    result = runner.invoke(cli.clips, ["gamesdonequick", "--json"])
+    assert_ok(result)
+    clips = json.loads(result.stdout)
+    clip = clips[0]
+
+    result = runner.invoke(cli.clips, "gamesdonequick")
+    assert_ok(result)
+
+    assert f"Clip {clip['slug']}" in result.stdout
+    assert clip["title"] in result.stdout
+
+    result = runner.invoke(cli.clips, ["gamesdonequick", "--compact"])
+    assert_ok(result)
+
+    assert clip["slug"] in result.stdout
+    assert clip["title"][:60] in result.stdout

tests/test_download.py

@@ -1,45 +0,0 @@
-import pytest
-
-from unittest.mock import patch
-from twitchdl.commands import download
-from collections import namedtuple
-
-Args = namedtuple("args", ["video"])
-
-
-TEST_VIDEO_PATTERNS = [
-    ("702689313", "702689313"),
-    ("702689313", "https://twitch.tv/videos/702689313"),
-    ("702689313", "https://www.twitch.tv/videos/702689313"),
-]
-
-TEST_CLIP_PATTERNS = {
-    ("AbrasivePlayfulMangoMau5", "AbrasivePlayfulMangoMau5"),
-    ("AbrasivePlayfulMangoMau5", "https://clips.twitch.tv/AbrasivePlayfulMangoMau5"),
-    ("AbrasivePlayfulMangoMau5", "https://www.twitch.tv/dracul1nx/clip/AbrasivePlayfulMangoMau5"),
-    ("AbrasivePlayfulMangoMau5", "https://twitch.tv/dracul1nx/clip/AbrasivePlayfulMangoMau5"),
-    ("HungryProudRadicchioDoggo", "HungryProudRadicchioDoggo"),
-    ("HungryProudRadicchioDoggo", "https://clips.twitch.tv/HungryProudRadicchioDoggo"),
-    ("HungryProudRadicchioDoggo", "https://www.twitch.tv/bananasaurus_rex/clip/HungryProudRadicchioDoggo?filter=clips&range=7d&sort=time"),
-    ("HungryProudRadicchioDoggo", "https://twitch.tv/bananasaurus_rex/clip/HungryProudRadicchioDoggo?filter=clips&range=7d&sort=time"),
-}
-
-
-@patch("twitchdl.commands._download_clip")
-@patch("twitchdl.commands._download_video")
-@pytest.mark.parametrize("expected,input", TEST_VIDEO_PATTERNS)
-def test_video_patterns(video_dl, clip_dl, expected, input):
-    args = Args(video=input)
-    download(args)
-    video_dl.assert_called_once_with(expected, args)
-    clip_dl.assert_not_called()
-
-
-@patch("twitchdl.commands._download_clip")
-@patch("twitchdl.commands._download_video")
-@pytest.mark.parametrize("expected,input", TEST_CLIP_PATTERNS)
-def test_clip_patterns(video_dl, clip_dl, expected, input):
-    args = Args(video=input)
-    download(args)
-    clip_dl.assert_called_once_with(expected, args)
-    video_dl.assert_not_called()

tests/test_patterns.py

@@ -0,0 +1,38 @@
+import pytest
+
+from twitchdl.utils import parse_clip_identifier, parse_video_identifier
+
+TEST_VIDEO_PATTERNS = [
+    ("702689313", "702689313"),
+    ("702689313", "https://twitch.tv/videos/702689313"),
+    ("702689313", "https://www.twitch.tv/videos/702689313"),
+    ("702689313", "https://m.twitch.tv/videos/702689313"),
+]
+
+TEST_CLIP_PATTERNS = {
+    ("AbrasivePlayfulMangoMau5", "AbrasivePlayfulMangoMau5"),
+    ("AbrasivePlayfulMangoMau5", "https://clips.twitch.tv/AbrasivePlayfulMangoMau5"),
+    ("AbrasivePlayfulMangoMau5", "https://www.twitch.tv/dracul1nx/clip/AbrasivePlayfulMangoMau5"),
+    ("AbrasivePlayfulMangoMau5", "https://m.twitch.tv/dracul1nx/clip/AbrasivePlayfulMangoMau5"),
+    ("AbrasivePlayfulMangoMau5", "https://twitch.tv/dracul1nx/clip/AbrasivePlayfulMangoMau5"),
+    ("HungryProudRadicchioDoggo", "HungryProudRadicchioDoggo"),
+    ("HungryProudRadicchioDoggo", "https://clips.twitch.tv/HungryProudRadicchioDoggo"),
+    ("HungryProudRadicchioDoggo", "https://www.twitch.tv/bananasaurus_rex/clip/HungryProudRadicchioDoggo?filter=clips&range=7d&sort=time"),
+    ("HungryProudRadicchioDoggo", "https://m.twitch.tv/bananasaurus_rex/clip/HungryProudRadicchioDoggo?filter=clips&range=7d&sort=time"),
+    ("HungryProudRadicchioDoggo", "https://twitch.tv/bananasaurus_rex/clip/HungryProudRadicchioDoggo?filter=clips&range=7d&sort=time"),
+    ("GloriousColdbloodedTortoiseRuleFive-E017utJ4DZmHVpfQ", "GloriousColdbloodedTortoiseRuleFive-E017utJ4DZmHVpfQ"),
+    ("GloriousColdbloodedTortoiseRuleFive-E017utJ4DZmHVpfQ", "https://twitch.tv/dracul1nx/clip/GloriousColdbloodedTortoiseRuleFive-E017utJ4DZmHVpfQ"),
+    ("GloriousColdbloodedTortoiseRuleFive-E017utJ4DZmHVpfQ", "https://twitch.tv/dracul1nx/clip/GloriousColdbloodedTortoiseRuleFive-E017utJ4DZmHVpfQ?filter=clips&range=7d&sort=time"),
+    ("GloriousColdbloodedTortoiseRuleFive-E017utJ4DZmHVpfQ", "https://www.twitch.tv/dracul1nx/clip/GloriousColdbloodedTortoiseRuleFive-E017utJ4DZmHVpfQ?filter=clips&range=7d&sort=time"),
+    ("GloriousColdbloodedTortoiseRuleFive-E017utJ4DZmHVpfQ", "https://m.twitch.tv/dracul1nx/clip/GloriousColdbloodedTortoiseRuleFive-E017utJ4DZmHVpfQ?filter=clips&range=7d&sort=time"),
+}
+
+
+@pytest.mark.parametrize("expected,input", TEST_VIDEO_PATTERNS)
+def test_video_patterns(expected: str, input: str):
+    assert parse_video_identifier(input) == expected
+
+
+@pytest.mark.parametrize("expected,input", TEST_CLIP_PATTERNS)
+def test_clip_patterns(expected: str, input: str):
+    assert parse_clip_identifier(input) == expected

tests/test_progress.py

@@ -0,0 +1,102 @@
+from twitchdl.progress import Progress
+
+
+def test_initial_values():
+    progress = Progress(10)
+    assert progress.downloaded == 0
+    assert progress.estimated_total is None
+    assert progress.progress_perc == 0
+    assert progress.remaining_time is None
+    assert progress.speed is None
+    assert progress.vod_count == 10
+    assert progress.vod_downloaded_count == 0
+
+
+def test_downloaded():
+    progress = Progress(3)
+    progress.start(1, 300)
+    progress.start(2, 300)
+    progress.start(3, 300)
+
+    assert progress.downloaded == 0
+    assert progress.progress_bytes == 0
+    assert progress.progress_perc == 0
+
+    progress.advance(1, 100)
+    assert progress.downloaded == 100
+    assert progress.progress_bytes == 100
+    assert progress.progress_perc == 11
+
+    progress.advance(2, 200)
+    assert progress.downloaded == 300
+    assert progress.progress_bytes == 300
+    assert progress.progress_perc == 33
+
+    progress.advance(3, 150)
+    assert progress.downloaded == 450
+    assert progress.progress_bytes == 450
+    assert progress.progress_perc == 50
+
+    progress.advance(1, 50)
+    assert progress.downloaded == 500
+    assert progress.progress_bytes == 500
+    assert progress.progress_perc == 55
+
+    progress.abort(2)
+    assert progress.downloaded == 500
+    assert progress.progress_bytes == 300
+    assert progress.progress_perc == 33
+
+    progress.start(2, 300)
+
+    progress.advance(1, 150)
+    progress.advance(2, 300)
+    progress.advance(3, 150)
+
+    assert progress.downloaded == 1100
+    assert progress.progress_bytes == 900
+    assert progress.progress_perc == 100
+
+    progress.end(1)
+    progress.end(2)
+    progress.end(3)
+
+    assert progress.downloaded == 1100
+    assert progress.progress_bytes == 900
+    assert progress.progress_perc == 100
+
+
+def test_estimated_total():
+    progress = Progress(3)
+    assert progress.estimated_total is None
+
+    progress.start(1, 12000)
+    assert progress.estimated_total == 12000 * 3
+
+    progress.start(2, 11000)
+    assert progress.estimated_total == 11500 * 3
+
+    progress.start(3, 10000)
+    assert progress.estimated_total == 11000 * 3
+
+
+def test_vod_downloaded_count():
+    progress = Progress(3)
+
+    progress.start(1, 100)
+    progress.start(2, 100)
+    progress.start(3, 100)
+
+    assert progress.vod_downloaded_count == 0
+
+    progress.advance(1, 100)
+    progress.end(1)
+    assert progress.vod_downloaded_count == 1
+
+    progress.advance(2, 100)
+    progress.end(2)
+    assert progress.vod_downloaded_count == 2
+
+    progress.advance(3, 100)
+    progress.end(3)
+    assert progress.vod_downloaded_count == 3

tests/test_utils.py

@@ -0,0 +1,13 @@
+from twitchdl.utils import titlify, slugify
+
+
+def test_titlify():
+    assert titlify("Foo Bar Baz.") == "Foo Bar Baz."
+    assert titlify("Foo (Bar) [Baz]") == "Foo (Bar) [Baz]"
+    assert titlify("Foo@{} Bar Baz!\"#$%&/=?*+'🔪") == "Foo Bar Baz"
+
+
+def test_slugify():
+    assert slugify("Foo Bar Baz") == "foo_bar_baz"
+    assert slugify("  Foo   Bar   Baz  ") == "foo_bar_baz"
+    assert slugify("Foo@{}[] Bar Baz!\"#$%&/()=?*+'🔪") == "foo_bar_baz"

twitchdl/__init__.py

@@ -1,3 +1,8 @@
-__version__ = "1.13.0"
+from importlib import metadata
 
-CLIENT_ID = "kimne78kx3ncx6brgo4mv6wki5h1ko"
+try:
+    __version__ = metadata.version("twitch-dl")
+except metadata.PackageNotFoundError:
+    __version__ = "0.0.0"
+
+CLIENT_ID = "kd1unb4b3q4t58fwlpcbzcbnm76a8fp"

twitchdl/__main__.py

@@ -1,3 +1,3 @@
-from twitchdl.console import main
+from twitchdl.cli import cli
 
-main()
+cli()

twitchdl/cli.py

@@ -0,0 +1,402 @@
+import logging
+import platform
+import re
+import sys
+from typing import Optional, Tuple
+
+import click
+
+from twitchdl import __version__
+from twitchdl.entities import DownloadOptions
+from twitchdl.twitch import ClipsPeriod, VideosSort, VideosType
+
+# Tweak the Click context
+# https://click.palletsprojects.com/en/8.1.x/api/#context
+CONTEXT = dict(
+    # Enable using environment variables to set options
+    auto_envvar_prefix="TWITCH_DL",
+    # Add shorthand -h for invoking help
+    help_option_names=["-h", "--help"],
+    # Always show default values for options
+    show_default=True,
+    # Make help a bit wider
+    max_content_width=100,
+)
+
+json_option = click.option(
+    "--json",
+    is_flag=True,
+    default=False,
+    help="Print data as JSON rather than human readable text",
+)
+
+
+def validate_positive(_ctx: click.Context, _param: click.Parameter, value: Optional[int]):
+    if value is not None and value <= 0:
+        raise click.BadParameter("must be greater than 0")
+    return value
+
+
+def validate_time(_ctx: click.Context, _param: click.Parameter, value: str) -> Optional[int]:
+    """Parse a time string (hh:mm or hh:mm:ss) to number of seconds."""
+    if not value:
+        return None
+
+    parts = [int(p) for p in value.split(":")]
+
+    if not 2 <= len(parts) <= 3:
+        raise click.BadParameter("invalid time")
+
+    hours = parts[0]
+    minutes = parts[1]
+    seconds = parts[2] if len(parts) > 2 else 0
+
+    if hours < 0 or not (0 <= minutes <= 59) or not (0 <= seconds <= 59):
+        raise click.BadParameter("invalid time")
+
+    return hours * 3600 + minutes * 60 + seconds
+
+
+def validate_rate(_ctx: click.Context, _param: click.Parameter, value: str) -> Optional[int]:
+    if not value:
+        return None
+
+    match = re.search(r"^([0-9]+)(k|m|)$", value, flags=re.IGNORECASE)
+
+    if not match:
+        raise click.BadParameter("must be an integer, followed by an optional 'k' or 'm'")
+
+    amount = int(match.group(1))
+    unit = match.group(2)
+
+    if unit == "k":
+        return amount * 1024
+
+    if unit == "m":
+        return amount * 1024 * 1024
+
+    return amount
+
+
+@click.group(context_settings=CONTEXT)
+@click.option("--debug/--no-debug", default=False, help="Log debug info to stderr")
+@click.option("--color/--no-color", default=sys.stdout.isatty(), help="Use ANSI color in output")
+@click.version_option(package_name="twitch-dl")
+@click.pass_context
+def cli(ctx: click.Context, color: bool, debug: bool):
+    """twitch-dl - twitch.tv downloader
+
+    https://twitch-dl.bezdomni.net/
+    """
+    ctx.color = color
+
+    if debug:
+        logging.basicConfig(level=logging.DEBUG)
+        logging.getLogger("httpx").setLevel(logging.WARN)
+        logging.getLogger("httpcore").setLevel(logging.WARN)
+
+
+@cli.command()
+@click.argument("channel_name")
+@click.option(
+    "-a",
+    "--all",
+    help="Fetch all clips, overrides --limit",
+    is_flag=True,
+)
+@click.option(
+    "-c",
+    "--compact",
+    help="Show clips in compact mode, one line per video",
+    is_flag=True,
+)
+@click.option(
+    "-d",
+    "--download",
+    help="Download clips in given period (in source quality)",
+    is_flag=True,
+)
+@click.option(
+    "-l",
+    "--limit",
+    help="Number of clips to fetch. Defaults to 40 in compact mode, 10 otherwise.",
+    type=int,
+    callback=validate_positive,
+)
+@click.option(
+    "-p",
+    "--pager",
+    help="Number of clips to show per page. Disabled by default.",
+    type=int,
+    callback=validate_positive,
+    is_flag=False,
+    flag_value=10,
+)
+@click.option(
+    "-P",
+    "--period",
+    help="Period from which to return clips",
+    default="all_time",
+    type=click.Choice(["last_day", "last_week", "last_month", "all_time"]),
+)
+@json_option
+def clips(
+    channel_name: str,
+    all: bool,
+    compact: bool,
+    download: bool,
+    json: bool,
+    limit: Optional[int],
+    pager: Optional[int],
+    period: ClipsPeriod,
+):
+    """List or download clips for given CHANNEL_NAME."""
+    from twitchdl.commands.clips import clips
+
+    clips(
+        channel_name,
+        all=all,
+        compact=compact,
+        download=download,
+        json=json,
+        limit=limit,
+        pager=pager,
+        period=period,
+    )
+
+
+@cli.command()
+@click.argument("ids", nargs=-1)
+@click.option(
+    "-a",
+    "--auth-token",
+    help="""Authentication token, passed to Twitch to access subscriber only
+         VODs. Can be copied from the `auth_token` cookie in any browser logged
+         in on Twitch.""",
+)
+@click.option(
+    "-c",
+    "--chapter",
+    help="""Download a single chapter of the video. Specify the chapter number
+         or use the flag without a number to display a chapter select prompt.
+         """,
+    type=int,
+    is_flag=False,
+    flag_value=0,
+)
+@click.option(
+    "--concat",
+    is_flag=True,
+    help="""Do not use ffmpeg to join files, concat them instead. This will
+         produce a .ts file by default.""",
+)
+@click.option(
+    "-d",
+    "--dry-run",
+    help="Simulate the download provcess without actually downloading any files.",
+    is_flag=True,
+)
+@click.option(
+    "-e",
+    "--end",
+    help="Download video up to this time (hh:mm or hh:mm:ss)",
+    callback=validate_time,
+)
+@click.option(
+    "-f",
+    "--format",
+    help="""Video format to convert into, passed to ffmpeg as the target file
+         extension. Defaults to `mkv`. If `--concat` is passed, defaults to
+         `ts`.""",
+)
+@click.option(
+    "-k",
+    "--keep",
+    help="Don't delete downloaded VODs and playlists after merging.",
+    is_flag=True,
+)
+@click.option(
+    "--no-join",
+    help="Don't run ffmpeg to join the downloaded vods, implies --keep.",
+    is_flag=True,
+)
+@click.option(
+    "--overwrite",
+    help="Overwrite the target file if it already exists without prompting.",
+    is_flag=True,
+)
+@click.option(
+    "-o",
+    "--output",
+    help="Output file name template. See docs for details.",
+    default="{date}_{id}_{channel_login}_{title_slug}.{format}",
+)
+@click.option(
+    "-q",
+    "--quality",
+    help="Video quality, e.g. `720p`. Set to `source` to get best quality.",
+)
+@click.option(
+    "-r",
+    "--rate-limit",
+    help="""Limit the maximum download speed in bytes per second. Use 'k' and
+         'm' suffixes for kbps and mbps.""",
+    callback=validate_rate,
+)
+@click.option(
+    "-s",
+    "--start",
+    help="Download video from this time (hh:mm or hh:mm:ss)",
+    callback=validate_time,
+)
+@click.option(
+    "-w",
+    "--max-workers",
+    help="Number of workers for downloading vods concurrently",
+    type=int,
+    default=5,
+)
+def download(
+    ids: Tuple[str, ...],
+    auth_token: Optional[str],
+    chapter: Optional[int],
+    concat: bool,
+    dry_run: bool,
+    end: Optional[int],
+    format: str,
+    keep: bool,
+    no_join: bool,
+    overwrite: bool,
+    output: str,
+    quality: Optional[str],
+    rate_limit: Optional[int],
+    start: Optional[int],
+    max_workers: int,
+):
+    """Download videos or clips.
+
+    Pass one or more video ID, clip slug or Twitch URL to download.
+    """
+    from twitchdl.commands.download import download
+
+    if not format:
+        format = "ts" if concat else "mkv"
+
+    options = DownloadOptions(
+        auth_token=auth_token,
+        chapter=chapter,
+        concat=concat,
+        dry_run=dry_run,
+        end=end,
+        format=format,
+        keep=keep,
+        no_join=no_join,
+        overwrite=overwrite,
+        output=output,
+        quality=quality,
+        rate_limit=rate_limit,
+        start=start,
+        max_workers=max_workers,
+    )
+
+    download(list(ids), options)
+
+
+@cli.command()
+def env():
+    """Print environment information for inclusion in bug reports."""
+    click.echo(f"twitch-dl {__version__}")
+    click.echo(f"Python {sys.version}")
+    click.echo(f"Platform: {platform.platform()}")
+
+
+@cli.command()
+@click.argument("id")
+@json_option
+def info(id: str, json: bool):
+    """Print information for a given Twitch URL, video ID or clip slug."""
+    from twitchdl.commands.info import info
+
+    info(id, json=json)
+
+
+@cli.command()
+@click.argument("channel_name")
+@click.option(
+    "-a",
+    "--all",
+    help="Fetch all clips, overrides --limit",
+    is_flag=True,
+)
+@click.option(
+    "-c",
+    "--compact",
+    help="Show videos in compact mode, one line per video",
+    is_flag=True,
+)
+@click.option(
+    "-l",
+    "--limit",
+    help="Number of videos to fetch. Defaults to 40 in compact mode, 10 otherwise.",
+    type=int,
+    callback=validate_positive,
+)
+@click.option(
+    "-p",
+    "--pager",
+    help="Number of videos to show per page. Disabled by default.",
+    type=int,
+    callback=validate_positive,
+    is_flag=False,
+    flag_value=10,
+)
+@click.option(
+    "-g",
+    "--game",
+    "games_tuple",
+    help="Show videos of given game (can be given multiple times)",
+    multiple=True,
+)
+@click.option(
+    "-s",
+    "--sort",
+    help="Sorting order of videos",
+    default="time",
+    type=click.Choice(["views", "time"]),
+)
+@click.option(
+    "-t",
+    "--type",
+    help="Broadcast type",
+    default="archive",
+    type=click.Choice(["archive", "highlight", "upload"]),
+)
+@json_option
+def videos(
+    channel_name: str,
+    all: bool,
+    compact: bool,
+    games_tuple: Tuple[str, ...],
+    json: bool,
+    limit: Optional[int],
+    pager: Optional[int],
+    sort: VideosSort,
+    type: VideosType,
+):
+    """List or download clips for given CHANNEL_NAME."""
+    from twitchdl.commands.videos import videos
+
+    # Click provides a tuple, make it a list instead
+    games = list(games_tuple)
+
+    videos(
+        channel_name,
+        all=all,
+        compact=compact,
+        games=games,
+        json=json,
+        limit=limit,
+        pager=pager,
+        sort=sort,
+        type=type,
+    )

twitchdl/commands.py

@@ -1,398 +0,0 @@
-import m3u8
-import re
-import requests
-import shutil
-import subprocess
-import tempfile
-
-from os import path
-from pathlib import Path
-from urllib.parse import urlparse
-
-from twitchdl import twitch, utils
-from twitchdl.download import download_file, download_files
-from twitchdl.exceptions import ConsoleError
-from twitchdl.output import print_out, print_clip, print_video, print_json
-
-
-def _continue():
-    print_out(
-        "\nThere are more videos. "
-        "Press <green><b>Enter</green> to continue, "
-        "<yellow><b>Ctrl+C</yellow> to break."
-    )
-
-    try:
-        input()
-    except KeyboardInterrupt:
-        return False
-
-    return True
-
-
-def _get_game_ids(names):
-    if not names:
-        return []
-
-    game_ids = []
-    for name in names:
-        print_out("<dim>Looking up game '{}'...</dim>".format(name))
-        game_id = twitch.get_game_id(name)
-        if not game_id:
-            raise ConsoleError("Game '{}' not found".format(name))
-        game_ids.append(int(game_id))
-
-    return game_ids
-
-
-def _clips_json(args):
-    clips = twitch.get_channel_clips(args.channel_name, args.period, args.limit)
-    nodes = list(edge["node"] for edge in clips["edges"])
-    print_json(nodes)
-
-
-def _clips_download(args):
-    generator = twitch.channel_clips_generator(args.channel_name, args.period, 100)
-    for clips, _ in generator:
-        for clip in clips["edges"]:
-            clip = clip["node"]
-            url = clip["videoQualities"][0]["sourceURL"]
-            target = _clip_target_filename(clip)
-            if path.exists(target):
-                print_out("Already downloaded: <green>{}</green>".format(target))
-            else:
-                print_out("Downloading: <yellow>{}</yellow>".format(target))
-                download_file(url, target)
-
-
-def clips(args):
-    if args.json:
-        return _clips_json(args)
-
-    if args.download:
-        return _clips_download(args)
-
-    print_out("<dim>Loading clips...</dim>")
-    generator = twitch.channel_clips_generator(args.channel_name, args.period, args.limit)
-
-    first = 1
-
-    for clips, has_more in generator:
-        count = len(clips["edges"]) if "edges" in clips else 0
-        last = first + count - 1
-
-        print_out("-" * 80)
-        print_out("<yellow>Showing clips {}-{} of ??</yellow>".format(first, last))
-
-        for clip in clips["edges"]:
-            print_clip(clip["node"])
-
-        if not args.pager:
-            print_out(
-                "\n<dim>There are more clips. "
-                "Increase the --limit or use --pager to see the rest.</dim>"
-            )
-            break
-
-        if not has_more or not _continue():
-            break
-
-        first += count
-    else:
-        print_out("<yellow>No clips found</yellow>")
-
-
-def videos(args):
-    game_ids = _get_game_ids(args.game)
-
-    print_out("<dim>Loading videos...</dim>")
-    generator = twitch.channel_videos_generator(
-        args.channel_name, args.limit, args.sort, args.type, game_ids=game_ids)
-
-    first = 1
-
-    for videos, has_more in generator:
-        count = len(videos["edges"]) if "edges" in videos else 0
-        total = videos["totalCount"]
-        last = first + count - 1
-
-        print_out("-" * 80)
-        print_out("<yellow>Showing videos {}-{} of {}</yellow>".format(first, last, total))
-
-        for video in videos["edges"]:
-            print_video(video["node"])
-
-        if not args.pager:
-            print_out(
-                "\n<dim>There are more videos. "
-                "Increase the --limit or use --pager to see the rest.</dim>"
-            )
-            break
-
-        if not has_more or not _continue():
-            break
-
-        first += count
-    else:
-        print_out("<yellow>No videos found</yellow>")
-
-
-def _parse_playlists(playlists_m3u8):
-    playlists = m3u8.loads(playlists_m3u8)
-
-    for p in playlists.playlists:
-        name = p.media[0].name if p.media else ""
-        resolution = "x".join(str(r) for r in p.stream_info.resolution)
-        yield name, resolution, p.uri
-
-
-def _get_playlist_by_name(playlists, quality):
-    if quality == "source":
-        _, _, uri = playlists[0]
-        return uri
-
-    for name, _, uri in playlists:
-        if name == quality:
-            return uri
-
-    available = ", ".join([name for (name, _, _) in playlists])
-    msg = "Quality '{}' not found. Available qualities are: {}".format(quality, available)
-    raise ConsoleError(msg)
-
-
-def _select_playlist_interactive(playlists):
-    print_out("\nAvailable qualities:")
-    for n, (name, resolution, uri) in enumerate(playlists):
-        print_out("{}) {} [{}]".format(n + 1, name, resolution))
-
-    no = utils.read_int("Choose quality", min=1, max=len(playlists) + 1, default=1)
-    _, _, uri = playlists[no - 1]
-    return uri
-
-
-def _join_vods(playlist_path, target, overwrite):
-    command = [
-        "ffmpeg",
-        "-i", playlist_path,
-        "-c", "copy",
-        target,
-        "-stats",
-        "-loglevel", "warning",
-    ]
-
-    if overwrite:
-        command.append("-y")
-
-    print_out("<dim>{}</dim>".format(" ".join(command)))
-    result = subprocess.run(command)
-    if result.returncode != 0:
-        raise ConsoleError("Joining files failed")
-
-
-def _video_target_filename(video, format):
-    match = re.search(r"^(\d{4})-(\d{2})-(\d{2})T", video['published_at'])
-    date = "".join(match.groups())
-
-    name = "_".join([
-        date,
-        video['_id'][1:],
-        video['channel']['name'],
-        utils.slugify(video['title']),
-    ])
-
-    return name + "." + format
-
-
-def _clip_target_filename(clip):
-    url = clip["videoQualities"][0]["sourceURL"]
-    _, ext = path.splitext(url)
-    ext = ext.lstrip(".")
-
-    match = re.search(r"^(\d{4})-(\d{2})-(\d{2})T", clip["createdAt"])
-    date = "".join(match.groups())
-
-    name = "_".join([
-        date,
-        clip["id"],
-        clip["broadcaster"]["channel"]["name"],
-        utils.slugify(clip["title"]),
-    ])
-
-    return "{}.{}".format(name, ext)
-
-
-def _get_vod_paths(playlist, start, end):
-    """Extract unique VOD paths for download from playlist."""
-    files = []
-    vod_start = 0
-    for segment in playlist.segments:
-        vod_end = vod_start + segment.duration
-
-        # `vod_end > start` is used here becuase it's better to download a bit
-        # more than a bit less, similar for the end condition
-        start_condition = not start or vod_end > start
-        end_condition = not end or vod_start < end
-
-        if start_condition and end_condition and segment.uri not in files:
-            files.append(segment.uri)
-
-        vod_start = vod_end
-
-    return files
-
-
-def _crete_temp_dir(base_uri):
-    """Create a temp dir to store downloads if it doesn't exist."""
-    path = urlparse(base_uri).path.lstrip("/")
-    temp_dir = Path(tempfile.gettempdir(), "twitch-dl", path)
-    temp_dir.mkdir(parents=True, exist_ok=True)
-    return temp_dir
-
-
-VIDEO_PATTERNS = [
-    r"^(?P<id>\d+)?$",
-    r"^https://(www.)?twitch.tv/videos/(?P<id>\d+)(\?.+)?$",
-]
-
-CLIP_PATTERNS = [
-    r"^(?P<slug>[A-Za-z0-9]+)$",
-    r"^https://(www.)?twitch.tv/\w+/clip/(?P<slug>[A-Za-z0-9]+)(\?.+)?$",
-    r"^https://clips.twitch.tv/(?P<slug>[A-Za-z0-9]+)(\?.+)?$",
-]
-
-
-def download(args):
-    for pattern in VIDEO_PATTERNS:
-        match = re.match(pattern, args.video)
-        if match:
-            video_id = match.group('id')
-            return _download_video(video_id, args)
-
-    for pattern in CLIP_PATTERNS:
-        match = re.match(pattern, args.video)
-        if match:
-            clip_slug = match.group('slug')
-            return _download_clip(clip_slug, args)
-
-    raise ConsoleError("Invalid video: {}".format(args.video))
-
-
-def _get_clip_url(clip, args):
-    qualities = clip["videoQualities"]
-
-    # Quality given as an argument
-    if args.quality:
-        if args.quality == "source":
-            return qualities[0]["sourceURL"]
-
-        selected_quality = args.quality.rstrip("p")  # allow 720p as well as 720
-        for q in qualities:
-            if q["quality"] == selected_quality:
-                return q["sourceURL"]
-
-        available = ", ".join([str(q["quality"]) for q in qualities])
-        msg = "Quality '{}' not found. Available qualities are: {}".format(args.quality, available)
-        raise ConsoleError(msg)
-
-    # Ask user to select quality
-    print_out("\nAvailable qualities:")
-    for n, q in enumerate(qualities):
-        print_out("{}) {} [{} fps]".format(n + 1, q["quality"], q["frameRate"]))
-    print_out()
-
-    no = utils.read_int("Choose quality", min=1, max=len(qualities), default=1)
-    selected_quality = qualities[no - 1]
-    return selected_quality["sourceURL"]
-
-
-def _download_clip(slug, args):
-    print_out("<dim>Looking up clip...</dim>")
-    clip = twitch.get_clip(slug)
-
-    if not clip:
-        raise ConsoleError("Clip '{}' not found".format(slug))
-
-    print_out("Found: <green>{}</green> by <yellow>{}</yellow>, playing <blue>{}</blue> ({})".format(
-        clip["title"],
-        clip["broadcaster"]["displayName"],
-        clip["game"]["name"],
-        utils.format_duration(clip["durationSeconds"])
-    ))
-
-    url = _get_clip_url(clip, args)
-    print_out("<dim>Selected URL: {}</dim>".format(url))
-
-    target = _clip_target_filename(clip)
-
-    print_out("Downloading clip...")
-    download_file(url, target)
-
-    print_out("Downloaded: {}".format(target))
-
-
-def _download_video(video_id, args):
-    if args.start and args.end and args.end <= args.start:
-        raise ConsoleError("End time must be greater than start time")
-
-    print_out("<dim>Looking up video...</dim>")
-    video = twitch.get_video(video_id)
-
-    print_out("Found: <blue>{}</blue> by <yellow>{}</yellow>".format(
-        video['title'], video['channel']['display_name']))
-
-    print_out("<dim>Fetching access token...</dim>")
-    access_token = twitch.get_access_token(video_id)
-
-    print_out("<dim>Fetching playlists...</dim>")
-    playlists_m3u8 = twitch.get_playlists(video_id, access_token)
-    playlists = list(_parse_playlists(playlists_m3u8))
-    playlist_uri = (_get_playlist_by_name(playlists, args.quality) if args.quality
-            else _select_playlist_interactive(playlists))
-
-    print_out("<dim>Fetching playlist...</dim>")
-    response = requests.get(playlist_uri)
-    response.raise_for_status()
-    playlist = m3u8.loads(response.text)
-
-    base_uri = re.sub("/[^/]+$", "/", playlist_uri)
-    target_dir = _crete_temp_dir(base_uri)
-    vod_paths = _get_vod_paths(playlist, args.start, args.end)
-
-    # Save playlists for debugging purposes
-    with open(path.join(target_dir, "playlists.m3u8"), "w") as f:
-        f.write(playlists_m3u8)
-    with open(path.join(target_dir, "playlist.m3u8"), "w") as f:
-        f.write(response.text)
-
-    print_out("\nDownloading {} VODs using {} workers to {}".format(
-        len(vod_paths), args.max_workers, target_dir))
-    path_map = download_files(base_uri, target_dir, vod_paths, args.max_workers)
-
-    # Make a modified playlist which references downloaded VODs
-    # Keep only the downloaded segments and skip the rest
-    org_segments = playlist.segments.copy()
-    playlist.segments.clear()
-    for segment in org_segments:
-        if segment.uri in path_map:
-            segment.uri = path_map[segment.uri]
-            playlist.segments.append(segment)
-
-    playlist_path = path.join(target_dir, "playlist_downloaded.m3u8")
-    playlist.dump(playlist_path)
-
-    if args.no_join:
-        print_out("\n\n<dim>Skipping joining files...</dim>")
-        print_out("VODs downloaded to:\n<blue>{}</blue>".format(target_dir))
-        return
-
-    print_out("\n\nJoining files...")
-    target = _video_target_filename(video, args.format)
-    _join_vods(playlist_path, target, args.overwrite)
-
-    if args.keep:
-        print_out("\n<dim>Temporary files not deleted: {}</dim>".format(target_dir))
-    else:
-        print_out("\n<dim>Deleting temporary files...</dim>")
-        shutil.rmtree(target_dir)
-
-    print_out("\nDownloaded: <green>{}</green>".format(target))

twitchdl/commands/clips.py

@@ -0,0 +1,95 @@
+import re
+import sys
+from os import path
+from typing import Callable, Generator, Optional
+
+import click
+
+from twitchdl import twitch, utils
+from twitchdl.commands.download import get_clip_authenticated_url
+from twitchdl.download import download_file
+from twitchdl.output import green, print_clip, print_clip_compact, print_json, print_paged, yellow
+from twitchdl.twitch import Clip, ClipsPeriod
+
+
+def clips(
+    channel_name: str,
+    *,
+    all: bool = False,
+    compact: bool = False,
+    download: bool = False,
+    json: bool = False,
+    limit: Optional[int] = None,
+    pager: Optional[int] = None,
+    period: ClipsPeriod = "all_time",
+):
+    # Set different defaults for limit for compact display
+    default_limit = 40 if compact else 10
+
+    # Ignore --limit if --pager or --all are given
+    limit = sys.maxsize if all or pager else (limit or default_limit)
+
+    generator = twitch.channel_clips_generator(channel_name, period, limit)
+
+    if json:
+        return print_json(list(generator))
+
+    if download:
+        return _download_clips(generator)
+
+    print_fn = print_clip_compact if compact else print_clip
+
+    if pager:
+        return print_paged("Clips", generator, print_fn, pager)
+
+    return _print_all(generator, print_fn, all)
+
+
+def _target_filename(clip: Clip):
+    url = clip["videoQualities"][0]["sourceURL"]
+    _, ext = path.splitext(url)
+    ext = ext.lstrip(".")
+
+    match = re.search(r"^(\d{4})-(\d{2})-(\d{2})T", clip["createdAt"])
+    if not match:
+        raise ValueError(f"Failed parsing date from: {clip['createdAt']}")
+    date = "".join(match.groups())
+
+    name = "_".join(
+        [
+            date,
+            clip["id"],
+            clip["broadcaster"]["login"],
+            utils.slugify(clip["title"]),
+        ]
+    )
+
+    return f"{name}.{ext}"
+
+
+def _download_clips(generator: Generator[Clip, None, None]):
+    for clip in generator:
+        target = _target_filename(clip)
+
+        if path.exists(target):
+            click.echo(f"Already downloaded: {green(target)}")
+        else:
+            url = get_clip_authenticated_url(clip["slug"], "source")
+            click.echo(f"Downloading: {yellow(target)}")
+            download_file(url, target)
+
+
+def _print_all(
+    generator: Generator[Clip, None, None],
+    print_fn: Callable[[Clip], None],
+    all: bool,
+):
+    for clip in generator:
+        print_fn(clip)
+
+    if not all:
+        click.secho(
+            "\nThere may be more clips. "
+            + "Increase the --limit, use --all or --pager to see the rest.",
+            dim=True,
+        )

twitchdl/commands/download.py

@@ -0,0 +1,365 @@
+import asyncio
+import os
+import platform
+import re
+import shutil
+import subprocess
+import tempfile
+from os import path
+from pathlib import Path
+from typing import Dict, List
+from urllib.parse import urlencode, urlparse
+
+import click
+import httpx
+
+from twitchdl import twitch, utils
+from twitchdl.download import download_file
+from twitchdl.entities import DownloadOptions
+from twitchdl.exceptions import ConsoleError
+from twitchdl.http import download_all
+from twitchdl.output import blue, bold, green, print_log, yellow
+from twitchdl.playlists import (
+    enumerate_vods,
+    load_m3u8,
+    make_join_playlist,
+    parse_playlists,
+    select_playlist,
+)
+from twitchdl.twitch import Chapter, Clip, ClipAccessToken, Video
+
+
+def download(ids: List[str], args: DownloadOptions):
+    for video_id in ids:
+        download_one(video_id, args)
+
+
+def download_one(video: str, args: DownloadOptions):
+    video_id = utils.parse_video_identifier(video)
+    if video_id:
+        return _download_video(video_id, args)
+
+    clip_slug = utils.parse_clip_identifier(video)
+    if clip_slug:
+        return _download_clip(clip_slug, args)
+
+    raise ConsoleError(f"Invalid input: {video}")
+
+
+def _join_vods(playlist_path: str, target: str, overwrite: bool, video: Video):
+    description = video["description"] or ""
+    description = description.strip()
+
+    command = [
+        "ffmpeg",
+        "-i",
+        playlist_path,
+        "-c",
+        "copy",
+        "-metadata",
+        f"artist={video['creator']['displayName']}",
+        "-metadata",
+        f"title={video['title']}",
+        "-metadata",
+        f"description={description}",
+        "-metadata",
+        "encoded_by=twitch-dl",
+        "-stats",
+        "-loglevel",
+        "warning",
+        f"file:{target}",
+    ]
+
+    if overwrite:
+        command.append("-y")
+
+    click.secho(f"{' '.join(command)}", dim=True)
+    result = subprocess.run(command)
+    if result.returncode != 0:
+        raise ConsoleError("Joining files failed")
+
+
+def _concat_vods(vod_paths: List[str], target: str):
+    tool = "type" if platform.system() == "Windows" else "cat"
+    command = [tool] + vod_paths
+
+    with open(target, "wb") as target_file:
+        result = subprocess.run(command, stdout=target_file)
+        if result.returncode != 0:
+            raise ConsoleError(f"Joining files failed: {result.stderr}")
+
+
+def get_video_placeholders(video: Video, format: str) -> Dict[str, str]:
+    date, time = video["publishedAt"].split("T")
+    game = video["game"]["name"] if video["game"] else "Unknown"
+
+    return {
+        "channel": video["creator"]["displayName"],
+        "channel_login": video["creator"]["login"],
+        "date": date,
+        "datetime": video["publishedAt"],
+        "format": format,
+        "game": game,
+        "game_slug": utils.slugify(game),
+        "id": video["id"],
+        "time": time,
+        "title": utils.titlify(video["title"]),
+        "title_slug": utils.slugify(video["title"]),
+    }
+
+
+def _video_target_filename(video: Video, args: DownloadOptions):
+    subs = get_video_placeholders(video, args.format)
+
+    try:
+        return args.output.format(**subs)
+    except KeyError as e:
+        supported = ", ".join(subs.keys())
+        raise ConsoleError(f"Invalid key {e} used in --output. Supported keys are: {supported}")
+
+
+def _clip_target_filename(clip: Clip, args: DownloadOptions):
+    date, time = clip["createdAt"].split("T")
+    game = clip["game"]["name"] if clip["game"] else "Unknown"
+
+    url = clip["videoQualities"][0]["sourceURL"]
+    _, ext = path.splitext(url)
+    ext = ext.lstrip(".")
+
+    subs = {
+        "channel": clip["broadcaster"]["displayName"],
+        "channel_login": clip["broadcaster"]["login"],
+        "date": date,
+        "datetime": clip["createdAt"],
+        "format": ext,
+        "game": game,
+        "game_slug": utils.slugify(game),
+        "id": clip["id"],
+        "slug": clip["slug"],
+        "time": time,
+        "title": utils.titlify(clip["title"]),
+        "title_slug": utils.slugify(clip["title"]),
+    }
+
+    try:
+        return args.output.format(**subs)
+    except KeyError as e:
+        supported = ", ".join(subs.keys())
+        raise ConsoleError(f"Invalid key {e} used in --output. Supported keys are: {supported}")
+
+
+def _crete_temp_dir(base_uri: str) -> str:
+    """Create a temp dir to store downloads if it doesn't exist."""
+    path = urlparse(base_uri).path.lstrip("/")
+    temp_dir = Path(tempfile.gettempdir(), "twitch-dl", path)
+    temp_dir.mkdir(parents=True, exist_ok=True)
+    return str(temp_dir)
+
+
+def _get_clip_url(access_token: ClipAccessToken, quality: str) -> str:
+    qualities = access_token["videoQualities"]
+
+    # Quality given as an argument
+    if quality:
+        if quality == "source":
+            return qualities[0]["sourceURL"]
+
+        selected_quality = quality.rstrip("p")  # allow 720p as well as 720
+        for q in qualities:
+            if q["quality"] == selected_quality:
+                return q["sourceURL"]
+
+        available = ", ".join([str(q["quality"]) for q in qualities])
+        msg = f"Quality '{quality}' not found. Available qualities are: {available}"
+        raise ConsoleError(msg)
+
+    # Ask user to select quality
+    click.echo("\nAvailable qualities:")
+    for n, q in enumerate(qualities):
+        click.echo(f"{n + 1}) {bold(q['quality'])} [{q['frameRate']} fps]")
+    click.echo()
+
+    no = utils.read_int("Choose quality", min=1, max=len(qualities), default=1)
+    selected_quality = qualities[no - 1]
+    return selected_quality["sourceURL"]
+
+
+def get_clip_authenticated_url(slug: str, quality: str):
+    print_log("Fetching access token...")
+    access_token = twitch.get_clip_access_token(slug)
+
+    if not access_token:
+        raise ConsoleError(f"Access token not found for slug '{slug}'")
+
+    url = _get_clip_url(access_token, quality)
+
+    query = urlencode(
+        {
+            "sig": access_token["playbackAccessToken"]["signature"],
+            "token": access_token["playbackAccessToken"]["value"],
+        }
+    )
+
+    return f"{url}?{query}"
+
+
+def _download_clip(slug: str, args: DownloadOptions) -> None:
+    print_log("Looking up clip...")
+    clip = twitch.get_clip(slug)
+
+    if not clip:
+        raise ConsoleError(f"Clip '{slug}' not found")
+
+    title = clip["title"]
+    user = clip["broadcaster"]["displayName"]
+    game = clip["game"]["name"] if clip["game"] else "Unknown"
+    duration = utils.format_duration(clip["durationSeconds"])
+    click.echo(f"Found: {green(title)} by {yellow(user)}, playing {blue(game)} ({duration})")
+
+    target = _clip_target_filename(clip, args)
+    click.echo(f"Target: {blue(target)}")
+
+    if not args.overwrite and path.exists(target):
+        response = click.prompt("File exists. Overwrite? [Y/n]", default="Y", show_default=False)
+        if response.lower().strip() != "y":
+            raise click.Abort()
+        args.overwrite = True
+
+    url = get_clip_authenticated_url(slug, args.quality)
+    print_log(f"Selected URL: {url}")
+
+    if args.dry_run:
+        click.echo("Dry run, clip not downloaded.")
+    else:
+        print_log("Downloading clip...")
+        download_file(url, target)
+        click.echo(f"Downloaded: {blue(target)}")
+
+
+def _download_video(video_id: str, args: DownloadOptions) -> None:
+    if args.start and args.end and args.end <= args.start:
+        raise ConsoleError("End time must be greater than start time")
+
+    print_log("Looking up video...")
+    video = twitch.get_video(video_id)
+
+    if not video:
+        raise ConsoleError(f"Video {video_id} not found")
+
+    click.echo(f"Found: {blue(video['title'])} by {yellow(video['creator']['displayName'])}")
+
+    target = _video_target_filename(video, args)
+    click.echo(f"Output: {blue(target)}")
+
+    if not args.overwrite and path.exists(target):
+        response = click.prompt("File exists. Overwrite? [Y/n]", default="Y", show_default=False)
+        if response.lower().strip() != "y":
+            raise click.Abort()
+        args.overwrite = True
+
+    # Chapter select or manual offset
+    start, end = _determine_time_range(video_id, args)
+
+    print_log("Fetching access token...")
+    access_token = twitch.get_access_token(video_id, auth_token=args.auth_token)
+
+    print_log("Fetching playlists...")
+    playlists_text = twitch.get_playlists(video_id, access_token)
+    playlists = parse_playlists(playlists_text)
+    playlist = select_playlist(playlists, args.quality)
+
+    print_log("Fetching playlist...")
+    vods_text = http_get(playlist.url)
+    vods_m3u8 = load_m3u8(vods_text)
+    vods = enumerate_vods(vods_m3u8, start, end)
+
+    if args.dry_run:
+        click.echo("Dry run, video not downloaded.")
+        return
+
+    base_uri = re.sub("/[^/]+$", "/", playlist.url)
+    target_dir = _crete_temp_dir(base_uri)
+
+    # Save playlists for debugging purposes
+    with open(path.join(target_dir, "playlists.m3u8"), "w") as f:
+        f.write(playlists_text)
+    with open(path.join(target_dir, "playlist.m3u8"), "w") as f:
+        f.write(vods_text)
+
+    click.echo(f"\nDownloading {len(vods)} VODs using {args.max_workers} workers to {target_dir}")
+
+    sources = [base_uri + vod.path for vod in vods]
+    targets = [os.path.join(target_dir, f"{vod.index:05d}.ts") for vod in vods]
+    asyncio.run(download_all(sources, targets, args.max_workers, rate_limit=args.rate_limit))
+
+    join_playlist = make_join_playlist(vods_m3u8, vods, targets)
+    join_playlist_path = path.join(target_dir, "playlist_downloaded.m3u8")
+    join_playlist.dump(join_playlist_path)  # type: ignore
+    click.echo()
+
+    if args.no_join:
+        print_log("Skipping joining files...")
+        click.echo(f"VODs downloaded to:\n{blue(target_dir)}")
+        return
+
+    if args.concat:
+        print_log("Concating files...")
+        _concat_vods(targets, target)
+    else:
+        print_log("Joining files...")
+        _join_vods(join_playlist_path, target, args.overwrite, video)
+
+    click.echo()
+
+    if args.keep:
+        click.echo(f"Temporary files not deleted: {target_dir}")
+    else:
+        print_log("Deleting temporary files...")
+        shutil.rmtree(target_dir)
+
+    click.echo(f"\nDownloaded: {green(target)}")
+
+
+def http_get(url: str) -> str:
+    response = httpx.get(url)
+    response.raise_for_status()
+    return response.text
+
+
+def _determine_time_range(video_id: str, args: DownloadOptions):
+    if args.start or args.end:
+        return args.start, args.end
+
+    if args.chapter is not None:
+        print_log("Fetching chapters...")
+        chapters = twitch.get_video_chapters(video_id)
+
+        if not chapters:
+            raise ConsoleError("This video has no chapters")
+
+        if args.chapter == 0:
+            chapter = _choose_chapter_interactive(chapters)
+        else:
+            try:
+                chapter = chapters[args.chapter - 1]
+            except IndexError:
+                raise ConsoleError(
+                    f"Chapter {args.chapter} does not exist. This video has {len(chapters)} chapters."
+                )
+
+        click.echo(f'Chapter selected: {blue(chapter["description"])}\n')
+        start = chapter["positionMilliseconds"] // 1000
+        duration = chapter["durationMilliseconds"] // 1000
+        return start, start + duration
+
+    return None, None
+
+
+def _choose_chapter_interactive(chapters: List[Chapter]):
+    click.echo("\nChapters:")
+    for index, chapter in enumerate(chapters):
+        duration = utils.format_time(chapter["durationMilliseconds"] // 1000)
+        click.echo(f'{index + 1}) {bold(chapter["description"])} ({duration})')
+    index = utils.read_int("Select a chapter", 1, len(chapters))
+    chapter = chapters[index - 1]
+    return chapter

twitchdl/commands/info.py

@@ -0,0 +1,102 @@
+from typing import List
+
+import click
+import m3u8
+
+from twitchdl import twitch, utils
+from twitchdl.commands.download import get_video_placeholders
+from twitchdl.exceptions import ConsoleError
+from twitchdl.output import bold, print_clip, print_json, print_log, print_table, print_video
+from twitchdl.playlists import parse_playlists
+from twitchdl.twitch import Chapter, Clip, Video
+
+
+def info(id: str, *, json: bool = False):
+    video_id = utils.parse_video_identifier(id)
+    if video_id:
+        print_log("Fetching video...")
+        video = twitch.get_video(video_id)
+
+        if not video:
+            raise ConsoleError(f"Video {video_id} not found")
+
+        print_log("Fetching access token...")
+        access_token = twitch.get_access_token(video_id)
+
+        print_log("Fetching playlists...")
+        playlists = twitch.get_playlists(video_id, access_token)
+
+        print_log("Fetching chapters...")
+        chapters = twitch.get_video_chapters(video_id)
+
+        if json:
+            video_json(video, playlists, chapters)
+        else:
+            video_info(video, playlists, chapters)
+        return
+
+    clip_slug = utils.parse_clip_identifier(id)
+    if clip_slug:
+        print_log("Fetching clip...")
+        clip = twitch.get_clip(clip_slug)
+        if not clip:
+            raise ConsoleError(f"Clip {clip_slug} not found")
+
+        if json:
+            print_json(clip)
+        else:
+            clip_info(clip)
+        return
+
+    raise ConsoleError(f"Invalid input: {id}")
+
+
+def video_info(video: Video, playlists: str, chapters: List[Chapter]):
+    click.echo()
+    print_video(video)
+
+    click.echo("Playlists:")
+    for p in parse_playlists(playlists):
+        click.echo(f"{bold(p.name)} {p.url}")
+
+    if chapters:
+        click.echo()
+        click.echo("Chapters:")
+        for chapter in chapters:
+            start = utils.format_time(chapter["positionMilliseconds"] // 1000, force_hours=True)
+            duration = utils.format_time(chapter["durationMilliseconds"] // 1000)
+            click.echo(f'{start} {bold(chapter["description"])} ({duration})')
+
+    placeholders = get_video_placeholders(video, format="mkv")
+    placeholders = [[f"{{{k}}}", v] for k, v in placeholders.items()]
+    click.echo("")
+    print_table(["Placeholder", "Value"], placeholders)
+
+
+def video_json(video: Video, playlists: str, chapters: List[Chapter]):
+    playlists = m3u8.loads(playlists).playlists
+
+    video["playlists"] = [
+        {
+            "bandwidth": p.stream_info.bandwidth,
+            "resolution": p.stream_info.resolution,
+            "codecs": p.stream_info.codecs,
+            "video": p.stream_info.video,
+            "uri": p.uri,
+        }
+        for p in playlists
+    ]
+
+    video["chapters"] = chapters
+
+    print_json(video)
+
+
+def clip_info(clip: Clip):
+    click.echo()
+    print_clip(clip)
+    click.echo()
+    click.echo("Download links:")
+
+    for q in clip["videoQualities"]:
+        click.echo(f"{bold(q['quality'])} [{q['frameRate']} fps] {q['sourceURL']}")

twitchdl/commands/videos.py

@@ -0,0 +1,79 @@
+import sys
+from typing import List, Optional
+
+import click
+
+from twitchdl import twitch
+from twitchdl.exceptions import ConsoleError
+from twitchdl.output import print_json, print_log, print_paged, print_video, print_video_compact
+
+
+def videos(
+    channel_name: str,
+    *,
+    all: bool,
+    compact: bool,
+    games: List[str],
+    json: bool,
+    limit: Optional[int],
+    pager: Optional[int],
+    sort: twitch.VideosSort,
+    type: twitch.VideosType,
+):
+    game_ids = get_game_ids(games)
+
+    # Set different defaults for limit for compact display
+    limit = limit or (40 if compact else 10)
+
+    # Ignore --limit if --pager or --all are given
+    max_videos = sys.maxsize if all or pager else limit
+
+    total_count, generator = twitch.channel_videos_generator(
+        channel_name, max_videos, sort, type, game_ids=game_ids
+    )
+
+    if json:
+        videos = list(generator)
+        print_json({"count": len(videos), "totalCount": total_count, "videos": videos})
+        return
+
+    if total_count == 0:
+        click.echo("No videos found")
+        return
+
+    if pager:
+        print_fn = print_video_compact if compact else print_video
+        print_paged("Videos", generator, print_fn, pager, total_count)
+        return
+
+    count = 0
+    for video in generator:
+        if compact:
+            print_video_compact(video)
+        else:
+            click.echo()
+            print_video(video)
+        count += 1
+
+    click.echo()
+    click.echo("-" * 80)
+    click.echo(f"Videos 1-{count} of {total_count}")
+
+    if total_count > count:
+        click.secho(
+            "\nThere are more videos. "
+            + "Increase the --limit, use --all or --pager to see the rest.",
+            dim=True,
+        )
+
+
+def get_game_ids(names: List[str]) -> List[str]:
+    return [get_game_id(name) for name in names]
+
+
+def get_game_id(name: str) -> str:
+    print_log(f"Looking up game '{name}'...")
+    game_id = twitch.get_game_id(name)
+    if not game_id:
+        raise ConsoleError(f"Game '{name}' not found")
+    return game_id

twitchdl/console.py

@@ -1,231 +0,0 @@
-# -*- coding: utf-8 -*-
-
-import sys
-
-from argparse import ArgumentParser, ArgumentTypeError
-from collections import namedtuple
-
-from twitchdl.exceptions import ConsoleError
-from twitchdl.output import print_err
-from twitchdl.twitch import GQLError
-from . import commands, __version__
-
-
-Command = namedtuple("Command", ["name", "description", "arguments"])
-
-CLIENT_WEBSITE = 'https://github.com/ihabunek/twitch-dl'
-
-
-def time(value):
-    """Parse a time string (hh:mm or hh:mm:ss) to number of seconds."""
-    parts = [int(p) for p in value.split(":")]
-
-    if not 2 <= len(parts) <= 3:
-        raise ArgumentTypeError()
-
-    hours = parts[0]
-    minutes = parts[1]
-    seconds = parts[2] if len(parts) > 2 else 0
-
-    if hours < 0 or not (0 <= minutes <= 59) or not (0 <= seconds <= 59):
-        raise ArgumentTypeError()
-
-    return hours * 3600 + minutes * 60 + seconds
-
-
-def limit(value):
-    """Validates the number of videos to fetch."""
-    try:
-        value = int(value)
-    except ValueError:
-        raise ArgumentTypeError("must be an integer")
-
-    if not 1 <= int(value) <= 100:
-        raise ArgumentTypeError("must be between 1 and 100")
-
-    return value
-
-
-COMMANDS = [
-    Command(
-        name="videos",
-        description="List videos from a channel",
-        arguments=[
-            (["channel_name"], {
-                "help": "channel name",
-                "type": str,
-            }),
-            (["-g", "--game"], {
-                "help": "Show videos of given game (can be given multiple times)",
-                "action": "append",
-                "type": str,
-            }),
-            (["-l", "--limit"], {
-                "help": "Number of videos to fetch (default 10, max 100)",
-                "type": limit,
-                "default": 10,
-            }),
-            (["-s", "--sort"], {
-                "help": "Sorting order of videos. (default: time)",
-                "type": str,
-                "choices": ["views", "time"],
-                "default": "time",
-            }),
-            (["-t", "--type"], {
-                "help": "Broadcast type. (default: archive)",
-                "type": str,
-                "choices": ["archive", "highlight", "upload"],
-                "default": "archive",
-            }),
-            (["-p", "--pager"], {
-                "help": "If there are more results than LIMIT, ask to show next page",
-                "action": "store_true",
-                "default": False,
-            }),
-        ],
-    ),
-    Command(
-        name="clips",
-        description="List clips",
-        arguments=[
-            (["channel_name"], {
-                "help": "channel name",
-                "type": str,
-            }),
-            (["-l", "--limit"], {
-                "help": "Number of videos to fetch (default 10, max 100)",
-                "type": limit,
-                "default": 10,
-            }),
-            (["-P", "--period"], {
-                "help": "Period from which to return clips. (default: 'all_time')",
-                "type": str,
-                "choices": ["last_day", "last_week", "last_month", "all_time"],
-                "default": "all_time",
-            }),
-            (["-j", "--json"], {
-                "help": "Show results as JSON",
-                "action": "store_true",
-                "default": False,
-            }),
-            (["-p", "--pager"], {
-                "help": "If there are more results than LIMIT, ask to show next page",
-                "action": "store_true",
-                "default": False,
-            }),
-            (["-d", "--download"], {
-                "help": "Download all videos in given period (in source quality)",
-                "action": "store_true",
-                "default": False,
-            }),
-        ],
-    ),
-    Command(
-        name="download",
-        description="Download a video",
-        arguments=[
-            (["video"], {
-                "help": "video ID, clip slug, or URL",
-                "type": str,
-            }),
-            (["-w", "--max-workers"], {
-                "help": "maximal number of threads for downloading vods "
-                        "concurrently (default 20)",
-                "type": int,
-                "default": 20,
-            }),
-            (["-s", "--start"], {
-                "help": "Download video from this time (hh:mm or hh:mm:ss)",
-                "type": time,
-                "default": None,
-            }),
-            (["-e", "--end"], {
-                "help": "Download video up to this time (hh:mm or hh:mm:ss)",
-                "type": time,
-                "default": None,
-            }),
-            (["-f", "--format"], {
-                "help": "Video format to convert into, passed to ffmpeg as the "
-                        "target file extension (default: mkv)",
-                "type": str,
-                "default": "mkv",
-            }),
-            (["-k", "--keep"], {
-                "help": "Don't delete downloaded VODs and playlists after merging.",
-                "action": "store_true",
-                "default": False,
-            }),
-            (["-q", "--quality"], {
-                "help": "Video quality, e.g. 720p. Set to 'source' to get best quality.",
-                "type": str,
-            }),
-            (["--no-join"], {
-                "help": "Don't run ffmpeg to join the downloaded vods, implies --keep.",
-                "action": "store_true",
-                "default": False,
-            }),
-            (["--overwrite"], {
-                "help": "Overwrite the target file if it already exists without prompting.",
-                "action": "store_true",
-                "default": False,
-            })
-        ],
-    ),
-]
-
-COMMON_ARGUMENTS = [
-    (["--debug"], {
-        "help": "show debug log in console",
-        "action": 'store_true',
-        "default": False,
-    }),
-    (["--no-color"], {
-        "help": "disable ANSI colors in output",
-        "action": 'store_true',
-        "default": False,
-    })
-]
-
-
-def get_parser():
-    description = "A script for downloading videos from Twitch"
-
-    parser = ArgumentParser(prog='twitch-dl', description=description, epilog=CLIENT_WEBSITE)
-    parser.add_argument("--version", help="show version number", action='store_true')
-
-    subparsers = parser.add_subparsers(title="commands")
-
-    for command in COMMANDS:
-        sub = subparsers.add_parser(command.name, help=command.description)
-
-        # Set the function to call to the function of same name in the "commands" package
-        sub.set_defaults(func=commands.__dict__.get(command.name))
-
-        for args, kwargs in command.arguments + COMMON_ARGUMENTS:
-            sub.add_argument(*args, **kwargs)
-
-    return parser
-
-
-def main():
-    parser = get_parser()
-    args = parser.parse_args()
-
-    if args.version:
-        print("twitch-dl v{}".format(__version__))
-        return
-
-    if "func" not in args:
-        parser.print_help()
-        return
-
-    try:
-        args.func(args)
-    except ConsoleError as e:
-        print_err(e)
-        sys.exit(1)
-    except GQLError as e:
-        print_err(e)
-        for err in e.errors:
-            print_err("*", err["message"])
-        sys.exit(1)

twitchdl/download.py

@@ -1,92 +1,37 @@
 import os
-import requests
 
-from collections import OrderedDict
-from concurrent.futures import ThreadPoolExecutor, as_completed
-from datetime import datetime
-from functools import partial
-from requests.exceptions import RequestException
-from twitchdl.output import print_out
-from twitchdl.utils import format_size, format_duration
+import httpx
 
+from twitchdl.exceptions import ConsoleError
 
 CHUNK_SIZE = 1024
 CONNECT_TIMEOUT = 5
 RETRY_COUNT = 5
 
 
-class DownloadFailed(Exception):
-    pass
-
-
-def _download(url, path):
+def _download(url: str, path: str):
     tmp_path = path + ".tmp"
-    response = requests.get(url, stream=True, timeout=CONNECT_TIMEOUT)
     size = 0
-    with open(tmp_path, 'wb') as target:
-        for chunk in response.iter_content(chunk_size=CHUNK_SIZE):
-            target.write(chunk)
-            size += len(chunk)
+    with httpx.stream("GET", url, timeout=CONNECT_TIMEOUT) as response:
+        with open(tmp_path, "wb") as target:
+            for chunk in response.iter_bytes(chunk_size=CHUNK_SIZE):
+                target.write(chunk)
+                size += len(chunk)
 
     os.rename(tmp_path, path)
     return size
 
 
-def download_file(url, path, retries=RETRY_COUNT):
+def download_file(url: str, path: str, retries: int = RETRY_COUNT):
     if os.path.exists(path):
-        return os.path.getsize(path)
+        from_disk = True
+        return (os.path.getsize(path), from_disk)
 
+    from_disk = False
     for _ in range(retries):
         try:
-            return _download(url, path)
-        except RequestException:
+            return (_download(url, path), from_disk)
+        except httpx.RequestError:
             pass
 
-    raise DownloadFailed(":(")
-
-
-def _print_progress(futures):
-    downloaded_count = 0
-    downloaded_size = 0
-    max_msg_size = 0
-    start_time = datetime.now()
-    total_count = len(futures)
-
-    for future in as_completed(futures):
-        size = future.result()
-        downloaded_count += 1
-        downloaded_size += size
-
-        percentage = 100 * downloaded_count // total_count
-        est_total_size = int(total_count * downloaded_size / downloaded_count)
-        duration = (datetime.now() - start_time).seconds
-        speed = downloaded_size // duration if duration else 0
-        remaining = (total_count - downloaded_count) * duration / downloaded_count
-
-        msg = " ".join([
-            "Downloaded VOD {}/{}".format(downloaded_count, total_count),
-            "({}%)".format(percentage),
-            "<cyan>{}</cyan>".format(format_size(downloaded_size)),
-            "of <cyan>~{}</cyan>".format(format_size(est_total_size)),
-            "at <cyan>{}/s</cyan>".format(format_size(speed)) if speed > 0 else "",
-            "remaining <cyan>~{}</cyan>".format(format_duration(remaining)) if speed > 0 else "",
-        ])
-
-        max_msg_size = max(len(msg), max_msg_size)
-        print_out("\r" + msg.ljust(max_msg_size), end="")
-
-
-def download_files(base_url, target_dir, vod_paths, max_workers):
-    """
-    Downloads a list of VODs defined by a common `base_url` and a list of
-    `vod_paths`, returning a dict which maps the paths to the downloaded files.
-    """
-    urls = [base_url + path for path in vod_paths]
-    targets = [os.path.join(target_dir, "{:05d}.ts".format(k)) for k, _ in enumerate(vod_paths)]
-    partials = (partial(download_file, url, path) for url, path in zip(urls, targets))
-
-    with ThreadPoolExecutor(max_workers=max_workers) as executor:
-        futures = [executor.submit(fn) for fn in partials]
-        _print_progress(futures)
-
-    return OrderedDict(zip(vod_paths, targets))
+    raise ConsoleError(f"Failed downloading after {retries} attempts: {url}")

twitchdl/entities.py

@@ -0,0 +1,25 @@
+from dataclasses import dataclass
+from typing import Any, Mapping, Optional
+
+
+@dataclass
+class DownloadOptions:
+    auth_token: Optional[str]
+    chapter: Optional[int]
+    concat: bool
+    dry_run: bool
+    end: Optional[int]
+    format: str
+    keep: bool
+    no_join: bool
+    overwrite: bool
+    output: str
+    quality: Optional[str]
+    rate_limit: Optional[int]
+    start: Optional[int]
+    max_workers: int
+
+
+# Type for annotating decoded JSON
+# TODO: make data classes for common structs
+Data = Mapping[str, Any]

twitchdl/exceptions.py

@@ -1,4 +1,7 @@
+import click
 
-class ConsoleError(Exception):
+
+class ConsoleError(click.ClickException):
     """Raised when an error occurs and script exectuion should halt."""
+
     pass

twitchdl/http.py

@@ -0,0 +1,144 @@
+import asyncio
+import logging
+import os
+import time
+from abc import ABC, abstractmethod
+from typing import List, Optional
+
+import httpx
+
+from twitchdl.progress import Progress
+
+logger = logging.getLogger(__name__)
+
+KB = 1024
+
+CHUNK_SIZE = 256 * KB
+"""How much of a VOD to download in each iteration"""
+
+RETRY_COUNT = 5
+"""Number of times to retry failed downloads before aborting."""
+
+TIMEOUT = 30
+"""
+Number of seconds to wait before aborting when there is no network activity.
+https://www.python-httpx.org/advanced/#timeout-configuration
+"""
+
+
+class TokenBucket(ABC):
+    @abstractmethod
+    def advance(self, size: int):
+        pass
+
+
+class LimitingTokenBucket(TokenBucket):
+    """Limit the download speed by strategically inserting sleeps."""
+
+    def __init__(self, rate: int, capacity: Optional[int] = None):
+        self.rate: int = rate
+        self.capacity: int = capacity or rate * 2
+        self.available: int = 0
+        self.last_refilled: float = time.time()
+
+    def advance(self, size: int):
+        """Called every time a chunk of data is downloaded."""
+        self._refill()
+
+        if self.available < size:
+            deficit = size - self.available
+            time.sleep(deficit / self.rate)
+
+        self.available -= size
+
+    def _refill(self):
+        """Increase available capacity according to elapsed time since last refill."""
+        now = time.time()
+        elapsed = now - self.last_refilled
+        refill_amount = int(elapsed * self.rate)
+        self.available = min(self.available + refill_amount, self.capacity)
+        self.last_refilled = now
+
+
+class EndlessTokenBucket(TokenBucket):
+    """Used when download speed is not limited."""
+
+    def advance(self, size: int):
+        pass
+
+
+async def download(
+    client: httpx.AsyncClient,
+    task_id: int,
+    source: str,
+    target: str,
+    progress: Progress,
+    token_bucket: TokenBucket,
+):
+    # Download to a temp file first, then copy to target when over to avoid
+    # getting saving chunks which may persist if canceled or --keep is used
+    tmp_target = f"{target}.tmp"
+    with open(tmp_target, "wb") as f:
+        async with client.stream("GET", source) as response:
+            size = int(response.headers.get("content-length"))
+            progress.start(task_id, size)
+            async for chunk in response.aiter_bytes(chunk_size=CHUNK_SIZE):
+                f.write(chunk)
+                size = len(chunk)
+                token_bucket.advance(size)
+                progress.advance(task_id, size)
+            progress.end(task_id)
+    os.rename(tmp_target, target)
+
+
+async def download_with_retries(
+    client: httpx.AsyncClient,
+    semaphore: asyncio.Semaphore,
+    task_id: int,
+    source: str,
+    target: str,
+    progress: Progress,
+    token_bucket: TokenBucket,
+):
+    async with semaphore:
+        if os.path.exists(target):
+            size = os.path.getsize(target)
+            progress.already_downloaded(task_id, size)
+            return
+
+        for n in range(RETRY_COUNT):
+            try:
+                return await download(client, task_id, source, target, progress, token_bucket)
+            except httpx.RequestError:
+                logger.exception("Task {task_id} failed. Retrying. Maybe.")
+                progress.abort(task_id)
+                if n + 1 >= RETRY_COUNT:
+                    raise
+
+        raise Exception("Should not happen")
+
+
+async def download_all(
+    sources: List[str],
+    targets: List[str],
+    workers: int,
+    *,
+    rate_limit: Optional[int] = None,
+):
+    progress = Progress(len(sources))
+    token_bucket = LimitingTokenBucket(rate_limit) if rate_limit else EndlessTokenBucket()
+    async with httpx.AsyncClient(timeout=TIMEOUT) as client:
+        semaphore = asyncio.Semaphore(workers)
+        tasks = [
+            download_with_retries(
+                client,
+                semaphore,
+                task_id,
+                source,
+                target,
+                progress,
+                token_bucket,
+            )
+            for task_id, (source, target) in enumerate(zip(sources, targets))
+        ]
+        await asyncio.gather(*tasks)

twitchdl/output.py

@@ -1,100 +1,177 @@
-# -*- coding: utf-8 -*-
-
 import json
-import sys
-import re
+from itertools import islice
+from typing import Any, Callable, Generator, List, Optional, TypeVar
+
+import click
 
 from twitchdl import utils
+from twitchdl.twitch import Clip, Video
+
+T = TypeVar("T")
 
 
-START_CODES = {
-    'b': '\033[1m',
-    'dim': '\033[2m',
-    'i': '\033[3m',
-    'u': '\033[4m',
-    'red': '\033[91m',
-    'green': '\033[92m',
-    'yellow': '\033[93m',
-    'blue': '\033[94m',
-    'magenta': '\033[95m',
-    'cyan': '\033[96m',
-}
+def truncate(string: str, length: int) -> str:
+    if len(string) > length:
+        return string[: length - 1] + "…"
 
-END_CODE = '\033[0m'
-
-START_PATTERN = "<(" + "|".join(START_CODES.keys()) + ")>"
-END_PATTERN = "</(" + "|".join(START_CODES.keys()) + ")>"
-
-USE_ANSI_COLOR = "--no-color" not in sys.argv
+    return string
 
 
-def start_code(match):
-    name = match.group(1)
-    return START_CODES[name]
+def print_json(data: Any):
+    click.echo(json.dumps(data))
 
 
-def colorize(text):
-    text = re.sub(START_PATTERN, start_code, text)
-    text = re.sub(END_PATTERN, END_CODE, text)
-
-    return text
+def print_log(message: Any):
+    click.secho(message, err=True, dim=True)
 
 
-def strip_tags(text):
-    text = re.sub(START_PATTERN, '', text)
-    text = re.sub(END_PATTERN, '', text)
+def print_table(headers: List[str], data: List[List[str]]):
+    widths = [[len(cell) for cell in row] for row in data + [headers]]
+    widths = [max(width) for width in zip(*widths)]
+    underlines = ["-" * width for width in widths]
 
-    return text
+    def print_row(row: List[str]):
+        for idx, cell in enumerate(row):
+            width = widths[idx]
+            click.echo(cell.ljust(width), nl=False)
+            click.echo("  ", nl=False)
+        click.echo()
+
+    print_row(headers)
+    print_row(underlines)
+
+    for row in data:
+        print_row(row)
 
 
-def print_out(*args, **kwargs):
-    args = [colorize(a) if USE_ANSI_COLOR else strip_tags(a) for a in args]
-    print(*args, **kwargs)
+def print_paged(
+    label: str,
+    generator: Generator[T, Any, Any],
+    print_fn: Callable[[T], None],
+    page_size: int,
+    total_count: Optional[int] = None,
+):
+    iterator = iter(generator)
+    page = list(islice(iterator, page_size))
+
+    first = 1
+    last = first + len(page) - 1
+
+    while True:
+        click.echo("-" * 80)
+
+        click.echo()
+        for item in page:
+            print_fn(item)
+
+        last = first + len(page) - 1
+
+        click.echo("-" * 80)
+        click.echo(f"{label} {first}-{last} of {total_count or '???'}")
+
+        first = first + len(page)
+        last = first + 1
+
+        page = list(islice(iterator, page_size))
+        if not page or not prompt_continue():
+            break
 
 
-def print_json(data):
-    print(json.dumps(data))
-
-
-def print_err(*args, **kwargs):
-    args = ["<red>{}</red>".format(a) for a in args]
-    args = [colorize(a) if USE_ANSI_COLOR else strip_tags(a) for a in args]
-    print(*args, file=sys.stderr, **kwargs)
-
-
-def print_video(video):
+def print_video(video: Video):
     published_at = video["publishedAt"].replace("T", " @ ").replace("Z", "")
     length = utils.format_duration(video["lengthSeconds"])
-    channel = video["creator"]["channel"]["displayName"]
-    playing = (
-        "playing <blue>{}</blue>".format(video["game"]["name"])
-        if video["game"] else ""
-    )
+
+    channel = blue(video["creator"]["displayName"]) if video["creator"] else ""
+    playing = f"playing {blue(video['game']['name'])}" if video["game"] else ""
 
     # Can't find URL in video object, strange
-    url = "https://www.twitch.tv/videos/{}".format(video["id"])
+    url = f"https://www.twitch.tv/videos/{video['id']}"
 
-    print_out("\n<b>{}</b>".format(video["id"]))
-    print_out("<green>{}</green>".format(video["title"]))
-    print_out("<blue>{}</blue> {}".format(channel, playing))
-    print_out("Published <blue>{}</blue>  Length: <blue>{}</blue> ".format(published_at, length))
-    print_out("<i>{}</i>".format(url))
+    click.secho(f"Video {video['id']}", bold=True)
+    click.secho(video["title"], fg="green")
+
+    if channel or playing:
+        click.echo(" ".join([channel, playing]))
+
+    if video["description"]:
+        click.echo(f"Description: {video['description']}")
+
+    click.echo(f"Published {blue(published_at)}  Length: {blue(length)} ")
+    click.secho(url, italic=True)
+    click.echo()
 
 
-def print_clip(clip):
+def print_video_compact(video: Video):
+    id = video["id"]
+    date = video["publishedAt"][:10]
+    game = video["game"]["name"] if video["game"] else ""
+    title = truncate(video["title"], 80).ljust(80)
+    click.echo(f"{bold(id)} {date} {green(title)} {blue(game)}")
+
+
+def print_clip(clip: Clip):
     published_at = clip["createdAt"].replace("T", " @ ").replace("Z", "")
     length = utils.format_duration(clip["durationSeconds"])
-    channel = clip["broadcaster"]["channel"]["displayName"]
-    playing = (
-        "playing <blue>{}</blue>".format(clip["game"]["name"])
-        if clip["game"] else ""
-    )
+    channel = clip["broadcaster"]["displayName"]
+    playing = f"playing {blue(clip['game']['name'])}" if clip["game"] else ""
 
-    print_out("\n<b>{}</b>".format(clip["slug"]))
-    print_out("<green>{}</green>".format(clip["title"]))
-    print_out("<blue>{}</blue> {}".format(channel, playing))
-    print_out(
-        "Published <blue>{}</blue>"
-        "  Length: <blue>{}</blue>"
-        "  Views: <blue>{}</blue>".format(published_at, length, clip["viewCount"]))
-    print_out("<i>{}</i>".format(clip["url"]))
+    click.echo(f"Clip {bold(clip['slug'])}")
+    click.secho(clip["title"], fg="green")
+    click.echo(f"{blue(channel)} {playing}")
+    click.echo(
+        f"Published {blue(published_at)}"
+        + f"  Length: {blue(length)}"
+        + f"  Views: {blue(clip['viewCount'])}"
+    )
+    click.secho(clip["url"], italic=True)
+    click.echo()
+
+
+def print_clip_compact(clip: Clip):
+    slug = clip["slug"]
+    date = clip["createdAt"][:10]
+    title = truncate(clip["title"], 50).ljust(50)
+    game = clip["game"]["name"] if clip["game"] else ""
+    game = truncate(game, 30).ljust(30)
+
+    click.echo(f"{date} {green(title)} {blue(game)} {bold(slug)}")
+
+
+def prompt_continue():
+    enter = click.style("Enter", bold=True, fg="green")
+    ctrl_c = click.style("Ctrl+C", bold=True, fg="yellow")
+    click.echo(f"Press {enter} to continue, {ctrl_c} to break.")
+
+    try:
+        input()
+    except KeyboardInterrupt:
+        return False
+
+    return True
+
+
+# Shorthand functions for coloring output
+
+
+def blue(text: Any) -> str:
+    return click.style(text, fg="blue")
+
+
+def cyan(text: Any) -> str:
+    return click.style(text, fg="cyan")
+
+
+def green(text: Any) -> str:
+    return click.style(text, fg="green")
+
+
+def yellow(text: Any) -> str:
+    return click.style(text, fg="yellow")
+
+
+def bold(text: Any) -> str:
+    return click.style(text, bold=True)
+
+
+def dim(text: Any) -> str:
+    return click.style(text, dim=True)

twitchdl/playlists.py

@@ -0,0 +1,131 @@
+"""
+Parse and manipulate m3u8 playlists.
+"""
+
+from dataclasses import dataclass
+from typing import Generator, List, Optional, OrderedDict
+
+import click
+import m3u8
+
+from twitchdl import utils
+from twitchdl.output import bold, dim
+
+
+@dataclass
+class Playlist:
+    name: str
+    resolution: Optional[str]
+    url: str
+
+
+@dataclass
+class Vod:
+    index: int
+    """Ordinal number of the VOD in the playlist"""
+    path: str
+    """Path part of the VOD URL"""
+    duration: int
+    """Segment duration in seconds"""
+
+
+def parse_playlists(playlists_m3u8: str) -> List[Playlist]:
+    def _parse(source: str) -> Generator[Playlist, None, None]:
+        document = load_m3u8(source)
+
+        for p in document.playlists:
+            if p.stream_info.resolution:
+                name = p.media[0].name
+                resolution = "x".join(str(r) for r in p.stream_info.resolution)
+            else:
+                name = p.media[0].group_id
+                resolution = None
+
+            yield Playlist(name, resolution, p.uri)
+
+    # Move audio to bottom, it has no resolution
+    return sorted(_parse(playlists_m3u8), key=lambda p: p.resolution is None)
+
+
+def load_m3u8(playlist_m3u8: str) -> m3u8.M3U8:
+    return m3u8.loads(playlist_m3u8)
+
+
+def enumerate_vods(
+    document: m3u8.M3U8,
+    start: Optional[int] = None,
+    end: Optional[int] = None,
+) -> List[Vod]:
+    """Extract VODs for download from document."""
+    vods = []
+    vod_start = 0
+
+    for index, segment in enumerate(document.segments):
+        vod_end = vod_start + segment.duration
+
+        # `vod_end > start` is used here becuase it's better to download a bit
+        # more than a bit less, similar for the end condition
+        start_condition = not start or vod_end > start
+        end_condition = not end or vod_start < end
+
+        if start_condition and end_condition:
+            vods.append(Vod(index, segment.uri, segment.duration))
+
+        vod_start = vod_end
+
+    return vods
+
+
+def make_join_playlist(
+    playlist: m3u8.M3U8,
+    vods: List[Vod],
+    targets: List[str],
+) -> m3u8.Playlist:
+    """
+    Make a modified playlist which references downloaded VODs
+    Keep only the downloaded segments and skip the rest
+    """
+    org_segments = playlist.segments.copy()
+
+    path_map = OrderedDict(zip([v.path for v in vods], targets))
+    playlist.segments.clear()
+    for segment in org_segments:
+        if segment.uri in path_map:
+            segment.uri = path_map[segment.uri]
+            playlist.segments.append(segment)
+
+    return playlist
+
+
+def select_playlist(playlists: List[Playlist], quality: Optional[str]) -> Playlist:
+    return (
+        select_playlist_by_name(playlists, quality)
+        if quality is not None
+        else select_playlist_interactive(playlists)
+    )
+
+
+def select_playlist_by_name(playlists: List[Playlist], quality: str) -> Playlist:
+    if quality == "source":
+        return playlists[0]
+
+    for playlist in playlists:
+        if playlist.name == quality:
+            return playlist
+
+    available = ", ".join([p.name for p in playlists])
+    msg = f"Quality '{quality}' not found. Available qualities are: {available}"
+    raise click.ClickException(msg)
+
+
+def select_playlist_interactive(playlists: List[Playlist]) -> Playlist:
+    click.echo("\nAvailable qualities:")
+    for n, playlist in enumerate(playlists):
+        if playlist.resolution:
+            click.echo(f"{n + 1}) {bold(playlist.name)} {dim(f'({playlist.resolution})')}")
+        else:
+            click.echo(f"{n + 1}) {bold(playlist.name)}")
+
+    no = utils.read_int("Choose quality", min=1, max=len(playlists) + 1, default=1)
+    playlist = playlists[no - 1]
+    return playlist

twitchdl/progress.py

@@ -0,0 +1,155 @@
+import logging
+import time
+from collections import deque
+from dataclasses import dataclass, field
+from statistics import mean
+from typing import Deque, Dict, NamedTuple, Optional
+
+import click
+
+from twitchdl.output import blue
+from twitchdl.utils import format_size, format_time
+
+logger = logging.getLogger(__name__)
+
+
+TaskId = int
+
+
+@dataclass
+class Task:
+    id: TaskId
+    size: int
+    downloaded: int = 0
+
+    def advance(self, size: int):
+        self.downloaded += size
+
+
+class Sample(NamedTuple):
+    downloaded: int
+    timestamp: float
+
+
+@dataclass
+class Progress:
+    vod_count: int
+    downloaded: int = 0
+    estimated_total: Optional[int] = None
+    last_printed: float = field(default_factory=time.time)
+    progress_bytes: int = 0
+    progress_perc: int = 0
+    remaining_time: Optional[int] = None
+    speed: Optional[float] = None
+    start_time: float = field(default_factory=time.time)
+    tasks: Dict[TaskId, Task] = field(default_factory=dict)
+    vod_downloaded_count: int = 0
+    samples: Deque[Sample] = field(default_factory=lambda: deque(maxlen=100))
+
+    def start(self, task_id: int, size: int):
+        if task_id in self.tasks:
+            raise ValueError(f"Task {task_id}: cannot start, already started")
+
+        self.tasks[task_id] = Task(task_id, size)
+        self._calculate_total()
+        self._calculate_progress()
+        self.print()
+
+    def advance(self, task_id: int, size: int):
+        if task_id not in self.tasks:
+            raise ValueError(f"Task {task_id}: cannot advance, not started")
+
+        self.downloaded += size
+        self.progress_bytes += size
+        self.tasks[task_id].advance(size)
+        self.samples.append(Sample(self.downloaded, time.time()))
+        self._calculate_progress()
+        self.print()
+
+    def already_downloaded(self, task_id: int, size: int):
+        if task_id in self.tasks:
+            raise ValueError(f"Task {task_id}: cannot mark as downloaded, already started")
+
+        self.tasks[task_id] = Task(task_id, size)
+        self.progress_bytes += size
+        self.vod_downloaded_count += 1
+        self._calculate_total()
+        self._calculate_progress()
+        self.print()
+
+    def abort(self, task_id: int):
+        if task_id not in self.tasks:
+            raise ValueError(f"Task {task_id}: cannot abort, not started")
+
+        del self.tasks[task_id]
+        self.progress_bytes = sum(t.downloaded for t in self.tasks.values())
+
+        self._calculate_total()
+        self._calculate_progress()
+        self.print()
+
+    def end(self, task_id: int):
+        if task_id not in self.tasks:
+            raise ValueError(f"Task {task_id}: cannot end, not started")
+
+        task = self.tasks[task_id]
+        if task.size != task.downloaded:
+            logger.warn(
+                f"Taks {task_id} ended with {task.downloaded}b downloaded, expected {task.size}b."
+            )
+
+        self.vod_downloaded_count += 1
+        self.print()
+
+    def _calculate_total(self):
+        self.estimated_total = (
+            int(mean(t.size for t in self.tasks.values()) * self.vod_count) if self.tasks else None
+        )
+
+    def _calculate_progress(self):
+        self.speed = self._calculate_speed()
+        self.progress_perc = (
+            int(100 * self.progress_bytes / self.estimated_total) if self.estimated_total else 0
+        )
+        self.remaining_time = (
+            int((self.estimated_total - self.progress_bytes) / self.speed)
+            if self.estimated_total and self.speed
+            else None
+        )
+
+    def _calculate_speed(self):
+        if len(self.samples) < 2:
+            return None
+
+        first_sample = self.samples[0]
+        last_sample = self.samples[-1]
+
+        size = last_sample.downloaded - first_sample.downloaded
+        duration = last_sample.timestamp - first_sample.timestamp
+
+        return size / duration if duration > 0 else None
+
+    def print(self):
+        now = time.time()
+
+        # Don't print more often than 10 times per second
+        if now - self.last_printed < 0.1:
+            return
+
+        click.echo(f"\rDownloaded {self.vod_downloaded_count}/{self.vod_count} VODs", nl=False)
+        click.secho(f" {self.progress_perc}%", fg="blue", nl=False)
+
+        if self.estimated_total is not None:
+            total = f"~{format_size(self.estimated_total)}"
+            click.echo(f" of {blue(total)}", nl=False)
+
+        if self.speed is not None:
+            speed = f"{format_size(self.speed)}/s"
+            click.echo(f" at {blue(speed)}", nl=False)
+
+        if self.remaining_time is not None:
+            click.echo(f" ETA {blue(format_time(self.remaining_time))}", nl=False)
+
+        click.echo("    ", nl=False)
+
+        self.last_printed = now

twitchdl/twitch.py

@@ -2,37 +2,109 @@
 Twitch API access.
 """
 
-import requests
+import json
+import logging
+import time
+from typing import Any, Dict, Generator, List, Literal, Mapping, Optional, Tuple, TypedDict, Union
+
+import click
+import httpx
 
 from twitchdl import CLIENT_ID
+from twitchdl.entities import Data
 from twitchdl.exceptions import ConsoleError
 
-
-class GQLError(Exception):
-    def __init__(self, errors):
-        super().__init__("GraphQL query failed")
-        self.errors = errors
+ClipsPeriod = Literal["last_day", "last_week", "last_month", "all_time"]
+VideosSort = Literal["views", "time"]
+VideosType = Literal["archive", "highlight", "upload"]
 
 
-def authenticated_get(url, params={}, headers={}):
-    headers['Client-ID'] = CLIENT_ID
-
-    response = requests.get(url, params, headers=headers)
-    if 400 <= response.status_code < 500:
-        data = response.json()
-        # TODO: this does not look nice in the console since data["message"]
-        # can contain a JSON encoded object.
-        raise ConsoleError(data["message"])
-
-    response.raise_for_status()
-
-    return response
+class AccessToken(TypedDict):
+    signature: str
+    value: str
 
 
-def authenticated_post(url, data=None, json=None, headers={}):
-    headers['Client-ID'] = CLIENT_ID
+class User(TypedDict):
+    login: str
+    displayName: str
 
-    response = requests.post(url, data=data, json=json, headers=headers)
+
+class Game(TypedDict):
+    id: str
+    name: str
+
+
+class VideoQuality(TypedDict):
+    frameRate: str
+    quality: str
+    sourceURL: str
+
+
+class ClipAccessToken(TypedDict):
+    id: str
+    playbackAccessToken: AccessToken
+    videoQualities: List[VideoQuality]
+
+
+class Clip(TypedDict):
+    id: str
+    slug: str
+    title: str
+    createdAt: str
+    viewCount: int
+    durationSeconds: int
+    url: str
+    videoQualities: List[VideoQuality]
+    game: Game
+    broadcaster: User
+
+
+class Video(TypedDict):
+    id: str
+    title: str
+    description: str
+    publishedAt: str
+    broadcastType: str
+    lengthSeconds: int
+    game: Game
+    creator: User
+
+
+class Chapter(TypedDict):
+    id: str
+    durationMilliseconds: int
+    positionMilliseconds: int
+    type: str
+    description: str
+    subDescription: str
+    thumbnailURL: str
+    game: Game
+
+
+class GQLError(click.ClickException):
+    def __init__(self, errors: List[str]):
+        message = "GraphQL query failed."
+        for error in errors:
+            message += f"\n* {error}"
+        super().__init__(message)
+
+
+Content = Union[str, bytes]
+Headers = Dict[str, str]
+
+
+def authenticated_post(
+    url: str,
+    *,
+    json: Any = None,
+    content: Optional[Content] = None,
+    auth_token: Optional[str] = None,
+):
+    headers = {"Client-ID": CLIENT_ID}
+    if auth_token is not None:
+        headers["authorization"] = f"OAuth {auth_token}"
+
+    response = request("POST", url, content=content, json=json, headers=headers)
     if response.status_code == 400:
         data = response.json()
         raise ConsoleError(data["message"])
@@ -42,61 +114,155 @@ def authenticated_post(url, data=None, json=None, headers={}):
     return response
 
 
-def kraken_get(url, params={}, headers={}):
-    """
-    Add accept header required by kraken API v5.
-    see: https://discuss.dev.twitch.tv/t/change-in-access-to-deprecated-kraken-twitch-apis/22241
-    """
-    headers["Accept"] = "application/vnd.twitchtv.v5+json"
-    return authenticated_get(url, params, headers)
+def request(
+    method: str,
+    url: str,
+    json: Any = None,
+    content: Optional[Content] = None,
+    headers: Optional[Mapping[str, str]] = None,
+):
+    with httpx.Client() as client:
+        request = client.build_request(method, url, json=json, content=content, headers=headers)
+        log_request(request)
+        start = time.time()
+        response = client.send(request)
+        duration = time.time() - start
+        log_response(response, duration)
+        return response
 
 
-def gql_query(query):
+logger = logging.getLogger(__name__)
+
+
+def log_request(request: httpx.Request):
+    logger.debug(f"--> {request.method} {request.url}")
+    if request.content:
+        logger.debug(f"--> {request.content}")
+
+
+def log_response(response: httpx.Response, duration: float):
+    request = response.request
+    duration_ms = int(1000 * duration)
+    logger.debug(f"<-- {request.method} {request.url} HTTP {response.status_code} {duration_ms}ms")
+    if response.content:
+        logger.debug(f"<-- {response.content}")
+
+
+def gql_post(query: str):
     url = "https://gql.twitch.tv/gql"
-    response = authenticated_post(url, json={"query": query}).json()
-
-    if "errors" in response:
-        raise GQLError(response["errors"])
-
-    return response
+    response = authenticated_post(url, content=query)
+    gql_raise_on_error(response)
+    return response.json()
 
 
-def get_video(video_id):
-    """
-    https://dev.twitch.tv/docs/v5/reference/videos#get-video
-    """
-    url = "https://api.twitch.tv/kraken/videos/{}".format(video_id)
-
-    return kraken_get(url).json()
+def gql_query(query: str, auth_token: Optional[str] = None):
+    url = "https://gql.twitch.tv/gql"
+    response = authenticated_post(url, json={"query": query}, auth_token=auth_token)
+    gql_raise_on_error(response)
+    return response.json()
 
 
-def get_clip(slug):
-    query = """
+def gql_raise_on_error(response: httpx.Response):
+    data = response.json()
+    if "errors" in data:
+        errors = [e["message"] for e in data["errors"]]
+        raise GQLError(errors)
+
+
+VIDEO_FIELDS = """
+    id
+    title
+    description
+    publishedAt
+    broadcastType
+    lengthSeconds
+    game {
+        id
+        name
+    }
+    creator {
+        login
+        displayName
+    }
+"""
+
+
+CLIP_FIELDS = """
+    id
+    slug
+    title
+    createdAt
+    viewCount
+    durationSeconds
+    url
+    videoQualities {
+        frameRate
+        quality
+        sourceURL
+    }
+    game {
+        id
+        name
+    }
+    broadcaster {
+        displayName
+        login
+    }
+"""
+
+
+def get_video(video_id: str) -> Optional[Video]:
+    query = f"""
     {{
-        clip(slug: "{}") {{
-            title
-            durationSeconds
-            game {{
-                name
-            }}
-            broadcaster {{
-                login
-                displayName
-            }}
-            videoQualities {{
-                frameRate
-                quality
-                sourceURL
+        video(id: "{video_id}") {{
+            {VIDEO_FIELDS}
+        }}
+    }}
+    """
+
+    response = gql_query(query)
+    return response["data"]["video"]
+
+
+def get_clip(slug: str) -> Optional[Clip]:
+    query = f"""
+    {{
+        clip(slug: "{slug}") {{
+            {CLIP_FIELDS}
+        }}
+    }}
+    """
+
+    response = gql_query(query)
+    return response["data"]["clip"]
+
+
+def get_clip_access_token(slug: str) -> ClipAccessToken:
+    query = f"""
+    {{
+        "operationName": "VideoAccessToken_Clip",
+        "variables": {{
+            "slug": "{slug}"
+        }},
+        "extensions": {{
+            "persistedQuery": {{
+                "version": 1,
+                "sha256Hash": "36b89d2507fce29e5ca551df756d27c1cfe079e2609642b4390aa4c35796eb11"
             }}
         }}
     }}
     """
 
-    response = gql_query(query.format(slug))
+    response = gql_post(query.strip())
     return response["data"]["clip"]
 
 
-def get_channel_clips(channel_id, period, limit, after=None):
+def get_channel_clips(
+    channel_id: str,
+    period: ClipsPeriod,
+    limit: int,
+    after: Optional[str] = None,
+):
     """
     List channel clips.
 
@@ -106,10 +272,10 @@ def get_channel_clips(channel_id, period, limit, after=None):
     * sorting by VIEWS_DESC and TRENDING returns the same results
     * there is no totalCount
     """
-    query = """
+    query = f"""
     {{
       user(login: "{channel_id}") {{
-        clips(first: {limit}, after: "{after}", criteria: {{ period: {period}, sort: VIEWS_DESC }}) {{
+        clips(first: {limit}, after: "{after or ''}", criteria: {{ period: {period.upper()}, sort: VIEWS_DESC }}) {{
           pageInfo {{
             hasNextPage
             hasPreviousPage
@@ -117,28 +283,7 @@ def get_channel_clips(channel_id, period, limit, after=None):
           edges {{
             cursor
             node {{
-              id
-              slug
-              title
-              createdAt
-              viewCount
-              durationSeconds
-              url
-              videoQualities {{
-                frameRate
-                quality
-                sourceURL
-              }}
-              game {{
-                id
-                name
-              }}
-              broadcaster {{
-                channel {{
-                  name
-                  displayName
-                }}
-              }}
+              {CLIP_FIELDS}
             }}
           }}
         }}
@@ -146,22 +291,44 @@ def get_channel_clips(channel_id, period, limit, after=None):
     }}
     """
 
-    query = query.format(**{
-        "channel_id": channel_id,
-        "after": after if after else "",
-        "limit": limit,
-        "period": period.upper(),
-    })
-
     response = gql_query(query)
+    user = response["data"]["user"]
+    if not user:
+        raise ConsoleError(f"Channel {channel_id} not found")
+
     return response["data"]["user"]["clips"]
 
 
-def channel_clips_generator(channel_id, period, limit):
+def channel_clips_generator(
+    channel_id: str,
+    period: ClipsPeriod,
+    limit: int,
+) -> Generator[Clip, None, None]:
+    def _generator(clips: Data, limit: int) -> Generator[Clip, None, None]:
+        for clip in clips["edges"]:
+            if limit < 1:
+                return
+            yield clip["node"]
+            limit -= 1
+
+        has_next = clips["pageInfo"]["hasNextPage"]
+        if limit < 1 or not has_next:
+            return
+
+        req_limit = min(limit, 100)
+        cursor = clips["edges"][-1]["cursor"]
+        clips = get_channel_clips(channel_id, period, req_limit, cursor)
+        yield from _generator(clips, limit)
+
+    req_limit = min(limit, 100)
+    clips = get_channel_clips(channel_id, period, req_limit)
+    return _generator(clips, limit)
+
+
+def channel_clips_generator_old(channel_id: str, period: ClipsPeriod, limit: int):
     cursor = ""
     while True:
-        clips = get_channel_clips(
-            channel_id, period, limit, after=cursor)
+        clips = get_channel_clips(channel_id, period, limit, after=cursor)
 
         if not clips["edges"]:
             break
@@ -175,17 +342,27 @@ def channel_clips_generator(channel_id, period, limit):
             break
 
 
-def get_channel_videos(channel_id, limit, sort, type="archive", game_ids=[], after=None):
-    query = """
+def get_channel_videos(
+    channel_id: str,
+    limit: int,
+    sort: str,
+    type: str = "archive",
+    game_ids: Optional[List[str]] = None,
+    after: Optional[str] = None,
+):
+    game_ids = game_ids or []
+    game_ids_str = f"[{','.join(game_ids)}]"
+
+    query = f"""
     {{
         user(login: "{channel_id}") {{
             videos(
                 first: {limit},
-                type: {type},
-                sort: {sort},
-                after: "{after}",
+                type: {type.upper()},
+                sort: {sort.upper()},
+                after: "{after or ''}",
                 options: {{
-                    gameIDs: {game_ids}
+                    gameIDs: {game_ids_str}
                 }}
             ) {{
                 totalCount
@@ -195,19 +372,7 @@ def get_channel_videos(channel_id, limit, sort, type="archive", game_ids=[], aft
                 edges {{
                     cursor
                     node {{
-                        id
-                        title
-                        publishedAt
-                        broadcastType
-                        lengthSeconds
-                        game {{
-                            name
-                        }}
-                        creator {{
-                            channel {{
-                                displayName
-                            }}
-                        }}
+                        {VIDEO_FIELDS}
                     }}
                 }}
             }}
@@ -215,69 +380,137 @@ def get_channel_videos(channel_id, limit, sort, type="archive", game_ids=[], aft
     }}
     """
 
-    query = query.format(**{
-        "channel_id": channel_id,
-        "game_ids": game_ids,
-        "after": after if after else "",
-        "limit": limit,
-        "sort": sort.upper(),
-        "type": type.upper(),
-    })
-
     response = gql_query(query)
+
+    if not response["data"]["user"]:
+        raise ConsoleError(f"Channel {channel_id} not found")
+
     return response["data"]["user"]["videos"]
 
 
-def channel_videos_generator(channel_id, limit, sort, type, game_ids=None):
-    cursor = ""
-    while True:
-        videos = get_channel_videos(
-            channel_id, limit, sort, type, game_ids=game_ids, after=cursor)
+def channel_videos_generator(
+    channel_id: str,
+    max_videos: int,
+    sort: VideosSort,
+    type: VideosType,
+    game_ids: Optional[List[str]] = None,
+) -> Tuple[int, Generator[Video, None, None]]:
+    game_ids = game_ids or []
 
-        if not videos["edges"]:
-            break
+    def _generator(videos: Data, max_videos: int) -> Generator[Video, None, None]:
+        for video in videos["edges"]:
+            if max_videos < 1:
+                return
+            yield video["node"]
+            max_videos -= 1
 
         has_next = videos["pageInfo"]["hasNextPage"]
-        cursor = videos["edges"][-1]["cursor"] if has_next else None
+        if max_videos < 1 or not has_next:
+            return
 
-        yield videos, has_next
+        limit = min(max_videos, 100)
+        cursor = videos["edges"][-1]["cursor"]
+        videos = get_channel_videos(channel_id, limit, sort, type, game_ids, cursor)
+        yield from _generator(videos, max_videos)
 
-        if not cursor:
-            break
+    limit = min(max_videos, 100)
+    videos = get_channel_videos(channel_id, limit, sort, type, game_ids)
+    return videos["totalCount"], _generator(videos, max_videos)
 
 
-def get_access_token(video_id):
-    url = "https://api.twitch.tv/api/vods/{}/access_token".format(video_id)
+def get_access_token(video_id: str, auth_token: Optional[str] = None) -> AccessToken:
+    query = f"""
+    {{
+        videoPlaybackAccessToken(
+            id: {video_id},
+            params: {{
+                platform: "web",
+                playerBackend: "mediaplayer",
+                playerType: "site"
+            }}
+        ) {{
+            signature
+            value
+        }}
+    }}
+    """
 
-    return authenticated_get(url).json()
+    try:
+        response = gql_query(query, auth_token=auth_token)
+        return response["data"]["videoPlaybackAccessToken"]
+    except httpx.HTTPStatusError as error:
+        # Provide a more useful error message when server returns HTTP 401
+        # Unauthorized while using a user-provided auth token.
+        if error.response.status_code == 401:
+            if auth_token:
+                raise ConsoleError("Unauthorized. The provided auth token is not valid.")
+            else:
+                raise ConsoleError(
+                    "Unauthorized. This video may be subscriber-only. See docs:\n"
+                    "https://twitch-dl.bezdomni.net/commands/download.html#downloading-subscriber-only-vods"
+                )
+
+        raise
 
 
-def get_playlists(video_id, access_token):
+def get_playlists(video_id: str, access_token: AccessToken) -> str:
     """
     For a given video return a playlist which contains possible video qualities.
     """
-    url = "http://usher.twitch.tv/vod/{}".format(video_id)
+    url = f"https://usher.ttvnw.net/vod/{video_id}"
 
-    response = requests.get(url, params={
-        "nauth": access_token['token'],
-        "nauthsig": access_token['sig'],
-        "allow_source": "true",
-        "player": "twitchweb",
-    })
+    response = httpx.get(
+        url,
+        params={
+            "nauth": access_token["value"],
+            "nauthsig": access_token["signature"],
+            "allow_audio_only": "true",
+            "allow_source": "true",
+            "player": "twitchweb",
+        },
+    )
     response.raise_for_status()
-    return response.content.decode('utf-8')
+    return response.content.decode("utf-8")
 
 
-def get_game_id(name):
-    query = """
+def get_game_id(name: str):
+    query = f"""
     {{
-        game(name: "{}") {{
+        game(name: "{name.strip()}") {{
             id
         }}
     }}
     """
 
-    response = gql_query(query.format(name.strip()))
+    response = gql_query(query)
     game = response["data"]["game"]
     if game:
         return game["id"]
+
+
+def get_video_chapters(video_id: str) -> List[Chapter]:
+    query = {
+        "operationName": "VideoPlayer_ChapterSelectButtonVideo",
+        "variables": {
+            "includePrivate": False,
+            "videoID": video_id,
+        },
+        "extensions": {
+            "persistedQuery": {
+                "version": 1,
+                "sha256Hash": "8d2793384aac3773beab5e59bd5d6f585aedb923d292800119e03d40cd0f9b41",
+            }
+        },
+    }
+
+    response = gql_post(json.dumps(query))
+    return list(_chapter_nodes(response["data"]["video"]["moments"]))
+
+
+def _chapter_nodes(moments: Data) -> Generator[Chapter, None, None]:
+    for edge in moments["edges"]:
+        node = edge["node"]
+        node["game"] = node["details"]["game"]
+        del node["details"]
+        del node["moments"]
+        yield node

twitchdl/utils.py

@@ -1,15 +1,18 @@
 import re
 import unicodedata
+from typing import Optional, Union
+
+import click
 
 
-def _format_size(value, digits, unit):
+def _format_size(value: float, digits: int, unit: str):
     if digits > 0:
-        return "{{:.{}f}}{}".format(digits, unit).format(value)
+        return f"{{:.{digits}f}}{unit}".format(value)
     else:
-        return "{{:d}}{}".format(unit).format(value)
+        return f"{int(value)}{unit}"
 
 
-def format_size(bytes_, digits=1):
+def format_size(bytes_: Union[int, float], digits: int = 1):
     if bytes_ < 1024:
         return _format_size(bytes_, digits, "B")
 
@@ -24,7 +27,7 @@ def format_size(bytes_, digits=1):
     return _format_size(mega / 1024, digits, "GB")
 
 
-def format_duration(total_seconds):
+def format_duration(total_seconds: Union[int, float]) -> str:
     total_seconds = int(total_seconds)
     hours = total_seconds // 3600
     remainder = total_seconds % 3600
@@ -32,21 +35,32 @@ def format_duration(total_seconds):
     seconds = total_seconds % 60
 
     if hours:
-        return "{} h {} min".format(hours, minutes)
+        return f"{hours} h {minutes} min"
 
     if minutes:
-        return "{} min {} sec".format(minutes, seconds)
+        return f"{minutes} min {seconds} sec"
 
-    return "{} sec".format(seconds)
+    return f"{seconds} sec"
 
 
-def read_int(msg, min, max, default):
-    msg = msg + " [default {}]: ".format(default)
+def format_time(total_seconds: Union[int, float], force_hours: bool = False) -> str:
+    total_seconds = int(total_seconds)
+    hours = total_seconds // 3600
+    remainder = total_seconds % 3600
+    minutes = remainder // 60
+    seconds = total_seconds % 60
 
+    if hours or force_hours:
+        return f"{hours:02}:{minutes:02}:{seconds:02}"
+
+    return f"{minutes:02}:{seconds:02}"
+
+
+def read_int(msg: str, min: int, max: int, default: Optional[int] = None) -> int:
     while True:
         try:
-            val = input(msg)
-            if not val:
+            val = click.prompt(msg, default=default, type=int)
+            if default and not val:
                 return default
             if min <= int(val) <= max:
                 return int(val)
@@ -54,10 +68,43 @@ def read_int(msg, min, max, default):
             pass
 
 
-def slugify(value):
-    re_pattern = re.compile(r'[^\w\s-]', flags=re.U)
-    re_spaces = re.compile(r'[-\s]+', flags=re.U)
-    value = str(value)
-    value = unicodedata.normalize('NFKC', value)
-    value = re_pattern.sub('', value).strip().lower()
-    return re_spaces.sub('_', value)
+def slugify(value: str) -> str:
+    value = unicodedata.normalize("NFKC", str(value))
+    value = re.sub(r"[^\w\s_-]", "", value)
+    value = re.sub(r"[\s_-]+", "_", value)
+    return value.strip("_").lower()
+
+
+def titlify(value: str) -> str:
+    value = unicodedata.normalize("NFKC", str(value))
+    value = re.sub(r"[^\w\s\[\]().-]", "", value)
+    value = re.sub(r"\s+", " ", value)
+    return value.strip()
+
+
+VIDEO_PATTERNS = [
+    r"^(?P<id>\d+)?$",
+    r"^https://(www\.|m\.)?twitch\.tv/videos/(?P<id>\d+)(\?.+)?$",
+]
+
+CLIP_PATTERNS = [
+    r"^(?P<slug>[A-Za-z0-9]+(?:-[A-Za-z0-9_-]{16})?)$",
+    r"^https://(www\.|m\.)?twitch\.tv/\w+/clip/(?P<slug>[A-Za-z0-9]+(?:-[A-Za-z0-9_-]{16})?)(\?.+)?$",
+    r"^https://clips\.twitch\.tv/(?P<slug>[A-Za-z0-9]+(?:-[A-Za-z0-9_-]{16})?)(\?.+)?$",
+]
+
+
+def parse_video_identifier(identifier: str) -> Optional[str]:
+    """Given a video ID or URL returns the video ID, or null if not matched"""
+    for pattern in VIDEO_PATTERNS:
+        match = re.match(pattern, identifier)
+        if match:
+            return match.group("id")
+
+
+def parse_clip_identifier(identifier: str) -> Optional[str]:
+    """Given a clip slug or URL returns the clip slug, or null if not matched"""
+    for pattern in CLIP_PATTERNS:
+        match = re.match(pattern, identifier)
+        if match:
+            return match.group("slug")