Compare commits

..

44 Commits

Author SHA1 Message Date
7ad574d103 Bump version 2022-02-05 15:08:51 +01:00
2fddb0c6a4 Add support for downloading audio only 2022-02-05 14:37:52 +01:00
fbc227017d Bump version, add changelog 2022-02-05 09:52:06 +01:00
b8aaa0c24c Fix issue with downloading clips with no game set
issue #78
2022-02-05 09:50:17 +01:00
0af472528d Remove unused function 2022-02-05 09:50:17 +01:00
fe9b69e1d4 Remove debian packaging stuff 2022-02-05 09:43:45 +01:00
fb73fe07c5 Add clip slug to formatting options 2022-02-05 09:36:50 +01:00
7eb50a0fa1 Fix output formatting when game is not set
issue #87
2022-02-05 09:34:28 +01:00
6ebe263f33 Fix issue with colons in file name
ffmpeg uses colons to define protocols, by prefixing the target with
file: this ambiguity is avoided.

issue #89
2022-02-05 09:26:10 +01:00
15654291d7 Remove double new line 2022-01-25 08:37:37 +01:00
28b4c6146b Add script for tagging versions, yaml changelog 2022-01-25 08:35:50 +01:00
310010363f Bump version, add changelog 2022-01-25 08:22:35 +01:00
4fd532f05d Ask overwrite confirmation early
Ask whether to overwrite before starting download, this way once
download starts, there will be no more prompts.
2022-01-25 08:08:17 +01:00
8156b18b37 Tweak colors 2022-01-25 07:59:54 +01:00
4ff4c8763c Ignore pyright config file 2022-01-24 10:11:16 +01:00
b24bc0eb29 Add option to define target filename 2022-01-24 09:06:02 +01:00
c5b5c49058 Delete bundles with make clean 2022-01-19 08:20:57 +01:00
b67ccc9dde Bump version, add changelog 2022-01-19 08:04:56 +01:00
192c2925b7 Allow newer versions of m3u8 2022-01-19 08:01:18 +01:00
2f977be161 Bump version 2021-12-03 10:26:17 +01:00
928c6d64cf Add metadata to the encoded video file 2021-12-03 10:25:04 +01:00
dd1f4e0d26 Better computation of speed
Distinct VODs took from disk and freshly downloaded
2021-09-16 08:29:33 +02:00
caabe3138c Remove old kraken requests 2021-09-16 08:01:31 +02:00
9c3cf11635 Dedupe clip fields 2021-09-16 07:59:07 +02:00
2f51b3821b Handle video not found gracefully 2021-09-16 07:58:43 +02:00
62092ee25f Add some tests 2021-09-16 07:57:55 +02:00
6f86aea493 Bump version, changelog 2021-07-31 11:44:56 +02:00
e3f66bda43 Fix compat with older versions of python
path.join doesn't handle PosixPath instances in older versions of python
which causes breakage.

fixes #71
2021-07-31 11:41:45 +02:00
5c3cebd0f3 Bump version, changelog 2021-06-09 15:10:39 +02:00
a49dcab419 Fix clips download by fetching access token
fixes #64
2021-06-09 15:08:40 +02:00
0dd04a7e2d Extract clips list method 2021-05-18 14:25:49 +02:00
5bd0747dde Respect --limit when downloading clips 2021-05-18 14:23:56 +02:00
63c2aff334 Handle keyboard interrupt 2021-05-18 14:00:50 +02:00
e95b430eec Remove unused namedtuple 2021-05-18 13:54:17 +02:00
8c582c600e Fix duplicate named test function 2021-05-18 13:52:34 +02:00
c0c5cbf2a8 Don't break if channel not found 2021-03-22 07:42:07 +01:00
3f143b0c84 Fix file naming not to strip first digit of id
fixes #60
2021-03-22 07:42:07 +01:00
2242af05fc Add changelog, bump version 2021-02-15 14:35:20 +01:00
9c901a21d9 Fix reference to invalid argument 2021-02-15 14:33:31 +01:00
270f53c3c1 Fix tests 2021-02-15 14:33:25 +01:00
e12dba26b4 Add dash as an allowed char, set length to 16. 2021-02-15 14:23:13 +01:00
3a61e61226 Add underscore as an allow char for clip extra ID 2021-02-15 14:23:13 +01:00
8ddfad51bc Update clip identifier patterns
New clips ID naming convention. ex: GloriousColdbloodedTortoiseRuleFive-E017utJ4DZmHVpfQ
2021-02-15 14:23:13 +01:00
d152cbff09 Update tests with new clip naming
New clips ID naming convention. ex: GloriousColdbloodedTortoiseRuleFive-E017utJ4DZmHVpfQ
2021-02-15 14:23:13 +01:00
22 changed files with 694 additions and 230 deletions

1
.gitignore vendored
View File

@ -13,3 +13,4 @@ tmp/
/twitch-dl.1.man
/bundle
/*.pyz
/pyrightconfig.json

View File

@ -1,35 +1,68 @@
Twitch Downloader change log
============================
twitch-dl changelog
===================
1.14.1 (2021-01-14)
-------------------
<!-- Do not edit. This file is automatically generated from changelog.yaml.-->
**1.19.0 (2022-02-05)**
* Add support for downloading audio only (#10)
**1.18.1 (2022-02-05)**
* Fix issues with output formats (#87, #89)
* Fix issues when downloading clip with no game set (#78)
* Add option to use clip slug in `--output` format
**1.18.0 (2022-01-25)**
* Add `--output` option to `download` command which allows setting output file
template (#70)
* Ask to overwrite before downloading to avoid later prompt
**1.17.1 (2022-01-19)**
* Upgrade m3u8 lib to 1.0.0+
**1.17.0 (2021-12-03)**
* Fix speed calculation when resuming download (#75, thanks CroquetteTheThe)
* Add artist and title metadata to resulting video (#80)
**1.16.1 (2021-07-31)**
* Fix compat with older versions of python (#71)
**1.16.0 (2021-06-09)**
* Fix clips download caused by Twitch changes (#64, thanks to all participants)
**1.15.0 (2021-02-15)**
* Add support for new format of clip slug (thanks @Loveangel1337)
**1.14.1 (2021-01-14)**
* Handle videos which don't exist more gracefully
1.14.0 (2021-01-14)
-------------------
**1.14.0 (2021-01-14)**
* Added `info` command for displaying video or clip info (#51)
* Don't show there are more videos when there aren't (#52, thanks @scottyallen)
* Fixed Twitch regression for getting the access token (#53)
1.13.1 (2020-11-23)
-------------------
**1.13.1 (2020-11-23)**
* Fixed clip download issue (#45)
1.13.0 (2020-11-10)
-------------------
**1.13.0 (2020-11-10)**
* Added `clips` command for listing and batch downloading clips (#26)
1.12.1 (2020-09-29)
-------------------
**1.12.1 (2020-09-29)**
* Fix bug introduced in previous version which broke joining
1.12.0 (2020-09-29)
-------------------
**1.12.0 (2020-09-29)**
* Added `source` as alias for best available quality (#33)
* Added `--no-join` option to `download` to skip ffmpeg join (#36)
@ -37,43 +70,37 @@ Twitch Downloader change log
for confirmation (#37)
* Added `--pager` option to `videos`, don't page by default (#30)
1.11.0 (2020-09-03)
-------------------
**1.11.0 (2020-09-03)**
* Make downloading more robust, fixes issues with some VODs (#35)
* Bundle twitch-dl to a standalone archive, simplifying installation, see
installation instructions in README
1.10.2 (2020-08-11)
-------------------
**1.10.2 (2020-08-11)**
* Fix version number displayed by `twitch-dl --version` (#29)
1.10.1 (2020-08-09)
-------------------
**1.10.1 (2020-08-09)**
* Fix videos incorrectly identified as clips (#28)
* Make download command work with video URLs lacking "www" before "twitch.tv"
* Make download command work with video URLs lacking 'www' before 'twitch.tv'
* Print an error when video or clip is not found instead of an exception trace
1.10.0 (2020-08-07)
-------------------
**1.10.0 (2020-08-07)**
* Add `--quality` option to `download` command, allows specifying the video
quality to download. In this case, twitch-dl will require no user input. (#22)
* Fix download of clips which contain numbers in their slug (#24)
* Fix URL to video displayed by `videos` command (it was missing /videos/)
1.9.0 (2020-06-10)
------------------
**1.9.0 (2020-06-10)**
* **Breaking**: wrongly named `--max_workers` option changed to `--max-workers`.
The shorthand option `-w` remains the same.
* Fix bug where `videos` command would crash if there was no game info (#21)
* Allow unicode characters in filenames, no longer strips e.g. cyrillic script
1.8.0 (2020-05-17)
------------------
**1.8.0 (2020-05-17)**
* Fix videos command (#18)
* **Breaking**: `videos` command no longer takes the `--offset` parameter due to
@ -81,58 +108,50 @@ Twitch Downloader change log
* Add paging to `videos` command to replace offset
* Add `--game` option to `videos` command to filter by game
1.7.0 (2020-04-25)
------------------
**1.7.0 (2020-04-25)**
* Support for specifying broadcast type when listing videos (#13)
1.6.0 (2020-04-11)
------------------
**1.6.0 (2020-04-11)**
* Support for downloading clips (#15)
1.5.1 (2020-04-11)
------------------
**1.5.1 (2020-04-11)**
* Fix VOD naming issue (#12)
* Nice console output while downloading
1.5.0 (2020-04-10)
------------------
**1.5.0 (2020-04-10)**
* Fix video downloads after Twitch deprecated access token access
* Don't print errors when retrying download, only if all fails
1.4.0 (2019-08-23)
------------------
**1.4.0 (2019-08-23)**
* Fix usage of deprecated v3 API
* Use m3u8 lib for parsing playlists
* Add `--keep` option not preserve downloaded VODs
1.3.1 (2019-08-13)
------------------
**1.3.1 (2019-08-13)**
* No changes, bumped to fix issue with pypi
1.3.0 (2019-08-13)
------------------
**1.3.0 (2019-08-13)**
* Add `--sort` and `--offset` options to `videos` command, allows paging (#7)
* Show video URL in `videos` command output
1.2.0 (2019-07-05)
------------------
**1.2.0 (2019-07-05)**
* Add `--format` option to `download` command for specifying the output format (#6)
* Add `--format` option to `download` command for specifying the output format
(#6)
* Add `--version` option for printing program version
1.1.0 (2019-06-06)
------------------
**1.1.0 (2019-06-06)**
* Allow limiting download by start and end time
1.0.0 (2019-04-30)
------------------
**1.0.0 (2019-04-30)**
* Initial release

View File

@ -6,7 +6,7 @@ dist :
clean :
find . -name "*pyc" | xargs rm -rf $1
rm -rf build dist bundle MANIFEST htmlcov deb_dist twitch-dl*.tar.gz twitch-dl.1.man
rm -rf build dist bundle MANIFEST htmlcov deb_dist twitch-dl.*.pyz twitch-dl.1.man
bundle:
mkdir bundle
@ -25,11 +25,11 @@ publish :
coverage:
py.test --cov=toot --cov-report html tests/
deb:
@python setup.py --command-packages=stdeb.command bdist_deb
man:
scdoc < twitch-dl.1.scd > twitch-dl.1.man
test:
pytest
changelog:
./scripts/generate_changelog > CHANGELOG.md

View File

@ -147,6 +147,48 @@ Setting quality to `source` will download the best available quality:
twitch-dl download -q source 221837124
```
Setting quality to `audio_only` will download only audio:
```
twitch-dl download -q audio_only 221837124
```
### Overriding file name
The target filename can be defined by passing the `--output` option followed by
the desired file name, e.g. `--output strim.mkv`.
The filename uses
[Python format string syntax](https://docs.python.org/3/library/string.html#format-string-syntax)
and may contain placeholders in curly braces which will be replaced with
relevant information tied to the downloaded video.
The supported placeholders are:
| Placeholder | Description | Example |
| ----------------- | ------------------------------ | ------------------------------ |
| `{id}` | Video ID | 1255522958 |
| `{title}` | Video title | Dark Souls 3 First playthrough |
| `{title_slug}` | Slugified video title | dark_souls_3_first_playthrough |
| `{datetime}` | Video date and time | 2022-01-07T04:00:27Z |
| `{date}` | Video date | 2022-01-07 |
| `{time}` | Video time | 04:00:27Z |
| `{channel}` | Channel name | KatLink |
| `{channel_login}` | Channel login | katlink |
| `{format}` | File extension, see `--format` | mkv |
| `{game}` | Game name | Dark Souls III |
| `{game_slug}` | Slugified game name | dark_souls_iii |
| `{slug}` | Clip slug (clips only) | AbrasivePlacidCatDxAbomb |
A couple of examples:
Pattern: `"{date}_{id}_{channel_login}_{title_slug}.{format}"`<br />
Expands to: `2022-01-07_1255522958_katlink_dark_souls_3_first_playthrough.mkv`<br />
*This is the default.*
Pattern: `"{channel} - {game} - {title}.{format}"`<br />
Expands to: `KatLink - Dark Souls III - Dark Souls 3 First playthrough.mkv`
### Listing clips
List clips for the given period:

174
changelog.yaml Normal file
View File

@ -0,0 +1,174 @@
1.19.0:
date: 2022-02-05
changes:
- "Add support for downloading audio only (#10)"
1.18.1:
date: 2022-02-05
changes:
- "Fix issues with output formats (#87, #89)"
- "Fix issues when downloading clip with no game set (#78)"
- "Add option to use clip slug in `--output` format"
1.18.0:
date: 2022-01-25
changes:
- "Add `--output` option to `download` command which allows setting output file template (#70)"
- "Ask to overwrite before downloading to avoid later prompt"
1.17.1:
date: 2022-01-19
changes:
- "Upgrade m3u8 lib to 1.0.0+"
1.17.0:
date: 2021-12-03
changes:
- "Fix speed calculation when resuming download (#75, thanks CroquetteTheThe)"
- "Add artist and title metadata to resulting video (#80)"
1.16.1:
date: 2021-07-31
changes:
- "Fix compat with older versions of python (#71)"
1.16.0:
date: 2021-06-09
changes:
- "Fix clips download caused by Twitch changes (#64, thanks to all participants)"
1.15.0:
date: 2021-02-15
changes:
- "Add support for new format of clip slug (thanks @Loveangel1337)"
1.14.1:
date: 2021-01-14
changes:
- "Handle videos which don't exist more gracefully"
1.14.0:
date: 2021-01-14
changes:
- "Added `info` command for displaying video or clip info (#51)"
- "Don't show there are more videos when there aren't (#52, thanks @scottyallen)"
- "Fixed Twitch regression for getting the access token (#53)"
1.13.1:
date: 2020-11-23
changes:
- "Fixed clip download issue (#45)"
1.13.0:
date: 2020-11-10
changes:
- "Added `clips` command for listing and batch downloading clips (#26)"
1.12.1:
date: 2020-09-29
changes:
- "Fix bug introduced in previous version which broke joining"
1.12.0:
date: 2020-09-29
changes:
- "Added `source` as alias for best available quality (#33)"
- "Added `--no-join` option to `download` to skip ffmpeg join (#36)"
- "Added `--overwrite` option to `download` to overwrite target without prompting for confirmation (#37)"
- "Added `--pager` option to `videos`, don't page by default (#30)"
1.11.0:
date: 2020-09-03
changes:
- "Make downloading more robust, fixes issues with some VODs (#35)"
- "Bundle twitch-dl to a standalone archive, simplifying installation, see installation instructions in README"
1.10.2:
date: 2020-08-11
changes:
- "Fix version number displayed by `twitch-dl --version` (#29)"
1.10.1:
date: 2020-08-09
changes:
- "Fix videos incorrectly identified as clips (#28)"
- "Make download command work with video URLs lacking 'www' before 'twitch.tv'"
- "Print an error when video or clip is not found instead of an exception trace"
1.10.0:
date: 2020-08-07
changes:
- "Add `--quality` option to `download` command, allows specifying the video quality to download. In this case, twitch-dl will require no user input. (#22)"
- "Fix download of clips which contain numbers in their slug (#24)"
- "Fix URL to video displayed by `videos` command (it was missing /videos/)"
1.9.0:
date: 2020-06-10
changes:
- "**Breaking**: wrongly named `--max_workers` option changed to `--max-workers`. The shorthand option `-w` remains the same."
- "Fix bug where `videos` command would crash if there was no game info (#21)"
- "Allow unicode characters in filenames, no longer strips e.g. cyrillic script"
1.8.0:
date: 2020-05-17
changes:
- "Fix videos command (#18)"
- "**Breaking**: `videos` command no longer takes the `--offset` parameter due to API changes"
- "Add paging to `videos` command to replace offset"
- "Add `--game` option to `videos` command to filter by game"
1.7.0:
date: 2020-04-25
changes:
- "Support for specifying broadcast type when listing videos (#13)"
1.6.0:
date: 2020-04-11
changes:
- "Support for downloading clips (#15)"
1.5.1:
date: 2020-04-11
changes:
- "Fix VOD naming issue (#12)"
- "Nice console output while downloading"
1.5.0:
date: 2020-04-10
changes:
- "Fix video downloads after Twitch deprecated access token access"
- "Don't print errors when retrying download, only if all fails"
1.4.0:
date: 2019-08-23
changes:
- "Fix usage of deprecated v3 API"
- "Use m3u8 lib for parsing playlists"
- "Add `--keep` option not preserve downloaded VODs"
1.3.1:
date: 2019-08-13
changes:
- "No changes, bumped to fix issue with pypi"
1.3.0:
date: 2019-08-13
changes:
- "Add `--sort` and `--offset` options to `videos` command, allows paging (#7)"
- "Show video URL in `videos` command output"
1.2.0:
date: 2019-07-05
changes:
- "Add `--format` option to `download` command for specifying the output format (#6)"
- "Add `--version` option for printing program version"
1.1.0:
date: 2019-06-06
changes:
- "Allow limiting download by start and end time"
1.0.0:
date: 2019-04-30
changes:
- "Initial release"

View File

@ -1,3 +1,4 @@
pytest
twine
wheel
pyyaml

33
scripts/generate_changelog Executable file
View File

@ -0,0 +1,33 @@
#!/usr/bin/env python3
"""
Generates a more user-readable changelog from changelog.yaml.
"""
import textwrap
import yaml
with open("changelog.yaml", "r") as f:
data = yaml.safe_load(f)
print("twitch-dl changelog")
print("===================")
print()
print("<!-- Do not edit. This file is automatically generated from changelog.yaml.-->")
print()
for version in data.keys():
date = data[version]["date"]
changes = data[version]["changes"]
print(f"**{version} ({date})**")
print()
for c in changes:
lines = textwrap.wrap(c, 78)
initial = True
for line in lines:
if initial:
print("* " + line)
initial = False
else:
print(" " + line)
print()

68
scripts/tag_version Executable file
View File

@ -0,0 +1,68 @@
#!/usr/bin/env python3
"""
Creates an annotated git tag for a given version number.
The tag will include the version number and changes for given version.
Usage: tag_version [version]
"""
import subprocess
import sys
import textwrap
import yaml
import twitchdl
from datetime import date
from os import path
from pkg_resources import get_distribution
path = path.join(path.dirname(path.dirname(path.abspath(__file__))), "changelog.yaml")
with open(path, "r") as f:
changelog = yaml.safe_load(f)
if len(sys.argv) != 2:
print("Wrong argument count", file=sys.stderr)
sys.exit(1)
version = sys.argv[1]
changelog_item = changelog.get(version)
if not changelog_item:
print(f"Version `{version}` not found in changelog.", file=sys.stderr)
sys.exit(1)
if twitchdl.__version__ != version:
print(f"twitchdl.__version__ is `{twitchdl.__version__}`, expected {version}.", file=sys.stderr)
sys.exit(1)
dist_version = get_distribution('twitch-dl').version
if dist_version != version:
print(f"Version in setup.py is `{dist_version}`, expected {version}.", file=sys.stderr)
sys.exit(1)
release_date = changelog_item["date"]
changes = changelog_item["changes"]
if not isinstance(release_date, date):
print(f"Release date not set for version `{version}` in the changelog.", file=sys.stderr)
sys.exit(1)
commit_message = f"twitch-dl {version}\n\n"
for c in changes:
lines = textwrap.wrap(c, 70)
initial = True
for line in lines:
lead = " *" if initial else " "
initial = False
commit_message += f"{lead} {line}\n"
proc = subprocess.run(["git", "tag", "-a", version, "-m", commit_message])
if proc.returncode != 0:
sys.exit(1)
print()
print(commit_message)
print()
print(f"Version {version} tagged \\o/")

View File

@ -11,7 +11,7 @@ makes it faster.
setup(
name='twitch-dl',
version='1.14.1',
version='1.19.0',
description='Twitch downloader',
long_description=long_description.strip(),
author='Ivan Habunek',
@ -30,7 +30,7 @@ setup(
packages=find_packages(),
python_requires='>=3.5',
install_requires=[
"m3u8>=0.3.12,<0.4",
"m3u8>=1.0.0,<2.0.0",
"requests>=2.13,<3.0",
],
entry_points={

View File

@ -1,3 +0,0 @@
[DEFAULT]
X-Python3-Version: >= 3.5
Copyright-File: LICENSE

30
tests/test_api.py Normal file
View File

@ -0,0 +1,30 @@
"""
These tests depend on the channel having some videos and clips published.
"""
from twitchdl import twitch
TEST_CHANNEL = "bananasaurus_rex"
def test_get_videos():
videos = twitch.get_channel_videos(TEST_CHANNEL, 3, "time")
assert videos["pageInfo"]
assert len(videos["edges"]) > 0
video_id = videos["edges"][0]["node"]["id"]
video = twitch.get_video(video_id)
assert video["id"] == video_id
def test_get_clips():
"""
This test depends on the channel having some videos published.
"""
clips = twitch.get_channel_clips(TEST_CHANNEL, "all_time", 3)
assert clips["pageInfo"]
assert len(clips["edges"]) > 0
clip_slug = clips["edges"][0]["node"]["slug"]
clip = twitch.get_clip(clip_slug)
assert clip["slug"] == clip_slug

View File

@ -1,10 +1,6 @@
import pytest
from unittest.mock import patch
from twitchdl.commands import download
from collections import namedtuple
Args = namedtuple("args", ["video"])
from twitchdl.utils import parse_video_identifier, parse_clip_identifier
TEST_VIDEO_PATTERNS = [
@ -22,24 +18,18 @@ TEST_CLIP_PATTERNS = {
("HungryProudRadicchioDoggo", "https://clips.twitch.tv/HungryProudRadicchioDoggo"),
("HungryProudRadicchioDoggo", "https://www.twitch.tv/bananasaurus_rex/clip/HungryProudRadicchioDoggo?filter=clips&range=7d&sort=time"),
("HungryProudRadicchioDoggo", "https://twitch.tv/bananasaurus_rex/clip/HungryProudRadicchioDoggo?filter=clips&range=7d&sort=time"),
("GloriousColdbloodedTortoiseRuleFive-E017utJ4DZmHVpfQ", "GloriousColdbloodedTortoiseRuleFive-E017utJ4DZmHVpfQ"),
("GloriousColdbloodedTortoiseRuleFive-E017utJ4DZmHVpfQ", "https://twitch.tv/dracul1nx/clip/GloriousColdbloodedTortoiseRuleFive-E017utJ4DZmHVpfQ"),
("GloriousColdbloodedTortoiseRuleFive-E017utJ4DZmHVpfQ", "https://twitch.tv/dracul1nx/clip/GloriousColdbloodedTortoiseRuleFive-E017utJ4DZmHVpfQ?filter=clips&range=7d&sort=time"),
("GloriousColdbloodedTortoiseRuleFive-E017utJ4DZmHVpfQ", "https://www.twitch.tv/dracul1nx/clip/GloriousColdbloodedTortoiseRuleFive-E017utJ4DZmHVpfQ?filter=clips&range=7d&sort=time"),
}
@patch("twitchdl.commands._download_clip")
@patch("twitchdl.commands._download_video")
@pytest.mark.parametrize("expected,input", TEST_VIDEO_PATTERNS)
def test_video_patterns(video_dl, clip_dl, expected, input):
args = Args(video=input)
download(args)
video_dl.assert_called_once_with(expected, args)
clip_dl.assert_not_called()
def test_video_patterns(expected, input):
assert parse_video_identifier(input) == expected
@patch("twitchdl.commands._download_clip")
@patch("twitchdl.commands._download_video")
@pytest.mark.parametrize("expected,input", TEST_CLIP_PATTERNS)
def test_clip_patterns(video_dl, clip_dl, expected, input):
args = Args(video=input)
download(args)
clip_dl.assert_called_once_with(expected, args)
video_dl.assert_not_called()
def test_clip_patterns(expected, input):
assert parse_clip_identifier(input) == expected

13
tests/test_utils.py Normal file
View File

@ -0,0 +1,13 @@
from twitchdl.utils import titlify, slugify
def test_titlify():
assert titlify("Foo Bar Baz.") == "Foo Bar Baz."
assert titlify("Foo (Bar) [Baz]") == "Foo (Bar) [Baz]"
assert titlify("Foo@{} Bar Baz!\"#$%&/=?*+'🔪") == "Foo Bar Baz"
def test_slugify():
assert slugify("Foo Bar Baz") == "foo_bar_baz"
assert slugify(" Foo Bar Baz ") == "foo_bar_baz"
assert slugify("Foo@{}[] Bar Baz!\"#$%&/()=?*+'🔪") == "foo_bar_baz"

View File

@ -1,3 +1,3 @@
__version__ = "1.14.1"
__version__ = "1.19.0"
CLIENT_ID = "kimne78kx3ncx6brgo4mv6wki5h1ko"

View File

@ -3,11 +3,22 @@ import re
from os import path
from twitchdl import twitch, utils
from twitchdl.commands.download import get_clip_authenticated_url
from twitchdl.download import download_file
from twitchdl.exceptions import ConsoleError
from twitchdl.output import print_out, print_clip, print_json
def clips(args):
if args.json:
return _clips_json(args)
if args.download:
return _clips_download(args)
return _clips_list(args)
def _continue():
print_out(
"\nThere are more clips. "
@ -63,26 +74,27 @@ def _clip_target_filename(clip):
def _clips_download(args):
downloaded_count = 0
generator = twitch.channel_clips_generator(args.channel_name, args.period, 100)
for clips, _ in generator:
for clip in clips["edges"]:
clip = clip["node"]
url = clip["videoQualities"][0]["sourceURL"]
url = get_clip_authenticated_url(clip["slug"], "source")
target = _clip_target_filename(clip)
if path.exists(target):
print_out("Already downloaded: <green>{}</green>".format(target))
else:
print_out("Downloading: <yellow>{}</yellow>".format(target))
download_file(url, target)
downloaded_count += 1
if args.limit and downloaded_count >= args.limit:
return
def clips(args):
if args.json:
return _clips_json(args)
if args.download:
return _clips_download(args)
def _clips_list(args):
print_out("<dim>Loading clips...</dim>")
generator = twitch.channel_clips_generator(args.channel_name, args.period, args.limit)

View File

@ -7,7 +7,7 @@ import tempfile
from os import path
from pathlib import Path
from urllib.parse import urlparse
from urllib.parse import urlparse, urlencode
from twitchdl import twitch, utils
from twitchdl.download import download_file, download_files
@ -18,10 +18,15 @@ from twitchdl.output import print_out
def _parse_playlists(playlists_m3u8):
playlists = m3u8.loads(playlists_m3u8)
for p in playlists.playlists:
name = p.media[0].name if p.media else ""
resolution = "x".join(str(r) for r in p.stream_info.resolution)
yield name, resolution, p.uri
for p in sorted(playlists.playlists, key=lambda p: p.stream_info.resolution is None):
if p.stream_info.resolution:
name = p.media[0].name
description = "x".join(str(r) for r in p.stream_info.resolution)
else:
name = p.media[0].group_id
description = None
yield name, description, p.uri
def _get_playlist_by_name(playlists, quality):
@ -41,21 +46,27 @@ def _get_playlist_by_name(playlists, quality):
def _select_playlist_interactive(playlists):
print_out("\nAvailable qualities:")
for n, (name, resolution, uri) in enumerate(playlists):
print_out("{}) {} [{}]".format(n + 1, name, resolution))
if resolution:
print_out("{}) {} [{}]".format(n + 1, name, resolution))
else:
print_out("{}) {}".format(n + 1, name))
no = utils.read_int("Choose quality", min=1, max=len(playlists) + 1, default=1)
_, _, uri = playlists[no - 1]
return uri
def _join_vods(playlist_path, target, overwrite):
def _join_vods(playlist_path, target, overwrite, video):
command = [
"ffmpeg",
"-i", playlist_path,
"-c", "copy",
target,
"-metadata", "artist={}".format(video["creator"]["displayName"]),
"-metadata", "title={}".format(video["title"]),
"-metadata", "encoded_by=twitch-dl",
"-stats",
"-loglevel", "warning",
"file:{}".format(target),
]
if overwrite:
@ -67,36 +78,59 @@ def _join_vods(playlist_path, target, overwrite):
raise ConsoleError("Joining files failed")
def _video_target_filename(video, format):
match = re.search(r"^(\d{4})-(\d{2})-(\d{2})T", video['publishedAt'])
date = "".join(match.groups())
def _video_target_filename(video, args):
date, time = video['publishedAt'].split("T")
game = video["game"]["name"] if video["game"] else "Unknown"
name = "_".join([
date,
video['id'][1:],
video['creator']['login'],
utils.slugify(video['title']),
])
subs = {
"channel": video["creator"]["displayName"],
"channel_login": video["creator"]["login"],
"date": date,
"datetime": video["publishedAt"],
"format": args.format,
"game": game,
"game_slug": utils.slugify(game),
"id": video["id"],
"time": time,
"title": utils.titlify(video["title"]),
"title_slug": utils.slugify(video["title"]),
}
return name + "." + format
try:
return args.output.format(**subs)
except KeyError as e:
supported = ", ".join(subs.keys())
raise ConsoleError("Invalid key {} used in --output. Supported keys are: {}".format(e, supported))
def _clip_target_filename(clip):
def _clip_target_filename(clip, args):
date, time = clip["createdAt"].split("T")
game = clip["game"]["name"] if clip["game"] else "Unknown"
url = clip["videoQualities"][0]["sourceURL"]
_, ext = path.splitext(url)
ext = ext.lstrip(".")
match = re.search(r"^(\d{4})-(\d{2})-(\d{2})T", clip["createdAt"])
date = "".join(match.groups())
subs = {
"channel": clip["broadcaster"]["displayName"],
"channel_login": clip["broadcaster"]["login"],
"date": date,
"datetime": clip["createdAt"],
"format": ext,
"game": game,
"game_slug": utils.slugify(game),
"id": clip["id"],
"slug": clip["slug"],
"time": time,
"title": utils.titlify(clip["title"]),
"title_slug": utils.slugify(clip["title"]),
}
name = "_".join([
date,
clip["id"],
clip["broadcaster"]["login"],
utils.slugify(clip["title"]),
])
return "{}.{}".format(name, ext)
try:
return args.output.format(**subs)
except KeyError as e:
supported = ", ".join(subs.keys())
raise ConsoleError("Invalid key {} used in --output. Supported keys are: {}".format(e, supported))
def _get_vod_paths(playlist, start, end):
@ -124,7 +158,7 @@ def _crete_temp_dir(base_uri):
path = urlparse(base_uri).path.lstrip("/")
temp_dir = Path(tempfile.gettempdir(), "twitch-dl", path)
temp_dir.mkdir(parents=True, exist_ok=True)
return temp_dir
return str(temp_dir)
def download(args):
@ -139,21 +173,21 @@ def download(args):
raise ConsoleError("Invalid input: {}".format(args.video))
def _get_clip_url(clip, args):
def _get_clip_url(clip, quality):
qualities = clip["videoQualities"]
# Quality given as an argument
if args.quality:
if args.quality == "source":
if quality:
if quality == "source":
return qualities[0]["sourceURL"]
selected_quality = args.quality.rstrip("p") # allow 720p as well as 720
selected_quality = quality.rstrip("p") # allow 720p as well as 720
for q in qualities:
if q["quality"] == selected_quality:
return q["sourceURL"]
available = ", ".join([str(q["quality"]) for q in qualities])
msg = "Quality '{}' not found. Available qualities are: {}".format(args.quality, available)
msg = "Quality '{}' not found. Available qualities are: {}".format(quality, available)
raise ConsoleError(msg)
# Ask user to select quality
@ -167,9 +201,27 @@ def _get_clip_url(clip, args):
return selected_quality["sourceURL"]
def get_clip_authenticated_url(slug, quality):
print_out("<dim>Fetching access token...</dim>")
access_token = twitch.get_clip_access_token(slug)
if not access_token:
raise ConsoleError("Access token not found for slug '{}'".format(slug))
url = _get_clip_url(access_token, quality)
query = urlencode({
"sig": access_token["playbackAccessToken"]["signature"],
"token": access_token["playbackAccessToken"]["value"],
})
return "{}?{}".format(url, query)
def _download_clip(slug, args):
print_out("<dim>Looking up clip...</dim>")
clip = twitch.get_clip(slug)
game = clip["game"]["name"] if clip["game"] else "Unknown"
if not clip:
raise ConsoleError("Clip '{}' not found".format(slug))
@ -177,19 +229,26 @@ def _download_clip(slug, args):
print_out("Found: <green>{}</green> by <yellow>{}</yellow>, playing <blue>{}</blue> ({})".format(
clip["title"],
clip["broadcaster"]["displayName"],
clip["game"]["name"],
game,
utils.format_duration(clip["durationSeconds"])
))
url = _get_clip_url(clip, args)
target = _clip_target_filename(clip, args)
print_out("Target: <blue>{}</blue>".format(target))
if not args.overwrite and path.exists(target):
response = input("File exists. Overwrite? [Y/n]: ")
if response.lower().strip() not in ["", "y"]:
raise ConsoleError("Aborted")
args.overwrite = True
url = get_clip_authenticated_url(slug, args.quality)
print_out("<dim>Selected URL: {}</dim>".format(url))
target = _clip_target_filename(clip)
print_out("Downloading clip...")
print_out("<dim>Downloading clip...</dim>")
download_file(url, target)
print_out("Downloaded: {}".format(target))
print_out("Downloaded: <blue>{}</blue>".format(target))
def _download_video(video_id, args):
@ -199,9 +258,21 @@ def _download_video(video_id, args):
print_out("<dim>Looking up video...</dim>")
video = twitch.get_video(video_id)
if not video:
raise ConsoleError("Video {} not found".format(video_id))
print_out("Found: <blue>{}</blue> by <yellow>{}</yellow>".format(
video['title'], video['creator']['displayName']))
target = _video_target_filename(video, args)
print_out("Output: <blue>{}</blue>".format(target))
if not args.overwrite and path.exists(target):
response = input("File exists. Overwrite? [Y/n]: ")
if response.lower().strip() not in ["", "y"]:
raise ConsoleError("Aborted")
args.overwrite = True
print_out("<dim>Fetching access token...</dim>")
access_token = twitch.get_access_token(video_id)
@ -248,8 +319,7 @@ def _download_video(video_id, args):
return
print_out("\n\nJoining files...")
target = _video_target_filename(video, args.format)
_join_vods(playlist_path, target, args.overwrite)
_join_vods(playlist_path, target, args.overwrite, video)
if args.keep:
print_out("\n<dim>Temporary files not deleted: {}</dim>".format(target_dir))

View File

@ -40,7 +40,7 @@ def info(args):
clip_info(clip)
return
raise ConsoleError("Invalid input: {}".format(args.video))
raise ConsoleError("Invalid input: {}".format(args.identifier))
def video_info(video, playlists):

View File

@ -168,6 +168,11 @@ COMMANDS = [
"help": "Overwrite the target file if it already exists without prompting.",
"action": "store_true",
"default": False,
}),
(["-o", "--output"], {
"help": "Output file name template. See docs for details.",
"type": str,
"default": "{date}_{id}_{channel_login}_{title_slug}.{format}"
})
],
),
@ -239,6 +244,9 @@ def main():
except ConsoleError as e:
print_err(e)
sys.exit(1)
except KeyboardInterrupt:
print_err("Operation canceled")
sys.exit(1)
except GQLError as e:
print_err(e)
for err in e.errors:

View File

@ -34,11 +34,13 @@ def _download(url, path):
def download_file(url, path, retries=RETRY_COUNT):
if os.path.exists(path):
return os.path.getsize(path)
from_disk = True
return (os.path.getsize(path), from_disk)
from_disk = False
for _ in range(retries):
try:
return _download(url, path)
return (_download(url, path), from_disk)
except RequestException:
pass
@ -51,17 +53,26 @@ def _print_progress(futures):
max_msg_size = 0
start_time = datetime.now()
total_count = len(futures)
current_download_size = 0
current_downloaded_count = 0
for future in as_completed(futures):
size = future.result()
size, from_disk = future.result()
downloaded_count += 1
downloaded_size += size
# If we find something on disk, we don't want to take it in account in
# the speed calculation
if not from_disk:
current_download_size += size
current_downloaded_count += 1
percentage = 100 * downloaded_count // total_count
est_total_size = int(total_count * downloaded_size / downloaded_count)
duration = (datetime.now() - start_time).seconds
speed = downloaded_size // duration if duration else 0
remaining = (total_count - downloaded_count) * duration / downloaded_count
speed = current_download_size // duration if duration else 0
remaining = (total_count - downloaded_count) * duration / current_downloaded_count \
if current_downloaded_count else 0
msg = " ".join([
"Downloaded VOD {}/{}".format(downloaded_count, total_count),
@ -69,7 +80,7 @@ def _print_progress(futures):
"<cyan>{}</cyan>".format(format_size(downloaded_size)),
"of <cyan>~{}</cyan>".format(format_size(est_total_size)),
"at <cyan>{}/s</cyan>".format(format_size(speed)) if speed > 0 else "",
"remaining <cyan>~{}</cyan>".format(format_duration(remaining)) if speed > 0 else "",
"remaining <cyan>~{}</cyan>".format(format_duration(remaining)) if remaining > 0 else "",
])
max_msg_size = max(len(msg), max_msg_size)

View File

@ -72,6 +72,7 @@ def print_video(video):
published_at = video["publishedAt"].replace("T", " @ ").replace("Z", "")
length = utils.format_duration(video["lengthSeconds"])
channel = video["creator"]["displayName"]
playing = (
"playing <blue>{}</blue>".format(video["game"]["name"])
if video["game"] else ""
@ -104,8 +105,3 @@ def print_clip(clip):
" Length: <blue>{}</blue>"
" Views: <blue>{}</blue>".format(published_at, length, clip["viewCount"]))
print_out("<i>{}</i>".format(clip["url"]))
def print_clip_urls(clip):
from pprint import pprint
pprint(clip)

View File

@ -42,13 +42,14 @@ def authenticated_post(url, data=None, json=None, headers={}):
return response
def kraken_get(url, params={}, headers={}):
"""
Add accept header required by kraken API v5.
see: https://discuss.dev.twitch.tv/t/change-in-access-to-deprecated-kraken-twitch-apis/22241
"""
headers["Accept"] = "application/vnd.twitchtv.v5+json"
return authenticated_get(url, params, headers)
def gql_post(query):
url = "https://gql.twitch.tv/gql"
response = authenticated_post(url, data=query).json()
if "errors" in response:
raise GQLError(response["errors"])
return response
def gql_query(query):
@ -61,15 +62,6 @@ def gql_query(query):
return response
def get_video_legacy(video_id):
"""
https://dev.twitch.tv/docs/v5/reference/videos#get-video
"""
url = "https://api.twitch.tv/kraken/videos/{}".format(video_id)
return kraken_get(url).json()
VIDEO_FIELDS = """
id
title
@ -86,6 +78,30 @@ VIDEO_FIELDS = """
"""
CLIP_FIELDS = """
id
slug
title
createdAt
viewCount
durationSeconds
url
videoQualities {
frameRate
quality
sourceURL
}
game {
id
name
}
broadcaster {
displayName
login
}
"""
def get_video(video_id):
query = """
{{
@ -105,31 +121,32 @@ def get_clip(slug):
query = """
{{
clip(slug: "{}") {{
id
slug
title
createdAt
viewCount
durationSeconds
url
videoQualities {{
frameRate
quality
sourceURL
}}
game {{
id
name
}}
broadcaster {{
displayName
login
{fields}
}}
}}
"""
response = gql_query(query.format(slug, fields=CLIP_FIELDS))
return response["data"]["clip"]
def get_clip_access_token(slug):
query = """
{{
"operationName": "VideoAccessToken_Clip",
"variables": {{
"slug": "{slug}"
}},
"extensions": {{
"persistedQuery": {{
"version": 1,
"sha256Hash": "36b89d2507fce29e5ca551df756d27c1cfe079e2609642b4390aa4c35796eb11"
}}
}}
}}
"""
response = gql_query(query.format(slug))
response = gql_post(query.format(slug=slug).strip())
return response["data"]["clip"]
@ -154,26 +171,7 @@ def get_channel_clips(channel_id, period, limit, after=None):
edges {{
cursor
node {{
id
slug
title
createdAt
viewCount
durationSeconds
url
videoQualities {{
frameRate
quality
sourceURL
}}
game {{
id
name
}}
broadcaster {{
displayName
login
}}
{fields}
}}
}}
}}
@ -181,12 +179,13 @@ def get_channel_clips(channel_id, period, limit, after=None):
}}
"""
query = query.format(**{
"channel_id": channel_id,
"after": after if after else "",
"limit": limit,
"period": period.upper(),
})
query = query.format(
channel_id=channel_id,
after=after if after else "",
limit=limit,
period=period.upper(),
fields=CLIP_FIELDS
)
response = gql_query(query)
user = response["data"]["user"]
@ -234,18 +233,7 @@ def get_channel_videos(channel_id, limit, sort, type="archive", game_ids=[], aft
edges {{
cursor
node {{
id
title
publishedAt
broadcastType
lengthSeconds
game {{
name
}}
creator {{
login
displayName
}}
{fields}
}}
}}
}}
@ -253,16 +241,21 @@ def get_channel_videos(channel_id, limit, sort, type="archive", game_ids=[], aft
}}
"""
query = query.format(**{
"channel_id": channel_id,
"game_ids": game_ids,
"after": after if after else "",
"limit": limit,
"sort": sort.upper(),
"type": type.upper(),
})
query = query.format(
channel_id=channel_id,
game_ids=game_ids,
after=after if after else "",
limit=limit,
sort=sort.upper(),
type=type.upper(),
fields=VIDEO_FIELDS
)
response = gql_query(query)
if not response["data"]["user"]:
raise ConsoleError("Channel {} not found".format(channel_id))
return response["data"]["user"]["videos"]
@ -316,6 +309,7 @@ def get_playlists(video_id, access_token):
response = requests.get(url, params={
"nauth": access_token['value'],
"nauthsig": access_token['signature'],
"allow_audio_only": "true",
"allow_source": "true",
"player": "twitchweb",
})

View File

@ -55,12 +55,17 @@ def read_int(msg, min, max, default):
def slugify(value):
re_pattern = re.compile(r'[^\w\s-]', flags=re.U)
re_spaces = re.compile(r'[-\s]+', flags=re.U)
value = str(value)
value = unicodedata.normalize('NFKC', value)
value = re_pattern.sub('', value).strip().lower()
return re_spaces.sub('_', value)
value = unicodedata.normalize('NFKC', str(value))
value = re.sub(r'[^\w\s_-]', '', value)
value = re.sub(r'[\s_-]+', '_', value)
return value.strip("_").lower()
def titlify(value):
value = unicodedata.normalize('NFKC', str(value))
value = re.sub(r'[^\w\s\[\]().-]', '', value)
value = re.sub(r'\s+', ' ', value)
return value.strip()
VIDEO_PATTERNS = [
@ -69,9 +74,9 @@ VIDEO_PATTERNS = [
]
CLIP_PATTERNS = [
r"^(?P<slug>[A-Za-z0-9]+)$",
r"^https://(www.)?twitch.tv/\w+/clip/(?P<slug>[A-Za-z0-9]+)(\?.+)?$",
r"^https://clips.twitch.tv/(?P<slug>[A-Za-z0-9]+)(\?.+)?$",
r"^(?P<slug>[A-Za-z0-9]+(?:-[A-Za-z0-9_-]{16})?)$",
r"^https://(www.)?twitch.tv/\w+/clip/(?P<slug>[A-Za-z0-9]+(?:-[A-Za-z0-9_-]{16})?)(\?.+)?$",
r"^https://clips.twitch.tv/(?P<slug>[A-Za-z0-9]+(?:-[A-Za-z0-9_-]{16})?)(\?.+)?$",
]