mirror of
https://github.com/Evil0ctal/Douyin_TikTok_Download_API.git
synced 2025-04-23 06:04:22 +08:00
Merge pull request #279 from Xuenew/xigua
This commit is contained in:
Evil0ctal
GitHub
commit
1eb4d5e0d7
363
README.en.md
363
README.en.md
@ -1,363 +0,0 @@
|
||||
<div align="center">
|
||||
<a href="https://douyin.wtf/" alt="logo" ><img src="./logo/logo192.png" width="120"/></a>
|
||||
</div>
|
||||
<h1 align="center">Douyin_TikTok_Download_API(抖音/TikTok API)</h1>
|
||||
|
||||
<div align="center">
|
||||
|
||||
[English](./README.en.md)\|[Simplified Chinese](./README.md)
|
||||
|
||||
🚀"Douyin_TikTok_Download_API" is a high-performance asynchronous API that can be used out of the box[Tik Tok](https://www.douyin.com)\|[TikTok](https://www.tiktok.com)\|[Bilibili](https://www.bilibili.com)Data crawling tool supports API calling, online batch analysis and downloading.
|
||||
|
||||
[](LICENSE)[](https://github.com/Evil0ctal/Douyin_TikTok_Download_API/releases/latest)[](https://github.com/Evil0ctal/Douyin_TikTok_Download_API/stargazers)[](https://github.com/Evil0ctal/Douyin_TikTok_Download_API/network/members)[](https://github.com/Evil0ctal/Douyin_TikTok_Download_API/issues)[](https://github.com/Evil0ctal/Douyin_TikTok_Download_API/issues?q=is%3Aissue+is%3Aclosed)<br>[](https://pypi.org/project/douyin-tiktok-scraper/)[](https://pypi.org/project/douyin-tiktok-scraper/#files)[](https://pypi.org/project/douyin-tiktok-scraper/)[](https://pypi.org/project/douyin-tiktok-scraper/)<br>[](https://api.douyin.wtf/docs)[](https://api.tikhub.io/docs)<br>[](https://afdian.net/@evil0ctal)[](https://ko-fi.com/evil0ctal)[](https://www.patreon.com/evil0ctal)
|
||||
|
||||
</div>
|
||||
|
||||
## 👻Introduction
|
||||
|
||||
> 🚨If you need to use a private server to run this project, please refer to the deployment method\[[Docker deployment](./README.md#%E9%83%A8%E7%BD%B2%E6%96%B9%E5%BC%8F%E4%BA%8C-docker),[One-click deployment](./README.md#%E9%83%A8%E7%BD%B2%E6%96%B9%E5%BC%8F%E4%B8%80-linux)]
|
||||
|
||||
This project is based on[PyWebIO](https://github.com/pywebio/PyWebIO)，[FastAPI](https://fastapi.tiangolo.com/)，[AIOHTTP](https://docs.aiohttp.org/), fast and asynchronous[Tik Tok](https://www.douyin.com/)/[TikTok](https://www.tiktok.com/)/[Bilibili](https://www.bilibili.com)Data crawling tool, and realizes online batch parsing and downloading of videos or photo albums without watermarks, data crawling API, and iOS shortcut commands without watermark downloads through the Web. You can deploy or modify this project yourself to achieve more functions, or you can call it directly in your project[scraper.py](https://github.com/Evil0ctal/Douyin_TikTok_Download_API/blob/Stable/scraper.py)or install an existing[pip package](https://pypi.org/project/douyin-tiktok-scraper/)As a parsing library, it is easy to crawl data, etc.....
|
||||
|
||||
_Some simple application scenarios:_
|
||||
|
||||
_Download prohibited videos, perform data analysis, download without watermark on iOS (with[Shortcut command APP that comes with iOS](https://apps.apple.com/cn/app/%E5%BF%AB%E6%8D%B7%E6%8C%87%E4%BB%A4/id915249334)Cooperate with the API of this project to achieve in-app downloads or read clipboard downloads), etc....._
|
||||
|
||||
## 🖥Public site: I am very vulnerable...please do not stress test (·•᷄ࡇ•᷅ )
|
||||
|
||||
> **Tikhub-Opi:**support`Douyin|TikTok`The user's homepage crawls the author's \[homepage video data (watermark removal link, liked video list (permission must be public), video comment data, background music video list data, etc...), please check TikHub- for details API documentation. In addition, compared with the API of this project, TikHub-API is faster when grabbing TikTok data.
|
||||
|
||||
🍔Web APP:<https://douyin.wtf/>
|
||||
|
||||
🍟API Document:<https://api.douyin.wtf/docs>
|
||||
|
||||
🌭TikHub API Document:<https://api.tikhub.io/docs>
|
||||
|
||||
💾iOS Shortcut (shortcut command):[Shortcut release](https://github.com/Evil0ctal/Douyin_TikTok_Download_API/discussions/104?sort=top)
|
||||
|
||||
📦️Desktop downloader (recommended by warehouse):
|
||||
|
||||
- [Johnserf-Seed/TikTokDownload](https://github.com/Johnserf-Seed/TikTokDownload)
|
||||
- [HFrost0/bilix](https://github.com/HFrost0/bilix)
|
||||
- [Tairraos/TikDown - \[needs update\]](https://github.com/Tairraos/TikDown/)
|
||||
|
||||
🛸Other repositories based on this project
|
||||
|
||||
- [TikHubIO/TikHub_API_PyPi](https://github.com/TikHubIO/TikHub_API_PyPi)
|
||||
- [Evil0ctal/Douyin_Tiktok_Scraper_PyPi](https://github.com/Evil0ctal/Douyin_Tiktok_Scraper_PyPi)
|
||||
|
||||
## ⚗️Technology stack
|
||||
|
||||
- [web_app.py](https://github.com/Evil0ctal/Douyin_TikTok_Download_API/blob/main/web_app.py)-[PyWebIO](https://www.pyweb.io/)
|
||||
- [web_api.py](https://github.com/Evil0ctal/Douyin_TikTok_Download_API/blob/main/web_api.py)-[FastAPI](https://fastapi.tiangolo.com/)
|
||||
- [scraper.py](https://github.com/Evil0ctal/Douyin_TikTok_Download_API/blob/main/scraper.py)-[AIOHTTP](https://docs.aiohttp.org/)
|
||||
|
||||
> **_scraper.py:_**
|
||||
|
||||
- Towards[Douyin|TikTok]The API submits a request and retrieves the data, and returns a dictionary (dict) after processing, supporting asynchronous.
|
||||
|
||||
> **_web_api.py:_**
|
||||
|
||||
- Get request parameters and use`Scraper()`The class processes the data and returns it in JSON form, downloads the video, and cooperates with iOS shortcut commands to achieve fast calling and supports asynchronous.
|
||||
|
||||
> **_web_app.py:_**
|
||||
|
||||
- for`web_api.py`as well as`scraper.py`A simple web program created to process the values entered on the web page and then use them`Scraper()`Class processing and cooperation`web_api.py`The interface output is on the web page (similar to front-end and back-end separation)
|
||||
|
||||
**_Most of the parameters of the above files can be found in[config.ini](https://github.com/Evil0ctal/Douyin_TikTok_Download_API/blob/main/config.ini)Make changes in_**
|
||||
|
||||
## 💡Project file structure
|
||||
|
||||
.
|
||||
└── Douyin_TikTok_Download_API/
|
||||
├── /static -> (PyWebIO static resources)
|
||||
├── web_app.py -> (Web APP)
|
||||
├── web_api.py -> (API)
|
||||
├── scraper.py -> (Parsing library)
|
||||
├── config.ini -> (Configuration file)
|
||||
├── install.sh -> (Installation bash script)
|
||||
|
||||
## ✨Features:
|
||||
|
||||
- Douyin (overseas version of Douyin: TikTok) video/picture analysis
|
||||
- Bilibili video analysis
|
||||
- Batch analysis on the web page (supports Douyin/TikTok mixed submission)
|
||||
- Batch download of non-watermarked videos from the web parsing result page (removed for V3.X and above versions, please deploy V2.X version by yourself)
|
||||
- API call to get link data
|
||||
- make[pip package](https://pypi.org/project/douyin-tiktok-scraper/)Conveniently and quickly import your projects
|
||||
- [iOS shortcut commands to quickly call API](https://apps.apple.com/cn/app/%E5%BF%AB%E6%8D%B7%E6%8C%87%E4%BB%A4/id915249334)Achieve in-app download of watermark-free videos/photo albums
|
||||
- Analyze all videos on the author's homepage ([Tikhub-opy](https://api.tikhub.io/docs)Support Douyin/TikTok)
|
||||
- Parse all comment information in the video ([Tikhub-opy](https://api.tikhub.io/docs)Support Douyin/TikTok)
|
||||
|
||||
* * *
|
||||
|
||||
## 🤦To-do list:
|
||||
|
||||
> 💡Welcome to make suggestions or submit PR directly to this warehouse ♪(･ω･)ﾉ)
|
||||
|
||||
- [ ] Write a desktop asynchronous downloader to implement local batch downloading
|
||||
- [ ] TikHub-API adds data crawling for hash_tag pages[#101](https://github.com/Evil0ctal/Douyin_TikTok_Download_API/issues/101)
|
||||
- [ ] Add support for other short video platforms, such as Douyin Huoshan Edition, Kuaishou, Xigua Video, and Bilibili
|
||||
|
||||
* * *
|
||||
|
||||
## 📦Call the parsing library:
|
||||
|
||||
> 💡PyPi：<https://pypi.org/project/douyin-tiktok-scraper/>
|
||||
|
||||
Install the parsing library:`pip install douyin-tiktok-scraper`
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from douyin_tiktok_scraper.scraper import Scraper
|
||||
|
||||
api = Scraper()
|
||||
|
||||
async def hybrid_parsing(url: str) -> dict:
|
||||
# Hybrid parsing(Douyin/TikTok URL)
|
||||
result = await api.hybrid_parsing(url)
|
||||
print(f"The hybrid parsing result:\n {result}")
|
||||
return result
|
||||
|
||||
asyncio.run(hybrid_parsing(url=input("Paste Douyin/TikTok/Bilibili share URL here: ")))
|
||||
```
|
||||
|
||||
## 🗺️Supported submission formats:
|
||||
|
||||
> 💡Tip: Including but not limited to the following examples, if you encounter link parsing failure, please open a new one[issue](https://github.com/Evil0ctal/Douyin_TikTok_Download_API/issues)
|
||||
|
||||
- Bilibili video link
|
||||
|
||||
```text
|
||||
https://www.bilibili.com/video/BV1Th411x7ii/
|
||||
```
|
||||
|
||||
- Douyin sharing password (copy in APP)
|
||||
|
||||
```text
|
||||
7.43 pda:/ 让你在几秒钟之内记住我 https://v.douyin.com/L5pbfdP/ 复制此链接，打开Dou音搜索，直接观看视频！
|
||||
```
|
||||
|
||||
- Douyin short URL (copy within APP)
|
||||
|
||||
```text
|
||||
https://v.douyin.com/L4FJNR3/
|
||||
```
|
||||
|
||||
- Douyin normal URL (copy from web version)
|
||||
|
||||
```text
|
||||
https://www.douyin.com/video/6914948781100338440
|
||||
```
|
||||
|
||||
- Douyin discovery page URL (APP copy)
|
||||
|
||||
```text
|
||||
https://www.douyin.com/discover?modal_id=7069543727328398622
|
||||
```
|
||||
|
||||
- TikTok short URL (copy within APP)
|
||||
|
||||
```text
|
||||
https://www.tiktok.com/t/ZTR9nDNWq/
|
||||
```
|
||||
|
||||
- TikTok normal URL (copy from web version)
|
||||
|
||||
```text
|
||||
https://www.tiktok.com/@evil0ctal/video/7156033831819037994
|
||||
```
|
||||
|
||||
- Douyin/TikTok batch URL (no need to use matching separation)
|
||||
|
||||
```text
|
||||
https://v.douyin.com/L4NpDJ6/
|
||||
https://www.douyin.com/video/7126745726494821640
|
||||
2.84 nqe:/ 骑白马的也可以是公主%%百万转场变身https://v.douyin.com/L4FJNR3/ 复制此链接，打开Dou音搜索，直接观看视频！
|
||||
https://www.tiktok.com/t/ZTR9nkkmL/
|
||||
https://www.tiktok.com/t/ZTR9nDNWq/
|
||||
https://www.tiktok.com/@evil0ctal/video/7156033831819037994
|
||||
```
|
||||
|
||||
## 🛰️API documentation
|
||||
|
||||
> 💡Tip: You can also view the interface documentation in the code comments of web_api.py
|
||||
|
||||
**_API documentation:_**
|
||||
|
||||
local:<http://localhost:8000/docs>
|
||||
|
||||
Online:<https://api.douyin.wtf/docs>
|
||||
|
||||
**_TikHub-API documentation:_**
|
||||
|
||||
Online:<https://api.tikhub.io/docs>
|
||||
|
||||
**_API demo:_**
|
||||
|
||||
- Crawl video data (TikTok or Douyin hybrid analysis)`https://api.douyin.wtf/api?url=[视频链接/Video URL]&minimal=false`
|
||||
- Download videos/photo albums (TikTok or Douyin hybrid analysis)`https://api.douyin.wtf/download?url=[视频链接/Video URL]&prefix=true&watermark=false`
|
||||
- Replace domain name to download videos/photo albums
|
||||
|
||||
|
||||
[抖音]
|
||||
原始链接:
|
||||
https://www.douyin.com/video/7159502929156705567
|
||||
替换域名:
|
||||
https://api.douyin.wtf/video/7159502929156705567
|
||||
# 返回无水印视频下载响应
|
||||
[TikTok]
|
||||
original link:
|
||||
https://www.tiktok.com/@evil0ctal/video/7156033831819037994
|
||||
Replace Domain:
|
||||
https://api.douyin.wtf/@evil0ctal/video/7156033831819037994
|
||||
# Return No Watermark Video Download Response
|
||||
|
||||
**_For more demonstrations, please view the document content..._**
|
||||
|
||||
## 💻Deployment (Method 1 Linux)
|
||||
|
||||
> 💡Tip: It is best to deploy this project to a server in the United States, otherwise strange BUGs may occur.
|
||||
|
||||
Recommended for everyone to use[Digitalocean](https://www.digitalocean.com/)servers, mainly because they are free.
|
||||
|
||||
Use my invitation link to sign up and you can get a $200 credit, and when you spend $25 on it, I can also get a $25 reward.
|
||||
|
||||
My invitation link:
|
||||
|
||||
<https://m.do.co/c/9f72a27dec35>
|
||||
|
||||
> Use script to deploy this project with one click
|
||||
|
||||
- Download using wget command[install.sh](https://raw.githubusercontent.com/Evil0ctal/Douyin_TikTok_Download_API/main/bash/install.sh)to the server and run
|
||||
|
||||
|
||||
wget -O install.sh https://raw.githubusercontent.com/Evil0ctal/Douyin_TikTok_Download_API/main/bash/install.sh && sudo bash install.sh
|
||||
|
||||
- It will be used automatically after running the Bash script[config.py](https://github.com/Evil0ctal/Douyin_TikTok_Download_API/blob/main/config.py)to help you modify[config.ini](https://github.com/Evil0ctal/Douyin_TikTok_Download_API/blob/main/config.ini)
|
||||
|
||||
```console
|
||||
Please edit config.ini, all input must be numbers!
|
||||
Default API port: 8000
|
||||
If you want use different port input new API port here: 80
|
||||
Use new port for web_api.py: 80
|
||||
Default API rate limit: 10/minute
|
||||
If you want use different rate limit input new rate limit here: 60
|
||||
Use new rate limit: 60/minute
|
||||
Default App port: 80
|
||||
If you want use different port input new App port here: 8080
|
||||
Use new port: 8080
|
||||
```
|
||||
|
||||
- The script will then ask you for the service you want to start.
|
||||
|
||||
api: start alone`web_api.py`
|
||||
|
||||
web: Start alone`web_app.py`
|
||||
|
||||
all: start at the same time`web_api.py`and`web_app.py`
|
||||
|
||||
```console
|
||||
Run API or Web? [api/web/all/quit] api
|
||||
Do you want to start the api service when system boot? [y/n] y
|
||||
Created symlink /etc/systemd/system/multi-user.target.wants/web_api.service → /etc/systemd/system/web_api.service.
|
||||
API service will start when system boot!
|
||||
Starting API...
|
||||
API is running! You can visit http://your_ip:port
|
||||
You can stop the api service by running: systemctl stop web_api.service
|
||||
```
|
||||
|
||||
> Start/stop service
|
||||
|
||||
- web service:`systemctl start/stop web_app.service`
|
||||
- api service:`systemctl start/stop web_api.service`
|
||||
|
||||
> Turn on/off automatic operation at startup
|
||||
|
||||
- web service:`systemctl enable/disable web_app.service`
|
||||
- api service:`systemctl enable/disable web_api.service`
|
||||
|
||||
> Update project
|
||||
|
||||
- `cd /www/wwwroot/Douyin_TikTok_Download_API/bash`
|
||||
- `sudo sh update.sh`
|
||||
|
||||
## 💽Deployment (Method 2 Docker)
|
||||
|
||||
> 💡Docker Image repo:[Docker Hub](https://hub.docker.com/repository/docker/evil0ctal/douyin_tiktok_download_api)
|
||||
|
||||
- Install docker
|
||||
|
||||
```yaml
|
||||
curl -fsSL get.docker.com -o get-docker.sh&&sh get-docker.sh &&systemctl enable docker&&systemctl start docker
|
||||
```
|
||||
|
||||
- Just leave the config.ini and docker-compose.yml files
|
||||
- Run the command to let the container run in the background
|
||||
|
||||
```yaml
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
- View container logs
|
||||
|
||||
```yaml
|
||||
docker logs -f douyin_tiktok_download_api
|
||||
```
|
||||
|
||||
- Delete container
|
||||
|
||||
```yaml
|
||||
docker rm -f douyin_tiktok_download_api
|
||||
```
|
||||
|
||||
- renew
|
||||
|
||||
```yaml
|
||||
docker-compose pull && docker-compose down && docker-compose up -d
|
||||
```
|
||||
|
||||
## ❤️ Contributor
|
||||
|
||||
[](https://github.com/Evil0ctal)[](https://github.com/jw-star)[](https://github.com/Jeffrey-deng)[](https://github.com/chris-ss)[](https://github.com/weixuan00)[](https://github.com/Tairraos)
|
||||
|
||||
## 📸Screenshot
|
||||
|
||||
**_API speed test (compared to official API)_**
|
||||
|
||||
<details><summary>🔎点击展开截图</summary>
|
||||
|
||||
Douyin official API:
|
||||
|
||||
API of this project:
|
||||
|
||||
TikTok official API:
|
||||
|
||||
API of this project:
|
||||
|
||||
</details>
|
||||
<hr>
|
||||
|
||||
**_Project interface_**
|
||||
|
||||
<details><summary>🔎点击展开截图</summary>
|
||||
|
||||
Web main interface:
|
||||
|
||||
|
||||
|
||||
Web main interface:
|
||||
|
||||
|
||||
|
||||
</details>
|
||||
<hr>
|
||||
|
||||
## 📜 Star History
|
||||
|
||||
[](https://star-history.com/#Evil0ctal/Douyin_TikTok_Download_API&Timeline)
|
||||
|
||||
[MY License](https://github.com/Evil0ctal/Douyin_TikTok_Download_API/blob/Stable/LICENSE)
|
||||
|
||||
> Start: 2021/11/06
|
||||
> GitHub:[@Evil0ctal](https://github.com/Evil0ctal)Contact:[Evil0ctal1985@gmail.com](mailto:Evil0ctal1985@gmail.com)
|
||||
@ -102,6 +102,7 @@
|
||||
|
||||
- 抖音（抖音海外版: TikTok）视频/图片解析
|
||||
- Bilibili视频解析
|
||||
- 西瓜视频解析
|
||||
- 网页端批量解析(支持抖音/TikTok混合提交)
|
||||
- 网页端解析结果页批量下载无水印视频(V3.X以上版本移除，请自行部署V2.X版本)
|
||||
- API调用获取链接数据
|
||||
@ -147,6 +148,13 @@ asyncio.run(hybrid_parsing(url=input("Paste Douyin/TikTok/Bilibili share URL her
|
||||
|
||||
> 💡提示：包含但不仅限于以下例子，如果遇到链接解析失败请开启一个新 [issue](https://github.com/Evil0ctal/Douyin_TikTok_Download_API/issues)
|
||||
|
||||
- 西瓜视频链接
|
||||
|
||||
```text
|
||||
https://www.ixigua.com/7270448082586698281/
|
||||
https://m.ixigua.com/video/7274710134306112054/
|
||||
```
|
||||
|
||||
- Bilibili视频链接
|
||||
|
||||
```text
|
||||
|
||||
146
scraper.py
146
scraper.py
@ -10,7 +10,7 @@
|
||||
# If this project is helpful to you, please give me a star, thank you!
|
||||
# @备注:
|
||||
# 核心代码，估值1块(๑•̀ㅂ•́)و✧
|
||||
# 用于爬取Douyin/TikTok/Bilibili的数据并以字典形式返回。
|
||||
# 用于爬取Douyin/TikTok/Bilibili/xigua的数据并以字典形式返回。
|
||||
# 如果本项目对您有帮助，请给我一个star，谢谢！
|
||||
|
||||
import re
|
||||
@ -23,7 +23,9 @@ import asyncio
|
||||
import traceback
|
||||
import configparser
|
||||
import urllib.parse
|
||||
import random
|
||||
|
||||
from zlib import crc32
|
||||
from typing import Union
|
||||
from tenacity import *
|
||||
|
||||
@ -47,6 +49,22 @@ class Scraper:
|
||||
self.bilibili_api_headers = {
|
||||
'User-Agent': 'com.ss.android.ugc.trill/494+Mozilla/5.0+(Linux;+Android+12;+2112123G+Build/SKQ1.211006.001;+wv)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Version/4.0+Chrome/107.0.5304.105+Mobile+Safari/537.36'
|
||||
}
|
||||
self.ixigua_api_headers = {
|
||||
'authority': 'ib.365yg.com',
|
||||
'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',
|
||||
'accept-language': 'zh-CN,zh;q=0.9',
|
||||
'cache-control': 'no-cache',
|
||||
'pragma': 'no-cache',
|
||||
'sec-ch-ua': '"Chromium";v="116", "Not)A;Brand";v="24", "Google Chrome";v="116"',
|
||||
'sec-ch-ua-mobile': '?0',
|
||||
'sec-ch-ua-platform': '"macOS"',
|
||||
'sec-fetch-dest': 'document',
|
||||
'sec-fetch-mode': 'navigate',
|
||||
'sec-fetch-site': 'none',
|
||||
'sec-fetch-user': '?1',
|
||||
'upgrade-insecure-requests': '1',
|
||||
'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36'
|
||||
}
|
||||
# 判断配置文件是否存在/Check if the configuration file exists
|
||||
if os.path.exists('config.ini'):
|
||||
self.config = configparser.ConfigParser()
|
||||
@ -210,6 +228,37 @@ class Scraper:
|
||||
else:
|
||||
print('该链接为原始链接,无需转换,原始链接为: {}'.format(url))
|
||||
return url
|
||||
elif 'ixigua.com' in url:
|
||||
"""
|
||||
西瓜视频链接类型(不全)：
|
||||
1. https://v.ixigua.com/ienrQ5bR/
|
||||
2. https://www.ixigua.com/7270448082586698281
|
||||
3. https://m.ixigua.com/video/7270448082586698281
|
||||
西瓜用户链接类型(不全)：
|
||||
1. https://www.ixigua.com/home/3189050062678823
|
||||
西瓜直播链接类型(不全)：
|
||||
"""
|
||||
if 'v.ixigua.com' in url:
|
||||
print('正在通过西瓜分享链接获取原始链接...')
|
||||
try:
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.get(url, headers=self.ixigua_api_headers, proxy=self.proxies, allow_redirects=False,
|
||||
timeout=10) as response:
|
||||
print("asdfasdf",response.headers)
|
||||
if response.status == 302:
|
||||
url = response.headers['Location'].split('?')[0] if '?' in response.headers[
|
||||
'Location'] else \
|
||||
response.headers['Location']
|
||||
print('获取原始链接成功, 原始链接为: {}'.format(url))
|
||||
return url
|
||||
except Exception as e:
|
||||
print('获取原始链接失败！')
|
||||
print(e)
|
||||
# return None
|
||||
raise e
|
||||
else:
|
||||
print('该链接为原始链接,无需转换,原始链接为: {}'.format(url))
|
||||
return url
|
||||
|
||||
"""__________________________________________⬇️Douyin methods(抖音方法)⬇️______________________________________"""
|
||||
|
||||
@ -365,7 +414,7 @@ class Scraper:
|
||||
|
||||
"""__________________________________________⬇️bilibili methods(Bilibili方法)⬇️______________________________________"""
|
||||
|
||||
# 获取TikTok视频ID/Get TikTok video ID
|
||||
# 获取bilibili视频ID/Get BiliBili video ID
|
||||
async def get_bilibili_video_id(self, original_url: str) -> Union[str, None]:
|
||||
"""
|
||||
获取视频id
|
||||
@ -427,6 +476,73 @@ class Scraper:
|
||||
except Exception as e:
|
||||
raise ValueError(f'获取BiliBili视频数据出错了:{e}')
|
||||
|
||||
|
||||
"""__________________________________________⬇️xigua methods(xigua方法)⬇️______________________________________"""
|
||||
# 获取西瓜拿播放地址的接口
|
||||
def get_xigua_json_url(self,video_id):
|
||||
# 获取json文件的地址
|
||||
r = str(random.random())[2:]
|
||||
url_part = "/video/urls/v/1/toutiao/mp4/{}?r={}".format(video_id, r)
|
||||
s = crc32(url_part.encode())
|
||||
json_url = "https://ib.365yg.com{}&s={}&nobase64=true".format(url_part, s)
|
||||
return json_url
|
||||
# 获取西瓜视频ID/Get xigua video ID
|
||||
async def get_ixigua_video_id(self, original_url: str) -> Union[str, None]:
|
||||
"""
|
||||
获取视频id
|
||||
:param original_url: 视频链接
|
||||
:return: 视频id
|
||||
"""
|
||||
try:
|
||||
# 转换链接/Convert link
|
||||
original_url = await self.convert_share_urls(original_url)
|
||||
# 获取视频ID/Get video ID
|
||||
if 'www.ixigua.com/' in original_url:
|
||||
video_id = re.findall('ixigua\.com/(\d+)', original_url)[0]
|
||||
elif 'm.ixigua.com/video' in original_url:
|
||||
video_id = re.findall('/video/(\d+)', original_url)[0]
|
||||
# 返回视频ID/Return video ID
|
||||
return video_id
|
||||
except Exception as e:
|
||||
raise ValueError(f'获取西瓜视频ID出错了:{e}')
|
||||
|
||||
@retry(stop=stop_after_attempt(4), wait=wait_fixed(7))
|
||||
async def get_ixigua_video_data(self, video_id: str) -> Union[dict, None]:
|
||||
"""
|
||||
获取单个视频信息
|
||||
:param video_id: 视频id
|
||||
:return: 视频信息
|
||||
"""
|
||||
print('正在获取西瓜视频数据...')
|
||||
try:
|
||||
# 构造访问链接/Construct the access link
|
||||
video_url = f'https://m.ixigua.com/video/{video_id}?wid_try=1'
|
||||
print("video_url",video_url)
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.get(video_url, headers=self.ixigua_api_headers, proxy=self.proxies,
|
||||
timeout=10) as response:
|
||||
response = await response.text()
|
||||
search = re.search("\"vid\":\"([^\"]+)\",", response)
|
||||
vid = search.group(1)
|
||||
print('获取视频vid信息成功！')
|
||||
play_url_api = self.get_xigua_json_url(vid)
|
||||
print(f"正在获取视频数据API: {play_url_api}")
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.get(play_url_api, headers=self.ixigua_api_headers, proxy=self.proxies,
|
||||
timeout=10) as response:
|
||||
response = await response.json()
|
||||
video_data = response.get("data",{}).get("video_list",{}).get("video_3",{}).get("main_url","")
|
||||
video_data = {
|
||||
'status': 'success',
|
||||
'message': "更多接口请查看(More API see): https://api.tikhub.io/",
|
||||
'type': 'video',
|
||||
'platform': '西瓜',
|
||||
'video_url': video_data,
|
||||
}
|
||||
return video_data
|
||||
except Exception as e:
|
||||
raise ValueError(f'获取西瓜视频数据出错了:{e}')
|
||||
|
||||
"""__________________________________________⬇️Hybrid methods(混合方法)⬇️______________________________________"""
|
||||
|
||||
# 判断链接平台/Judge link platform
|
||||
@ -435,6 +551,8 @@ class Scraper:
|
||||
url_platform = 'douyin'
|
||||
elif 'bilibili' in video_url:
|
||||
url_platform = 'bilibili'
|
||||
elif 'xigua' in video_url:
|
||||
url_platform = 'xigua'
|
||||
elif 'tiktok' in video_url:
|
||||
url_platform = 'tiktok'
|
||||
else:
|
||||
@ -456,6 +574,7 @@ class Scraper:
|
||||
video_id = await self.get_douyin_video_id(video_url) if url_platform == 'douyin' \
|
||||
else await self.get_tiktok_video_id(video_url) if url_platform == 'tiktok' \
|
||||
else await self.get_bilibili_video_id(video_url) if url_platform == 'bilibili' \
|
||||
else await self.get_ixigua_video_id(video_url) if url_platform == 'xigua' \
|
||||
else None
|
||||
|
||||
# 如果获取不到视频ID抛出异常/If the video ID cannot be obtained, an exception is thrown
|
||||
@ -467,6 +586,7 @@ class Scraper:
|
||||
data = await self.get_douyin_video_data(video_id) if url_platform == 'douyin' \
|
||||
else await self.get_tiktok_video_data(video_id) if url_platform == 'tiktok' \
|
||||
else await self.get_bilibili_video_data(video_id) if url_platform == 'bilibili' \
|
||||
else await self.get_ixigua_video_data(video_id) if url_platform == 'xigua' \
|
||||
else None
|
||||
|
||||
if data:
|
||||
@ -475,6 +595,10 @@ class Scraper:
|
||||
if url_platform == 'bilibili':
|
||||
print("获取Bilibili视频数据成功！")
|
||||
return data
|
||||
# 如果是西瓜平台则返回视频数据/If it is a ixigua platform, return video data
|
||||
if url_platform == 'xigua':
|
||||
print("获取西瓜视频数据成功！")
|
||||
return data
|
||||
|
||||
# 如果是抖音/TikTok平台则继续进行数据解析/If it is a Douyin/TikTok platform, continue to parse the data
|
||||
print(f"获取**{url_platform}**视频数据成功，正在判断数据类型...")
|
||||
@ -654,11 +778,19 @@ class Scraper:
|
||||
"""__________________________________________⬇️Test methods(测试方法)⬇️______________________________________"""
|
||||
|
||||
|
||||
async def async_test(_douyin_url: str = None, _tiktok_url: str = None, _bilibili_url: str = None) -> None:
|
||||
async def async_test(_douyin_url: str = None, _tiktok_url: str = None, _bilibili_url: str = None, _ixigua_url: str = None) -> None:
|
||||
# 异步测试/Async test
|
||||
start_time = time.time()
|
||||
print("<异步测试/Async test>")
|
||||
|
||||
print('\n--------------------------------------------------')
|
||||
print("正在测试异步获取西瓜视频ID方法...")
|
||||
ixigua_id = await api.get_ixigua_video_id(_ixigua_url)
|
||||
print(f"西瓜视频ID: {ixigua_id}")
|
||||
print("正在测试异步获取西瓜视频数据方法...")
|
||||
ixigua_data = await api.get_ixigua_video_data(ixigua_id)
|
||||
print(f"西瓜视频数据: {str(ixigua_data)[:100]}")
|
||||
|
||||
print('\n--------------------------------------------------')
|
||||
print("正在测试异步获取哔哩哔哩视频ID方法...")
|
||||
bilibili_id = await api.get_bilibili_video_id(_bilibili_url)
|
||||
@ -688,7 +820,8 @@ async def async_test(_douyin_url: str = None, _tiktok_url: str = None, _bilibili
|
||||
douyin_hybrid_data = await api.hybrid_parsing(_douyin_url)
|
||||
tiktok_hybrid_data = await api.hybrid_parsing(_tiktok_url)
|
||||
bilibili_hybrid_data = await api.hybrid_parsing(_bilibili_url)
|
||||
print(f"抖音、TikTok、哔哩哔哩混合解析全部成功！")
|
||||
xigua_hybrid_data = await api.hybrid_parsing(_ixigua_url)
|
||||
print(f"抖音、TikTok、哔哩哔哩、西瓜混合解析全部成功！")
|
||||
|
||||
print('\n--------------------------------------------------')
|
||||
# 总耗时/Total time
|
||||
@ -704,4 +837,7 @@ if __name__ == '__main__':
|
||||
douyin_url = 'https://v.douyin.com/rLyrQxA/6.66'
|
||||
tiktok_url = 'https://www.tiktok.com/@evil0ctal/video/7217027383390555438'
|
||||
bilibili_url = "https://www.bilibili.com/video/BV1Th411x7ii/"
|
||||
asyncio.run(async_test(_douyin_url=douyin_url, _tiktok_url=tiktok_url, _bilibili_url=bilibili_url))
|
||||
ixigua_url = "https://www.ixigua.com/7270448082586698281"
|
||||
# ixigua_url = "ttps://v.ixigua.com/ienrQ5bR/" # convert_share_urls 这里有bug 如果抖音的口令解析的出来其他的都是none
|
||||
asyncio.run(async_test(_douyin_url=douyin_url, _tiktok_url=tiktok_url, _bilibili_url=bilibili_url, _ixigua_url=ixigua_url))
|
||||
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user