2025-04-22 07:06:37 +08:00
9 changed files with 82 additions and 135 deletions
--- a/33
+++ b/33
@ -1,26 +1,33 @@
-# 使用官方 Python 3.11 的轻量版镜像
-FROM python:3.11-slim
-
+# Use the official Ubuntu base image
+FROM ubuntu:jammy
 LABEL maintainer="Evil0ctal"

-# 设置非交互模式，避免 Docker 构建时的交互问题
+# Set non-interactive frontend (useful for Docker builds)
 ENV DEBIAN_FRONTEND=noninteractive

-# 设置工作目录
+# Update the package list and install Python and pip
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    python3.11 \
+    python3-pip \
+    python3.11-dev \
+    && apt-get clean \
+    && rm -rf /var/lib/apt/lists/*
+
+# Set a working directory
 WORKDIR /app

-# 复制应用代码到容器
+# Copy the application source code to the container
 COPY . /app

-# 使用 Aliyun 镜像源加速 pip
-RUN pip install -i https://mirrors.aliyun.com/pypi/simple/ -U pip \
-    && pip config set global.index-url https://mirrors.aliyun.com/pypi/simple/
+# Install pip and set the PyPI mirror (Aliyun)
+RUN pip3 install -i https://mirrors.aliyun.com/pypi/simple/ -U pip \
+    && pip3 config set global.index-url https://mirrors.aliyun.com/pypi/simple/

-# 安装依赖
-RUN pip install --no-cache-dir -r requirements.txt
+# Install dependencies directly
+RUN pip3 install --no-cache-dir -r requirements.txt

-# 确保启动脚本可执行
+# Make the start script executable
 RUN chmod +x start.sh

-# 设置容器启动命令
+# Command to run on container start
 CMD ["./start.sh"]
--- a/README.en.md
+++ b/README.en.md
@ -50,8 +50,8 @@ _Download videos that are prohibited from being downloaded, perform data analysi

 ## 🔊 V4 version notes

-   If you are interested in writing this project, please add WeChat`Evil0ctal`Note: Github project reconstruction, everyone can communicate and learn from each other in the group, and do not allow advertisements or illegal things to be made purely friends and technical communication.
-   This project uses`X-Bogus`Algorithm and`A_Bogus`The algorithm requests TikTok and TikTok's Web API.
+-   If you are interested in writing this project, please add WeChat.`Evil0ctal`Note: Github project reconstruction, everyone can communicate and learn from each other in the group, and do not allow advertisements or illegal things to be made purely friends and technical communication.
+-   This project uses`X-Bogus`Algorithm and`A_Bogus`The algorithm requests the web APIs of TikTok and TikTok.
 -   Due to Douyin's risk control, please go to**Get the Douyin website cookies in the browser and replace them in config.yaml.**
 -   Please read the document below before asking for an issue, and most solutions to the problem will be included in the document.
 -   This project is completely free, but please follow it when using it:[Apache-2.0 license](https://github.com/Evil0ctal/Douyin_TikTok_Download_API?tab=Apache-2.0-1-ov-file#readme)
@ -64,11 +64,11 @@ _Download videos that are prohibited from being downloaded, perform data analysi

 > 📦 Out of the box

-Simplify the usage process and quickly carry out development work using the encapsulated SDK. All API interfaces are designed according to the RESTful architecture and are described and documented using the OpenAPI specification, accompanied by example parameters to ensure that calls are easier.
+Simplify the usage process and quickly carry out development work using the encapsulated SDK. All API interfaces are designed according to the RESTful architecture and are described and documented using the OpenAPI specification, with example parameters to ensure that calls are easier.

 > 💰 Cost Advantage

-There is no preset package limit, no monthly usage threshold, all consumption is billed instantly based on the actual usage, and is billed step by step based on the user's daily request volume. At the same time, you can check in in the user's background through daily check-in, and these free amounts will not expire.
+There is no preset package limit, no monthly usage threshold, all consumption is billed instantly based on the actual usage, and the user's daily request volume is charged step by step. At the same time, you can check in in the user's background by checking in every day. , and these free amounts will not expire.

 > ⚡️ Quick support

@ -101,7 +101,7 @@ Some of the source code of TikHub will be open sourced on Github and will sponso

 -   [Johnserf-Seed/Tiktokdownload](https://github.com/Johnserf-Seed/TikTokDownload)
 -   [HFrost0/bilix](https://github.com/HFrost0/bilix)
-   [Tairraos/TikDown - \[Updated to be\]](https://github.com/Tairraos/TikDown/)
+-   [Tairraos/TikDown - \[Updated required\]](https://github.com/Tairraos/TikDown/)

 ## ⚗️Technology Stack

@ -148,8 +148,8 @@ Some of the source code of TikHub will be open sourced on Github and will sponso

 -   Batch analysis on the web side (supports Douyin/TikTok hybrid analysis)
 -   Download videos or albums online.
-   Production[pip package](https://pypi.org/project/douyin-tiktok-scraper/)方便快速导入你的项目
-   [iOS shortcuts to quickly call API](https://apps.apple.com/cn/app/%E5%BF%AB%E6%8D%B7%E6%8C%87%E4%BB%A4/id915249334)Implement watermark-free videos/pictures in-app download
+-   Production[pip package](https://pypi.org/project/douyin-tiktok-scraper/)Easy and quick import of your project
+-   [iOS shortcuts to quickly call API](https://apps.apple.com/cn/app/%E5%BF%AB%E6%8D%B7%E6%8C%87%E4%BB%A4/id915249334)Implement watermark-free video/picture collection in-app download
 -   Complete API documentation ([Demo/Demo](https://api.douyin.wtf/docs))
 -   Rich API interfaces:
    -   TikTok web version API
@ -184,8 +184,8 @@ Some of the source code of TikHub will be open sourced on Github and will sponso
        -   [x] Get user homepage information
        -   [x] Get the user's homepage fan data
        -   [x] Get user's homepage follow data
-        -   [x] 获取用户主页合辑作品数据
-        -   [x] Get search data for users' homepage
+        -   [x] Obtain data on the collection of works by users on the homepage
+        -   [x] Get user home page search data
        -   [x] Get user homepage playlist data
        -   [x] Get individual video comment data
        -   [x] Get comments and response data for specified videos
@ -314,7 +314,7 @@ Online:<https://api.douyin.wtf/docs>
    -   <https://github.com/Evil0ctal/Douyin_TikTok_Download_API/blob/30e56e5a7f97f87d60b1045befb1f6db147f8590/crawlers/tiktok/web/config.yaml#L6>
 -   I turned off the online download function of the demonstration site. Someone downloaded a huge video and it crashed directly on my server. You can right-click to save the video on the web parsing result page...
 -   The cookies on the demo site are my own and are not guaranteed to be valid for a long time. They only serve as a demonstration. If you deploy it yourself, please get the cookies yourself.
-   HTTP 403 error will occur if you need to access the video link returned by TikTok Web API. Please use the API in this project`/api/download`The interface downloads TikTok videos. This interface has been manually closed in the demonstration site, and you need to deploy this project yourself.
+-   HTTP 403 error occurs if you need to access the video link returned by TikTok Web API. Please use the API in this project`/api/download`The interface downloads TikTok videos. This interface has been manually closed in the demonstration site, and you need to deploy this project yourself.
 -   There is one here**Video tutorial**You can refer to:**_<https://www.bilibili.com/video/BV1vE421j7NR/>_**

 ## 💻Deployment (Method 1 Linux)
@ -397,7 +397,7 @@ Use the following command to check if your container is running:
 docker ps
 ```

-这将列出所有活动容器。查找`douyin_tiktok_api `to confirm its normal operation.
+This lists all active containers. Find`douyin_tiktok_api `to confirm its normal operation.

 ### Step 4: Access the application

--- a/app/api/endpoints/download.py
+++ b/app/api/endpoints/download.py
@ -4,7 +4,7 @@ import zipfile
 import aiofiles
 import httpx
 import yaml
-from fastapi import APIRouter, Request, Query, HTTPException  # 导入FastAPI组件
+from fastapi import APIRouter, Request, Query  # 导入FastAPI组件
 from starlette.responses import FileResponse

 from app.api.models.APIResponseModel import ErrorResponseModel  # 导入响应模型
@ -18,6 +18,7 @@ config_path = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(os.pa
 with open(config_path, 'r', encoding='utf-8') as file:
    config = yaml.safe_load(file)

+
 async def fetch_data(url: str, headers: dict = None):
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
@ -27,26 +28,6 @@ async def fetch_data(url: str, headers: dict = None):
        response.raise_for_status()  # 确保响应是成功的
        return response

-# 下载视频专用
-async def fetch_data_stream(url: str, request:Request , headers: dict = None, file_path: str = None):
-    headers = {
-        'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
-    } if headers is None else headers.get('headers')
-    async with httpx.AsyncClient() as client:
-        # 启用流式请求
-        async with client.stream("GET", url, headers=headers) as response:
-            response.raise_for_status()
-
-            # 流式保存文件
-            async with aiofiles.open(file_path, 'wb') as out_file:
-                async for chunk in response.aiter_bytes():
-                    if await request.is_disconnected():
-                        print("客户端断开连接，清理未完成的文件")
-                        await out_file.close()
-                        os.remove(file_path)
-                        return False
-                    await out_file.write(chunk)
-            return True

@router.get("/download", summary="在线下载抖音|TikTok视频/图片/Online download Douyin|TikTok video/image")
 async def download_file_hybrid(request: Request,
@ -123,18 +104,11 @@ async def download_file_hybrid(request: Request,

            # 获取视频文件
            __headers = await HybridCrawler.TikTokWebCrawler.get_tiktok_headers() if platform == 'tiktok' else await HybridCrawler.DouyinWebCrawler.get_douyin_headers()
-            # response = await fetch_data(url, headers=__headers)
+            response = await fetch_data(url, headers=__headers)

-            success = await fetch_data_stream(url, request, headers=__headers, file_path=file_path)
-            if not success:
-                raise HTTPException(
-                    status_code=500,
-                    detail="An error occurred while fetching data"
-                )
-
-            # # 保存文件
-            # async with aiofiles.open(file_path, 'wb') as out_file:
-            #     await out_file.write(response.content)
+            # 保存文件
+            async with aiofiles.open(file_path, 'wb') as out_file:
+                await out_file.write(response.content)

            # 返回文件内容
            return FileResponse(path=file_path, filename=file_name, media_type="video/mp4")
--- a/config.yaml
+++ b/config.yaml
@ -30,8 +30,8 @@ API:
  Redoc_URL: /redoc    # API documentation URL | API文档URL

  # API Information
-  Version: V4.1.2    # API version | API版本
-  Update_Time: 2025/03/16    # API update time | API更新时间
+  Version: V4.1.0    # API version | API版本
+  Update_Time: 2025/02/15    # API update time | API更新时间
  Environment: Demo    # API environment | API环境

  # Download Configuration
--- a/crawlers/douyin/web/config.yaml
+++ b/crawlers/douyin/web/config.yaml
@ -2,21 +2,15 @@ TokenManager:
  douyin:
    headers:
      Accept-Language: zh-CN,zh;q=0.8,zh-TW;q=0.7,zh-HK;q=0.5,en-US;q=0.3,en;q=0.2
-      # 不要这里的修改User-Agent，请保持默认，否则会导致请求失败。
-      # Do not modify User-Agent here, please keep the default, otherwise it will cause request failure.
      User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36
      Referer: https://www.douyin.com/
-      # 你唯一需要修改的地方就是这里的Cookie，然后保存后重启程序即可。
-        # The only place you need to modify is the Cookie here, and then save and restart the program.
-      Cookie: __ac_nonce=067d687ac00d70af16eab; __ac_signature=_02B4Z6wo00f018O6kmgAAIDAR1H8JrcivBPDi5bAAJdBcf; ttwid=1%7C46sVJ6G5zO0ZRKBqbFef2B13U3CqP9gLwQEH8IV2y6A%7C1742112685%7Cae649397cca7dde21884d5f8e3e3d53eb2361aa64af04cd6889fa71d7f23344b; UIFID_TEMP=986fab8dfc2c74111fac2b883dbdee67777473ded35e2c4bebbf68cc8b91739da61f6b365ad9795b0aa3a8bddce6cc3e39c5d4fd4bad667aaefd3d3ec08baac66fe3b215343f12d8aae84e0a24048f44; douyin.com; device_web_cpu_core=16; device_web_memory_size=-1; architecture=amd64; hevc_supported=true; IsDouyinActive=true; home_can_add_dy_2_desktop=%220%22; dy_swidth=1835; dy_sheight=1147; stream_recommend_feed_params=%22%7B%5C%22cookie_enabled%5C%22%3Atrue%2C%5C%22screen_width%5C%22%3A1835%2C%5C%22screen_height%5C%22%3A1147%2C%5C%22browser_online%5C%22%3Atrue%2C%5C%22cpu_core_num%5C%22%3A16%2C%5C%22device_memory%5C%22%3A0%2C%5C%22downlink%5C%22%3A%5C%22%5C%22%2C%5C%22effective_type%5C%22%3A%5C%22%5C%22%2C%5C%22round_trip_time%5C%22%3A0%7D%22; strategyABtestKey=%221742112685.842%22; volume_info=%7B%22isUserMute%22%3Afalse%2C%22isMute%22%3Afalse%2C%22volume%22%3A0.5%7D; stream_player_status_params=%22%7B%5C%22is_auto_play%5C%22%3A0%2C%5C%22is_full_screen%5C%22%3A0%2C%5C%22is_full_webscreen%5C%22%3A0%2C%5C%22is_mute%5C%22%3A0%2C%5C%22is_speed%5C%22%3A1%2C%5C%22is_visible%5C%22%3A1%7D%22; xgplayer_user_id=835787001711; fpk1=U2FsdGVkX19Ke0llbjXpGOOr1Jeel/2GnaSJz41VO3mAFs271jC0hG7gdWlk+2pYLM4GF8TVGtwClCJIXsTKUw==; fpk2=2333b8d335abc6e14aef1caed0ae26fc; s_v_web_id=verify_m8bcww86_XfwSCnmj_5i3F_4Joq_8edO_9gRH9JENh07f; csrf_session_id=6f34e666e71445c9d39d8d06a347a13f; FORCE_LOGIN=%7B%22videoConsumedRemainSeconds%22%3A180%7D; biz_trace_id=c34e5eaf; passport_csrf_token=ab84b3e39ad78e719b236035a27379c0; passport_csrf_token_default=ab84b3e39ad78e719b236035a27379c0; __security_mc_1_s_sdk_crypt_sdk=ac2d56c3-44cd-a161; __security_mc_1_s_sdk_cert_key=ccf2bd2d-4718-b8de; __security_mc_1_s_sdk_sign_data_key_web_protect=9995d368-4e45-b17f; bd_ticket_guard_client_data=eyJiZC10aWNrZXQtZ3VhcmQtdmVyc2lvbiI6MiwiYmQtdGlja2V0LWd1YXJkLWl0ZXJhdGlvbi12ZXJzaW9uIjoxLCJiZC10aWNrZXQtZ3VhcmQtcmVlLXB1YmxpYy1rZXkiOiJCUHR2ZDlUeGU4UlhPaWdIczFqaStJWityQlF4UWZMKytiL2drbXlYUmNrelNua1lQUjJTRVZHVlo4MWFCU0EvSW4xSnBmbzN3TFlvSnhIZTZTV29DTmc9IiwiYmQtdGlja2V0LWd1YXJkLXdlYi12ZXJzaW9uIjoyfQ%3D%3D; bd_ticket_guard_client_web_domain=2; xg_device_score=8.208487995540095; sdk_source_info=7e276470716a68645a606960273f276364697660272927676c715a6d6069756077273f276364697660272927666d776a68605a607d71606b766c6a6b5a7666776c7571273f275e58272927666a6b766a69605a696c6061273f27636469766027292762696a6764695a7364776c6467696076273f275e5827292771273f27303035353c3337343437313234272927676c715a75776a716a666a69273f2763646976602778; bit_env=LVdHnIescW9BCGpo5gGuqIlwNfgj757SBdMhdZXBSWjPWbxp9Nv_B2vUt_LtEvr-ioRv0E9b8N8HWiOHe20JqcUhO4YmpIM6gB83hjgqZfmAhYEbzJR7z2bRViJaPg4xeoyGhwdjwK_Bzogp6uoUs4ov-P4JYzMh78i7jaY5Pzd6h3CaVO-eUKnTiFfUlJo_jmhSfHXGdwkurXwR4lO_UnU4Loqa0YlmDiyi0fPxURFIN5t4Ny6Ua8LLSYcUrBXHlXoQ5G4bQN4XqwuWwT9YauexXbkotU1Jv8pMJUiAhlFIMjbvfTutTSnOXJLoH_JsR_doifURl0wf8CIa_OcYw-A2VglrpbaFU6HDVTKbSRKovzIMY9bUwl_4EAiLBf87g2BU0Uz1MHd_lGNdH3ImEWpLtdRvUsW_KD7q87rPsEGVTceyQ5U3ZlETqoEOwOiggNGu5lL_1O8lt8_7eydeGA%3D%3D; gulu_source_res=eyJwX2luIjoiM2Y3NGJhZDgxMzc3OThkNmVkN2U5ZjM3NDMzNGJkYjMwNzRhYjI0ZWJhMDZkMzdmYWNiNjgzNTY2ZjY0OGUyNCJ9; passport_auth_mix_state=c534f2qcgpohqv4juisp74wq28e90snz
+      Cookie: __ac_nonce=067b05bb4001bbfaa98d; __ac_signature=_02B4Z6wo00f01rhd1MAAAIDCpLyUsaBwu2a4bNBAAMmz80; ttwid=1%7C1Ahlpf0Qc1ppzov9_hNnTCygaUPOPOs3UvMydUd_6HU%7C1739611060%7C3a557fe631bbc1e74c85d8a483a81997064fc373a2ca7e682fa4b06d73a828b4; UIFID_TEMP=209b86ca33829c7d9c7b7c40c5eb89f829cca190227d7391efa903940cb34a3cd87c24c7642ef0293896efa854abc537f39465d8bd4b9983ec9c483a5895894b1988ee54c22558e836c75e7d7bc8e638; douyin.com; device_web_cpu_core=16; device_web_memory_size=-1; architecture=amd64; hevc_supported=true; IsDouyinActive=true; home_can_add_dy_2_desktop=%220%22; dy_swidth=1835; dy_sheight=1147; stream_recommend_feed_params=%22%7B%5C%22cookie_enabled%5C%22%3Atrue%2C%5C%22screen_width%5C%22%3A1835%2C%5C%22screen_height%5C%22%3A1147%2C%5C%22browser_online%5C%22%3Atrue%2C%5C%22cpu_core_num%5C%22%3A16%2C%5C%22device_memory%5C%22%3A0%2C%5C%22downlink%5C%22%3A%5C%22%5C%22%2C%5C%22effective_type%5C%22%3A%5C%22%5C%22%2C%5C%22round_trip_time%5C%22%3A0%7D%22; strategyABtestKey=%221739611062.948%22; volume_info=%7B%22isUserMute%22%3Afalse%2C%22isMute%22%3Afalse%2C%22volume%22%3A0.5%7D; stream_player_status_params=%22%7B%5C%22is_auto_play%5C%22%3A0%2C%5C%22is_full_screen%5C%22%3A0%2C%5C%22is_full_webscreen%5C%22%3A0%2C%5C%22is_mute%5C%22%3A0%2C%5C%22is_speed%5C%22%3A1%2C%5C%22is_visible%5C%22%3A1%7D%22; csrf_session_id=6f34e666e71445c9d39d8d06a347a13f; fpk1=U2FsdGVkX1+2RhnaraBP7MvLFGHizn1/4jXb26GNeMYX+5InWD2xSi+Nxlid7bvkc/8M93cY00/6N7eqKGHmQA==; fpk2=b0fc1a0934e7ea864f39ca0a0b863cfa; FORCE_LOGIN=%7B%22videoConsumedRemainSeconds%22%3A180%7D; s_v_web_id=verify_m75zigjp_tbNk5QQC_Noqt_4v2V_AUnV_kN8fMu23eUGR; biz_trace_id=b97111de; passport_csrf_token=0b71f941cda73bc8b22eaf9e9c4d2e40; passport_csrf_token_default=0b71f941cda73bc8b22eaf9e9c4d2e40; __security_mc_1_s_sdk_crypt_sdk=f51837e1-4b8d-94df; __security_mc_1_s_sdk_cert_key=0c011109-446b-be55; __security_mc_1_s_sdk_sign_data_key_web_protect=abb27eaf-4917-a163; bd_ticket_guard_client_data=eyJiZC10aWNrZXQtZ3VhcmQtdmVyc2lvbiI6MiwiYmQtdGlja2V0LWd1YXJkLWl0ZXJhdGlvbi12ZXJzaW9uIjoxLCJiZC10aWNrZXQtZ3VhcmQtcmVlLXB1YmxpYy1rZXkiOiJCQS84MEZDcW14NzhZck1UaGFrb0JISlJvQk9vbmlLcWdiZTV6cVdyODBoQ1R3ZWxnbmJVTUxMcjVOSkxlMDgyNlc1ZGdsNCtreEZvcVdJNWZPRmNUM289IiwiYmQtdGlja2V0LWd1YXJkLXdlYi12ZXJzaW9uIjoyfQ%3D%3D; bd_ticket_guard_client_web_domain=2; sdk_source_info=7e276470716a68645a606960273f276364697660272927676c715a6d6069756077273f276364697660272927666d776a68605a607d71606b766c6a6b5a7666776c7571273f275e58272927666a6b766a69605a696c6061273f27636469766027292762696a6764695a7364776c6467696076273f275e5827292771273f2730313d3c33353434333c363234272927676c715a75776a716a666a69273f2763646976602778; bit_env=mGFpFZLi300fY_vW8MzaQj6m1TdQG9NP6-OJOAerJcquX959KddzpxfoQ-I8Z7nF0ABMXRBCcpGqZUPRiQ8wUjjVl26ODhla6gV2Yl_uU2CA6hjBoQTsKZVhsNHoN2dCQXimgmxIMxrW_L3oYbCQxaz-Tkb5Ivk9FEZJF5lzAW7MbsMtWZcZWMG00U81pnXMp0W_Pe4PzpKOoQlJyLjkzZg2HoFGxhXswJ73jyoGc_amdhqk7PAufAOnPvPKN7ISZxVaFih54sSCkmxC7Pv_v70iZX3XS6HmYjjS4sPZbZrhsM-KjDxDgUFC73AXX-ZH9zXmFL-0GzUJ5OjqdkTHIuAtyqasWopmANJ5C5xAkSqgmHnjq8tKYsvduXm4EBQKmvyDZqLYoPrPJVXJhvTdI3c3YIKi9k4xUvVvhwLywoaAt0JHq8XIDYXPmQyuf7LlrKrJKqoDKugidn7x_jCcdg%3D%3D; gulu_source_res=eyJwX2luIjoiZTgyMzRmN2M0N2RiYzI0ZWY1ZWYwMzMyY2U5YTI3MjllNjhmMGUyOTc0NTdiOTNjNDJkNzQ0ZGEyYTBiNWU4ZCJ9; passport_auth_mix_state=0ucvtyinbma6mx3rkqbnf8p7tslrg76mnee54d1250ugzi8a

    proxies:
      http:
      https:

    msToken:
-        # 不要修改下面的内容。
-        # Do not modify the content below.
      url: https://mssdk.bytedance.com/web/report
      magic: 538969122
      version: 1
@ -25,7 +19,5 @@ TokenManager:
      User-Agent: 5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36 Edg/117.0.2045.47

    ttwid:
-        # 不要修改下面的内容。
-        # Do not modify the content below.
      url: https://ttwid.bytedance.com/ttwid/union/register/
      data: '{"region":"cn","aid":1768,"needFid":false,"service":"www.ixigua.com","migrate_info":{"ticket":"","source":"node"},"cbUrlProtocol":"https","union":true}'
--- a/crawlers/douyin/web/utils.py
+++ b/crawlers/douyin/web/utils.py
@ -406,7 +406,6 @@ class SecUserIdFetcher:
 class AwemeIdFetcher:
    # 预编译正则表达式
    _DOUYIN_VIDEO_URL_PATTERN = re.compile(r"video/([^/?]*)")
-    _DOUYIN_VIDEO_URL_PATTERN_NEW = re.compile(r"[?&]vid=(\d+)")
    _DOUYIN_NOTE_URL_PATTERN = re.compile(r"note/([^/?]*)")
    _DOUYIN_DISCOVER_URL_PATTERN = re.compile(r"modal_id=([0-9]+)")

@ -419,44 +418,62 @@ class AwemeIdFetcher:
            url (str): 输入的url (Input url)

        Returns:
-            str: 匹配到的aweme_id (Matched aweme_id)
+            str: 匹配到的aweme_id (Matched aweme_id)。
        """

        if not isinstance(url, str):
            raise TypeError("参数必须是字符串类型")

+        # 提取有效URL
+        url = extract_valid_urls(url)
+
+        if url is None:
+            raise (
+                APINotFoundError("输入的URL不合法。类名：{0}".format(cls.__name__))
+            )
+
        # 重定向到完整链接
        transport = httpx.AsyncHTTPTransport(retries=5)
        async with httpx.AsyncClient(
-                transport=transport, proxy=None, timeout=10
+                transport=transport, proxies=TokenManager.proxies, timeout=10
        ) as client:
            try:
                response = await client.get(url, follow_redirects=True)
                response.raise_for_status()

-                response_url = str(response.url)
+                video_pattern = cls._DOUYIN_VIDEO_URL_PATTERN
+                note_pattern = cls._DOUYIN_NOTE_URL_PATTERN
+                discover_pattern = cls._DOUYIN_DISCOVER_URL_PATTERN

-                # 按顺序尝试匹配视频ID
-                for pattern in [
-                    cls._DOUYIN_VIDEO_URL_PATTERN,
-                    cls._DOUYIN_VIDEO_URL_PATTERN_NEW,
-                    cls._DOUYIN_NOTE_URL_PATTERN,
-                    cls._DOUYIN_DISCOVER_URL_PATTERN
-                ]:
-                    match = pattern.search(response_url)
+                # 2024-4-22
+                # 嵌套如果超过3层需要修改此处代码 (If the nesting exceeds 3 layers, you need to modify this code)
+                match = video_pattern.search(str(response.url))
+                if video_pattern.search(str(response.url)):
+                    aweme_id = match.group(1)
+                else:
+                    match = note_pattern.search(str(response.url))
                    if match:
-                        return match.group(1)
-
-                raise APIResponseError("未在响应的地址中找到 aweme_id，检查链接是否为作品页")
+                        aweme_id = match.group(1)
+                    else:
+                        match = discover_pattern.search(str(response.url))
+                        if match:
+                            aweme_id = match.group(1)
+                        else:
+                            raise APIResponseError(
+                                "未在响应的地址中找到aweme_id，检查链接是否为作品页"
+                            )
+                return aweme_id

            except httpx.RequestError as exc:
-                raise APIConnectionError(
-                    f"请求端点失败，请检查当前网络环境。链接：{url}，代理：{TokenManager.proxies}，异常类名：{cls.__name__}，异常详细信息：{exc}"
-                )
+                # 捕获所有与 httpx 请求相关的异常情况 (Captures all httpx request-related exceptions)
+                raise APIConnectionError("请求端点失败，请检查当前网络环境。 链接：{0}，代理：{1}，异常类名：{2}，异常详细信息：{3}"
+                                         .format(url, TokenManager.proxies, cls.__name__, exc)
+                                         )

            except httpx.HTTPStatusError as e:
-                raise APIResponseError(
-                    f"链接：{e.response.url}，状态码 {e.response.status_code}：{e.response.text}"
+                raise APIResponseError("链接：{0}，状态码 {1}：{2} ".format(
+                    e.response.url, e.response.status_code, e.response.text
+                )
                )

    @classmethod
--- a/crawlers/tiktok/web/config.yaml
+++ b/crawlers/tiktok/web/config.yaml
@ -3,8 +3,6 @@ TokenManager:
    headers:
      User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36
      Referer: https://www.tiktok.com/
-      # 你唯一需要修改的地方就是这里的Cookie，然后保存后重启程序即可。
-        # The only place you need to modify is the Cookie here, and then save and restart the program.
      Cookie: tt_csrf_token=bwnaRGd9-B-0ce8ntqw9jtGzAdvzTRKNpBl0; ak_bmsc=75A1956756DE42FD14ED069AAE7A8780~000000000000000000000000000000~YAAQXCw+F8jpmBGQAQAAIfGsFBj+ZEGzR/ZeiuPpMtItu0QQUQRmjBX2kADliy6QA9rZSfrxRUZc9zuRrI4/xbIrAwA/nkdguGpa+v3QSn/1sk5uP2aqLVm0eYB/SGNafa2h2QvIPbLNiSCRhgq1GalZJL4+udqDnyBRJWE74nin74bZwrVDvCX1s8M2hWqZ9/jTkdm4sfwON9MdJIEtjAPlddQ4gxoqjPoWhfnrm24dhPT4OjL1B8QP1mgurj7zJGspqD53VcjkAl65gHVxp3dwZ5WbPYpqrh9j8wo2u/Wh6uhX+0HWmkv5yVZyTyYQTl3/ilPp9G4CuIUi84gaPLjNYea9AEnphNX0ywzDa6/yegfqyE6r3wqBBDCrR1xRM98YEB4A5PV7pw==; tt_chain_token=ljZFLdRDfyfDflXMg5XGpg==; tiktok_webapp_theme_auto_dark_ab=1; tiktok_webapp_theme=dark; perf_feed_cache={%22expireTimestamp%22:1718503200000%2C%22itemIds%22:[%227348816520216186158%22%2C%227356022137678810410%22%2C%227349561209340857630%22]}; s_v_web_id=verify_lxe3l432_JnDE5WWo_URef_4WrS_88IM_fd1CqEXZs4dZ; passport_csrf_token=af197f073ed95f4dc2636f24d55566a6; passport_csrf_token_default=af197f073ed95f4dc2636f24d55566a6; ttwid=1%7CuNT4GcgvvOjH8rTETh9d9xti_QDJjlcnSK2V7djIpuc%7C1718333954%7Cf81b989a495aedff91302da4d0a3ab6055dea486fb203a4326b37d5a5346ad0c; msToken=1Mhpyi8MlaZjM6bbLDVUhCj_6C0kEO_1_Nb62ByXLg7wy_vLnBxdMFpKclhf4HYnEjCghk2Gq47ZM5jPj3L1yFxQUZvq4oPLo1b2Wfe_33RE94uIxdiR-eSueWbcYDDgOj1Pn9Wyid5Uf5fzBQ7xxFA=; bm_sv=9ADBA7BE06EC41817F117E2279F1410C~YAAQXCw+F8bsmBGQAQAAzSewFBg2fP3Zd0aky2x7S13D97O64xi8EXhoKORBnPQyCHlh0iSlh63FFjoy6peDWaF3lkWaTly3Z7I7WvWk1GCntnYzpJaSCE5EO2OL38zPWpHcgGWuekluvptHXsheedNEefN4SUHVMt4jJynWNeTKrao0RmNLkH4zGs7QO6+MPCt94QFvNfLjBRr0wVcXlN/hx9m6kcvCyzsBBqEnpugoYvZ0SMA+INsKI5PDfQz1~1; msToken=449_l3kdcLmnEHdDP0uACa5EcPVL1NbpjyVv8yah61EwxIPZRDlGwpGIkpIjH0Tk-CDtoKwFrDdP1v2AOpwmdoIz5oQzPEXCdyfGzcVXCHbwMX1fwPxMHpea5yFPUYEDlNWaCFlgLnejRdWeN5sB_lE=

    proxies:
@ -12,8 +10,6 @@ TokenManager:
      https:

    msToken:
-        # 不要修改下面的内容。
-        # Do not modify the content below.
      url: https://mssdk.tiktokw.us/web/report?msToken=1Ab-7YxR9lUHSem0PraI_XzdKmpHb6j50L8AaXLAd2aWTdoJCYLfX_67rVQFE4UwwHVHmyG_NfIipqrlLT3kCXps-5PYlNAqtdwEg7TrDyTAfCKyBrOLmhMUjB55oW8SPZ4_EkNxNFUdV7MquA==
      magic: 538969122
      version: 1
@ -22,8 +18,6 @@ TokenManager:
      User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36

    ttwid:
-        # 不要修改下面的内容。
-        # Do not modify the content below.
      url: https://www.tiktok.com/ttwid/check/
      data: '{"aid":1988,"service":"www.tiktok.com","union":false,"unionHost":"","needFid":false,"fid":"","migrate_priority":0}'
      cookie: tt_csrf_token=YmksDB6a-h4cT2fF7JpORI2O9UBMCWjsntIc; ttwid=1%7C0FVb9fFc-sjDG2UdJwdC1AirqYozQ0xfbAS4N72vN2Y%7C1713886256%7C78a9d83445b82b73ca8d4e0cf024ea6cdf1329b7f3866c826b0a69a300ebce46; ak_bmsc=51B1D53481A3A4E4D0CEFF2BCF622DA2~000000000000000000000000000000~YAAQ7uIsF6c4j+SOAQAAANmUCxfRGVXZ4D9xnO97l1yDw0OWyomnVkNY7IUKaggUja0kQzFQ+WG4xaxBcPt0AN0n26KeHXGGKgHYpHPUMUBHGHQGDtE4RLyy7U+LPbSJCqVaSDiPuzxHht0YUIbWogvrFmBfkP4ohcmjkZxWtEI9qQ4Whaobb2CFHGdKNt0zlVNBjJQ3uYRAvUe12zSBynQB18y6QhE8goneRkCEw9VIeft2pFIwNQ8tkWWEjDt6wHNaqeND7eASg5WLzYskWbTt6bPAOhSNRLJ38HZrOB5QNg+xxN5uuCSYmjMXCl8SkvQr91pInmOng+V898FLLBQtefs95whvbpfE0mKwBk5Cz2TkkHcUJa/IoC0CLmNqoEk3AtKxpw/J; tt_chain_token=46Xkv2ukMzyJ2e7XU7y0AQ==; bm_sv=A2E67B998DE8E6A4F1C2C02485467446~YAAQ7uIsF6g4j+SOAQAABdqUCxf1J/K4dYG0k7bbw2m5rFujdlSqMoCKDubu4R602nFvbY6zWC5puJczBv3IXwJJRpQxxR03wDCMVlKTCqjQvgDs8BoCuoNQxfY2fdS+F3bKut2lxXPQ2qctqz4kHBrgspJArHn/zu/IuKCIeSzmV4KcyxW6Zvw3/xMRA0MeHgyuHsTRBS+VrFk8Ju2NbJWWC8uSHbLCM/dhFT7/ktw8RE30r24XpQmhLpVTsUSC~1; tiktok_webapp_theme=light; msToken=ySXERzKCE0QUG0cCg6TWLw3wfEB-6kh6kAfuzhzjcQvmV1jBFloSgIsT9xk-QXFVdI99U1Fqb9mhUpIOldoDkjdZwskB8rvt66MHZaHnvBRZRtOKtTYsWT8osDyQXDVZWdPkvyE598h9; passport_csrf_token=1a47d95ebf68fc3648b0018ee75afc9f; passport_csrf_token_default=1a47d95ebf68fc3648b0018ee75afc9f; perf_feed_cache={%22expireTimestamp%22:1714057200000%2C%22itemIds%22:[%227346425092966206766%22%2C%227353812964207594795%22%2C%227343343741916171563%22]}; msToken=yWwG-ITrCnjJbx5ltBa9FTXdCImOJrl-wtQJSQH3afeEumWZcbo_qcrF6F7-NjYcrG6JVxtJiOU208REZeCSgXEZrrs5_65K741fQ7PSzCGOhz6vUyycq3Xvj4Mu-S0kJ6SqyltHnpJp
--- a/crawlers/tiktok/web/models.py
+++ b/crawlers/tiktok/web/models.py
@ -23,7 +23,6 @@ class BaseRequestModel(BaseModel):
    channel: str = "tiktok_web"
    cookie_enabled: str = "true"
    device_id: int = 7380187414842836523
-    odinId: int = 7404669909585003563
    device_platform: str = "web_pc"
    focus_state: str = "true"
    from_page: str = "user"
@ -50,47 +49,11 @@ class UserProfile(BaseRequestModel):
    uniqueId: str


-class UserPost(BaseModel):
-    WebIdLastTime: str = "1714385892"
-    aid: str = "1988"
-    app_language: str = "zh-Hans"
-    app_name: str = "tiktok_web"
-    browser_language: str = "zh-CN"
-    browser_name: str = "Mozilla"
-    browser_online: str = "true"
-    browser_platform: str = "Win32"
-    browser_version: str = "5.0%20%28Windows%29"
-    channel: str = "tiktok_web"
-    cookie_enabled: str = "true"
-    count: int = 20
+class UserPost(BaseRequestModel):
    coverFormat: int = 2
+    count: int = 35
    cursor: int = 0
-    data_collection_enabled: str = "true"
-    device_id: str = "7380187414842836523"
-    device_platform: str = "web_pc"
-    focus_state: str = "true"
-    from_page: str = "user"
-    history_len: str = "3"
-    is_fullscreen: str = "false"
-    is_page_visible: str = "true"
-    language: str = "zh-Hans"
-    locate_item_id: str = ""
-    needPinnedItemIds: str = "true"
-    odinId: str = "7404669909585003563"
-    os: str = "windows"
-    # 0：默认排序，1：热门排序，2：最旧排序
-    post_item_list_request_type: int = 0
-    priority_region: str = "US"
-    referer: str = ""
-    region: str = "US"
-    screen_height: str = "827"
-    screen_width: str = "1323"
    secUid: str
-    tz_name: str = "America%2FLos_Angeles"
-    user_is_login: str = "true"
-    webcast_language: str = "zh-Hans"
-    msToken: str = "SXtP7K0MMFlQmzpuWfZoxAlAaKqt-2p8oAbOHFBw-k3TA2g4jE_FXrFKf3i38lR-xNh_bV1_qfTPRnj4PXbkBfrVD2iAazeUkASIASHT0pu-Bx2_POx7O3nBBHZe2SI7CPsanerdclxHht1hcoUTlg%3D%3D"
-    _signature: str = "_02B4Z6wo000017oyWOQAAIDD9xNhTSnfaDu6MFxAAIlj23"


 class UserLike(BaseRequestModel):
--- a/crawlers/tiktok/web/web_crawler.py
+++ b/crawlers/tiktok/web/web_crawler.py
@ -344,9 +344,9 @@ class TikTokWebCrawler:

    async def main(self):
        # 获取单个作品数据
-        # item_id = "7369296852669205791"
-        # response = await self.fetch_one_video(item_id)
-        # print(response)
+        item_id = "7369296852669205791"
+        response = await self.fetch_one_video(item_id)
+        print(response)

        # 获取用户的个人信息
        # secUid = "MS4wLjABAAAAfDPs6wbpBcMMb85xkvDGdyyyVAUS2YoVCT9P6WQ1bpuwEuPhL9eFtTmGvxw1lT2C"