spider/EasySpider
1
0
Fork 0
mirror of https://github.com/NaiboWang/EasySpider.git synced 2025-04-22 12:09:35 +08:00
Code Issues Packages Projects Releases Wiki Activity

构建说明更新

Browse Source
This commit is contained in:
NaiboWang-Alienware 2023-07-03 04:42:58 +08:00
parent 8238ad676a
commit 03ac3c5927
5 changed files with 25 additions and 21 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
ElectronJS/tasks/50.json
View File

File diff suppressed because one or more lines are too long

2
ExecuteStage/.vscode/launch.json vendored
View File

@ -10,7 +10,7 @@
"program": "${file}",
"console": "integratedTerminal",
"justMyCode": true,
"args": ["--id", "[117]", "--read_type", "remote", "--headless", "0"]
"args": ["--id", "[1]", "--read_type", "remote", "--headless", "0"]
// "args": ["--id", "[44]", "--headless", "0", "--user_data", "1"]
}
]

2
ExecuteStage/Readme.md
View File

@ -64,7 +64,7 @@ The above is an example command to run a task with the ID of `1`. For more infor
### VS Code调试/VS Code Debug
可以用VS Code打开此文件夹即可调试程序，可修改`.vscode`下的`launch.json`文件中的调试参数，调试说明参考：https://zhuanlan.zhihu.com/p/41189402
可以用VS Code打开此文件夹即可调试程序，可修改`.vscode`下的`launch.json`文件中的调试参数，调试说明参考：[https://zhuanlan.zhihu.com/p/41189402](https://zhuanlan.zhihu.com/p/41189402)。
You can use VS Code to open this folder and debug the program. You can modify the debugging parameters in the launch.json file located under the .vscode folder. For instructions on debugging with VSCode, you can refer to this guide: [Debugging Python with Visual Studio Code](https://code.visualstudio.com/docs/python/debugging).

36
ExecuteStage/easyspider_executestage.py
View File

@ -45,9 +45,9 @@ desired_capabilities["pageLoadStrategy"] = "none"
# 控制流程的暂停和继续
def check_pause(event):
def check_pause(key, event):
while True:
if keyboard.is_pressed('p'): # 按下p键，暂停程序
if keyboard.is_pressed(key): # 按下p键，暂停程序
if event._flag == False:
print("任务执行中，长按p键暂停执行。")
print("Task is running, long press 'p' to pause.")
@ -1106,19 +1106,23 @@ class BrowserThread(Thread):
for result in content if result.strip())
else:
content = p["default"]
try:
if not self.dataNotFoundKeys[p["name"]]:
print('Element %s not found with parameter name %s when extracting data, use default, this error will only show once' % (
p["relativeXPath"], p["name"]))
print("提取数据操作时，字段名 %s 对应XPath %s 未找到，使用默认值，本字段将不再重复报错" % (
p["name"], p["relativeXPath"]))
self.dataNotFoundKeys[p["name"]] = True
self.recordLog(
'Element %s not found, use default' % p["relativeXPath"])
except:
pass
if not self.dataNotFoundKeys[p["name"]]:
print('Element %s not found with parameter name %s when extracting data, use default, this error will only show once' % (
p["relativeXPath"], p["name"]))
print("提取数据操作时，字段名 %s 对应XPath %s 未找到，使用默认值，本字段将不再重复报错" % (
p["name"], p["relativeXPath"]))
self.dataNotFoundKeys[p["name"]] = True
self.recordLog(
'Element %s not found, use default' % p["relativeXPath"])
except Exception as e:
print(e)
if not self.dataNotFoundKeys[p["name"]]:
print('Element %s not found with parameter name %s when extracting data, use default, this error will only show once' % (
p["relativeXPath"], p["name"]))
print("提取数据操作时，字段名 %s 对应XPath %s 未找到，使用默认值，本字段将不再重复报错" % (
p["name"], p["relativeXPath"]))
self.dataNotFoundKeys[p["name"]] = True
self.recordLog(
'Element %s not found, use default' % p["relativeXPath"])
self.outputParameters[p["name"]] = content
# 对于不能优化的操作，使用selenium执行
@ -1350,7 +1354,7 @@ if __name__ == '__main__':
if not os.path.exists("Data/" + str(i)):
os.mkdir("Data/" + str(i))
if not os.path.exists("Data/" + str(i) + "/" + saveName):
os.mkdir("Data/" + saveName) # 创建保存文件夹用来保存截图
os.mkdir("Data/" + str(i) + "/" + saveName) # 创建保存文件夹用来保存截图
if c.read_type == "remote":
print("remote")
content = requests.get(
@ -1405,7 +1409,7 @@ if __name__ == '__main__':
print("Thread with task id: ", i, " is created")
threads.append(thread)
thread.start()
Thread(target=check_pause, args=(event)).start()
Thread(target=check_pause, args=("p", event)).start()
time.sleep(5)
print("\n\n----------------------------------")
print("正在运行任务，长按键盘p键可暂停任务的执行以便手工操作浏览器如输入验证码；如果想恢复任务的执行，请再次长按p键。")

4
Extension/README.md
View File

@ -44,7 +44,7 @@ npm install
npm run dev
```
打开一个Chrome浏览器窗口，然后在浏览器地址栏输入`chrome://extensions/`，在打开的页面中，打开右上角的`开发者模式`，点击`加载已解压的扩展程序`，选择`manifest_v3\dist`文件夹，即可加载扩展。
打开一个Chrome浏览器窗口，然后在浏览器地址栏输入`chrome://extensions/`，在打开的页面中，打开右上角的`开发者模式`，点击`加载已解压的扩展程序`，选择`manifest_v3/dist`文件夹，即可加载扩展。
-----
@ -54,7 +54,7 @@ Run the following command to hot reload the extension:
npm run dev
```
Open a Chrome browser window, then enter `chrome://extensions/` in the browser address bar. On the opened page, open the `Developer mode` in the upper right corner, click `Load unpacked` and select the `manifest_v3\dist` folder to load the extension.
Open a Chrome browser window, then enter `chrome://extensions/` in the browser address bar. On the opened page, open the `Developer mode` in the upper right corner, click `Load unpacked` and select the `manifest_v3/dist` folder to load the extension.
## 打包扩展/Package the extension