init upload
This commit is contained in:
guyue
parent
38549643bd
commit
2fb1cf1c28
11
DONATE.md
Normal file
11
DONATE.md
Normal file
@ -0,0 +1,11 @@
|
|||||||
|
如果觉得SoulBook对您有帮助，那么您可以给开发者一些支持：
|
||||||
|
|
||||||
|
- 开源
|
||||||
|
- 服务器的费用也是一笔开销
|
||||||
|
- 为用户提供更好的搜索环境
|
||||||
|
- 一杯咖啡
|
||||||
|
|
||||||
|
请扫一扫微信支付（留言希望你带上自己的名字^_^）：
|
||||||
|
|
||||||
|
<img src="https://ws1.sinaimg.cn/large/007i3XCUgy1fydjg9e5fsj30lq0lmgoj.jpg" width = "400" height = "400" alt="donate" align=center />
|
||||||
|
|
||||||
11
Dockerfile
Normal file
11
Dockerfile
Normal file
@ -0,0 +1,11 @@
|
|||||||
|
# 基于python3.6镜像来构建owllook镜像
|
||||||
|
FROM python:3.6
|
||||||
|
MAINTAINER howie6879 <howie6879@gmail.com>
|
||||||
|
ENV TIME_ZONE=Asia/Shanghai
|
||||||
|
RUN echo "${TIME_ZONE}" > /etc/timezone \
|
||||||
|
&& ln -sf /usr/share/zoneinfo/${TIME_ZONE} /etc/localtime
|
||||||
|
ADD . /owllook
|
||||||
|
WORKDIR /owllook
|
||||||
|
RUN pip install --no-cache-dir --trusted-host mirrors.aliyun.com -i http://mirrors.aliyun.com/pypi/simple/ pipenv
|
||||||
|
RUN pipenv install --skip-lock
|
||||||
|
RUN find . -name "*.pyc" -delete
|
||||||
13
HISTORY.md
Normal file
13
HISTORY.md
Normal file
@ -0,0 +1,13 @@
|
|||||||
|
## 更新历史记录
|
||||||
|
|
||||||
|
### 0.1.0 / 2017-03-05：
|
||||||
|
---
|
||||||
|
- 小说的基本搜索解析功能
|
||||||
|
- 搜索记录
|
||||||
|
- 缓存
|
||||||
|
- 书架
|
||||||
|
- 书签
|
||||||
|
- 登录（暂时不开放注册，可与我申请体验）
|
||||||
|
- 初步兼容手机（后续跟进）
|
||||||
|
|
||||||
|
|
||||||
201
LICENSE.txt
Normal file
201
LICENSE.txt
Normal file
@ -0,0 +1,201 @@
|
|||||||
|
Apache License
|
||||||
|
Version 2.0, January 2004
|
||||||
|
http://www.apache.org/licenses/
|
||||||
|
|
||||||
|
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||||
|
|
||||||
|
1. Definitions.
|
||||||
|
|
||||||
|
"License" shall mean the terms and conditions for use, reproduction,
|
||||||
|
and distribution as defined by Sections 1 through 9 of this document.
|
||||||
|
|
||||||
|
"Licensor" shall mean the copyright owner or entity authorized by
|
||||||
|
the copyright owner that is granting the License.
|
||||||
|
|
||||||
|
"Legal Entity" shall mean the union of the acting entity and all
|
||||||
|
other entities that control, are controlled by, or are under common
|
||||||
|
control with that entity. For the purposes of this definition,
|
||||||
|
"control" means (i) the power, direct or indirect, to cause the
|
||||||
|
direction or management of such entity, whether by contract or
|
||||||
|
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||||
|
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||||
|
|
||||||
|
"You" (or "Your") shall mean an individual or Legal Entity
|
||||||
|
exercising permissions granted by this License.
|
||||||
|
|
||||||
|
"Source" form shall mean the preferred form for making modifications,
|
||||||
|
including but not limited to software source code, documentation
|
||||||
|
source, and configuration files.
|
||||||
|
|
||||||
|
"Object" form shall mean any form resulting from mechanical
|
||||||
|
transformation or translation of a Source form, including but
|
||||||
|
not limited to compiled object code, generated documentation,
|
||||||
|
and conversions to other media types.
|
||||||
|
|
||||||
|
"Work" shall mean the work of authorship, whether in Source or
|
||||||
|
Object form, made available under the License, as indicated by a
|
||||||
|
copyright notice that is included in or attached to the work
|
||||||
|
(an example is provided in the Appendix below).
|
||||||
|
|
||||||
|
"Derivative Works" shall mean any work, whether in Source or Object
|
||||||
|
form, that is based on (or derived from) the Work and for which the
|
||||||
|
editorial revisions, annotations, elaborations, or other modifications
|
||||||
|
represent, as a whole, an original work of authorship. For the purposes
|
||||||
|
of this License, Derivative Works shall not include works that remain
|
||||||
|
separable from, or merely link (or bind by name) to the interfaces of,
|
||||||
|
the Work and Derivative Works thereof.
|
||||||
|
|
||||||
|
"Contribution" shall mean any work of authorship, including
|
||||||
|
the original version of the Work and any modifications or additions
|
||||||
|
to that Work or Derivative Works thereof, that is intentionally
|
||||||
|
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||||
|
or by an individual or Legal Entity authorized to submit on behalf of
|
||||||
|
the copyright owner. For the purposes of this definition, "submitted"
|
||||||
|
means any form of electronic, verbal, or written communication sent
|
||||||
|
to the Licensor or its representatives, including but not limited to
|
||||||
|
communication on electronic mailing lists, source code control systems,
|
||||||
|
and issue tracking systems that are managed by, or on behalf of, the
|
||||||
|
Licensor for the purpose of discussing and improving the Work, but
|
||||||
|
excluding communication that is conspicuously marked or otherwise
|
||||||
|
designated in writing by the copyright owner as "Not a Contribution."
|
||||||
|
|
||||||
|
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||||
|
on behalf of whom a Contribution has been received by Licensor and
|
||||||
|
subsequently incorporated within the Work.
|
||||||
|
|
||||||
|
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||||
|
this License, each Contributor hereby grants to You a perpetual,
|
||||||
|
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||||
|
copyright license to reproduce, prepare Derivative Works of,
|
||||||
|
publicly display, publicly perform, sublicense, and distribute the
|
||||||
|
Work and such Derivative Works in Source or Object form.
|
||||||
|
|
||||||
|
3. Grant of Patent License. Subject to the terms and conditions of
|
||||||
|
this License, each Contributor hereby grants to You a perpetual,
|
||||||
|
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||||
|
(except as stated in this section) patent license to make, have made,
|
||||||
|
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||||
|
where such license applies only to those patent claims licensable
|
||||||
|
by such Contributor that are necessarily infringed by their
|
||||||
|
Contribution(s) alone or by combination of their Contribution(s)
|
||||||
|
with the Work to which such Contribution(s) was submitted. If You
|
||||||
|
institute patent litigation against any entity (including a
|
||||||
|
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||||
|
or a Contribution incorporated within the Work constitutes direct
|
||||||
|
or contributory patent infringement, then any patent licenses
|
||||||
|
granted to You under this License for that Work shall terminate
|
||||||
|
as of the date such litigation is filed.
|
||||||
|
|
||||||
|
4. Redistribution. You may reproduce and distribute copies of the
|
||||||
|
Work or Derivative Works thereof in any medium, with or without
|
||||||
|
modifications, and in Source or Object form, provided that You
|
||||||
|
meet the following conditions:
|
||||||
|
|
||||||
|
(a) You must give any other recipients of the Work or
|
||||||
|
Derivative Works a copy of this License; and
|
||||||
|
|
||||||
|
(b) You must cause any modified files to carry prominent notices
|
||||||
|
stating that You changed the files; and
|
||||||
|
|
||||||
|
(c) You must retain, in the Source form of any Derivative Works
|
||||||
|
that You distribute, all copyright, patent, trademark, and
|
||||||
|
attribution notices from the Source form of the Work,
|
||||||
|
excluding those notices that do not pertain to any part of
|
||||||
|
the Derivative Works; and
|
||||||
|
|
||||||
|
(d) If the Work includes a "NOTICE" text file as part of its
|
||||||
|
distribution, then any Derivative Works that You distribute must
|
||||||
|
include a readable copy of the attribution notices contained
|
||||||
|
within such NOTICE file, excluding those notices that do not
|
||||||
|
pertain to any part of the Derivative Works, in at least one
|
||||||
|
of the following places: within a NOTICE text file distributed
|
||||||
|
as part of the Derivative Works; within the Source form or
|
||||||
|
documentation, if provided along with the Derivative Works; or,
|
||||||
|
within a display generated by the Derivative Works, if and
|
||||||
|
wherever such third-party notices normally appear. The contents
|
||||||
|
of the NOTICE file are for informational purposes only and
|
||||||
|
do not modify the License. You may add Your own attribution
|
||||||
|
notices within Derivative Works that You distribute, alongside
|
||||||
|
or as an addendum to the NOTICE text from the Work, provided
|
||||||
|
that such additional attribution notices cannot be construed
|
||||||
|
as modifying the License.
|
||||||
|
|
||||||
|
You may add Your own copyright statement to Your modifications and
|
||||||
|
may provide additional or different license terms and conditions
|
||||||
|
for use, reproduction, or distribution of Your modifications, or
|
||||||
|
for any such Derivative Works as a whole, provided Your use,
|
||||||
|
reproduction, and distribution of the Work otherwise complies with
|
||||||
|
the conditions stated in this License.
|
||||||
|
|
||||||
|
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||||
|
any Contribution intentionally submitted for inclusion in the Work
|
||||||
|
by You to the Licensor shall be under the terms and conditions of
|
||||||
|
this License, without any additional terms or conditions.
|
||||||
|
Notwithstanding the above, nothing herein shall supersede or modify
|
||||||
|
the terms of any separate license agreement you may have executed
|
||||||
|
with Licensor regarding such Contributions.
|
||||||
|
|
||||||
|
6. Trademarks. This License does not grant permission to use the trade
|
||||||
|
names, trademarks, service marks, or product names of the Licensor,
|
||||||
|
except as required for reasonable and customary use in describing the
|
||||||
|
origin of the Work and reproducing the content of the NOTICE file.
|
||||||
|
|
||||||
|
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||||
|
agreed to in writing, Licensor provides the Work (and each
|
||||||
|
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||||
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||||
|
implied, including, without limitation, any warranties or conditions
|
||||||
|
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||||
|
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||||
|
appropriateness of using or redistributing the Work and assume any
|
||||||
|
risks associated with Your exercise of permissions under this License.
|
||||||
|
|
||||||
|
8. Limitation of Liability. In no event and under no legal theory,
|
||||||
|
whether in tort (including negligence), contract, or otherwise,
|
||||||
|
unless required by applicable law (such as deliberate and grossly
|
||||||
|
negligent acts) or agreed to in writing, shall any Contributor be
|
||||||
|
liable to You for damages, including any direct, indirect, special,
|
||||||
|
incidental, or consequential damages of any character arising as a
|
||||||
|
result of this License or out of the use or inability to use the
|
||||||
|
Work (including but not limited to damages for loss of goodwill,
|
||||||
|
work stoppage, computer failure or malfunction, or any and all
|
||||||
|
other commercial damages or losses), even if such Contributor
|
||||||
|
has been advised of the possibility of such damages.
|
||||||
|
|
||||||
|
9. Accepting Warranty or Additional Liability. While redistributing
|
||||||
|
the Work or Derivative Works thereof, You may choose to offer,
|
||||||
|
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||||
|
or other liability obligations and/or rights consistent with this
|
||||||
|
License. However, in accepting such obligations, You may act only
|
||||||
|
on Your own behalf and on Your sole responsibility, not on behalf
|
||||||
|
of any other Contributor, and only if You agree to indemnify,
|
||||||
|
defend, and hold each Contributor harmless for any liability
|
||||||
|
incurred by, or claims asserted against, such Contributor by reason
|
||||||
|
of your accepting any such warranty or additional liability.
|
||||||
|
|
||||||
|
END OF TERMS AND CONDITIONS
|
||||||
|
|
||||||
|
APPENDIX: How to apply the Apache License to your work.
|
||||||
|
|
||||||
|
To apply the Apache License to your work, attach the following
|
||||||
|
boilerplate notice, with the fields enclosed by brackets "{}"
|
||||||
|
replaced with your own identifying information. (Don't include
|
||||||
|
the brackets!) The text should be enclosed in the appropriate
|
||||||
|
comment syntax for the file format. We also recommend that a
|
||||||
|
file or class name and description of purpose be included on the
|
||||||
|
same "printed page" as the copyright notice for easier
|
||||||
|
identification within third-party archives.
|
||||||
|
|
||||||
|
Copyright 2017 howie.hu
|
||||||
|
|
||||||
|
Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
you may not use this file except in compliance with the License.
|
||||||
|
You may obtain a copy of the License at
|
||||||
|
|
||||||
|
http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
|
||||||
|
Unless required by applicable law or agreed to in writing, software
|
||||||
|
distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
See the License for the specific language governing permissions and
|
||||||
|
limitations under the License.
|
||||||
35
Pipfile
Normal file
35
Pipfile
Normal file
@ -0,0 +1,35 @@
|
|||||||
|
[[source]]
|
||||||
|
verify_ssl = true
|
||||||
|
name = "pypi"
|
||||||
|
url = "https://pypi.douban.com/simple/"
|
||||||
|
|
||||||
|
[requires]
|
||||||
|
python_version = "3.6"
|
||||||
|
|
||||||
|
[dev-packages]
|
||||||
|
|
||||||
|
[packages]
|
||||||
|
aiohttp = "*"
|
||||||
|
arrow = "*"
|
||||||
|
async_timeout = "*"
|
||||||
|
"bs4" = "*"
|
||||||
|
cchardet = "*"
|
||||||
|
gunicorn = "*"
|
||||||
|
"html5lib" = "*"
|
||||||
|
Jinja2 = "==2.11.3"
|
||||||
|
motor = "==1.1"
|
||||||
|
numpy = "*"
|
||||||
|
requests = ">=2.20.0"
|
||||||
|
Sanic = "==0.5.1"
|
||||||
|
sanic_session = "==0.1.2"
|
||||||
|
schedule = "*"
|
||||||
|
uvloop = "*"
|
||||||
|
pytest = "*"
|
||||||
|
pymongo = "*"
|
||||||
|
aiofiles = "*"
|
||||||
|
pyquery = "*"
|
||||||
|
ruia-ua = "*"
|
||||||
|
ruia = ">=0.5.8"
|
||||||
|
aiocache = "==0.3.2"
|
||||||
|
aioredis = "==0.3.3"
|
||||||
|
asyncio_redis = "==0.14.3"
|
||||||
184
README.md
184
README.md
@ -1,2 +1,182 @@
|
|||||||
# SoulBook
|
## 基于大佬的项目修改
|
||||||
在线（解析）阅读小说
|
早先使用时基于大佬的项目owllook做了一点点修改，更贴近自己的习惯，最近有空正好上传下，方便闲暇时优化，原说明在下面。
|
||||||
|
|
||||||
|
## owllook - 在线小说搜索引擎
|
||||||
|
|
||||||
|
[](https://github.com/howie6879/owllook) [](https://github.com/howie6879/owllook)
|
||||||
|
|
||||||
|
`owllook`是一个基于其他搜索引擎构建的垂直小说搜索引擎，owllook目的是让阅读更简单、优雅，让每位读者都有舒适的阅读体验，如**搜书、阅读、收藏、追更、推荐等功能**：
|
||||||
|
|
||||||
|
- ~~演示网址（不维护了，有兴趣自己搭建）~~：[https://www.owllook.net/](https://www.owllook.net/)
|
||||||
|
- 公众号：[**粮草小说**](https://www.owllook.net/static/novels/img/lcxs_show.jpg)，有兴趣的话可以关注下
|
||||||
|
- 详细安装介绍：[详细安装介绍](https://mp.weixin.qq.com/s/0CqLiKsyDQ-pVmeo3R-UlA)
|
||||||
|
- 个人公众号：[老胡的储物柜](https://raw.githubusercontent.com/howie6879/howie6879.github.io/img/pictures/20190529083905.png)
|
||||||
|
|
||||||
|
<div align=center><img width="400" height="400" src="https://raw.githubusercontent.com/howie6879/howie6879.github.io/img/pictures/20190529083905.png" /></div>
|
||||||
|
|
||||||
|
衍生项目：
|
||||||
|
|
||||||
|
- 终端看小说： [NIYT](https://github.com/howie6879/NIYT)
|
||||||
|
- 小说监控工具：[owllook_gui](https://github.com/howie6879/owllook_gui)
|
||||||
|
- 小说接口：[owllook_api](https://github.com/howie6879/owllook_api)
|
||||||
|
|
||||||
|
**注意**：若将本项目部署并发行，请**声明来源**，本项目纯属**共享学习之用，不得用于商业！**
|
||||||
|
|
||||||
|
### Overview
|
||||||
|
|
||||||
|
`owllook`使用了mongodb储存了用户使用过程中的产生的基本信息，诸如注册信息、搜索小说信息、收藏小说数据等，对于某些必要的缓存，则利用redis进行缓存处理，如小说缓存、session缓存，注意，对于限制数据：都将在24小时删除
|
||||||
|
|
||||||
|
对于不同网站的小说，页面规则都不尽相同，我希望能够在代码解析后再统一展示出来，这样方便且美观，而不是仅仅跳转到对应网站就完事，清新简洁的阅读体验才是最重要的
|
||||||
|
|
||||||
|
目前采用的是直接在搜索引擎上进行结果检索，我尽量写少量的规则来完成解析，具体见[规则定义](./docs/规则定义.md)，遇到自己喜欢的小说网站，你也可以自己添加解析，`owllook`目前解析了超过 **200+** 网站，追更网站解析了**50+**
|
||||||
|
|
||||||
|
有一些地方需要用到爬虫，比如说排行榜，一些书籍信息等，我不想动用重量级爬虫框架来写，于是我在owllook里面编写了一个很轻量的爬虫框架来做这件事，见 **[ruia](https://github.com/howie6879/ruia)**
|
||||||
|
|
||||||
|
BTW，sanic写界面确实不是很方便，至于为什么写这个，一是想利用`sanic`尽量做成异步服务，二是想就此练习下推荐系统，顺便作为毕业设计
|
||||||
|
|
||||||
|
#### Installation
|
||||||
|
|
||||||
|
请先装好mongo以及redis，不会安装mongo看[这里](https://www.digitalocean.com/community/tutorials/how-to-install-mongodb-on-centos-7)，`owllook`使用`pipenv`管理虚拟环境，所以使用前请确保安装一个`Python3.6`环境（比如我这里环境路径在`/Users/howie/anaconda3/envs/python36/bin/python3.6`），然后安装好`pipenv`
|
||||||
|
|
||||||
|
mongo以及redis装好后，进入项目目录，依照步骤执行：
|
||||||
|
|
||||||
|
```shell
|
||||||
|
# 下载代码
|
||||||
|
git clone https://github.com/howie6879/owllook
|
||||||
|
cd owllook
|
||||||
|
pip install pipenv
|
||||||
|
# 请先提前创建好Python3.6环境
|
||||||
|
pipenv install --python /Users/howie/anaconda3/envs/python36/bin/python3.6 --skip-lock
|
||||||
|
# 进入虚拟环境
|
||||||
|
pipenv shell
|
||||||
|
|
||||||
|
# 先配置好相关数据库配置，具体看`config/dev_config.py`
|
||||||
|
|
||||||
|
# 方案一
|
||||||
|
# 运行：
|
||||||
|
cd owllook
|
||||||
|
pipenv run gunicorn -c owllook/config/gunicorn.py --worker-class sanic.worker.GunicornWorker owllook.server:app
|
||||||
|
|
||||||
|
# 方案二 推荐
|
||||||
|
# 安装docker并打包镜像
|
||||||
|
docker build -t howie6879/owllook .
|
||||||
|
# 实际运行
|
||||||
|
docker-compose up -d
|
||||||
|
# 退出
|
||||||
|
docker-compose down
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Features
|
||||||
|
|
||||||
|
- 多搜索源
|
||||||
|
- 丰富的解析源
|
||||||
|
- 界面统一解析
|
||||||
|
- 完善的阅读体验
|
||||||
|
- 搜索记录
|
||||||
|
- 缓存
|
||||||
|
- 书架
|
||||||
|
- 书签
|
||||||
|
- 登录
|
||||||
|
- 初步兼容手机（后续跟进）
|
||||||
|
- 注册（开放注册）
|
||||||
|
- 上次阅读记录
|
||||||
|
- 最新章节
|
||||||
|
- 书友推荐（简单地基于相似度进行推荐）
|
||||||
|
- 目录获取
|
||||||
|
- 翻页
|
||||||
|
- 搜索排行
|
||||||
|
- 章节异步加载 感谢@[mscststs](https://github.com/mscststs)
|
||||||
|
- 排行榜 - 起点+owllook
|
||||||
|
- 自带爬虫框架，统一爬虫规范，对爬虫感兴趣的可以看看 - [ruia](https://github.com/howie6879/ruia)
|
||||||
|
|
||||||
|
#### TODO
|
||||||
|
|
||||||
|
- [ ] 评分
|
||||||
|
- [ ] 阅读书单
|
||||||
|
- [ ] 推荐
|
||||||
|
- [ ] 小说圈 - 小说界的消息
|
||||||
|
|
||||||
|
### Screenshots
|
||||||
|
|
||||||
|
首页:
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
搜索：
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
榜单：
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
书架：
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
目录解析页：
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
阅读：
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
### FAQ
|
||||||
|
|
||||||
|
**为什么首页榜单为空白？**
|
||||||
|
|
||||||
|
这个是根据小说搜索次数显示的，每天刷新一次，使用多了就会有
|
||||||
|
|
||||||
|
**为什么会出现302跳转？**
|
||||||
|
|
||||||
|
为了防止直接运行服务被恶意域名绑定，所以作出如下修改：
|
||||||
|
|
||||||
|
```shell
|
||||||
|
vim config/config.py
|
||||||
|
# 将 true 改为 false
|
||||||
|
VAL_HOST = os.getenv('VAL_HOST', 'true')
|
||||||
|
VAL_HOST = os.getenv('VAL_HOST', 'false')
|
||||||
|
```
|
||||||
|
|
||||||
|
**小说榜单页面为什么没有内容？**
|
||||||
|
|
||||||
|
需要运行`owllook/spiders/spider_console.py`，
|
||||||
|
目前代码中是设定60分钟运行一次，运行的时候请酌情更改，
|
||||||
|
`os.environ['MODE'] = 'PRO'` 这段代码也请注释掉
|
||||||
|
|
||||||
|
### Reference
|
||||||
|
|
||||||
|
**owllook使用了以下第三方包:**
|
||||||
|
|
||||||
|
- sanic：基于Python 3.5+的异步web服务器
|
||||||
|
|
||||||
|
- motor：异步的mongodb驱动
|
||||||
|
|
||||||
|
- Jinja2：基于python的模板引擎
|
||||||
|
|
||||||
|
- aiohttp：异步请求
|
||||||
|
|
||||||
|
- caddy：基于go的web服务器
|
||||||
|
|
||||||
|
…...更多见Pipfile，感谢开发者
|
||||||
|
|
||||||
|
**web框架：**
|
||||||
|
|
||||||
|
- [bootstrap](https://github.com/twbs/bootstrap)：Sleek, intuitive, and powerful front-end framework for faster and easier web development.
|
||||||
|
- [mdui](https://github.com/zdhxiong/mdui )：MDUI 是一个基于 Material Design 的前端框架
|
||||||
|
|
||||||
|
### License
|
||||||
|
|
||||||
|
`owllook` is offered under the Apache 2 license.
|
||||||
|
|
||||||
|
### Donate
|
||||||
|
|
||||||
|
**捐赠：**
|
||||||
|
|
||||||
|
> 美酒加咖啡，我只要喝一杯～
|
||||||
|
|
||||||
|
<img src="https://raw.githubusercontent.com/howie6879/howie6879.github.io/img/pictures/20190904201512.png" width = "400" height = "400" alt="donate" align=center />
|
||||||
|
|
||||||
|
|
||||||
|
感谢以下捐赠者，具体见[捐赠名单](./DONATE.md) ^_^
|
||||||
|
|||||||
11
__init__.py
Normal file
11
__init__.py
Normal file
@ -0,0 +1,11 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""
|
||||||
|
Created by howie.hu at 17-9-22.
|
||||||
|
"""
|
||||||
|
from sys import path as path_sys
|
||||||
|
from os import path as path_os
|
||||||
|
# from multiprocessing.managers import BaseManager
|
||||||
|
# from multiprocessing import freeze_support
|
||||||
|
|
||||||
|
# environment variable
|
||||||
|
path_sys.append(path_os.dirname(__file__))
|
||||||
9
docker-compose.yml
Normal file
9
docker-compose.yml
Normal file
@ -0,0 +1,9 @@
|
|||||||
|
version: '3'
|
||||||
|
services:
|
||||||
|
web:
|
||||||
|
image: howie6879/owllook
|
||||||
|
command: pipenv run gunicorn -c soulbook/config/gunicorn.py --worker-class sanic.worker.GunicornWorker soulbook.server:app
|
||||||
|
ports:
|
||||||
|
- "8001:8001"
|
||||||
|
volumes:
|
||||||
|
- .:/soulbook
|
||||||
BIN
docs/imgs/book.jpeg
Normal file
BIN
docs/imgs/book.jpeg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 316 KiB |
BIN
docs/imgs/chapter.png
Normal file
BIN
docs/imgs/chapter.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 433 KiB |
BIN
docs/imgs/content.png
Normal file
BIN
docs/imgs/content.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 1.9 MiB |
BIN
docs/imgs/similar.png
Normal file
BIN
docs/imgs/similar.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 203 KiB |
BIN
docs/imgs/the_latest_chapter.jpeg
Normal file
BIN
docs/imgs/the_latest_chapter.jpeg
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 266 KiB |
102
docs/简介.md
Normal file
102
docs/简介.md
Normal file
@ -0,0 +1,102 @@
|
|||||||
|
|
||||||
|
## 0.前言
|
||||||
|
事情的起因是这样的，前段时间在把之前写的一些同步网络请求代码改成异步的，用的是`aiohttp`，既然是请求，那请求什么呢，就百度吧，请求什么内容呢，我随手一输自己最近看的一本小说，然后作为练手用，就做了这个，我将其取名为`owllook`，准备做成一个推荐系统，所以我就将其部署上去，这才有了这个网站。
|
||||||
|
|
||||||
|
上一篇介绍了自己在使用`sanic`过程中遇到的一些问题，这次就想介绍下这个[owllook--novels_search](https://github.com/howie6879/novels-search)，上面是演示demo，具体可以见[http://www.owllook.net/](http://www.owllook.net/)
|
||||||
|
**本项目纯属共享学习之用，不得用于商业！**
|
||||||
|
首先我想说下目前的项目进度：
|
||||||
|
|
||||||
|
v0.1已经上线：
|
||||||
|
|
||||||
|
- 小说的基本搜索解析功能
|
||||||
|
- 搜索记录
|
||||||
|
- 缓存
|
||||||
|
- 书架
|
||||||
|
- 书签
|
||||||
|
- 登录（**暂时不开放注册，可与我申请体验**）
|
||||||
|
|
||||||
|
TODO:
|
||||||
|
|
||||||
|
- [ ] 注册
|
||||||
|
- [ ] 阅读书单
|
||||||
|
- [ ] 手机端兼容
|
||||||
|
- [ ] 推荐
|
||||||
|
- [ ] 排行榜
|
||||||
|
- [ ] 个人中心
|
||||||
|
|
||||||
|
## 1.介绍
|
||||||
|
`owllook`的思路很简单，利用百度检索出来的结果，进行过滤解析后再展示，使用的技术如下：
|
||||||
|
|
||||||
|
- sanic：基于Python 3.5+的异步web服务器，快快快
|
||||||
|
- sanic_session：sanic的持续会话插件
|
||||||
|
- vloop：sanic默认使用uvloop，替代asyncio本身的loop
|
||||||
|
- motor：异步的mongodb驱动
|
||||||
|
- aiohttp：异步请求
|
||||||
|
- aiocache：异步缓存，本项目改用了其中的decorator部分，缓存数据库使用redis
|
||||||
|
|
||||||
|
对于用户的一系列操作信息，使用`mongodb`进行存储，而缓存则使用`redis`。
|
||||||
|
|
||||||
|
对于不同网站的小说，页面规则都不尽相同，我希望能够在代码解析后再统一展示出来，这样方便且美观，而不是仅仅跳转到对应网站就完事，清新简洁的阅读体验才是最重要的。
|
||||||
|
|
||||||
|
目前采用的是直接在百度上进行结果检索，也不是不能做的更大更全，只是觉得没什么意义，目前的检索结果已经很足够。
|
||||||
|
|
||||||
|
我尽量写少量的规则来完成解析，具体见[规则定义](https://github.com/howie6879/novels-search/blob/master/docs/%E8%A7%84%E5%88%99%E5%AE%9A%E4%B9%89.md)，遇到自己喜欢的小说网站，方便诸位添加解析。
|
||||||
|
|
||||||
|
一般都是在下班时间编写这个项目，目前`v0.1`版本大概实现。
|
||||||
|
|
||||||
|
后期我准备将基础功能写好之后，能够实现小说与kindle之间的对接。
|
||||||
|
|
||||||
|
## 2.解析
|
||||||
|
这个项目的思路与技术都比较简单，但是小说网站的解析工作是很难全部解析完毕，下面我将举个例子怎么具体的解析一个网站，欢迎各位添砖加瓦，请先看一遍[规则定义](https://github.com/howie6879/novels-search/blob/master/docs/%E8%A7%84%E5%88%99%E5%AE%9A%E4%B9%89.md)。
|
||||||
|
解析也很简单，只要求有点html以及css基础即可。
|
||||||
|
|
||||||
|
首先进入网站http://www.owllook.net/。
|
||||||
|
搜索：
|
||||||
|
|
||||||
|
|
||||||
|
注意第一条结果显示未解析，这就是我们要解析的对象了，点击进入源网站，审查小说目录对应的元素，这里显示的是：`class="mulu_list`，然后注意其`content_url`类型是`0`
|
||||||
|
|
||||||
|
|
||||||
|
进入小说章节内容页面，审查元素可以看到`id=htmlContent`，所以`content_selector=htmlContent`
|
||||||
|
|
||||||
|
接下来进入项目，打开`config/rules.py`文件，在RULES字典中加入：
|
||||||
|
|
||||||
|
``` python
|
||||||
|
'www.ybdu.com': Rules('0', {'class': 'mulu_list'}, {'id': 'htmlContent'}),
|
||||||
|
```
|
||||||
|
这时候再重启服务，刷新页面：
|
||||||
|
|
||||||
|
可以看到未解析变成了已解析，这时候点击进入，会出现两种情况：
|
||||||
|
|
||||||
|
- 排版完整，不需要再写样式，此时请直接解析章节内容
|
||||||
|
- 排版不行，需要写样
|
||||||
|
|
||||||
|
现在我演示的这个链接排版就是显示不行，请打开`/static/novels/css/chapter.css`文件，在编写之前，请先看下面这张图：
|
||||||
|
|
||||||
|
|
||||||
|
图中.mulu_list1：打错了表示区域1，.mulu_list1：表示区域3.
|
||||||
|
|
||||||
|
首先更改区域1的样式，我已经在代码注释写了各个区域样式代码在哪，直接更改就行，比如：
|
||||||
|
|
||||||
|
``` css
|
||||||
|
/* 区域1 */
|
||||||
|
.mulu_list, .acss, .list, #xslist ul, .dirlist, .list-chapter, .list_box ul, #defaulthtml4 table, .article-list > dl, .update, #list .box, .bookcontent > dl, .listmain > dl, .ml_main > dl, #list dl, #chapter_list, .chapterlist, tbody, .mt10, .catalog, #readerlists {
|
||||||
|
float: left;
|
||||||
|
overflow: hidden;
|
||||||
|
padding-bottom: 1px;
|
||||||
|
margin: auto;
|
||||||
|
background: #F6F4EC;
|
||||||
|
border-radius: 15px;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
可以看到加上了`.mulu_list`，以此类推，最后解析完毕之后如下图：
|
||||||
|
|
||||||
|
|
||||||
|
注意章节内容里面一些无关的链接以及原本网页自带的上一章下一章需要隐藏掉哦。
|
||||||
|
|
||||||
|
## 3.总结
|
||||||
|
功能还很简单，解析的网站有很多，希望各位添砖加瓦，容我奉上项目地址：[owllook--novels_search](https://github.com/howie6879/novels-search)
|
||||||
|
|
||||||
|
**仅供分享交流 不可用于商业用途**
|
||||||
|
|
||||||
|
|
||||||
64
docs/规则定义.md
Normal file
64
docs/规则定义.md
Normal file
@ -0,0 +1,64 @@
|
|||||||
|
## 规则定义说明：
|
||||||
|
|
||||||
|
对于各类小说网站的解析，只需按照这个要求进行定义，解析便会尽量自动进行，总规则如下：
|
||||||
|
|
||||||
|
```python
|
||||||
|
# 在项目下config/config.py文件中可以看到
|
||||||
|
Rules = namedtuple('Rules', 'content_url chapter_selector content_selector')
|
||||||
|
# 在自定义字典RULES下面可以看到各个网站的规则，仅限class和id
|
||||||
|
# demo 'name': Rules('content_url', {chapter_selector}, {content_selector})
|
||||||
|
```
|
||||||
|
|
||||||
|
对于各个`namedtuple`里面的`content_url，chapter_selector，content_selector`，下面通过不同类型的网站一一进行解释：
|
||||||
|
|
||||||
|
- http://www.bxwx9.org/b/57/57181/
|
||||||
|
|
||||||
|
打开这个网站，审查元素后你会发现：
|
||||||
|
|
||||||
|
- url输入栏显示内容为`http://www.bxwx9.org/b/57/57181/`
|
||||||
|
- 主要内容包含在`.TabCss`内
|
||||||
|
- 第一章的href值为`8845907.html`，由此可知该章的链接为`http://www.bxwx9.org/b/57/57181/8845907.html`
|
||||||
|
|
||||||
|
最后写出规则：
|
||||||
|
|
||||||
|
```python
|
||||||
|
RULES={
|
||||||
|
'www.bxwx9.org': Rules('0', {'class': 'TabCss'}, {})
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
`key`值起判断该网站是否被解析，注意此时`content_url`为0，这是表示在解析小说目录的时候，每章节对应的url在拼接过程中需要的是当前的url。对于`content_url`，请看下表：
|
||||||
|
|
||||||
|
| 值 | 说明 |
|
||||||
|
| :----: | :------------------: |
|
||||||
|
| -1 | 不解析，跳转到原本网页 |
|
||||||
|
| 0 | 表示章节网页需要当前页面url拼接 |
|
||||||
|
| 1 | 表示章节链接使用本身自带的链接，不用拼接 |
|
||||||
|
| netloc | 用域名进行拼接 |
|
||||||
|
|
||||||
|
`chapter_selector`解析章节目录的规则
|
||||||
|
|
||||||
|
`content_selector`解析目录内容的规则
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
- http://www.23us.la/html/0/42/
|
||||||
|
|
||||||
|
```python
|
||||||
|
RULES={
|
||||||
|
'www.23us.la': Rules('http://www.23us.la', {'class': 'inner'}, {})
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
- http://zetianjiba.net
|
||||||
|
|
||||||
|
```python
|
||||||
|
RULES={
|
||||||
|
'zetianjiba.net': Rules('1', {'class': 'bg'}, {})
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
`content_selector`值的获取方式同上
|
||||||
34
examples/latest_chapter.py
Normal file
34
examples/latest_chapter.py
Normal file
@ -0,0 +1,34 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
import asyncio
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
import uvloop
|
||||||
|
|
||||||
|
from pprint import pprint
|
||||||
|
|
||||||
|
os.environ['MODE'] = 'PRO'
|
||||||
|
from soulbook.fetcher.cache import get_the_latest_chapter, update_all_books
|
||||||
|
|
||||||
|
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
|
||||||
|
|
||||||
|
|
||||||
|
def latest_chapter_task(url):
|
||||||
|
loop = asyncio.get_event_loop()
|
||||||
|
task = asyncio.ensure_future(get_the_latest_chapter(url))
|
||||||
|
loop.run_until_complete(task)
|
||||||
|
return task.result()
|
||||||
|
|
||||||
|
|
||||||
|
def update_all():
|
||||||
|
loop = asyncio.get_event_loop()
|
||||||
|
task = asyncio.ensure_future(update_all_books())
|
||||||
|
loop.run_until_complete(task)
|
||||||
|
return task.result()
|
||||||
|
|
||||||
|
|
||||||
|
start = time.time()
|
||||||
|
chapter_url = "/chapter?url=http://www.8535.org/xuanhuan/139075/&novels_name=圣墟"
|
||||||
|
|
||||||
|
result = latest_chapter_task(chapter_url)
|
||||||
|
pprint(result)
|
||||||
|
print(time.time() - start)
|
||||||
173
examples/novels/baidu_novels.py
Normal file
173
examples/novels/baidu_novels.py
Normal file
@ -0,0 +1,173 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
import aiohttp
|
||||||
|
import arrow
|
||||||
|
import asyncio
|
||||||
|
import async_timeout
|
||||||
|
import re
|
||||||
|
|
||||||
|
from bs4 import BeautifulSoup
|
||||||
|
from urllib.parse import urlparse
|
||||||
|
|
||||||
|
from soulbook.fetcher.function import get_random_user_agent
|
||||||
|
from soulbook.config import CONFIG, LOGGER, BLACK_DOMAIN, RULES, LATEST_RULES
|
||||||
|
|
||||||
|
|
||||||
|
async def fetch(client, url, name, is_web):
|
||||||
|
with async_timeout.timeout(15):
|
||||||
|
try:
|
||||||
|
headers = {'user-agent': await get_random_user_agent()}
|
||||||
|
if is_web:
|
||||||
|
params = {'wd': name, 'ie': 'utf-8', 'rn': CONFIG.BAIDU_RN, 'vf_bl': 1}
|
||||||
|
else:
|
||||||
|
params = {'word': name}
|
||||||
|
async with client.get(url, params=params, headers=headers) as response:
|
||||||
|
assert response.status == 200
|
||||||
|
LOGGER.info('Task url: {}'.format(response.url))
|
||||||
|
try:
|
||||||
|
text = await response.text()
|
||||||
|
except:
|
||||||
|
text = await response.read()
|
||||||
|
return text
|
||||||
|
except Exception as e:
|
||||||
|
LOGGER.exception(e)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
async def get_real_url(client, url):
|
||||||
|
with async_timeout.timeout(5):
|
||||||
|
try:
|
||||||
|
headers = {'user-agent': await get_random_user_agent()}
|
||||||
|
async with client.head(url, headers=headers, allow_redirects=True) as response:
|
||||||
|
assert response.status == 200
|
||||||
|
LOGGER.info('Parse url: {}'.format(response.url))
|
||||||
|
# text = ""
|
||||||
|
# try:
|
||||||
|
# text = await response.text()
|
||||||
|
# except:
|
||||||
|
# text = await response.read()
|
||||||
|
# if text:
|
||||||
|
# print(text)
|
||||||
|
# text = re.findall(r'replace\(\"(.*?)\"\)', str(text))
|
||||||
|
# text = text[0] if text[0] else ""
|
||||||
|
url = response.url if response.url else None
|
||||||
|
return url
|
||||||
|
except Exception as e:
|
||||||
|
LOGGER.exception(e)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
async def data_extraction_for_phone(html):
|
||||||
|
with async_timeout.timeout(10):
|
||||||
|
try:
|
||||||
|
# Get title
|
||||||
|
data_log = eval(html['data-log'])
|
||||||
|
url = data_log.get('mu', None)
|
||||||
|
if not url:
|
||||||
|
return None
|
||||||
|
# Get title
|
||||||
|
title = html.find('h3').get_text()
|
||||||
|
# Get author and update_time (option)
|
||||||
|
novel_mess = html.findAll(class_='c-gap-right-large')
|
||||||
|
basic_mess = [i.get_text() for i in novel_mess] if novel_mess else None
|
||||||
|
return {'title': title, 'url': url, 'basic_mess': basic_mess}
|
||||||
|
except Exception as e:
|
||||||
|
LOGGER.exception(e)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
async def data_extraction_for_web(html):
|
||||||
|
with async_timeout.timeout(10):
|
||||||
|
try:
|
||||||
|
url = html.find('a').get('href', None)
|
||||||
|
if not url or 'baidu' in url or urlparse(url).netloc in BLACK_DOMAIN:
|
||||||
|
return None
|
||||||
|
netloc = urlparse(url).netloc
|
||||||
|
is_parse = 1 if netloc in RULES.keys() else 0
|
||||||
|
title = html.select('font[size="3"]')[0].get_text()
|
||||||
|
source = html.select('font[color="#008000"]')[0].get_text()
|
||||||
|
time = re.findall(r'\d+-\d+-\d+', source)
|
||||||
|
time = time[0] if time else None
|
||||||
|
timestamp = 0
|
||||||
|
if time:
|
||||||
|
try:
|
||||||
|
time_list = [int(i) for i in time.split('-')]
|
||||||
|
timestamp = arrow.get(time_list[0], time_list[1], time_list[2]).timestamp
|
||||||
|
except Exception as e:
|
||||||
|
LOGGER.exception(e)
|
||||||
|
timestamp = 0
|
||||||
|
return {'title': title, 'url': url.replace('index.html', '').replace('Index.html', ''), 'time': time,
|
||||||
|
'is_parse': is_parse,
|
||||||
|
'timestamp': timestamp,
|
||||||
|
'netloc': netloc}
|
||||||
|
except Exception as e:
|
||||||
|
LOGGER.exception(e)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
async def data_extraction_for_web_baidu(client, html):
|
||||||
|
with async_timeout.timeout(20):
|
||||||
|
try:
|
||||||
|
url = html.select('h3.t a')[0].get('href', None)
|
||||||
|
real_url = await get_real_url(client=client, url=url) if url else None
|
||||||
|
if real_url:
|
||||||
|
real_str_url = str(real_url)
|
||||||
|
netloc = urlparse(real_str_url).netloc
|
||||||
|
if "http://" + netloc + "/" == real_str_url:
|
||||||
|
return None
|
||||||
|
if 'baidu' in real_str_url or netloc in BLACK_DOMAIN:
|
||||||
|
return None
|
||||||
|
is_parse = 1 if netloc in RULES.keys() else 0
|
||||||
|
title = html.select('h3.t a')[0].get_text()
|
||||||
|
is_recommend = 1 if netloc in LATEST_RULES.keys() else 0
|
||||||
|
# time = re.findall(r'\d+-\d+-\d+', source)
|
||||||
|
# time = time[0] if time else None
|
||||||
|
timestamp = 0
|
||||||
|
time = ""
|
||||||
|
# if time:
|
||||||
|
# try:
|
||||||
|
# time_list = [int(i) for i in time.split('-')]
|
||||||
|
# timestamp = arrow.get(time_list[0], time_list[1], time_list[2]).timestamp
|
||||||
|
# except Exception as e:
|
||||||
|
# LOGGER.exception(e)
|
||||||
|
# timestamp = 0
|
||||||
|
return {'title': title, 'url': real_str_url.replace('index.html', ''), 'time': time,
|
||||||
|
'is_parse': is_parse,
|
||||||
|
'is_recommend': is_recommend,
|
||||||
|
'timestamp': timestamp,
|
||||||
|
'netloc': netloc}
|
||||||
|
else:
|
||||||
|
return None
|
||||||
|
except Exception as e:
|
||||||
|
LOGGER.exception(e)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
async def baidu_search(name, is_web=1):
|
||||||
|
url = CONFIG.URL_PC if is_web else CONFIG.URL_PHONE
|
||||||
|
async with aiohttp.ClientSession() as client:
|
||||||
|
html = await fetch(client=client, url=url, name=name, is_web=is_web)
|
||||||
|
if html:
|
||||||
|
soup = BeautifulSoup(html, 'html5lib')
|
||||||
|
if is_web:
|
||||||
|
# result = soup.find_all(class_='f')
|
||||||
|
result = soup.find_all(class_='result')
|
||||||
|
extra_tasks = [data_extraction_for_web_baidu(client=client, html=i) for i in result]
|
||||||
|
tasks = [asyncio.ensure_future(i) for i in extra_tasks]
|
||||||
|
else:
|
||||||
|
result = soup.find_all(class_='result c-result c-clk-recommend')
|
||||||
|
extra_tasks = [data_extraction_for_phone(i) for i in result]
|
||||||
|
tasks = [asyncio.ensure_future(i) for i in extra_tasks]
|
||||||
|
# return await asyncio.gather(*tasks)
|
||||||
|
done_list, pending_list = await asyncio.wait(tasks)
|
||||||
|
res = []
|
||||||
|
for task in done_list:
|
||||||
|
res.append(task.result())
|
||||||
|
return res
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
import time
|
||||||
|
|
||||||
|
start = time.time()
|
||||||
|
print(asyncio.get_event_loop().run_until_complete(baidu_search('雪中悍刀行')))
|
||||||
|
print(time.time() - start)
|
||||||
114
examples/novels/bing_novels.py
Normal file
114
examples/novels/bing_novels.py
Normal file
@ -0,0 +1,114 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""
|
||||||
|
Created by howie.hu at 21/11/2017.
|
||||||
|
"""
|
||||||
|
import aiohttp
|
||||||
|
import asyncio
|
||||||
|
import async_timeout
|
||||||
|
|
||||||
|
from bs4 import BeautifulSoup
|
||||||
|
from urllib.parse import urlparse
|
||||||
|
|
||||||
|
from soulbook.fetcher.function import get_random_user_agent
|
||||||
|
from soulbook.config import CONFIG, LOGGER, BLACK_DOMAIN, RULES, LATEST_RULES
|
||||||
|
|
||||||
|
|
||||||
|
async def fetch(client, url, novels_name):
|
||||||
|
with async_timeout.timeout(20):
|
||||||
|
try:
|
||||||
|
headers = {
|
||||||
|
'user-agent': await get_random_user_agent(),
|
||||||
|
'referer': "https://www.bing.com/"
|
||||||
|
}
|
||||||
|
print(headers)
|
||||||
|
params = {'q': novels_name, 'ensearch': 0}
|
||||||
|
async with client.get(url, params=params, headers=headers) as response:
|
||||||
|
assert response.status == 200
|
||||||
|
LOGGER.info('Task url: {}'.format(response.url))
|
||||||
|
try:
|
||||||
|
text = await response.text()
|
||||||
|
except:
|
||||||
|
text = await response.read()
|
||||||
|
return text
|
||||||
|
except Exception as e:
|
||||||
|
LOGGER.exception(e)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
async def data_extraction_for_web_bing(client, html):
|
||||||
|
with async_timeout.timeout(15):
|
||||||
|
try:
|
||||||
|
try:
|
||||||
|
title = html.select('h2 a')[0].get_text()
|
||||||
|
url = html.select('h2 a')[0].get('href', None)
|
||||||
|
netloc = urlparse(url).netloc
|
||||||
|
url = url.replace('index.html', '').replace('Index.html', '')
|
||||||
|
if not url or 'baidu' in url or 'baike.so.com' in url or netloc in BLACK_DOMAIN or '.html' in url:
|
||||||
|
return None
|
||||||
|
is_parse = 1 if netloc in RULES.keys() else 0
|
||||||
|
is_recommend = 1 if netloc in LATEST_RULES.keys() else 0
|
||||||
|
# time = html.select('div.b_attribution')[0].get_text()
|
||||||
|
# time = re.findall(r'\d+-\d+-\d+', time)
|
||||||
|
# time = time[0] if time else ''
|
||||||
|
timestamp = 0
|
||||||
|
time = ''
|
||||||
|
# if time:
|
||||||
|
# try:
|
||||||
|
# time_list = [int(i) for i in time.split('-')]
|
||||||
|
# years = str(time_list[0])[-4:]
|
||||||
|
# timestamp = arrow.get(int(years), time_list[1], time_list[2]).timestamp
|
||||||
|
# time = years + "-" + str(time_list[1]) + "-" + str(time_list[2])
|
||||||
|
# except Exception as e:
|
||||||
|
# LOGGER.exception(e)
|
||||||
|
# timestamp = 0
|
||||||
|
return {'title': title,
|
||||||
|
'url': url,
|
||||||
|
'time': time,
|
||||||
|
'is_parse': is_parse,
|
||||||
|
'is_recommend': is_recommend,
|
||||||
|
'timestamp': timestamp,
|
||||||
|
'netloc': netloc}
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
LOGGER.exception(e)
|
||||||
|
url, title = None, None
|
||||||
|
return None
|
||||||
|
except Exception as e:
|
||||||
|
LOGGER.exception(e)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
async def bing_search(novels_name):
|
||||||
|
url = CONFIG.BY_URL
|
||||||
|
async with aiohttp.ClientSession() as client:
|
||||||
|
html = await fetch(client=client, url=url, novels_name=novels_name)
|
||||||
|
if html:
|
||||||
|
soup = BeautifulSoup(html, 'html5lib')
|
||||||
|
result = soup.find_all(class_='b_algo')
|
||||||
|
extra_tasks = [data_extraction_for_web_bing(client=client, html=i) for i in result]
|
||||||
|
tasks = [asyncio.ensure_future(i) for i in extra_tasks]
|
||||||
|
return await asyncio.gather(*tasks)
|
||||||
|
else:
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
import uvloop
|
||||||
|
import time
|
||||||
|
|
||||||
|
from pprint import pprint
|
||||||
|
|
||||||
|
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
|
||||||
|
|
||||||
|
|
||||||
|
def novel_task(name):
|
||||||
|
loop = asyncio.get_event_loop()
|
||||||
|
task = asyncio.ensure_future(bing_search(name))
|
||||||
|
loop.run_until_complete(task)
|
||||||
|
return task.result()
|
||||||
|
|
||||||
|
|
||||||
|
start = time.time()
|
||||||
|
result = novel_task('牧神记 小说 阅读 最新章节')
|
||||||
|
pprint(result)
|
||||||
|
print(time.time() - start)
|
||||||
106
examples/novels/duck_go_novels.py
Normal file
106
examples/novels/duck_go_novels.py
Normal file
@ -0,0 +1,106 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""
|
||||||
|
Created by howie.hu at 23/11/2017.
|
||||||
|
"""
|
||||||
|
import aiohttp
|
||||||
|
import asyncio
|
||||||
|
import async_timeout
|
||||||
|
|
||||||
|
from bs4 import BeautifulSoup
|
||||||
|
from urllib.parse import parse_qs, urlparse
|
||||||
|
|
||||||
|
from soulbook.fetcher.function import get_random_user_agent
|
||||||
|
from soulbook.config import CONFIG, LOGGER, BLACK_DOMAIN, RULES, LATEST_RULES
|
||||||
|
|
||||||
|
|
||||||
|
async def fetch(client, url, novels_name):
|
||||||
|
with async_timeout.timeout(20):
|
||||||
|
try:
|
||||||
|
headers = {
|
||||||
|
'user-agent': await get_random_user_agent(),
|
||||||
|
'referer': "https://duckduckgo.com/"
|
||||||
|
}
|
||||||
|
params = {'q': novels_name}
|
||||||
|
# , proxy="http://127.0.0.1:8118"
|
||||||
|
async with client.get(url, params=params, headers=headers) as response:
|
||||||
|
assert response.status == 200
|
||||||
|
LOGGER.info('Task url: {}'.format(response.url))
|
||||||
|
try:
|
||||||
|
text = await response.text()
|
||||||
|
except:
|
||||||
|
text = await response.read()
|
||||||
|
return text
|
||||||
|
except Exception as e:
|
||||||
|
LOGGER.exception(e)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
async def data_extraction_for_web_duck(client, html):
|
||||||
|
with async_timeout.timeout(15):
|
||||||
|
try:
|
||||||
|
try:
|
||||||
|
title = html.select('h2 a')[0].get_text()
|
||||||
|
url = html.select('h2 a')[0].get('href', None)
|
||||||
|
url = parse_qs(url).get('uddg', ['#'])[0]
|
||||||
|
netloc = urlparse(url).netloc
|
||||||
|
url = url.replace('index.html', '').replace('Index.html', '')
|
||||||
|
if not url or 'baidu' in url or 'baike.so.com' in url or netloc in BLACK_DOMAIN or '.html' in url:
|
||||||
|
return None
|
||||||
|
is_parse = 1 if netloc in RULES.keys() else 0
|
||||||
|
is_recommend = 1 if netloc in LATEST_RULES.keys() else 0
|
||||||
|
# time = html.select('div.b_attribution')[0].get_text()
|
||||||
|
# time = re.findall(r'\d+-\d+-\d+', time)
|
||||||
|
# time = time[0] if time else ''
|
||||||
|
timestamp = 0
|
||||||
|
time = ''
|
||||||
|
return {'title': title,
|
||||||
|
'url': url,
|
||||||
|
'time': time,
|
||||||
|
'is_parse': is_parse,
|
||||||
|
'is_recommend': is_recommend,
|
||||||
|
'timestamp': timestamp,
|
||||||
|
'netloc': netloc}
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
LOGGER.exception(e)
|
||||||
|
url, title = None, None
|
||||||
|
return None
|
||||||
|
except Exception as e:
|
||||||
|
LOGGER.exception(e)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
async def duck_search(novels_name):
|
||||||
|
url = CONFIG.DUCKGO_URL
|
||||||
|
async with aiohttp.ClientSession() as client:
|
||||||
|
html = await fetch(client=client, url=url, novels_name=novels_name)
|
||||||
|
if html:
|
||||||
|
soup = BeautifulSoup(html, 'html5lib')
|
||||||
|
result = soup.find_all(class_='result')
|
||||||
|
extra_tasks = [data_extraction_for_web_duck(client=client, html=i) for i in result]
|
||||||
|
tasks = [asyncio.ensure_future(i) for i in extra_tasks]
|
||||||
|
return await asyncio.gather(*tasks)
|
||||||
|
else:
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
import uvloop
|
||||||
|
import time
|
||||||
|
|
||||||
|
from pprint import pprint
|
||||||
|
|
||||||
|
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
|
||||||
|
|
||||||
|
|
||||||
|
def novel_task(name):
|
||||||
|
loop = asyncio.get_event_loop()
|
||||||
|
task = asyncio.ensure_future(duck_search(name))
|
||||||
|
loop.run_until_complete(task)
|
||||||
|
return task.result()
|
||||||
|
|
||||||
|
|
||||||
|
start = time.time()
|
||||||
|
result = novel_task('圣墟 小说 阅读 最新章节')
|
||||||
|
pprint(result)
|
||||||
|
print(time.time() - start)
|
||||||
20
examples/novels/novels_schedule.py
Normal file
20
examples/novels/novels_schedule.py
Normal file
@ -0,0 +1,20 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
import asyncio
|
||||||
|
import os
|
||||||
|
import uvloop
|
||||||
|
|
||||||
|
os.environ['MODE'] = 'PRO'
|
||||||
|
from soulbook.fetcher.cache import update_all_books
|
||||||
|
|
||||||
|
|
||||||
|
def update_all():
|
||||||
|
# asyncio.get_event_loop().run_until_complete(update_all_books())
|
||||||
|
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
|
||||||
|
loop = asyncio.get_event_loop()
|
||||||
|
task = asyncio.ensure_future(update_all_books(loop=loop, timeout=15))
|
||||||
|
loop.run_until_complete(task)
|
||||||
|
return task.result() or None
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
print(update_all())
|
||||||
26
examples/novels/parse.py
Normal file
26
examples/novels/parse.py
Normal file
@ -0,0 +1,26 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
import aiohttp
|
||||||
|
import asyncio
|
||||||
|
import time
|
||||||
|
import uvloop
|
||||||
|
|
||||||
|
from pprint import pprint
|
||||||
|
|
||||||
|
from soulbook.fetcher.function import target_fetch
|
||||||
|
|
||||||
|
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
|
||||||
|
|
||||||
|
|
||||||
|
def novel_task(url):
|
||||||
|
loop = asyncio.get_event_loop()
|
||||||
|
|
||||||
|
task = asyncio.ensure_future(target_fetch(url, {}))
|
||||||
|
loop.run_until_complete(task)
|
||||||
|
return task.result()
|
||||||
|
|
||||||
|
|
||||||
|
start = time.time()
|
||||||
|
# result = novel_task('http://www.shuge.net/html/98/98044')
|
||||||
|
result = novel_task('http://www.biqugexsw.com/1_1020/')
|
||||||
|
pprint(result)
|
||||||
|
print(time.time() - start)
|
||||||
119
examples/novels/so_novels.py
Normal file
119
examples/novels/so_novels.py
Normal file
@ -0,0 +1,119 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
import aiohttp
|
||||||
|
import asyncio
|
||||||
|
import async_timeout
|
||||||
|
|
||||||
|
from bs4 import BeautifulSoup
|
||||||
|
from urllib.parse import parse_qs, urlparse
|
||||||
|
|
||||||
|
from soulbook.fetcher.function import get_random_user_agent
|
||||||
|
from soulbook.config import CONFIG, LOGGER, BLACK_DOMAIN, RULES, LATEST_RULES
|
||||||
|
|
||||||
|
|
||||||
|
async def fetch(client, url, novels_name):
|
||||||
|
with async_timeout.timeout(20):
|
||||||
|
try:
|
||||||
|
headers = {
|
||||||
|
'User-Agent': await get_random_user_agent(),
|
||||||
|
'Referer': "http://www.so.com/haosou.html?src=home"
|
||||||
|
}
|
||||||
|
params = {'ie': 'utf-8', 'src': 'noscript_home', 'shb': 1, 'q': novels_name, }
|
||||||
|
async with client.get(url, params=params, headers=headers) as response:
|
||||||
|
assert response.status == 200
|
||||||
|
LOGGER.info('Task url: {}'.format(response.url))
|
||||||
|
try:
|
||||||
|
text = await response.text()
|
||||||
|
except:
|
||||||
|
text = await response.read()
|
||||||
|
return text
|
||||||
|
except Exception as e:
|
||||||
|
LOGGER.exception(e)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
async def data_extraction_for_web_so(client, html):
|
||||||
|
with async_timeout.timeout(15):
|
||||||
|
try:
|
||||||
|
# 2017.09.09 修改 更加全面地获取title && url
|
||||||
|
try:
|
||||||
|
title = html.select('h3 a')[0].get_text()
|
||||||
|
url = html.select('h3 a')[0].get('href', None)
|
||||||
|
except Exception as e:
|
||||||
|
LOGGER.exception(e)
|
||||||
|
url, title = None, None
|
||||||
|
return None
|
||||||
|
# 针对不同的请进行url的提取
|
||||||
|
if "www.so.com/link?m=" in url:
|
||||||
|
url = html.select('h3 a')[0].get('data-url', None)
|
||||||
|
if "www.so.com/link?url=" in url:
|
||||||
|
url = parse_qs(urlparse(url).query).get('url', None)
|
||||||
|
url = url[0] if url else None
|
||||||
|
|
||||||
|
# try:
|
||||||
|
# url = html.select('h3.res-title a')[0].get('data-url', None)
|
||||||
|
# title = html.select('h3.res-title a')[0].get_text()
|
||||||
|
# except IndexError:
|
||||||
|
# url = html.select('h3.title a')[0].get('href', None)
|
||||||
|
# url = parse_qs(urlparse(url).query).get('url', None)
|
||||||
|
# url = url[0] if url else None
|
||||||
|
# title = html.select('h3.title a')[0].get_text()
|
||||||
|
# except Exception as e:
|
||||||
|
# LOGGER.exception(e)
|
||||||
|
# url, title = None, None
|
||||||
|
# return None
|
||||||
|
|
||||||
|
# 2017.07.09 此处出现bug url展示形式发生变化 因此对于h3.title a形式依旧不变 但是h3.res-title a则取属性data-url
|
||||||
|
# url = parse_qs(urlparse(url).query).get('url', None)
|
||||||
|
# url = url[0] if url else None
|
||||||
|
|
||||||
|
netloc = urlparse(url).netloc
|
||||||
|
if not url or 'baidu' in url or 'baike.so.com' in url or netloc in BLACK_DOMAIN:
|
||||||
|
return None
|
||||||
|
is_parse = 1 if netloc in RULES.keys() else 0
|
||||||
|
is_recommend = 1 if netloc in LATEST_RULES.keys() else 0
|
||||||
|
time = ''
|
||||||
|
timestamp = 0
|
||||||
|
return {'title': title, 'url': url.replace('index.html', '').replace('Index.html', ''), 'time': time,
|
||||||
|
'is_parse': is_parse,
|
||||||
|
'is_recommend': is_recommend,
|
||||||
|
'timestamp': timestamp,
|
||||||
|
'netloc': netloc}
|
||||||
|
except Exception as e:
|
||||||
|
LOGGER.exception(e)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
async def so_search(novels_name):
|
||||||
|
url = CONFIG.SO_URL
|
||||||
|
async with aiohttp.ClientSession() as client:
|
||||||
|
html = await fetch(client=client, url=url, novels_name=novels_name)
|
||||||
|
if html:
|
||||||
|
soup = BeautifulSoup(html, 'html5lib')
|
||||||
|
result = soup.find_all(class_='res-list')
|
||||||
|
extra_tasks = [data_extraction_for_web_so(client=client, html=i) for i in result]
|
||||||
|
tasks = [asyncio.ensure_future(i) for i in extra_tasks]
|
||||||
|
return await asyncio.gather(*tasks)
|
||||||
|
else:
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
import uvloop
|
||||||
|
import time
|
||||||
|
|
||||||
|
from pprint import pprint
|
||||||
|
|
||||||
|
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
|
||||||
|
|
||||||
|
|
||||||
|
def novel_task(name):
|
||||||
|
loop = asyncio.get_event_loop()
|
||||||
|
task = asyncio.ensure_future(so_search(name))
|
||||||
|
loop.run_until_complete(task)
|
||||||
|
return task.result()
|
||||||
|
|
||||||
|
|
||||||
|
start = time.time()
|
||||||
|
result = novel_task('圣墟 小说 阅读 最新章节')
|
||||||
|
pprint(result)
|
||||||
|
print(time.time() - start)
|
||||||
72
examples/user_recommend.py
Normal file
72
examples/user_recommend.py
Normal file
@ -0,0 +1,72 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
import asyncio
|
||||||
|
import collections
|
||||||
|
import os
|
||||||
|
from operator import itemgetter
|
||||||
|
|
||||||
|
import uvloop
|
||||||
|
|
||||||
|
os.environ['MODE'] = 'DEV'
|
||||||
|
|
||||||
|
from pprint import pprint
|
||||||
|
from copy import deepcopy
|
||||||
|
from soulbook.database.mongodb import MotorBase
|
||||||
|
from soulbook.recommend.cosinesimilarity import CosineSimilarity
|
||||||
|
from soulbook.fetcher.function import get_time
|
||||||
|
|
||||||
|
|
||||||
|
async def get_user_tag():
|
||||||
|
motor_db = MotorBase().get_db()
|
||||||
|
user_tag_cursor = motor_db.user_tag.find({}, {'data': 1, 'user': 1, '_id': 0})
|
||||||
|
result = {}
|
||||||
|
user_book_dict = {}
|
||||||
|
|
||||||
|
async for document in user_tag_cursor:
|
||||||
|
if document['data']:
|
||||||
|
user_book_dict[document['user'].replace('.', '·')] = document['data']
|
||||||
|
result[document['user'].replace('.', '·')] = document['data']['user_tag']
|
||||||
|
|
||||||
|
for key, value in result.items():
|
||||||
|
if not value:
|
||||||
|
continue
|
||||||
|
print("\nUser:", key)
|
||||||
|
print("User tags:", set(value))
|
||||||
|
print("User books:", user_book_dict[key]["user_novels"])
|
||||||
|
result_copy = deepcopy(result)
|
||||||
|
del result_copy[key]
|
||||||
|
cos = CosineSimilarity(value, result_copy)
|
||||||
|
vector = cos.create_vector()
|
||||||
|
resultDic = cos.calculate(vector)
|
||||||
|
print("相似用户:")
|
||||||
|
pprint(resultDic[:10])
|
||||||
|
# pprint(type(resultList[1]))
|
||||||
|
booksDic = collections.defaultdict(float)
|
||||||
|
for userBooks in resultDic:
|
||||||
|
for simuser, simrate in userBooks.items():
|
||||||
|
# print(simuser, simrate)
|
||||||
|
for book in user_book_dict[simuser]["user_novels"]:
|
||||||
|
booksDic[book] += simrate
|
||||||
|
# print(booksDic)
|
||||||
|
# 推荐20本书，注意这里的推荐并没有去除用户收藏的书籍
|
||||||
|
recommend = sorted(booksDic.items(), key=itemgetter(1), reverse=True)[0:20]
|
||||||
|
print("书籍推荐：")
|
||||||
|
pprint(recommend)
|
||||||
|
recommend_novels = [book for book, simrate in recommend]
|
||||||
|
await motor_db.user_recommend.update_one(
|
||||||
|
{"user": key},
|
||||||
|
{'$set': {'similar_user': resultDic, 'user_tag': result[key], 'recommend_novels': recommend_novels,
|
||||||
|
"updated_at": get_time()}}, upsert=True)
|
||||||
|
|
||||||
|
|
||||||
|
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
|
||||||
|
|
||||||
|
|
||||||
|
def get_user_tag_test():
|
||||||
|
loop = asyncio.get_event_loop()
|
||||||
|
task = asyncio.ensure_future(get_user_tag())
|
||||||
|
loop.run_until_complete(task)
|
||||||
|
return task.result()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
get_user_tag_test()
|
||||||
90
examples/user_tag.py
Normal file
90
examples/user_tag.py
Normal file
@ -0,0 +1,90 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
import asyncio
|
||||||
|
import os
|
||||||
|
import pymongo
|
||||||
|
import uvloop
|
||||||
|
|
||||||
|
os.environ['MODE'] = 'PRO'
|
||||||
|
|
||||||
|
from pprint import pprint
|
||||||
|
from urllib.parse import parse_qs, urlparse
|
||||||
|
|
||||||
|
from soulbook.database.mongodb import MotorBase
|
||||||
|
from soulbook.fetcher.function import get_time
|
||||||
|
|
||||||
|
# mongo
|
||||||
|
MONGODB_SERVER = "localhost"
|
||||||
|
MONGODB_PORT = 27017
|
||||||
|
MONGODB_USERNAME = ""
|
||||||
|
MONGODB_PASSWORD = ""
|
||||||
|
MONGODB_DB = "soulbook"
|
||||||
|
MONGODB_COLLECTION = "all_novels_info"
|
||||||
|
|
||||||
|
|
||||||
|
class Novels(object):
|
||||||
|
def __init__(self):
|
||||||
|
_mongo_uri = 'mongodb://{account}{host}:{port}/{database}'.format(
|
||||||
|
account='{username}:{password}@'.format(
|
||||||
|
username=MONGODB_USERNAME,
|
||||||
|
password=MONGODB_PASSWORD) if MONGODB_USERNAME else '',
|
||||||
|
host='localhost',
|
||||||
|
port=27017,
|
||||||
|
database=MONGODB_DB)
|
||||||
|
connection = pymongo.MongoClient(_mongo_uri)
|
||||||
|
db = connection[MONGODB_DB]
|
||||||
|
self.collection = db[MONGODB_COLLECTION]
|
||||||
|
self.result_dict = dict()
|
||||||
|
|
||||||
|
def search_name(self, name):
|
||||||
|
if name in self.result_dict:
|
||||||
|
return self.result_dict[name] if self.result_dict[name] else False
|
||||||
|
result = self.collection.find_one({'novel_name': name})
|
||||||
|
if not result:
|
||||||
|
self.result_dict[name] = False
|
||||||
|
return False
|
||||||
|
self.result_dict[name] = {'author': result['author'], 'novels_type': result['novels_type'].split('#')}
|
||||||
|
return self.result_dict[name]
|
||||||
|
|
||||||
|
|
||||||
|
async def get_tag():
|
||||||
|
motor_db = MotorBase().get_db()
|
||||||
|
novels = Novels()
|
||||||
|
# 获取所有书架链接游标
|
||||||
|
books_url_cursor = motor_db.user_message.find({}, {'books_url.book_url': 1, 'user': 1, '_id': 0})
|
||||||
|
async for document in books_url_cursor:
|
||||||
|
if document:
|
||||||
|
books_url = document.get('books_url', None)
|
||||||
|
user = document['user']
|
||||||
|
if books_url:
|
||||||
|
all_user = {user + '_novels': [], user + '_tag': [], user + '_author': []}
|
||||||
|
for book_url in books_url:
|
||||||
|
chapter_url = book_url['book_url']
|
||||||
|
novels_name = parse_qs(urlparse(chapter_url).query).get('novels_name', '')[0]
|
||||||
|
all_user[user + '_novels'].append(novels_name)
|
||||||
|
novels_info = novels.search_name(novels_name)
|
||||||
|
if novels_info:
|
||||||
|
all_user[user + '_author'].append(novels_info['author'])
|
||||||
|
all_user[user + '_tag'].extend(novels_info['novels_type'])
|
||||||
|
data = {
|
||||||
|
'user_novels': all_user[user + '_novels'],
|
||||||
|
'user_tag': all_user[user + '_tag'],
|
||||||
|
'user_author': all_user[user + '_author'],
|
||||||
|
}
|
||||||
|
await motor_db.user_tag.update_one(
|
||||||
|
{"user": user},
|
||||||
|
{'$set': {'data': data, "updated_at": get_time()}}, upsert=True)
|
||||||
|
pprint(data)
|
||||||
|
|
||||||
|
|
||||||
|
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
|
||||||
|
|
||||||
|
|
||||||
|
def tag_test():
|
||||||
|
loop = asyncio.get_event_loop()
|
||||||
|
task = asyncio.ensure_future(get_tag())
|
||||||
|
loop.run_until_complete(task)
|
||||||
|
return task.result()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
tag_test()
|
||||||
67
requirements.txt
Normal file
67
requirements.txt
Normal file
@ -0,0 +1,67 @@
|
|||||||
|
# sentry_sdk==1.5.10
|
||||||
|
# async_timeou
|
||||||
|
# requests>=2.20.0
|
||||||
|
# Sanic==0.5.1
|
||||||
|
# sanic_session==0.1.2
|
||||||
|
# pyquery
|
||||||
|
# asyncio_redis==0.14.3
|
||||||
|
|
||||||
|
|
||||||
|
aiocache==0.3.2
|
||||||
|
aiofiles==0.8.0
|
||||||
|
aiohttp==3.8.1
|
||||||
|
aiomcache==0.6.0
|
||||||
|
aioredis==0.3.3
|
||||||
|
aiosignal==1.2.0
|
||||||
|
arrow==1.2.2
|
||||||
|
async-timeout==4.0.2
|
||||||
|
asyncio-redis==0.14.3
|
||||||
|
asynctest==0.13.0
|
||||||
|
attrs==21.4.0
|
||||||
|
beautifulsoup4==4.11.1
|
||||||
|
bs4==0.0.1
|
||||||
|
cchardet==2.1.7
|
||||||
|
certifi==2021.10.8
|
||||||
|
charset-normalizer==2.0.12
|
||||||
|
cssselect==1.1.0
|
||||||
|
frozenlist==1.2.0
|
||||||
|
gunicorn==20.1.0
|
||||||
|
hiredis==2.0.0
|
||||||
|
html5lib==1.1
|
||||||
|
httptools==0.4.0
|
||||||
|
idna==3.3
|
||||||
|
idna-ssl==1.1.0
|
||||||
|
importlib-metadata==4.8.3
|
||||||
|
iniconfig==1.1.1
|
||||||
|
Jinja2==2.11.3
|
||||||
|
lxml==4.8.0
|
||||||
|
MarkupSafe==2.0.1
|
||||||
|
motor==1.1
|
||||||
|
multidict==5.2.0
|
||||||
|
numpy==1.19.5
|
||||||
|
packaging==21.3
|
||||||
|
pluggy==1.0.0
|
||||||
|
py==1.11.0
|
||||||
|
pymongo==3.4.0
|
||||||
|
pyparsing==3.0.8
|
||||||
|
pyquery==1.4.3
|
||||||
|
pytest==7.0.1
|
||||||
|
python-dateutil==2.8.2
|
||||||
|
requests==2.27.1
|
||||||
|
ruia==0.8.4
|
||||||
|
ruia-ua==0.0.2
|
||||||
|
sanic==0.5.1
|
||||||
|
sanic-session==0.1.2
|
||||||
|
schedule==1.1.0
|
||||||
|
sentry-sdk==1.5.10
|
||||||
|
six==1.16.0
|
||||||
|
soupsieve==2.3.2.post1
|
||||||
|
tomli==1.2.3
|
||||||
|
typing-extensions==4.1.1
|
||||||
|
ujson==4.3.0
|
||||||
|
urllib3==1.26.9
|
||||||
|
uvloop==0.14.0
|
||||||
|
webencodings==0.5.1
|
||||||
|
websockets==9.1
|
||||||
|
yarl==1.7.2
|
||||||
|
zipp==3.6.0
|
||||||
58
requirements2.txt
Normal file
58
requirements2.txt
Normal file
@ -0,0 +1,58 @@
|
|||||||
|
aiocache==0.3.2
|
||||||
|
aiofiles==0.8.0
|
||||||
|
aiohttp==3.8.1
|
||||||
|
aiomcache==0.6.0
|
||||||
|
aioredis==0.3.3
|
||||||
|
aiosignal==1.2.0
|
||||||
|
arrow==1.2.2
|
||||||
|
async-timeout==4.0.2
|
||||||
|
asyncio-redis==0.14.3
|
||||||
|
asynctest==0.13.0
|
||||||
|
attrs==21.4.0
|
||||||
|
beautifulsoup4==4.11.1
|
||||||
|
bs4==0.0.1
|
||||||
|
cchardet==2.1.7
|
||||||
|
certifi==2021.10.8
|
||||||
|
charset-normalizer==2.0.12
|
||||||
|
cssselect==1.1.0
|
||||||
|
frozenlist==1.2.0
|
||||||
|
gunicorn==20.1.0
|
||||||
|
hiredis==2.0.0
|
||||||
|
html5lib==1.1
|
||||||
|
httptools==0.4.0
|
||||||
|
idna==3.3
|
||||||
|
idna-ssl==1.1.0
|
||||||
|
importlib-metadata==4.8.3
|
||||||
|
iniconfig==1.1.1
|
||||||
|
Jinja2==2.11.3
|
||||||
|
lxml==4.8.0
|
||||||
|
MarkupSafe==2.0.1
|
||||||
|
motor==1.1
|
||||||
|
multidict==5.2.0
|
||||||
|
numpy==1.19.5
|
||||||
|
packaging==21.3
|
||||||
|
pluggy==1.0.0
|
||||||
|
py==1.11.0
|
||||||
|
pymongo==3.4.0
|
||||||
|
pyparsing==3.0.8
|
||||||
|
pyquery==1.4.3
|
||||||
|
pytest==7.0.1
|
||||||
|
python-dateutil==2.8.2
|
||||||
|
requests==2.27.1
|
||||||
|
ruia==0.8.4
|
||||||
|
ruia-ua==0.0.2
|
||||||
|
sanic==0.5.1
|
||||||
|
sanic-session==0.1.2
|
||||||
|
schedule==1.1.0
|
||||||
|
sentry-sdk==1.5.10
|
||||||
|
six==1.16.0
|
||||||
|
soupsieve==2.3.2.post1
|
||||||
|
tomli==1.2.3
|
||||||
|
typing-extensions==4.1.1
|
||||||
|
ujson==4.3.0
|
||||||
|
urllib3==1.26.9
|
||||||
|
uvloop==0.14.0
|
||||||
|
webencodings==0.5.1
|
||||||
|
websockets==9.1
|
||||||
|
yarl==1.7.2
|
||||||
|
zipp==3.6.0
|
||||||
11
soulbook/__init__.py
Normal file
11
soulbook/__init__.py
Normal file
@ -0,0 +1,11 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""
|
||||||
|
Created by howie.hu at 17-9-22.
|
||||||
|
"""
|
||||||
|
from sys import path as path_sys
|
||||||
|
from os import path as path_os
|
||||||
|
# from multiprocessing.managers import BaseManager
|
||||||
|
# from multiprocessing import freeze_support
|
||||||
|
|
||||||
|
# environment variable
|
||||||
|
path_sys.append(path_os.dirname(__file__))
|
||||||
43
soulbook/config/__init__.py
Normal file
43
soulbook/config/__init__.py
Normal file
@ -0,0 +1,43 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
import os
|
||||||
|
import logging
|
||||||
|
|
||||||
|
from .rules import *
|
||||||
|
|
||||||
|
logging_format = "[%(asctime)s] %(process)d-%(levelname)s "
|
||||||
|
logging_format += "%(module)s::%(funcName)s():l%(lineno)d: "
|
||||||
|
logging_format += "%(message)s"
|
||||||
|
|
||||||
|
logging.basicConfig(
|
||||||
|
format=logging_format,
|
||||||
|
level=logging.DEBUG
|
||||||
|
)
|
||||||
|
LOGGER = logging.getLogger()
|
||||||
|
|
||||||
|
|
||||||
|
def load_config():
|
||||||
|
"""
|
||||||
|
Load a config class
|
||||||
|
"""
|
||||||
|
|
||||||
|
os.environ['MODE'] = 'PRO'
|
||||||
|
mode = os.environ.get('MODE', 'DEV')
|
||||||
|
LOGGER.info('soulbook 启动模式：{}'.format(mode))
|
||||||
|
try:
|
||||||
|
# if mode == 'PRO':
|
||||||
|
# from .pro_config import ProConfig
|
||||||
|
# return ProConfig
|
||||||
|
# elif mode == 'DEV':
|
||||||
|
# from .dev_config import DevConfig
|
||||||
|
# return DevConfig
|
||||||
|
# else:
|
||||||
|
# from .dev_config import DevConfig
|
||||||
|
# return DevConfig
|
||||||
|
from .dev_config import DevConfig
|
||||||
|
return DevConfig
|
||||||
|
except ImportError:
|
||||||
|
from .config import Config
|
||||||
|
return Config
|
||||||
|
|
||||||
|
|
||||||
|
CONFIG = load_config()
|
||||||
40
soulbook/config/config.py
Normal file
40
soulbook/config/config.py
Normal file
@ -0,0 +1,40 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
import os
|
||||||
|
|
||||||
|
|
||||||
|
class Config():
|
||||||
|
"""
|
||||||
|
Basic config for soulbook
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Application config
|
||||||
|
DEBUG = True
|
||||||
|
VAL_HOST = os.getenv('VAL_HOST', 'true')
|
||||||
|
FORBIDDEN = ['139.199.198.228']
|
||||||
|
HOST = ['127.0.0.1:8001', '0.0.0.0:8001', '127.0.0.1:8002', '0.0.0.0:8002']
|
||||||
|
TIMEZONE = 'Asia/Shanghai'
|
||||||
|
BASE_DIR = os.path.dirname(os.path.dirname(__file__))
|
||||||
|
USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36'
|
||||||
|
WEBSITE = dict(
|
||||||
|
IS_RUNNING=True,
|
||||||
|
TOKEN='',
|
||||||
|
AUTHOR_LATEST_COUNT=5,
|
||||||
|
)
|
||||||
|
|
||||||
|
SCHEDULED_DICT = dict(
|
||||||
|
SPIDER_INTERVAL=int(os.getenv('SPIDER_INTERVAL', 120)),
|
||||||
|
NOVELS_INTERVAL=int(os.getenv('SPIDER_INTERVAL', 180)),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Engine config
|
||||||
|
URL_PHONE = 'https://m.baidu.com/s'
|
||||||
|
URL_PC = 'http://www.baidu.com/s'
|
||||||
|
# BAIDU_RN = 15
|
||||||
|
BAIDU_RN = 15
|
||||||
|
SO_URL = "https://www.so.com/s"
|
||||||
|
BY_URL = "https://www.bing.com/search"
|
||||||
|
DUCKGO_URL = "https://duckduckgo.com/html"
|
||||||
|
|
||||||
|
REMOTE_SERVER = {
|
||||||
|
'proxy_server': 'http://0.0.0.0:8002/'
|
||||||
|
}
|
||||||
41
soulbook/config/dev_config.py
Normal file
41
soulbook/config/dev_config.py
Normal file
@ -0,0 +1,41 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
import os
|
||||||
|
|
||||||
|
from .config import Config
|
||||||
|
|
||||||
|
|
||||||
|
class DevConfig(Config):
|
||||||
|
"""
|
||||||
|
Dev config for soulbook
|
||||||
|
"""
|
||||||
|
|
||||||
|
VAL_HOST = os.getenv('VAL_HOST', 'false')
|
||||||
|
|
||||||
|
# Database config
|
||||||
|
REDIS_DICT = dict(
|
||||||
|
IS_CACHE=True,
|
||||||
|
REDIS_ENDPOINT=os.getenv('REDIS_ENDPOINT', "localhost"),
|
||||||
|
REDIS_PORT=int(os.getenv('REDIS_PORT', 6379)),
|
||||||
|
REDIS_PASSWORD=os.getenv('REDIS_PASSWORD', None),
|
||||||
|
REDIS_PORT=6379,
|
||||||
|
CACHE_DB=0,
|
||||||
|
SESSION_DB=1,
|
||||||
|
POOLSIZE=10,
|
||||||
|
)
|
||||||
|
MONGODB = dict(
|
||||||
|
MONGO_HOST=os.getenv('MONGO_HOST', "127.0.0.1"),
|
||||||
|
MONGO_PORT=int(os.getenv('MONGO_PORT', 27017)),
|
||||||
|
MONGO_USERNAME="",
|
||||||
|
MONGO_PASSWORD="",
|
||||||
|
DATABASE='soulbook',
|
||||||
|
)
|
||||||
|
|
||||||
|
# website
|
||||||
|
WEBSITE = dict(
|
||||||
|
IS_RUNNING=os.getenv('OWLLOOK_IS_RUNNING', True),
|
||||||
|
TOKEN=os.getenv('OWLLOOK_TOKEN', '')
|
||||||
|
)
|
||||||
|
|
||||||
|
AUTH = {
|
||||||
|
"Owllook-Api-Key": os.getenv('OWLLOOK_API_KEY', "your key")
|
||||||
|
}
|
||||||
21
soulbook/config/gunicorn.py
Normal file
21
soulbook/config/gunicorn.py
Normal file
@ -0,0 +1,21 @@
|
|||||||
|
# gunicorn config
|
||||||
|
# gunicorn -c config/gunicorn.py --worker-class sanic.worker.GunicornWorker server:app
|
||||||
|
import os
|
||||||
|
|
||||||
|
os.environ['MODE'] = 'PRO'
|
||||||
|
|
||||||
|
WORKERS = os.getenv('WORKERS', 2)
|
||||||
|
TIMEOUT = os.getenv('TIMEOUT', 60)
|
||||||
|
|
||||||
|
bind = '0.0.0.0:8001'
|
||||||
|
backlog = 2048
|
||||||
|
|
||||||
|
workers = WORKERS
|
||||||
|
worker_connections = 1000
|
||||||
|
keepalive = 2
|
||||||
|
|
||||||
|
spew = False
|
||||||
|
daemon = False
|
||||||
|
umask = 0
|
||||||
|
timeout = TIMEOUT
|
||||||
|
preload = True
|
||||||
672
soulbook/config/rules.py
Normal file
672
soulbook/config/rules.py
Normal file
@ -0,0 +1,672 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
from collections import namedtuple
|
||||||
|
|
||||||
|
####################################### 规则 ###########################################
|
||||||
|
# DOMAIN
|
||||||
|
BLACK_DOMAIN = ['www.17k.com', 'mm.17k.com', 'www.xs8.cn', 'www.zongheng.com', 'yunqi.qq.com', 'chuangshi.qq.com',
|
||||||
|
'book.qidian.com', 'www.soduso.com', 'pages.book.qq.com', 'book.km.com', 'www.lread.net',
|
||||||
|
'www.0dsw.com', 'www.5200xsb.com', 'www.80txt.com', 'www.sodu.tw', 'www.shuquge.com',
|
||||||
|
'www.shenmanhua.com', 'xiaoshuo.sogou.com', 'www.999wx.com', 'zetianji8.com', 'www.bookso.net',
|
||||||
|
'm.23us.com', 'www.qbxsw.com', 'www.zhuzhudao.com', 'www.shengyan.org', 'www.360doc.com',
|
||||||
|
'www.ishuo.cn', 'read.qidian.com', 'www.yunlaige.com', 'www.qidian.com', 'www.sodu888.com',
|
||||||
|
'www.siluke.cc', 'read.10086.cn', 'www.pbtxt.com', 'c4txt.com', 'www.bokon.net', 'www.sikushu.net',
|
||||||
|
'www.is028.cn', 'www.tadu.com', 'www.kudu8.com', 'www.bmwen.com', 'www.5858xs.com', 'www.yiwan.com',
|
||||||
|
'www.x81zw.com', 'www.123du.cc', 'www.chashu.cc', '20xs.com', 'www.haxwx.net', 'www.dushiwenxue.com',
|
||||||
|
"www.yxdown.com", 'www.jingcaiyuedu.com', 'www.zhetian.org', 'www.xiaoshuo02.com', 'www.xiaoshuo77.com',
|
||||||
|
'www.868xh.com', 'dp.changyou.com', 'www.iyouman.com', 'www.qq717.com', 'www.yznn.com', "www.69w.cc",
|
||||||
|
"www.doupocangqiong1.com", "www.manhuatai.com", "www.5wxs.com", "www.ggshuji.com", "www.msxf.net",
|
||||||
|
"www.mianhuatang.la", "www.boluoxs.com", "www.lbiquge.top", "www.69shu.com", "www.qingkan520.com",
|
||||||
|
"book.douban.com", "movie.douban.com", "www.txshuku.com", "lz.book.sohu.com", "www.3gsc.com.cn",
|
||||||
|
"www.txtshu365.com", "www.517yuedu.com", "www.baike.com", "read.jd.com", "www.zhihu.com", "wshuyi.com",
|
||||||
|
"www.19lou.tw", "www.chenwangbook.com", "www.aqtxt.com", "book.114la.com", "www.niepo.net",
|
||||||
|
"me.qidian.com", "www.gengd.com", "www.77l.com", "www.geilwx.com", "www.97xiao.com", "www.anqu.com",
|
||||||
|
"www.wuxiaxs.com", "yuedu.163.com", "b.faloo.com", "bbs.qidian.com", "jingji.qidian.com", "www.sodu.cc",
|
||||||
|
"forum.qdmm.com", "www.qdmm.com", "game.91.com", "www.11773.com", "mt.sohu.com", "book.dajianet.com",
|
||||||
|
"haokan.17k.com", "www.qmdsj.com", "www.jjwxc.net", "ishare.iask.sina.com.cn", "www.cmread.com",
|
||||||
|
"www.52ranwen.net", "www.dingdianzw.com", "www.topber.com", "www.391k.com", "www.qqxzb.com",
|
||||||
|
"www.zojpw.com", "www.pp8.com", "www.bxwx.org", "www.hrsxb.com", "www.497.com", "www.d8qu.com",
|
||||||
|
"www.duwanjuan.com", "www.05935.com", "book.zongheng.com", "www.55x.cn", "www.freexs.cn",
|
||||||
|
"xiaoshuo.360.cn", "www.3kw.cc", "www.gzbpi.com", "book.sina.com.cn", "www.vodtw.com", "wenda.so.com",
|
||||||
|
"product.dangdang.com", "www.chuiyao.com", "novel.slieny.com", "www.bilibili.com", "donghua.dmzj.com",
|
||||||
|
"www.yaojingweiba.com", "www.qb5200.com", "www.520tingshu.com", "www.567zw.com", "www.zjrxz.com",
|
||||||
|
"v.qq.com", "blog.sina.com.cn", "www.hackhome.com", "news.fznews.com.cn", "www.jingyu.com",
|
||||||
|
"news.so.com", "www.sodu3.com", "vipreader.qidian.com", "www.mozhua9.com", "www.iqiyi.com",
|
||||||
|
"xs.sogou.com"]
|
||||||
|
|
||||||
|
# 针对某些网站检索出来的地址和真正的目录地址不一样从而进行替换
|
||||||
|
REPLACE_RULES = {
|
||||||
|
"www.miaobige.com": {
|
||||||
|
'old': 'miaobige.com/book/',
|
||||||
|
'new': 'miaobige.com/read/'
|
||||||
|
},
|
||||||
|
"www.5ccc.net": {
|
||||||
|
'old': '5ccc.net/wksz_info/',
|
||||||
|
'new': '5ccc.net/xiaoshuo/'
|
||||||
|
},
|
||||||
|
"www.7kankan.com": {
|
||||||
|
'old': '7kankan.com/files/article/info/',
|
||||||
|
'new': '7kankan.com/files/article/html/'
|
||||||
|
},
|
||||||
|
"www.xqingdou.net": {
|
||||||
|
'old': 'xqingdou.net/book_',
|
||||||
|
'new': 'xqingdou.net/chapter_'
|
||||||
|
},
|
||||||
|
"www.wuyanxia.net": {
|
||||||
|
'old': 'wuyanxia.net/book/',
|
||||||
|
'new': 'wuyanxia.net/read/'
|
||||||
|
},
|
||||||
|
"www.263zw.com": {
|
||||||
|
'old': '263zw.com/402770/',
|
||||||
|
'new': '263zw.com/402770/list/'
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
# 搜索引擎检索优先级
|
||||||
|
ENGINE_PRIORITY = ['bing', '360', 'baidu', 'bing', 'duck_go']
|
||||||
|
# ENGINE_PRIORITY = ['baidu', '360', 'bing', 'duck_go']
|
||||||
|
|
||||||
|
# Rules
|
||||||
|
Rules = namedtuple('Rules', 'content_url chapter_selector content_selector')
|
||||||
|
LatestRules = namedtuple('LatestRules', 'plan meta_value selector')
|
||||||
|
|
||||||
|
# 获取小说最新章节
|
||||||
|
PLAN_01 = LatestRules(
|
||||||
|
True,
|
||||||
|
{'latest_chapter_name': 'og:novel:latest_chapter_name', 'latest_chapter_url': 'og:novel:latest_chapter_url'},
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
|
||||||
|
LATEST_RULES = {
|
||||||
|
"www.biqugex.com": PLAN_01,
|
||||||
|
"www.x23us.com": PLAN_01,
|
||||||
|
"www.23us.la": PLAN_01,
|
||||||
|
"www.sqsxs.com": PLAN_01,
|
||||||
|
"www.nuomi9.com": PLAN_01,
|
||||||
|
"www.biquge.info": PLAN_01,
|
||||||
|
"www.biquge.tw": PLAN_01,
|
||||||
|
"www.qu.la": PLAN_01,
|
||||||
|
"www.ybdu.com": PLAN_01,
|
||||||
|
"www.wenxuemi.com": PLAN_01,
|
||||||
|
"www.biquge.com": PLAN_01,
|
||||||
|
"www.23us.cc": PLAN_01,
|
||||||
|
"www.xs222.com": PLAN_01,
|
||||||
|
"www.lewen8.com": PLAN_01,
|
||||||
|
"www.bqg5200.com": PLAN_01,
|
||||||
|
"www.vodtw.com": PLAN_01,
|
||||||
|
"www.6mao.com": PLAN_01,
|
||||||
|
"www.biquge.sh": PLAN_01,
|
||||||
|
"www.touxiang.la": PLAN_01,
|
||||||
|
"www.bxquge.com": PLAN_01,
|
||||||
|
"www.beidouxin.com": PLAN_01,
|
||||||
|
"www.biquge.lu": PLAN_01,
|
||||||
|
"www.263zw.com": PLAN_01,
|
||||||
|
"www.3qzone.com": PLAN_01,
|
||||||
|
"wwww.yooread.com": PLAN_01,
|
||||||
|
# "www.suimeng.la": PLAN_01,
|
||||||
|
"www.bequge.com": PLAN_01,
|
||||||
|
"www.biquku.co": PLAN_01,
|
||||||
|
"www.xbqge.com": PLAN_01,
|
||||||
|
"www.aiquxs.com": PLAN_01,
|
||||||
|
"www.23us.com": PLAN_01,
|
||||||
|
"www.biqiuge.com": PLAN_01,
|
||||||
|
"www.ddbiquge.com": PLAN_01,
|
||||||
|
"www.abocms.cn": PLAN_01,
|
||||||
|
"www.a306.com": PLAN_01,
|
||||||
|
"www.liewen.cc": PLAN_01,
|
||||||
|
"www.8535.org": PLAN_01,
|
||||||
|
"www.dingdianzw.com": PLAN_01,
|
||||||
|
"www.biquge.cc": PLAN_01,
|
||||||
|
"www.111bz.org": PLAN_01,
|
||||||
|
"www.biqugebook.com": PLAN_01,
|
||||||
|
"www.e8zw.com": PLAN_01,
|
||||||
|
"www.xqqxs.com": PLAN_01,
|
||||||
|
"tianyibook.la": PLAN_01,
|
||||||
|
"www.lingdianksw.com": PLAN_01,
|
||||||
|
"www.qb5.tw": PLAN_01,
|
||||||
|
"www.quanben.com": PLAN_01,
|
||||||
|
"www.58xs.com": PLAN_01,
|
||||||
|
"www.biqukan.com": PLAN_01,
|
||||||
|
"www.yssm.org": PLAN_01,
|
||||||
|
"www.81zw.com": PLAN_01,
|
||||||
|
"www.ymoxuan.com": PLAN_01,
|
||||||
|
"www.mytxt.cc": PLAN_01,
|
||||||
|
"www.woquge.com": PLAN_01,
|
||||||
|
"www.biquguo.com": PLAN_01,
|
||||||
|
"www.8jzw.cc": PLAN_01,
|
||||||
|
"www.biquge.tv": PLAN_01,
|
||||||
|
"www.biquge5200.com": PLAN_01,
|
||||||
|
"www.8jzw.com": PLAN_01,
|
||||||
|
"www.23xsw.cc": PLAN_01,
|
||||||
|
"www.miaobige.com": PLAN_01,
|
||||||
|
"www.xs.la": PLAN_01,
|
||||||
|
"www.44pq.co": PLAN_01,
|
||||||
|
"www.50zw.la": PLAN_01,
|
||||||
|
"www.33xs.com": PLAN_01,
|
||||||
|
"www.zwdu.com": PLAN_01,
|
||||||
|
"www.ttzw.com": PLAN_01,
|
||||||
|
"www.zanghaihuatxt.com": PLAN_01,
|
||||||
|
"www.kuxiaoshuo.com": PLAN_01,
|
||||||
|
"www.biqudu.com": PLAN_01,
|
||||||
|
"www.biqugeg.com": PLAN_01,
|
||||||
|
"www.23txt.com": PLAN_01,
|
||||||
|
"www.baquge.tw": PLAN_01,
|
||||||
|
"www.23qb.com": PLAN_01,
|
||||||
|
"www.lread.cc": PLAN_01,
|
||||||
|
"www.biqudao.com": PLAN_01,
|
||||||
|
"www.laidudu.com": PLAN_01,
|
||||||
|
"www.kxs7.com": PLAN_01,
|
||||||
|
"www.biquguan.com": PLAN_01,
|
||||||
|
"www.biquta.com": PLAN_01,
|
||||||
|
"www.xs98.com": PLAN_01,
|
||||||
|
"www.bqge.org": PLAN_01,
|
||||||
|
"www.58xs.tw": PLAN_01,
|
||||||
|
"www.187ks.com": PLAN_01,
|
||||||
|
"www.yikanxiaoshuo.com": PLAN_01,
|
||||||
|
"www.23zw.me": PLAN_01,
|
||||||
|
"www.37zw.net": PLAN_01,
|
||||||
|
"www.biquge.cm": PLAN_01,
|
||||||
|
"www.kanshu58.com": PLAN_01,
|
||||||
|
"www.biqumo.com": PLAN_01,
|
||||||
|
"www.mpxiaoshuo.com": PLAN_01,
|
||||||
|
"www.23wx.cm": PLAN_01,
|
||||||
|
"www.biquge.jp": PLAN_01,
|
||||||
|
"www.biqugexsw.com": PLAN_01,
|
||||||
|
"www.biqu6.com": PLAN_01,
|
||||||
|
"www.xiuxs.com": PLAN_01,
|
||||||
|
"www.booktxt.net": PLAN_01,
|
||||||
|
"www.biqule.com": PLAN_01,
|
||||||
|
"www.biquzi.com": PLAN_01,
|
||||||
|
"www.biquku.la": PLAN_01,
|
||||||
|
"www.00ksw.org": PLAN_01,
|
||||||
|
"www.bqg.cc": PLAN_01,
|
||||||
|
"www.biqugezw.com": PLAN_01,
|
||||||
|
"www.bbiquge.com": PLAN_01,
|
||||||
|
"www.aikantxt.la": PLAN_01,
|
||||||
|
"www.biquge.com.tw": PLAN_01,
|
||||||
|
"www.xxbiquge.com": PLAN_01,
|
||||||
|
"www.biquwo.com": PLAN_01,
|
||||||
|
"www.ixs.cc": PLAN_01,
|
||||||
|
|
||||||
|
# 其他规则
|
||||||
|
# "www.50331.net": LatestRules(
|
||||||
|
# False,
|
||||||
|
# None,
|
||||||
|
# {'content_url': "http://www.50331.net/", 'tag': 'span.zzjie a'}
|
||||||
|
# )
|
||||||
|
}
|
||||||
|
|
||||||
|
RULES = {
|
||||||
|
# demo 'name': Rules('content_url', {chapter_selector}, {content_selector})
|
||||||
|
# content_url=1 表示章节链接使用本身自带的链接，不用拼接
|
||||||
|
# content_url=0 表示章节网页需要当前页面url拼接
|
||||||
|
# 'www.biqule.com': Rules('www.biqule.com', {'class': 'box_con'},{}),
|
||||||
|
# 'www.lingdiankanshu.com': Rules('www.lingdiankanshu.com', {'class': 'box_con'}, {}),
|
||||||
|
# 'www.hhlwx.com': Rules('www.hhlwx.co', {'class': 'chapterlist'},{}),
|
||||||
|
'www.biquwu.cc': Rules('https://www.biquwu.cc/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
'www.quyuege.com': Rules('0', {'class': 'mod-article-list'}, {'class': 'page-content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biqugex.com': Rules('http://www.biqugex.com/', {'class': 'listmain'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.bbiquge.com': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biquge.info': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.37zw.net': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biquku.la': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biquge.sh': Rules('1', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biquge.co': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.00ksw.org': Rules('0', {'class': 'ml_list'}, {'id': 'articlecontent'}),
|
||||||
|
# 已解析
|
||||||
|
'www.bqge.org': Rules('http://www.bqge.org/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.aikantxt.la': Rules('http://www.aikantxt.la/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biquzi.com': Rules('http://www.biquzi.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.bqg.cc': Rules('http://www.bqg.cc/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biquge.jp': Rules('0', {'id': 'list'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.vipzw.com': Rules('http://www.vipzw.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biquge5200.com': Rules('1', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.zanghaihuatxt.com': Rules('http://www.zanghaihuatxt.com/', {'class': 'listmain'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.xiuxs.com': Rules('http://www.xiuxs.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.1biquge.com': Rules('http://www.1biquge.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.xiaoshuowan.com': Rules('http://www.xiaoshuowan.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biqugela.com': Rules('http://www.biqugela.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biqu6.com': Rules('http://www.biqu6.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.zhuaji.org': Rules('0', {'tag': 'dd'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.sqsxs.com': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biquge.tv': Rules('http://www.biquge.tv/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biquta.com': Rules('https://www.biquta.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# # 已解析
|
||||||
|
'www.xbiquge.la': Rules('http://www.xbiquge.la/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biquge.cm': Rules('http://www.biquge.cm/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.23qb.com': Rules('https://www.23qb.com/', {'id': 'chapterList'}, {'id': 'TextContent'}),
|
||||||
|
# 已解析
|
||||||
|
# 'www.txtwan.com': Rules('http://www.txtwan.com/', {'id': 'chapterList'}, {'id': 'txt'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biqugexsw.com': Rules('http://www.biqugexsw.com/', {'class': 'listmain'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.kuxiaoshuo.com': Rules('1', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.laidudu.com': Rules('http://www.laidudu.com/', {'class': 'listmain'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.kanshu58.com': Rules('0', {'class': 'chapters'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.mpxiaoshuo.com': Rules('0', {'class': 'mulu_list'}, {'id': 'htmlContent'}),
|
||||||
|
# 已解析
|
||||||
|
'www.23zw.me': Rules('0', {'id': 'chapter_list'}, {'id': 'text_area'}),
|
||||||
|
# 已解析
|
||||||
|
'www.187ks.com': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.58xs.tw': Rules('http://www.58xs.tw/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biquguan.com': Rules('http://www.biquguan.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.xs98.com': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.kxs7.com': Rules('http://www.kxs7.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biqudao.com': Rules('https://www.biqudao.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.nuomi9.com': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'book.sfacg.com': Rules('http://book.sfacg.com/', {'class': 'story-catalog'}, {'tag': 'p'}),
|
||||||
|
# 已解析
|
||||||
|
'www.7kshu.com': Rules('0', {'id': 'chapterlist'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.lread.cc': Rules('http://www.lread.cc/', {'class': 'box_con'}, {'id': 'booktext'}),
|
||||||
|
# 已解析
|
||||||
|
'www.baquge.tw': Rules('http://www.baquge.tw/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biqudu.com': Rules('https://www.biqudu.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biqugeg.com': Rules('http://www.biqugeg.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.23txt.com': Rules('http://www.23txt.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.ttzw.com': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.zwdu.com': Rules('http://www.zwdu.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.33xs.com': Rules('http://www.33xs.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.50zw.la': Rules('0', {'class': 'chapterlist'}, {'id': 'htmlContent'}),
|
||||||
|
# 已解析
|
||||||
|
'www.44pq.co': Rules('0', {'class': 'chapterlist'}, {'id': 'BookText'}),
|
||||||
|
# 已解析
|
||||||
|
'www.wddsnxn.org': Rules('1', {'class': 'booklist'}, {'id': 'BookText'}),
|
||||||
|
# 已解析
|
||||||
|
'mianzhuan.wddsnxn.org': Rules('1', {'class': 'booklist'}, {'id': 'BookText'}),
|
||||||
|
# 已解析
|
||||||
|
'www.a306.com': Rules('http://www.a306.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.xs52.com': Rules('0', {'id': 'chapter_list'}, {'id': 'text_c'}),
|
||||||
|
# 已解析
|
||||||
|
'www.xs.la': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.23xsw.cc': Rules('http://www.23xsw.cc/', {'class': 'listmain'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.8jzw.com': Rules('http://www.8jzw.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biquguo.com': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.woquge.com': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.zhonghuawuxia.com': Rules('1', {'class': 'list'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.mytxt.cc': Rules('http://www.mytxt.cc/', {'class': 'story_list_m62topxs'}, {'class': 'detail_con_m62topxs'}),
|
||||||
|
# 已解析
|
||||||
|
'www.136txt.com': Rules('1', {'class': 'directory_con'}, {'id': 'chapterContent'}),
|
||||||
|
# 已解析
|
||||||
|
'www.xs74.com': Rules('1', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.yssm.org': Rules('0', {'class': 'chapterlist'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.luoxia.com': Rules('1', {'class': 'book-list'}, {'tag': 'p'}),
|
||||||
|
# 已解析
|
||||||
|
'www.sbkk88.com': Rules('http://www.sbkk88.com/', {'class': 'leftList'}, {'id': 'f_article'}),
|
||||||
|
# 已解析
|
||||||
|
'www.dxsxs.com': Rules('http://www.dxsxs.com/', {'id': 'yuedu'}, {'class': 'zw'}),
|
||||||
|
# 已解析
|
||||||
|
'www.wenku8.com': Rules('0', {'class': 'css'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.xqingdou.net': Rules('http://www.xqingdou.net/', {'class': 'dirconone'}, {'id': 'chapter_content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.zuowe.com': Rules('http://www.zuowe.com/', {'class': 'book_list'}, {'id': 'htmlContent'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biqugek.com': Rules('1', {'class': 'book_list'}, {'id': 'htmlContent'}),
|
||||||
|
# 已解析
|
||||||
|
'www.wuyanxia.net': Rules('http://www.wuyanxia.net/', {'class': 'zjlist4'}, {'id': 'htmlContent'}),
|
||||||
|
# 已解析
|
||||||
|
'www.50331.net': Rules('http://www.50331.net/', {'class': 'listmain'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.wenxuemi.com': Rules('http://www.wenxuemi.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.xs222.com': Rules('http://www.xs222.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.lewen8.com': Rules('http://www.lewen8.com/', {'id': 'chapterlist'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.5ccc.net': Rules('0', {'class': 'acss'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
# 'www.suimeng.la': Rules('0', {'class': 'acss'}, {'id': 'ccontent'}),
|
||||||
|
# 已解析
|
||||||
|
'www.bqg5200.com': Rules('http://www.bqg5200.com/', {'id': 'readerlist'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.vodtw.com': Rules('0', {'class': 'insert_list'}, {'class': 'contentbox'}),
|
||||||
|
# 已解析
|
||||||
|
'www.6mao.com': Rules('http://www.6mao.com/', {'class': 'liebiao_bottom'}, {'id': 'neirong'}),
|
||||||
|
# 已解析
|
||||||
|
'www.touxiang.la': Rules('http://www.touxiang.la/', {'class': 'listmain'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.7kankan.com': Rules('0', {'class': 'uclist'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biqugetw.com': Rules('http://www.biqugetw.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'gdbzkz.com': Rules('1', {'class': 'mulu'}, {'class': 'content-body'}),
|
||||||
|
# 已解析
|
||||||
|
'www.gdbzkz.com': Rules('1', {'class': 'mulu'}, {'class': 'content-body'}),
|
||||||
|
# 已解析
|
||||||
|
'www.freexs.cn': Rules('0', {'class': 'readout'}, {'class': 'shuneirong'}),
|
||||||
|
# 已解析
|
||||||
|
'www.bxquge.com': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.beidouxin.com': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.3qzone.com': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.97xs.net': Rules('1', {'class': 'box'}, {'id': 'htmlContent'}),
|
||||||
|
# 已解析
|
||||||
|
'www.7dsw.com': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.263zw.com': Rules('1', {'class': 'chapter'}, {'id': 'chapterContent'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biquge5.com': Rules('1', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.yooread.com': Rules('http://www.yooread.com', {'id': 'chapterList'}, {'tag': 'p'}),
|
||||||
|
# 已解析
|
||||||
|
'www.xs82.com': Rules('0', {'class': 'chapterlist'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.kanshuhai.com': Rules('0', {'id': 'book'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.bequge.com': Rules('https://www.bequge.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析 请求失败
|
||||||
|
# 'www.biquge5200.com': Rules('1', {'id': 'list'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biquku.co': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.xbqge.com': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.aiquxs.com': Rules('0', {'id': 'list'}, {'id': 'booktext'}),
|
||||||
|
# 已解析
|
||||||
|
# 'www.piaotian.com': Rules('0', {'class': 'centent'}, {'class': 'fonts_mesne'}),
|
||||||
|
# 已解析
|
||||||
|
'www.ttshu.com': Rules('http://www.ttshu.com', {'class': 'border'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.23us.com': Rules('0', {'id': 'at'}, {'id': 'contents'}),
|
||||||
|
# 已解析
|
||||||
|
'www.x23us.com': Rules('0', {'id': 'at'}, {'id': 'contents'}),
|
||||||
|
# 已解析
|
||||||
|
'www.23wx.cc': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.23wx.cm': Rules('0', {'class': 'book_list'}, {'id': 'htmlContent'}),
|
||||||
|
# 已解析
|
||||||
|
'www.ddbiquge.com': Rules('http://www.ddbiquge.com', {'class': 'listmain'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.abocms.cn': Rules('http://www.abocms.cn/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.liewen.cc': Rules('https://www.liewen.cc/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.heiyange.com': Rules('http://www.heiyange.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.8535.org': Rules('0', {'class': 'booklist'}, {'class': 'txtc'}),
|
||||||
|
# 已解析
|
||||||
|
'www.dingdianzw.com': Rules('http://www.dingdianzw.com/', {'id': 'bgdiv'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biquge.cc': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.lewenxiaoshuo.com': Rules('1', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.111bz.org': Rules('http://www.111bz.org/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biqugebook.com': Rules('http://www.biqugebook.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.e8zw.com': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.xqqxs.com': Rules('0', {'class': 'box_con'}, {'class': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.139book.com': Rules('http://www.139book.com/', {'class': 'list_box'}, {'class': 'box_box'}),
|
||||||
|
# 已解析
|
||||||
|
'www.jcdf99.com': Rules('0', {'class': 'list_box'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.tianzeba.com': Rules('http://www.tianzeba.com/', {'class': 'chapterlist'}, {'id': 'BookText'}),
|
||||||
|
# 已解析
|
||||||
|
'www.kanshuwangzhan.com': Rules('0', {'id': 'chapterlist'}, {'id': 'booktext'}),
|
||||||
|
# 已解析
|
||||||
|
'tianyibook.la': Rules('http://tianyibook.la/', {'class': 'chapterlist'}, {'id': 'BookText'}),
|
||||||
|
# 已解析
|
||||||
|
'www.quanben.net': Rules('http://www.quanben.net/', {'class': 'chapterlist'}, {'id': 'BookText'}),
|
||||||
|
# 已解析
|
||||||
|
# 'www.zhetian.org': Rules('http://www.zhetian.org', {'class': 'body '}, {'class': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.lingdianksw.com': Rules('0', {'class': 'acss'}, {'id': 'ccontent'}),
|
||||||
|
# 已解析
|
||||||
|
'www.qb5.tw': Rules('http://www.qb5.tw/', {'class': 'zjbox'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.ybdu.com': Rules('0', {'class': 'mulu_list'}, {'id': 'htmlContent'}),
|
||||||
|
# 已解析
|
||||||
|
'www.quanben.com': Rules('0', {'class': 'mulu_list'}, {'id': 'htmlContent'}),
|
||||||
|
# 已解析
|
||||||
|
'www.fhxs.com': Rules('1', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biquge.biz': Rules('http://www.biquge.biz/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.58xs.com': Rules('http://www.58xs.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biqukan.com': Rules('http://www.biqukan.com/', {'class': 'listmain'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.shuyuelou.com': Rules('http://www.shuyuelou.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.mangg.com': Rules('http://www.mangg.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.50zw.com': Rules('0', {'class': 'chapterlist'}, {'id': 'htmlContent'}),
|
||||||
|
# 已解析
|
||||||
|
'www.lingdiankanshu.co': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biqiku.com': Rules('http://www.biqiku.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.duilianku.com': Rules('http://www.duilianku.com/', {'id': 'list'}, {'class': 'chapter'}),
|
||||||
|
# 已解析
|
||||||
|
'www.5xiaxiaoshuo.com': Rules('http://www.5xiaxiaoshuo.com/', {'class': 'art_listmain_main'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.81xsw.com': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.wxguan.com': Rules('http://www.wxguan.com/', {'class': 'listmain'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.qb5200.tw': Rules('http://www.qb5200.tw/', {'class': 'listmain'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.fox2008.cn': Rules('http://www.fox2008.cn/', {'class': 'book'}, {'id': 'chapterContent'}),
|
||||||
|
# 已解析
|
||||||
|
'www.22zw.com': Rules('0', {'class': 'acss'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.k6uk.com': Rules('0', {'class': 'acss'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.126shu.com': Rules('http://www.126shu.com/', {'id': 'list'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.kooxs.com': Rules('0', {'class': 'list'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.shubaotxt.com': Rules('0', {'class': 'list'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.muyuge.com': Rules('1', {'id': 'xslist'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
# 'www.daizhuzai.com': Rules('http://www.daizhuzai.com', {'class': 'dirlist'}, {'class': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biqu.la': Rules('0', {'class': 'book_list'}, {'id': 'htmlContent'}),
|
||||||
|
# 已解析
|
||||||
|
'shushu.com.cn': Rules('http://shushu.com.cn/', {'id': 'dirsort01'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.shuhai.com': Rules('0', {'class': 'box_chap'}, {'id': 'readcon'}),
|
||||||
|
# 已解析
|
||||||
|
'www.37yue.com': Rules('0', {'class': 'list-chapter'}, {'class': 'chapter'}),
|
||||||
|
# 已解析
|
||||||
|
'www.35zw.com': Rules('0', {'class': 'book_list'}, {'id': 'htmlContent'}),
|
||||||
|
# 已解析
|
||||||
|
'www.xinshu.in': Rules('http://www.xinshu.in/', {'class': 'list_box'}, {'class': 'box_box'}),
|
||||||
|
# 已解析
|
||||||
|
'www.lwxs520.com': Rules('0', {'class': 'dccss'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.lwxs.la': Rules('http://www.lwxs.la/', {'id': 'defaulthtml4'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biqule.com': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.33yq.com': Rules('1', {'class': 'box_con'}, {'class': 'zhangjieTXT'}),
|
||||||
|
# 已解析
|
||||||
|
'www.dishuge.com': Rules('1', {'class': 'update'}, {'tag': 'p'}),
|
||||||
|
# 已解析
|
||||||
|
'www.qu.la': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.shuge.net': Rules('http://www.shuge.ne/t', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.daomengren.com': Rules('http://www.daomengren.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.81zw.net': Rules('http://www.81zw.net/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.09xs.com': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.fhxiaoshuo.com': Rules('1', {'class': 'box_con'}, {'class': 'zhangjieTXT'}),
|
||||||
|
# 已解析
|
||||||
|
'www.yikanxiaoshuo.com': Rules('http://www.yikanxiaoshuo.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.1xiaoshuo.com': Rules('http://www.1xiaoshuo.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.kanshu.la': Rules('http://www.kanshu.la/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.kbiquge.com': Rules('http://www.kbiquge.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.00ksw.net': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.booktxt.net': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'wanmeishijiexiaoshuo.org': Rules('1', {'class': 'bg'}, {'class': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.sosoxiaoshuo.cc': Rules('http://www.sosoxiaoshuo.cc/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.ciluke.com': Rules('0', {'id': 'list'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.81zw.com': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.cilook.net': Rules('0', {'id': 'cl_content'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.baoliny.com': Rules('http://www.baoliny.com/', {'class': 'readerListShow'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biquge.tw': Rules('http://www.biquge.tw/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.7788xs.net': Rules('http://www.7788xs.net/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.06sy.com': Rules('http://www.06sy.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biqumo.com': Rules('https://www.biqumo.com/', {'class': 'listmain'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.kanshuzhe.com': Rules('http://www.kanshuzhe.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biqiuge.com': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.lwxs.com': Rules('0', {'class': 'box_con'}, {'id': 'TXT'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biqugezw.com': Rules('http://www.biqugezw.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析 经常跳到无法预料的网站 故禁止
|
||||||
|
# 'www.is028.cn': Rules('http://www.biquge.com.tw', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# www.is028.cn会跳转到http://www.biquge.com.tw
|
||||||
|
'www.biquge.com.tw': Rules('http://www.biquge.com.tw/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 'www.xs82.com': Rules('-1', {'class': 'chapterlist'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.shuqizw.com': Rules('http://www.shuqizw.com/', {'class': 'article_texttitleb'}, {'id': 'book_text'}),
|
||||||
|
# 已解析
|
||||||
|
'read.ixdzs.com': Rules('0', {'class': 'catalog'}, {'class': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.shumilou.net': Rules('0', {'class': 'chapterlist'}, {'id': 'BookText'}),
|
||||||
|
# 已解析
|
||||||
|
'www.8shuw.com': Rules('1', {'class': 'chapterlist'}, {'id': 'readtext'}),
|
||||||
|
# 已解析
|
||||||
|
# 'www.ttshu.com': Rules('http://www.ttshu.com', {'class': 'border'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.heiyan.la': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.bbsa5.com': Rules('1', {'class': 'panel'}, {'class': 'content-body'}),
|
||||||
|
# 已解析
|
||||||
|
'www.tycqxs.com': Rules('http://www.tycqxs.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.miaobige.com': Rules('https://www.miaobige.com/', {'id': 'readerlists'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.dashubao.net': Rules('0', {'class': 'ml_main'}, {'class': 'yd_text2'}),
|
||||||
|
# 已解析
|
||||||
|
'www.23zw.com': Rules('0', {'id': 'chapter_list'}, {'id': 'text_area'}),
|
||||||
|
# 已解析
|
||||||
|
'www.23us.la': Rules('http://www.23us.la/', {'class': 'inner'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.2952.cc': Rules('0', {'class': 'inner'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.23us.cc': Rules('0', {'class': 'inner'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.13xs.com': Rules('0', {'class': 'box_con'}, {'id': 'booktext'}),
|
||||||
|
# 已解析
|
||||||
|
'www.tsxsw.com': Rules('0', {'class': 'bdsub'}, {'id': 'contents'}),
|
||||||
|
# 已解析
|
||||||
|
'www.ymoxuan.com': Rules('1', {'class': 'mulu'}, {'id': 'content'}),
|
||||||
|
'www.ixs.cc': Rules('https:', {'class': 'mulu'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'zetianjiba.net': Rules('1', {'class': 'bg'}, {'class': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.37zw.com': Rules('0', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.555zw.com': Rules('0', {'class': 'dir'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.jueshitangmen.info': Rules('1', {'class': 'bg'}, {'class': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.bxwx9.org': Rules('0', {'class': 'TabCss'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.xxbiquge.com': Rules('https://www.xxbiquge.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biquwo.com': Rules('https://www.biquwo.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.fs23.com': Rules('http://www.fs23.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.longtengx.com': Rules('http://www.longtengx.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.lingyu.org': Rules('http://www.lingyu.org/', {'class': 'mt10'}, {'id': 'htmlContent'}),
|
||||||
|
# 已解析
|
||||||
|
'www.aszw8.com': Rules('0', {'id': 'at'}, {'id': 'contents'}),
|
||||||
|
# 已解析
|
||||||
|
'www.23us.so': Rules('1', {'id': 'at'}, {'id': 'contents'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biquge.lu': Rules('http://www.biquge.lu/', {'class': 'listmain'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.3zm.net': Rules('http://www.3zm.net/', {'class': 'listmain'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.biquge.com': Rules('http://www.biquge.com/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
'www.kanshuzhong.com': Rules('0', {'class': 'bookcontent'}, {'class': 'textcontent'}),
|
||||||
|
# 已解析
|
||||||
|
'www.siluke.tw': Rules('http://www.siluke.tw/', {'class': 'box_con'}, {'id': 'content'}),
|
||||||
|
# 已解析
|
||||||
|
# 'www.ttshu.com': Rules('http://www.ttshu.com', {'class': 'border'}, {'id': 'content'}),
|
||||||
|
|
||||||
|
}
|
||||||
899
soulbook/data/user_agents.txt
Normal file
899
soulbook/data/user_agents.txt
Normal file
@ -0,0 +1,899 @@
|
|||||||
|
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0)
|
||||||
|
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)
|
||||||
|
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/5.0)
|
||||||
|
Mozilla/4.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/5.0)
|
||||||
|
Mozilla/1.22 (compatible; MSIE 10.0; Windows 3.1)
|
||||||
|
Mozilla/5.0 (Windows; U; MSIE 9.0; WIndows NT 9.0; en-US))
|
||||||
|
Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)
|
||||||
|
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 7.1; Trident/5.0)
|
||||||
|
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; Media Center PC 6.0; InfoPath.3; MS-RTC LM 8; Zune 4.7)
|
||||||
|
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; Media Center PC 6.0; InfoPath.3; MS-RTC LM 8; Zune 4.7
|
||||||
|
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Zune 4.0; InfoPath.3; MS-RTC LM 8; .NET4.0C; .NET4.0E)
|
||||||
|
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC 6.0)
|
||||||
|
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC 6.0)
|
||||||
|
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 2.0.50727; SLCC2; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Zune 4.0; Tablet PC 2.0; InfoPath.3; .NET4.0C; .NET4.0E)
|
||||||
|
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0
|
||||||
|
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; chromeframe/11.0.696.57)
|
||||||
|
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0) chromeframe/10.0.648.205
|
||||||
|
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.0; Trident/5.0; chromeframe/11.0.696.57)
|
||||||
|
Mozilla/5.0 ( ; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)
|
||||||
|
Mozilla/4.0 (compatible; MSIE 9.0; Windows NT 5.1; Trident/5.0)
|
||||||
|
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 7.1; Trident/5.0; .NET CLR 2.0.50727; SLCC2; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.3; .NET4.0C)
|
||||||
|
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; AskTB5.5)
|
||||||
|
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; InfoPath.2; .NET4.0C; .NET4.0E)
|
||||||
|
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 2.0.50727; SLCC2; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.3; .NET4.0C)
|
||||||
|
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; FDM; .NET CLR 1.1.4322; .NET4.0C; .NET4.0E; Tablet PC 2.0)
|
||||||
|
Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.2; Trident/4.0; Media Center PC 4.0; SLCC1; .NET CLR 3.0.04320)
|
||||||
|
Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; SLCC1; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 1.1.4322)
|
||||||
|
Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; InfoPath.2; SLCC1; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 2.0.50727)
|
||||||
|
Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727)
|
||||||
|
Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.0; Trident/4.0; InfoPath.1; SV1; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 3.0.04506.30)
|
||||||
|
Mozilla/5.0 (compatible; MSIE 7.0; Windows NT 5.0; Trident/4.0; FBSMTWB; .NET CLR 2.0.34861; .NET CLR 3.0.3746.3218; .NET CLR 3.5.33652; msn OptimizedIE8;ENUS)
|
||||||
|
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.2; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)
|
||||||
|
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; Media Center PC 6.0; InfoPath.2; MS-RTC LM 8)
|
||||||
|
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; Media Center PC 6.0; InfoPath.2; MS-RTC LM 8
|
||||||
|
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; Media Center PC 6.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C)
|
||||||
|
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; InfoPath.3; .NET4.0C; .NET4.0E; .NET CLR 3.5.30729; .NET CLR 3.0.30729; MS-RTC LM 8)
|
||||||
|
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; InfoPath.2)
|
||||||
|
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Zune 3.0)
|
||||||
|
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; msn OptimizedIE8;ZHCN)
|
||||||
|
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MS-RTC LM 8; InfoPath.3; .NET4.0C; .NET4.0E) chromeframe/8.0.552.224
|
||||||
|
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MS-RTC LM 8; .NET4.0C; .NET4.0E; Zune 4.7; InfoPath.3)
|
||||||
|
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MS-RTC LM 8; .NET4.0C; .NET4.0E; Zune 4.7)
|
||||||
|
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MS-RTC LM 8)
|
||||||
|
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.3; Zune 4.0)
|
||||||
|
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.3; .NET4.0C; .NET4.0E; MS-RTC LM 8; Zune 4.7)
|
||||||
|
Mozilla/5.0 (X11; Linux x86_64; rv:2.2a1pre) Gecko/20110324 Firefox/4.2a1pre
|
||||||
|
Mozilla/5.0 (X11; Linux x86_64; rv:2.2a1pre) Gecko/20100101 Firefox/4.2a1pre
|
||||||
|
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.2a1pre) Gecko/20110324 Firefox/4.2a1pre
|
||||||
|
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.2a1pre) Gecko/20110323 Firefox/4.2a1pre
|
||||||
|
Mozilla/5.0 (X11; Linux x86_64; rv:2.0b9pre) Gecko/20110111 Firefox/4.0b9pre
|
||||||
|
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.0b9pre) Gecko/20101228 Firefox/4.0b9pre
|
||||||
|
Mozilla/5.0 (Windows NT 5.1; rv:2.0b9pre) Gecko/20110105 Firefox/4.0b9pre
|
||||||
|
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b8pre) Gecko/20101114 Firefox/4.0b8pre
|
||||||
|
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.0b8pre) Gecko/20101213 Firefox/4.0b8pre
|
||||||
|
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.0b8pre) Gecko/20101128 Firefox/4.0b8pre
|
||||||
|
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.0b8pre) Gecko/20101114 Firefox/4.0b8pre
|
||||||
|
Mozilla/5.0 (Windows NT 5.1; rv:2.0b8pre) Gecko/20101127 Firefox/4.0b8pre
|
||||||
|
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0b8) Gecko/20100101 Firefox/4.0b8
|
||||||
|
Mozilla/5.0 (Windows NT 6.1; rv:2.0b7pre) Gecko/20100921 Firefox/4.0b7pre
|
||||||
|
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b7) Gecko/20101111 Firefox/4.0b7
|
||||||
|
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b7) Gecko/20100101 Firefox/4.0b7
|
||||||
|
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b6pre) Gecko/20100903 Firefox/4.0b6pre
|
||||||
|
Mozilla/5.0 (Windows NT 6.1; rv:2.0b6pre) Gecko/20100903 Firefox/4.0b6pre Firefox/4.0b6pre
|
||||||
|
Mozilla/5.0 (X11; Linux x86_64; rv:2.0b4) Gecko/20100818 Firefox/4.0b4
|
||||||
|
Mozilla/5.0 (X11; Linux i686; rv:2.0b3pre) Gecko/20100731 Firefox/4.0b3pre
|
||||||
|
Mozilla/5.0 (Windows NT 5.2; rv:2.0b13pre) Gecko/20110304 Firefox/4.0b13pre
|
||||||
|
Mozilla/5.0 (Windows NT 5.1; rv:2.0b13pre) Gecko/20110223 Firefox/4.0b13pre
|
||||||
|
Mozilla/5.0 (X11; Linux i686; rv:2.0b12pre) Gecko/20110204 Firefox/4.0b12pre
|
||||||
|
Mozilla/5.0 (X11; Linux i686; rv:2.0b12pre) Gecko/20100101 Firefox/4.0b12pre
|
||||||
|
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b11pre) Gecko/20110128 Firefox/4.0b11pre
|
||||||
|
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.0b11pre) Gecko/20110131 Firefox/4.0b11pre
|
||||||
|
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.0b11pre) Gecko/20110129 Firefox/4.0b11pre
|
||||||
|
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.0b11pre) Gecko/20110128 Firefox/4.0b11pre
|
||||||
|
Mozilla/5.0 (Windows NT 6.1; rv:2.0b11pre) Gecko/20110126 Firefox/4.0b11pre
|
||||||
|
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:2.0b11pre) Gecko/20110126 Firefox/4.0b11pre
|
||||||
|
Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:2.0b10pre) Gecko/20110118 Firefox/4.0b10pre
|
||||||
|
Mozilla/5.0 (Windows NT 6.1; rv:2.0b10pre) Gecko/20110113 Firefox/4.0b10pre
|
||||||
|
Mozilla/5.0 (X11; Linux i686; rv:2.0b10) Gecko/20100101 Firefox/4.0b10
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:2.0b10) Gecko/20110126 Firefox/4.0b10
|
||||||
|
Mozilla/5.0 (Windows NT 6.1; rv:2.0b10) Gecko/20110126 Firefox/4.0b10
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; pl-PL; rv:2.0) Gecko/20110307 Firefox/4.0
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:2.0) Gecko/20110404 Fedora/16-dev Firefox/4.0
|
||||||
|
Mozilla/5.0 (X11; Arch Linux i686; rv:2.0) Gecko/20110321 Firefox/4.0
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; ru; rv:1.9.2.3) Gecko/20100401 Firefox/4.0 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows NT 6.1; rv:2.0) Gecko/20110319 Firefox/4.0
|
||||||
|
Mozilla/5.0 (Windows NT 6.1; rv:1.9) Gecko/20100101 Firefox/4.0
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; pl-PL; rv:1.9.0.2) Gecko/20121223 Ubuntu/9.25 (jaunty) Firefox/3.8
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; pl-PL; rv:1.9.0.2) Gecko/2008092313 Ubuntu/9.25 (jaunty) Firefox/3.8
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; it-IT; rv:1.9.0.2) Gecko/2008092313 Ubuntu/9.25 (jaunty) Firefox/3.8
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Mozilla/5.0 (X11; U; Linux i686; it-IT; rv:1.9.0.2) Gecko/2008092313 Ubuntu/9.25 (jaunty) Firefox/3.8
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; ru; rv:1.9.3a5pre) Gecko/20100526 Firefox/3.7a5pre
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; ru; rv:1.9.2b5) Gecko/20091204 Firefox/3.6b5
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2b5) Gecko/20091204 Firefox/3.6b5
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.2b5) Gecko/20091204 Firefox/3.6b5
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20091218 Firefox 3.6b5
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.2b4) Gecko/20091124 Firefox/3.6b4 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2b4) Gecko/20091124 Firefox/3.6b4
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2b1) Gecko/20091014 Firefox/3.6b1 GTB5
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2a1pre) Gecko/20090428 Firefox/3.6a1pre
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2a1pre) Gecko/20090405 Firefox/3.6a1pre
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; ru-RU; rv:1.9.2a1pre) Gecko/20090405 Ubuntu/9.04 (jaunty) Firefox/3.6a1pre
|
||||||
|
Mozilla/5.0 (Windows; Windows NT 5.1; es-ES; rv:1.9.2a1pre) Gecko/20090402 Firefox/3.6a1pre
|
||||||
|
Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:1.9.2a1pre) Gecko/20090402 Firefox/3.6a1pre
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; ja; rv:1.9.2a1pre) Gecko/20090402 Firefox/3.6a1pre (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.9) Gecko/20100915 Gentoo Firefox/3.6.9
|
||||||
|
Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.9.2.9) Gecko/20100913 Firefox/3.6.9
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.2.9) Gecko/20100824 Firefox/3.6.9 ( .NET CLR 3.5.30729; .NET CLR 4.0.20506)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.2; en-GB; rv:1.9.2.9) Gecko/20100824 Firefox/3.6.9
|
||||||
|
Mozilla/5.0 (X11; U; OpenBSD i386; en-US; rv:1.9.2.8) Gecko/20101230 Firefox/3.6.8
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.8) Gecko/20100804 Gentoo Firefox/3.6.8
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.8) Gecko/20100723 SUSE/3.6.8-0.1.1 Firefox/3.6.8
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; zh-CN; rv:1.9.2.8) Gecko/20100722 Ubuntu/10.04 (lucid) Firefox/3.6.8
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; ru; rv:1.9.2.8) Gecko/20100723 Ubuntu/10.04 (lucid) Firefox/3.6.8
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; fi-FI; rv:1.9.2.8) Gecko/20100723 Ubuntu/10.04 (lucid) Firefox/3.6.8
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.8) Gecko/20100727 Firefox/3.6.8
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; de-DE; rv:1.9.2.8) Gecko/20100725 Gentoo Firefox/3.6.8
|
||||||
|
Mozilla/5.0 (X11; U; FreeBSD i386; de-CH; rv:1.9.2.8) Gecko/20100729 Firefox/3.6.8
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; zh-CN; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; pt-BR; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8 GTB7.1
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; it; rv:1.9.2.8) Gecko/20100722 AskTbADAP/3.9.1.14019 Firefox/3.6.8
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; he; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; fr; rv:1.9.2.8) Gecko/20100722 Firefox 3.6.8 GTB7.1
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8 ( .NET CLR 3.5.30729; .NET4.0C)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.2.8) Gecko/20100722 Firefox 3.6.8
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.2.3) Gecko/20121221 Firefox/3.6.8
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.2; zh-TW; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; tr; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8 ( .NET CLR 3.5.30729; .NET4.0E
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.7) Gecko/20100809 Fedora/3.6.7-1.fc14 Firefox/3.6.7
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.7) Gecko/20100723 Fedora/3.6.7-1.fc13 Firefox/3.6.7
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.7) Gecko/20100726 CentOS/3.6-3.el5.centos Firefox/3.6.7
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; hu; rv:1.9.2.7) Gecko/20100713 Firefox/3.6.7 GTB7.1
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.7 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; pt-PT; rv:1.9.2.7) Gecko/20100713 Firefox/3.6.7 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.6) Gecko/20100628 Ubuntu/10.04 (lucid) Firefox/3.6.6 GTB7.1
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.6) Gecko/20100628 Ubuntu/10.04 (lucid) Firefox/3.6.6 GTB7.0
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.6) Gecko/20100628 Ubuntu/10.04 (lucid) Firefox/3.6.6 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.6) Gecko/20100628 Ubuntu/10.04 (lucid) Firefox/3.6.6
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; pt-PT; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; it; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6 ( .NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; zh-CN; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6 GTB7.1
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; nl; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; it; rv:1.9.2.6) Gecko/20100625 Firefox/3.6.6 ( .NET CLR 3.5.30729; .NET4.0E)
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.4) Gecko/20100614 Ubuntu/10.04 (lucid) Firefox/3.6.4
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.4) Gecko/20100625 Gentoo Firefox/3.6.4
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; zh-TW; rv:1.9.2.4) Gecko/20100611 Firefox/3.6.4 ( .NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; ru; rv:1.9.2.4) Gecko/20100513 Firefox/3.6.4
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; ja; rv:1.9.2.4) Gecko/20100611 Firefox/3.6.4 GTB7.1
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; cs; rv:1.9.2.4) Gecko/20100513 Firefox/3.6.4 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; zh-CN; rv:1.9.2.4) Gecko/20100513 Firefox/3.6.4
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; ja; rv:1.9.2.4) Gecko/20100513 Firefox/3.6.4 ( .NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; fr; rv:1.9.2.4) Gecko/20100523 Firefox/3.6.4 ( .NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.4) Gecko/20100527 Firefox/3.6.4 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.4) Gecko/20100527 Firefox/3.6.4
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.4) Gecko/20100523 Firefox/3.6.4 ( .NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.4) Gecko/20100513 Firefox/3.6.4 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.2; en-CA; rv:1.9.2.4) Gecko/20100523 Firefox/3.6.4
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-TW; rv:1.9.2.4) Gecko/20100611 Firefox/3.6.4 GTB7.0 ( .NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9.2.4) Gecko/20100513 Firefox/3.6.4 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9.2.4) Gecko/20100503 Firefox/3.6.4 ( .NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; nb-NO; rv:1.9.2.4) Gecko/20100611 Firefox/3.6.4 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; ko; rv:1.9.2.4) Gecko/20100523 Firefox/3.6.4
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; cs; rv:1.9.2.4) Gecko/20100611 Firefox/3.6.4
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.3pre) Gecko/20100405 Firefox/3.6.3plugin1 ( .NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; fr; rv:1.9.2.3) Gecko/20100403 Fedora/3.6.3-4.fc13 Firefox/3.6.3
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.3) Gecko/20100403 Firefox/3.6.3
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; de; rv:1.9.2.3) Gecko/20100401 SUSE/3.6.3-1.1 Firefox/3.6.3
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; ko-KR; rv:1.9.2.3) Gecko/20100423 Ubuntu/10.04 (lucid) Firefox/3.6.3
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.3) Gecko/20100404 Ubuntu/10.04 (lucid) Firefox/3.6.3
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 GTB7.1
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.3
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.2.3) Gecko/20100423 Ubuntu/10.04 (lucid) Firefox/3.6.3
|
||||||
|
Mozilla/5.0 (X11; U; Linux AMD64; en-US; rv:1.9.2.3) Gecko/20100403 Ubuntu/10.10 (maverick) Firefox/3.6.3
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; zh-CN; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; ru; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; pl; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; it; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; hu; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 GTB7.1
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; es-ES; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 GTB7.1
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; es-ES; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 GTB7.0 ( .NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; es-ES; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; cs; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 ( .NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; ca; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; fr; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; fr; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2 GTB7.0
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.2) Gecko/20100316 AskTbSPC2/3.9.1.14019 Firefox/3.6.2
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2 ( .NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; pl; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2 GTB6 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2 ( .NET CLR 3.0.04506.648)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2 ( .NET CLR 3.0.04506.30)
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.7; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.10pre) Gecko/20100902 Ubuntu/9.10 (karmic) Firefox/3.6.1pre
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; ja-JP; rv:1.9.2.16) Gecko/20110323 Ubuntu/10.10 (maverick) Firefox/3.6.16
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.16) Gecko/20110323 Ubuntu/9.10 (karmic) Firefox/3.6.16 FirePHP/0.5
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; fr; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; pl; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; ko; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16 ( .NET CLR 3.5.30729; .NET4.0E)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.9.2.16) Gecko/20110319 Firefox/3.6.16 ( .NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en; rv:1.9.1.13) Gecko/20100914 Firefox/3.6.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.16) Gecko/20110319 AskTbUTR/3.11.3.15590 Firefox/3.6.16
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.16pre) Gecko/20110304 Ubuntu/10.10 (maverick) Firefox/3.6.15pre
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 FirePHP/0.5
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.2.15) Gecko/20110330 CentOS/3.6-1.el5.centos Firefox/3.6.15
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; es-ES; rv:1.9.2.15) Gecko/20110303 Firefox/3.6.15
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.15) Gecko/20110303 Firefox/3.6.15 ( .NET CLR 3.5.30729; .NET4.0C) FirePHP/0.5
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.2.15) Gecko/20110303 AskTbBT4/3.11.3.15590 Firefox/3.6.15 ( .NET CLR 3.5.30729; .NET4.0C)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.15) Gecko/20110303 Firefox/3.6.15 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.14pre) Gecko/20110105 Firefox/3.6.14pre
|
||||||
|
Mozilla/5.0 (X11; U; Linux armv7l; en-US; rv:1.9.2.14) Gecko/20110224 Firefox/3.6.14 MB860/Version.0.43.3.MB860.AmericaMovil.en.MX
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; zh-CN; rv:1.9.2.14) Gecko/20110218 Firefox/3.6.14
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-AU; rv:1.9.2.14) Gecko/20110218 Firefox/3.6.14
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.14) Gecko/20110218 Firefox/3.6.14 GTB7.1 ( .NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.2.13) Firefox/3.6.13
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; pl-PL; rv:1.9.2.13) Gecko/20101206 Ubuntu/10.04 (lucid) Firefox/3.6.13
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; nb-NO; rv:1.9.2.13) Gecko/20101206 Ubuntu/10.04 (lucid) Firefox/3.6.13
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; it; rv:1.9.2.13) Gecko/20101206 Ubuntu/10.04 (lucid) Firefox/3.6.13 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; fr; rv:1.9.2.13) Gecko/20110103 Fedora/3.6.13-1.fc14 Firefox/3.6.13
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; fr; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101223 Gentoo Firefox/3.6.13
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101219 Gentoo Firefox/3.6.13
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101206 Red Hat/3.6-3.el4 Firefox/3.6.13
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101206 Firefox/3.6.13
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-NZ; rv:1.9.2.13) Gecko/20101206 Ubuntu/10.10 (maverick) Firefox/3.6.13
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-GB; rv:1.9.2.13) Gecko/20101206 Ubuntu/9.10 (karmic) Firefox/3.6.13
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-GB; rv:1.9.2.13) Gecko/20101206 Red Hat/3.6-2.el5 Firefox/3.6.13
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; da-DK; rv:1.9.2.13) Gecko/20101206 Ubuntu/10.10 (maverick) Firefox/3.6.13
|
||||||
|
Mozilla/5.0 (X11; U; Linux MIPS32 1074Kf CPS QuadCore; en-US; rv:1.9.2.13) Gecko/20110103 Fedora/3.6.13-1.fc14 Firefox/3.6.13
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; ru; rv:1.9.2.13) Gecko/20101206 Ubuntu/10.10 (maverick) Firefox/3.6.13
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; pt-BR; rv:1.9.2.13) Gecko/20101209 Fedora/3.6.13-1.fc13 Firefox/3.6.13
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; es-ES; rv:1.9.2.13) Gecko/20101206 Ubuntu/9.10 (karmic) Firefox/3.6.13
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.2.13) Gecko/20101209 CentOS/3.6-2.el5.centos Firefox/3.6.13
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.2.13) Gecko/20101206 Ubuntu/10.10 (maverick) Firefox/3.6.13
|
||||||
|
Mozilla/5.0 (X11; U; NetBSD i386; en-US; rv:1.9.2.12) Gecko/20101030 Firefox/3.6.12
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; es-MX; rv:1.9.2.12) Gecko/20101027 Ubuntu/10.04 (lucid) Firefox/3.6.12
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; es-ES; rv:1.9.2.12) Gecko/20101027 Fedora/3.6.12-1.fc13 Firefox/3.6.12
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; es-ES; rv:1.9.2.12) Gecko/20101026 SUSE/3.6.12-0.7.1 Firefox/3.6.12
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.12) Gecko/20101102 Gentoo Firefox/3.6.12
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.12) Gecko/20101102 Firefox/3.6.12
|
||||||
|
Mozilla/5.0 (X11; U; Linux ppc; fr; rv:1.9.2.12) Gecko/20101027 Ubuntu/10.10 (maverick) Firefox/3.6.12
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; ko-KR; rv:1.9.2.12) Gecko/20101027 Ubuntu/10.10 (maverick) Firefox/3.6.12
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.9.2.12) Gecko/20101027 Ubuntu/10.10 (maverick) Firefox/3.6.12 GTB7.1
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.2.12) Gecko/20101027 Fedora/3.6.12-1.fc13 Firefox/3.6.12
|
||||||
|
Mozilla/5.0 (X11; FreeBSD x86_64; rv:2.0) Gecko/20100101 Firefox/3.6.12
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; zh-CN; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 ( .NET CLR 3.5.30729; .NET4.0E)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; sv-SE; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 (.NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729; .NET CLR 3.5.21022)
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; de; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 GTB5
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; ru; rv:1.9.2.11) Gecko/20101028 CentOS/3.6-2.el5.centos Firefox/3.6.11
|
||||||
|
Mozilla/5.0 (X11; U; Linux armv7l; en-GB; rv:1.9.2.3pre) Gecko/20100723 Firefox/3.6.11
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.2; ru; rv:1.9.2.11) Gecko/20101012 Firefox/3.6.11
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; it; rv:1.9.2.11) Gecko/20101012 Firefox/3.6.11 ( .NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; zh-CN; rv:1.9.2.10) Gecko/20100922 Ubuntu/10.10 (maverick) Firefox/3.6.10
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; pt-BR; rv:1.9.2.10) Gecko/20100922 Ubuntu/10.10 (maverick) Firefox/3.6.10
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; pl-PL; rv:1.9.2.10) Gecko/20100922 Ubuntu/10.10 (maverick) Firefox/3.6.10
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.10) Gecko/20100922 Ubuntu/10.10 (maverick) Firefox/3.6.10 GTB7.1
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; el-GR; rv:1.9.2.10) Gecko/20100922 Ubuntu/10.10 (maverick) Firefox/3.6.10
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; de; rv:1.9.2.10) Gecko/20100922 Ubuntu/10.10 (maverick) Firefox/3.6.10 GTB7.1
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; cs-CZ; rv:1.9.2.10) Gecko/20100915 Ubuntu/10.04 (lucid) Firefox/3.6.10
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; pl-PL; rv:1.9.2.10) Gecko/20100915 Ubuntu/10.04 (lucid) Firefox/3.6.10
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; fr-FR; rv:1.9.2.10) Gecko/20100914 Firefox/3.6.10
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; es-AR; rv:1.9.2.10) Gecko/20100922 Ubuntu/10.10 (maverick) Firefox/3.6.10
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.10) Gecko/20100915 Ubuntu/9.04 (jaunty) Firefox/3.6.10
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.9.2.11) Gecko/20101013 Ubuntu/10.10 (maverick) Firefox/3.6.10
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-CA; rv:1.9.2.10) Gecko/20100922 Ubuntu/10.10 (maverick) Firefox/3.6.10
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.2.10) Gecko/20100922 Ubuntu/10.10 (maverick) Firefox/3.6.10
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.2.10) Gecko/20100915 Ubuntu/9.10 (karmic) Firefox/3.6.10
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.2.10) Gecko/20100915 Ubuntu/10.04 (lucid) Firefox/3.6.10
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.2.10) Gecko/20100914 SUSE/3.6.10-0.3.1 Firefox/3.6.10
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; ro; rv:1.9.2.10) Gecko/20100914 Firefox/3.6.10
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; nl; rv:1.9.2.10) Gecko/20100914 Firefox/3.6.10 ( .NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; fr; rv:1.9.2.10) Gecko/20100914 Firefox/3.6.10 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.1) Gecko/20100122 firefox/3.6.1
|
||||||
|
Mozilla/5.0(Windows; U; Windows NT 7.0; rv:1.9.2) Gecko/20100101 Firefox/3.6
|
||||||
|
Mozilla/5.0(Windows; U; Windows NT 5.2; rv:1.9.2) Gecko/20100101 Firefox/3.6
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2) Gecko/20100222 Ubuntu/10.04 (lucid) Firefox/3.6
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2) Gecko/20100130 Gentoo Firefox/3.6
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; de; rv:1.9.2) Gecko/20100308 Ubuntu/10.04 (lucid) Firefox/3.6
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.2pre) Gecko/20100312 Ubuntu/9.04 (jaunty) Firefox/3.6
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2) Gecko/20100128 Gentoo Firefox/3.6
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2) Gecko/20100115 Ubuntu/10.04 (lucid) Firefox/3.6
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 FirePHP/0.4
|
||||||
|
Mozilla/5.0 (X11; Linux i686; rv:2.0) Gecko/20100101 Firefox/3.6
|
||||||
|
Mozilla/5.0 (X11; FreeBSD i686) Firefox/3.6
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; ru-RU; rv:1.9.2) Gecko/20100105 MRA 5.6 (build 03278) Firefox/3.6 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; lt; rv:1.9.2) Gecko/20100115 Firefox/3.6
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.3a3pre) Gecko/20100306 Firefox3.6 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.8) Gecko/20100806 Firefox/3.6
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.3) Gecko/20100401 Firefox/3.6;MEGAUPLOAD 1.0
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; ar; rv:1.9.2) Gecko/20100115 Firefox/3.6
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; ru; rv:1.9.2) Gecko/20100115 Firefox/3.6
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; ru; rv:1.9.2) Gecko/20100105 Firefox/3.6 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; pl; rv:1.9.2) Gecko/20100115 Firefox/3.6 ( .NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b5pre) Gecko/20090517 Firefox/3.5b4pre (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b4pre) Gecko/20090409 Firefox/3.5b4pre
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b4pre) Gecko/20090401 Firefox/3.5b4pre
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; nl-NL; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4 GTB5 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; fr; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1b4) Gecko/20090423 Firefox/3.5b4 GTB5
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; it; rv:1.9.1.9) Gecko/20100402 Ubuntu/9.10 (karmic) Firefox/3.5.9 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; it; rv:1.9.1.9) Gecko/20100330 Fedora/3.5.9-2.fc12 Firefox/3.5.9
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; fr; rv:1.9.1.9) Gecko/20100317 SUSE/3.5.9-0.1.1 Firefox/3.5.9 GTB7.0
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; es-CL; rv:1.9.1.9) Gecko/20100402 Ubuntu/9.10 (karmic) Firefox/3.5.9
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; cs-CZ; rv:1.9.1.9) Gecko/20100317 SUSE/3.5.9-0.1.1 Firefox/3.5.9
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; nl; rv:1.9.1.9) Gecko/20100401 Ubuntu/9.10 (karmic) Firefox/3.5.9
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; hu-HU; rv:1.9.1.9) Gecko/20100330 Fedora/3.5.9-1.fc12 Firefox/3.5.9
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; es-ES; rv:1.9.1.9) Gecko/20100317 SUSE/3.5.9-0.1 Firefox/3.5.9
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.9) Gecko/20100401 Ubuntu/9.10 (karmic) Firefox/3.5.9 GTB7.1
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.9) Gecko/20100315 Ubuntu/9.10 (karmic) Firefox/3.5.9
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.4) Gecko/20091028 Ubuntu/9.10 (karmic) Firefox/3.5.9
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; tr; rv:1.9.1.9) Gecko/20100315 Firefox/3.5.9 GTB7.1
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; hu; rv:1.9.1.9) Gecko/20100315 Firefox/3.5.9 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; fr; rv:1.9.1.9) Gecko/20100315 Firefox/3.5.9
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; et; rv:1.9.1.9) Gecko/20100315 Firefox/3.5.9
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.9) Gecko/20100315 Firefox/3.5.9
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; nl; rv:1.9.1.9) Gecko/20100315 Firefox/3.5.9 ( .NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; es-ES; rv:1.9.1.9) Gecko/20100315 Firefox/3.5.9 GTB5 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; de; rv:1.9.2.13) Gecko/20101203 Firefox/3.5.9 (de)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; de; rv:1.9.1.9) Gecko/20100315 Firefox/3.5.9 GTB7.0 (.NET CLR 3.0.30618)
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; ru; rv:1.9.1.8) Gecko/20100216 Fedora/3.5.8-1.fc12 Firefox/3.5.8
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; es-ES; rv:1.9.1.8) Gecko/20100216 Fedora/3.5.8-1.fc11 Firefox/3.5.8
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.8) Gecko/20100318 Gentoo Firefox/3.5.8
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; zh-CN; rv:1.9.1.8) Gecko/20100216 Fedora/3.5.8-1.fc12 Firefox/3.5.8
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; ja-JP; rv:1.9.1.8) Gecko/20100216 Fedora/3.5.8-1.fc12 Firefox/3.5.8
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; es-AR; rv:1.9.1.8) Gecko/20100214 Ubuntu/9.10 (karmic) Firefox/3.5.8
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.1.8) Gecko/20100214 Ubuntu/9.10 (karmic) Firefox/3.5.8
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.1.8) Gecko/20100202 Firefox/3.5.8
|
||||||
|
Mozilla/5.0 (X11; U; FreeBSD i386; ja-JP; rv:1.9.1.8) Gecko/20100305 Firefox/3.5.8
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; sl; rv:1.9.1.8) Gecko/20100202 Firefox/3.5.8
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.8) Gecko/20100202 Firefox/3.5.8 (.NET CLR 3.5.30729) FirePHP/0.4
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-TW; rv:1.9.1.8) Gecko/20100202 Firefox/3.5.8 GTB6
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; ja; rv:1.9.1.8) Gecko/20100202 Firefox/3.5.8 GTB7.0 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2) Gecko/20100305 Gentoo Firefox/3.5.7
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; cs-CZ; rv:1.9.1.7) Gecko/20100106 Ubuntu/9.10 (karmic) Firefox/3.5.7
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; es-ES; rv:1.9.1.7) Gecko/20091222 SUSE/3.5.7-1.1.1 Firefox/3.5.7
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; ja; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 (.NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.30729; .NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.2; fr; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 (.NET CLR 3.0.04506.648)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; fa; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.7) Gecko/20091221 MRA 5.5 (build 02842) Firefox/3.5.7 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; fr; rv:1.9.1.6) Gecko/20091215 Ubuntu/9.10 (karmic) Firefox/3.5.6
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.6) Gecko/20100117 Gentoo Firefox/3.5.6
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; zh-CN; rv:1.9.1.6) Gecko/20091216 Fedora/3.5.6-1.fc11 Firefox/3.5.6 GTB6
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; es-ES; rv:1.9.1.6) Gecko/20091201 SUSE/3.5.6-1.1.1 Firefox/3.5.6 GTB6
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.6) Gecko/20100118 Gentoo Firefox/3.5.6
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.9.1.6) Gecko/20091215 Ubuntu/9.10 (karmic) Firefox/3.5.6 GTB6
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.1.6) Gecko/20091215 Ubuntu/9.10 (karmic) Firefox/3.5.6 GTB7.0
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.1.6) Gecko/20091215 Ubuntu/9.10 (karmic) Firefox/3.5.6
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.1.6) Gecko/20091201 SUSE/3.5.6-1.1.1 Firefox/3.5.6
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; cs-CZ; rv:1.9.1.6) Gecko/20100107 Fedora/3.5.6-1.fc12 Firefox/3.5.6
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; ca; rv:1.9.1.6) Gecko/20091215 Ubuntu/9.10 (karmic) Firefox/3.5.6
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; it; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 ( .NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; id; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.6) Gecko/20091201 MRA 5.4 (build 02647) Firefox/3.5.6 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 (.NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.6) Gecko/20091201 MRA 5.5 (build 02842) Firefox/3.5.6 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.6) Gecko/20091201 MRA 5.5 (build 02842) Firefox/3.5.6
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 GTB6 (.NET CLR 3.5.30729) FBSMTWB
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 (.NET CLR 3.5.30729) FBSMTWB
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; fr; rv:1.9.1.5) Gecko/20091109 Ubuntu/9.10 (karmic) Firefox/3.5.5
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.8pre) Gecko/20091227 Ubuntu/9.10 (karmic) Firefox/3.5.5
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.5) Gecko/20091114 Gentoo Firefox/3.5.5
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; uk; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.5) Gecko/20091102 MRA 5.5 (build 02842) Firefox/3.5.5
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; ru; rv:1.9.1.5) Gecko/20091102 MRA 5.5 (build 02842) Firefox/3.5.5
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5 ( .NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.2; zh-CN; rv:1.9.1.5) Gecko/Firefox/3.5.5
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091102 MRA 5.5 (build 02842) Firefox/3.5.5 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.5) Gecko/20091102 MRA 5.5 (build 02842) Firefox/3.5.5
|
||||||
|
Mozilla/5.0 (Windows NT 5.1; U; zh-cn; rv:1.8.1) Gecko/20091102 Firefox/3.5.5
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; pl; rv:1.9.1.5) Gecko/20091102 Firefox/3.5.5 FBSMTWB
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; ja; rv:1.9.1.4) Gecko/20091016 SUSE/3.5.4-1.1.2 Firefox/3.5.4
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.4) Gecko/20091016 Firefox/3.5.4 (.NET CLR 3.5.30729) FBSMTWB
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.1.4) Gecko/20091007 Firefox/3.5.4
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; ru-RU; rv:1.9.1.4) Gecko/20091016 Firefox/3.5.4 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.1.4) Gecko/20091016 Firefox/3.5.4 ( .NET CLR 3.5.30729; .NET4.0E)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.4) Gecko/20091007 Firefox/3.5.4
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; fr; rv:1.9.1.5) Gecko/20091109 Ubuntu/9.10 (karmic) Firefox/3.5.3pre
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.3) Gecko/20090914 Slackware/13.0_stable Firefox/3.5.3
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.3) Gecko/20090913 Firefox/3.5.3
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; ru; rv:1.9.1.3) Gecko/20091020 Ubuntu/9.10 (karmic) Firefox/3.5.3
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; fr; rv:1.9.1.3) Gecko/20090913 Firefox/3.5.3
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.3) Gecko/20090919 Firefox/3.5.3
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.3) Gecko/20090912 Gentoo Firefox/3.5.3 FirePHP/0.3
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 GTB5
|
||||||
|
Mozilla/5.0 (X11; U; FreeBSD i386; ru-RU; rv:1.9.1.3) Gecko/20090913 Firefox/3.5.3
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; zh-CN; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; fr; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 ( .NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.5.3;MEGAUPLOAD 1.0 ( .NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; de-DE; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; ko; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; fi; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 (.NET CLR 2.0.50727; .NET CLR 3.0.30618; .NET CLR 3.5.21022; .NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; bg; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; ko; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; pl; rv:1.9.1.2) Gecko/20090911 Slackware Firefox/3.5.2
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.2) Gecko/20090803 Slackware Firefox/3.5.2
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.2) Gecko/20090803 Firefox/3.5.2 Slackware
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; ru-RU; rv:1.9.1.2) Gecko/20090804 Firefox/3.5.2
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.2) Gecko/20090729 Slackware/13.0 Firefox/3.5.2
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686 (x86_64); fr; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; zh-CN; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; pl; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 GTB7.1 ( .NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; es-MX; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; de; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; de; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-TW; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; uk; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; pt-BR; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; ja; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; ja; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; es-ES; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.16) Gecko/20101130 Firefox/3.5.16 FirePHP/0.4
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.1.16) Gecko/20101130 AskTbMYC/3.9.1.14019 Firefox/3.5.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; it; rv:1.9.1.16) Gecko/20101130 Firefox/3.5.16 GTB7.1 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.1.16) Gecko/20101130 MRA 5.4 (build 02647) Firefox/3.5.16 ( .NET CLR 3.5.30729; .NET4.0C)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.16) Gecko/20101130 Firefox/3.5.16 GTB7.1
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.16) Gecko/20101130 AskTbPLTV5/3.8.0.12304 Firefox/3.5.16 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.1.16) Gecko/20101130 Firefox/3.5.16 GTB7.1 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.1.16) Gecko/20101130 Firefox/3.5.16 GTB7.1
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; it; rv:1.9.1.15) Gecko/20101027 Fedora/3.5.15-1.fc12 Firefox/3.5.15
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.9.1.15) Gecko/20101027 Fedora/3.5.15-1.fc12 Firefox/3.5.15
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.0; ru; rv:1.9.1.13) Gecko/20100914 Firefox/3.5.13
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.12) Gecko/2009070611 Firefox/3.5.12
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; ru; rv:1.9.1.12) Gecko/20100824 MRA 5.7 (build 03755) Firefox/3.5.12
|
||||||
|
Mozilla/5.0 (X11; U; Linux; en-US; rv:1.9.1.11) Gecko/20100720 Firefox/3.5.11
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.1.11) Gecko/20100701 Firefox/3.5.11 ( .NET CLR 3.5.30729; .NET4.0C)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; pt-BR; rv:1.9.1.11) Gecko/20100701 Firefox/3.5.11 ( .NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; hu; rv:1.9.1.11) Gecko/20100701 Firefox/3.5.11
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.10) Gecko/20100504 Firefox/3.5.11 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; de; rv:1.9.1.10) Gecko/20100506 SUSE/3.5.10-0.1.1 Firefox/3.5.10
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.1.10) Gecko/20100504 Firefox/3.5.10 GTB7.0 ( .NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; rv:1.9.1.1) Gecko/20090716 Linux Firefox/3.5.1
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.3) Gecko/20100524 Firefox/3.5.1
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.1) Gecko/20090716 Linux Mint/7 (Gloria) Firefox/3.5.1
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.1) Gecko/20090716 Firefox/3.5.1
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.1) Gecko/20090714 SUSE/3.5.1-1.1 Firefox/3.5.1
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86; rv:1.9.1.1) Gecko/20090716 Linux Firefox/3.5.1
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; nl; rv:1.9.1.1) Gecko/20090715 Firefox/3.5.1
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.2pre) Gecko/20090729 Ubuntu/9.04 (jaunty) Firefox/3.5.1
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.1) Gecko/20090715 Firefox/3.5.1 GTB5
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.1.1) Gecko/20090722 Gentoo Firefox/3.5.1
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.1.1) Gecko/20090714 SUSE/3.5.1-1.1 Firefox/3.5.1
|
||||||
|
Mozilla/5.0 (X11; U; DragonFly i386; de; rv:1.9.1) Gecko/20090720 Firefox/3.5.1
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.1) Gecko/20090718 Firefox/3.5.1
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.1.1) Gecko/20090715 Firefox/3.5.1
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; tr; rv:1.9.1.1) Gecko/20090715 Firefox/3.5.1 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; sv-SE; rv:1.9.1.1) Gecko/20090715 Firefox/3.5.1 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; ja; rv:1.9.1.1) Gecko/20090715 Firefox/3.5.1
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.1.1) Gecko/20090715 Firefox/3.5.1 GTB5 (.NET CLR 4.0.20506)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.1.1) Gecko/20090715 Firefox/3.5.1 GTB5 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; de; rv:1.9.1.1) Gecko/20090715 Firefox/3.5.1 GTB5 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (X11;U; Linux i686; en-GB; rv:1.9.1) Gecko/20090624 Ubuntu/9.04 (jaunty) Firefox/3.5
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1) Gecko/20090630 Firefox/3.5 GTB6
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; ja; rv:1.9.1) Gecko/20090624 Firefox/3.5 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; it-IT; rv:1.9.0.2) Gecko/2008092313 Ubuntu/9.04 (jaunty) Firefox/3.5
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; fr; rv:1.9.1) Gecko/20090624 Firefox/3.5
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; fr-FR; rv:1.9.1) Gecko/20090624 Ubuntu/9.04 (jaunty) Firefox/3.5
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1) Gecko/20090701 Ubuntu/9.04 (jaunty) Firefox/3.5
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-us; rv:1.9.0.2) Gecko/2008092313 Ubuntu/9.04 (jaunty) Firefox/3.5
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.1) Gecko/20090624 Ubuntu/8.04 (hardy) Firefox/3.5
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; de; rv:1.9.1) Gecko/20090624 Firefox/3.5
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686 (x86_64); de; rv:1.9.1) Gecko/20090624 Firefox/3.5
|
||||||
|
Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.9.1) Gecko/20090703 Firefox/3.5
|
||||||
|
Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.9.0.10) Gecko/20090624 Firefox/3.5
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; pl; rv:1.9.1) Gecko/20090624 Firefox/3.5 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; es-ES; rv:1.9.1) Gecko/20090624 Firefox/3.5 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1) Gecko/20090612 Firefox/3.5 (.NET CLR 4.0.20506)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1) Gecko/20090612 Firefox/3.5
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.1) Gecko/20090624 Firefox/3.5 (.NET CLR 4.0.20506)
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.1) Gecko/20090624 Firefox/3.5
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; zh-TW; rv:1.9.1) Gecko/20090624 Firefox/3.5 (.NET CLR 3.5.30729)
|
||||||
|
Mozilla/5.0 (Windows NT 5.1) AppleWebKit/534.25 (KHTML, like Gecko) Chrome/12.0.706.0 Safari/534.25
|
||||||
|
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.24 (KHTML, like Gecko) Ubuntu/10.10 Chromium/12.0.703.0 Chrome/12.0.703.0 Safari/534.24
|
||||||
|
Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.24 (KHTML, like Gecko) Ubuntu/10.10 Chromium/12.0.702.0 Chrome/12.0.702.0 Safari/534.24
|
||||||
|
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/12.0.702.0 Safari/534.24
|
||||||
|
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/12.0.702.0 Safari/534.24
|
||||||
|
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.699.0 Safari/534.24
|
||||||
|
Mozilla/5.0 (Windows NT 6.0; WOW64) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.699.0 Safari/534.24
|
||||||
|
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_6) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.698.0 Safari/534.24
|
||||||
|
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.697.0 Safari/534.24
|
||||||
|
Mozilla/5.0 (Windows NT 5.1) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.43 Safari/534.24
|
||||||
|
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.34 Safari/534.24
|
||||||
|
Mozilla/5.0 (Windows NT 6.0; WOW64) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.34 Safari/534.24
|
||||||
|
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.3 Safari/534.24
|
||||||
|
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.3 Safari/534.24
|
||||||
|
Mozilla/5.0 (Windows NT 6.0) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.3 Safari/534.24
|
||||||
|
Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.14 Safari/534.24
|
||||||
|
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.12 Safari/534.24
|
||||||
|
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_6) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.12 Safari/534.24
|
||||||
|
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.24 (KHTML, like Gecko) Ubuntu/10.04 Chromium/11.0.696.0 Chrome/11.0.696.0 Safari/534.24
|
||||||
|
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_0) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.0 Safari/534.24
|
||||||
|
Mozilla/5.0 (Windows NT 6.1) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.694.0 Safari/534.24
|
||||||
|
Mozilla/5.0 (X11; Linux i686) AppleWebKit/534.23 (KHTML, like Gecko) Chrome/11.0.686.3 Safari/534.23
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.21 (KHTML, like Gecko) Chrome/11.0.682.0 Safari/534.21
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.21 (KHTML, like Gecko) Chrome/11.0.678.0 Safari/534.21
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_7_0; en-US) AppleWebKit/534.21 (KHTML, like Gecko) Chrome/11.0.678.0 Safari/534.21
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/534.20 (KHTML, like Gecko) Chrome/11.0.672.2 Safari/534.20
|
||||||
|
Mozilla/5.0 (Windows NT) AppleWebKit/534.20 (KHTML, like Gecko) Chrome/11.0.672.2 Safari/534.20
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-US) AppleWebKit/534.20 (KHTML, like Gecko) Chrome/11.0.672.2 Safari/534.20
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.20 (KHTML, like Gecko) Chrome/11.0.669.0 Safari/534.20
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.19 (KHTML, like Gecko) Chrome/11.0.661.0 Safari/534.19
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.18 (KHTML, like Gecko) Chrome/11.0.661.0 Safari/534.18
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-US) AppleWebKit/534.18 (KHTML, like Gecko) Chrome/11.0.660.0 Safari/534.18
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.17 (KHTML, like Gecko) Chrome/11.0.655.0 Safari/534.17
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-US) AppleWebKit/534.17 (KHTML, like Gecko) Chrome/11.0.655.0 Safari/534.17
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.17 (KHTML, like Gecko) Chrome/11.0.654.0 Safari/534.17
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US) AppleWebKit/534.17 (KHTML, like Gecko) Chrome/11.0.652.0 Safari/534.17
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.17 (KHTML, like Gecko) Chrome/10.0.649.0 Safari/534.17
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; de-DE) AppleWebKit/534.17 (KHTML, like Gecko) Chrome/10.0.649.0 Safari/534.17
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.82 Safari/534.16
|
||||||
|
Mozilla/5.0 (X11; U; Linux armv7l; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.204 Safari/534.16
|
||||||
|
Mozilla/5.0 (X11; U; FreeBSD x86_64; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.204 Safari/534.16
|
||||||
|
Mozilla/5.0 (X11; U; FreeBSD i386; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.204 Safari/534.16
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_5; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.204
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.134 Safari/534.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.134 Safari/534.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.134 Safari/534.16
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.134 Safari/534.16
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Ubuntu/10.10 Chromium/10.0.648.133 Chrome/10.0.648.133 Safari/534.16
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.133 Safari/534.16
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Ubuntu/10.10 Chromium/10.0.648.133 Chrome/10.0.648.133 Safari/534.16
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.133 Safari/534.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.133 Safari/534.16
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.133 Safari/534.16
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_2; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.133 Safari/534.16
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Ubuntu/10.10 Chromium/10.0.648.127 Chrome/10.0.648.127 Safari/534.16
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.127 Safari/534.16
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.127 Safari/534.16
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.127 Safari/534.16
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.11 Safari/534.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; ru-RU) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.11 Safari/534.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.11 Safari/534.16
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Ubuntu/10.10 Chromium/10.0.648.0 Chrome/10.0.648.0 Safari/534.16
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Ubuntu/10.10 Chromium/10.0.648.0 Chrome/10.0.648.0 Safari/534.16
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.648.0 Safari/534.16
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Ubuntu/10.10 Chromium/10.0.642.0 Chrome/10.0.642.0 Safari/534.16
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_5; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.639.0 Safari/534.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.638.0 Safari/534.16
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.634.0 Safari/534.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.16 (KHTML, like Gecko) Chrome/10.0.634.0 Safari/534.16
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.16 SUSE/10.0.626.0 (KHTML, like Gecko) Chrome/10.0.626.0 Safari/534.16
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.15 (KHTML, like Gecko) Chrome/10.0.613.0 Safari/534.15
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.15 (KHTML, like Gecko) Ubuntu/10.10 Chromium/10.0.613.0 Chrome/10.0.613.0 Safari/534.15
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.15 (KHTML, like Gecko) Ubuntu/10.04 Chromium/10.0.612.3 Chrome/10.0.612.3 Safari/534.15
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.15 (KHTML, like Gecko) Chrome/10.0.612.1 Safari/534.15
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.15 (KHTML, like Gecko) Ubuntu/10.10 Chromium/10.0.611.0 Chrome/10.0.611.0 Safari/534.15
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.14 (KHTML, like Gecko) Chrome/10.0.602.0 Safari/534.14
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.14 (KHTML, like Gecko) Chrome/10.0.601.0 Safari/534.14
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.14 (KHTML, like Gecko) Chrome/10.0.601.0 Safari/534.14
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/540.0 (KHTML,like Gecko) Chrome/9.1.0.0 Safari/540.0
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/540.0 (KHTML, like Gecko) Ubuntu/10.10 Chrome/9.1.0.0 Safari/540.0
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/534.14 (KHTML, like Gecko) Chrome/9.0.601.0 Safari/534.14
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.14 (KHTML, like Gecko) Ubuntu/10.10 Chromium/9.0.600.0 Chrome/9.0.600.0 Safari/534.14
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.14 (KHTML, like Gecko) Chrome/9.0.600.0 Safari/534.14
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.599.0 Safari/534.13
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.84 Safari/534.13
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.44 Safari/534.13
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.19 Safari/534.13
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.15 Safari/534.13
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_5; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.15 Safari/534.13
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.0 Safari/534.13
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.0 Safari/534.13
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.0 Safari/534.13
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.0 Safari/534.13
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_5; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.0 Safari/534.13
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.0 Safari/534.13
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.596.0 Safari/534.13
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Ubuntu/10.04 Chromium/9.0.595.0 Chrome/9.0.595.0 Safari/534.13
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Ubuntu/9.10 Chromium/9.0.592.0 Chrome/9.0.592.0 Safari/534.13
|
||||||
|
Mozilla/5.0 (X11; U; Windows NT 6; en-US) AppleWebKit/534.12 (KHTML, like Gecko) Chrome/9.0.587.0 Safari/534.12
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.12 (KHTML, like Gecko) Chrome/9.0.579.0 Safari/534.12
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US) AppleWebKit/534.12 (KHTML, like Gecko) Chrome/9.0.576.0 Safari/534.12
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/540.0 (KHTML, like Gecko) Ubuntu/10.10 Chrome/8.1.0.0 Safari/540.0
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.558.0 Safari/534.10
|
||||||
|
Mozilla/5.0 (X11; U; CrOS i686 0.9.130; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.344 Safari/534.10
|
||||||
|
Mozilla/5.0 (X11; U; CrOS i686 0.9.128; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.343 Safari/534.10
|
||||||
|
Mozilla/5.0 (X11; U; CrOS i686 0.9.128; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.341 Safari/534.10
|
||||||
|
Mozilla/5.0 (X11; U; CrOS i686 0.9.128; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.339 Safari/534.10
|
||||||
|
Mozilla/5.0 (X11; U; CrOS i686 0.9.128; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.339
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Ubuntu/10.10 Chromium/8.0.552.237 Chrome/8.0.552.237 Safari/534.10
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; de-DE) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.224 Safari/534.10
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/533.3 (KHTML, like Gecko) Chrome/8.0.552.224 Safari/533.3
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.224 Safari/534.10
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.215 Safari/534.10
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.215 Safari/534.10
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.215 Safari/534.10
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.210 Safari/534.10
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.200 Safari/534.10
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.551.0 Safari/534.10
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/7.0.548.0 Safari/534.10
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/7.0.544.0 Safari/534.10
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.15) Gecko/20101027 Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/7.0.540.0 Safari/534.10
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/7.0.540.0 Safari/534.10
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; de-DE) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/7.0.540.0 Safari/534.10
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/7.0.540.0 Safari/534.10
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.9 (KHTML, like Gecko) Chrome/7.0.531.0 Safari/534.9
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/534.8 (KHTML, like Gecko) Chrome/7.0.521.0 Safari/534.8
|
||||||
|
Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.24 Safari/534.7
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; fr-FR) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.514.0 Safari/534.7
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.514.0 Safari/534.7
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.514.0 Safari/534.7
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/534.6 (KHTML, like Gecko) Chrome/7.0.500.0 Safari/534.6
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; tr-TR) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; ko-KR) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; fr-FR) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; cs-CZ) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; ja-JP) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
|
||||||
|
Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10_5_8; zh-cn) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
|
||||||
|
Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10_5_8; ja-jp) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_7; ja-jp) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; zh-cn) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; sv-se) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; ko-kr) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; ja-jp) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; it-it) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; fr-fr) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; es-es) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-us) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-gb) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; de-de) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; sv-SE) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; ja-JP) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; de-DE) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; hu-HU) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; de-DE) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; ru-RU) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; ja-JP) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; it-IT) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_7; en-us) AppleWebKit/534.16+ (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; fr-ch) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_5; de-de) AppleWebKit/534.15+ (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_5; ar) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4
|
||||||
|
Mozilla/5.0 (Android 2.2; Windows; U; Windows NT 6.1; en-US) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; zh-HK) AppleWebKit/533.18.1 (KHTML, like Gecko) Version/5.0.2 Safari/533.18.5
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/5.0.2 Safari/533.18.5
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; tr-TR) AppleWebKit/533.18.1 (KHTML, like Gecko) Version/5.0.2 Safari/533.18.5
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; nb-NO) AppleWebKit/533.18.1 (KHTML, like Gecko) Version/5.0.2 Safari/533.18.5
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; fr-FR) AppleWebKit/533.18.1 (KHTML, like Gecko) Version/5.0.2 Safari/533.18.5
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-TW) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/5.0.2 Safari/533.18.5
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; ru-RU) AppleWebKit/533.18.1 (KHTML, like Gecko) Version/5.0.2 Safari/533.18.5
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; zh-cn) AppleWebKit/533.18.1 (KHTML, like Gecko) Version/5.0.2 Safari/533.18.5
|
||||||
|
Mozilla/5.0 (iPod; U; CPU iPhone OS 4_3_1 like Mac OS X; zh-cn) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8G4 Safari/6533.18.5
|
||||||
|
Mozilla/5.0 (iPod; U; CPU iPhone OS 4_2_1 like Mac OS X; he-il) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5
|
||||||
|
Mozilla/5.0 (iPhone; U; fr; CPU iPhone OS 4_2_1 like Mac OS X; fr) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148a Safari/6533.18.5
|
||||||
|
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_1 like Mac OS X; zh-tw) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8G4 Safari/6533.18.5
|
||||||
|
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3 like Mac OS X; pl-pl) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8F190 Safari/6533.18.5
|
||||||
|
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3 like Mac OS X; fr-fr) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8F190 Safari/6533.18.5
|
||||||
|
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3 like Mac OS X; en-gb) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8F190 Safari/6533.18.5
|
||||||
|
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_2_1 like Mac OS X; nb-no) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148a Safari/6533.18.5
|
||||||
|
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_2_1 like Mac OS X; it-it) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148a Safari/6533.18.5
|
||||||
|
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_2_1 like Mac OS X; fr) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148a Safari/6533.18.5
|
||||||
|
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_2_1 like Mac OS X; fi-fi) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148a Safari/6533.18.5
|
||||||
|
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_2_1 like Mac OS X; fi-fi) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8C148 Safari/6533.18.5
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US) AppleWebKit/533.17.8 (KHTML, like Gecko) Version/5.0.1 Safari/533.17.8
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; th-th) AppleWebKit/533.17.8 (KHTML, like Gecko) Version/5.0.1 Safari/533.17.8
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-us) AppleWebKit/531.2+ (KHTML, like Gecko) Version/5.0 Safari/531.2+
|
||||||
|
Mozilla/5.0 (X11; U; Linux x86_64; en-ca) AppleWebKit/531.2+ (KHTML, like Gecko) Version/5.0 Safari/531.2+
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; ja-JP) AppleWebKit/533.16 (KHTML, like Gecko) Version/5.0 Safari/533.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; es-ES) AppleWebKit/533.18.1 (KHTML, like Gecko) Version/5.0 Safari/533.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/533.18.1 (KHTML, like Gecko) Version/5.0 Safari/533.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; ja-JP) AppleWebKit/533.16 (KHTML, like Gecko) Version/5.0 Safari/533.16
|
||||||
|
Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10_5_8; ja-jp) AppleWebKit/533.16 (KHTML, like Gecko) Version/5.0 Safari/533.16
|
||||||
|
Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10_4_11; fr) AppleWebKit/533.16 (KHTML, like Gecko) Version/5.0 Safari/533.16
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3; zh-cn) AppleWebKit/533.16 (KHTML, like Gecko) Version/5.0 Safari/533.16
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3; ru-ru) AppleWebKit/533.16 (KHTML, like Gecko) Version/5.0 Safari/533.16
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3; ko-kr) AppleWebKit/533.16 (KHTML, like Gecko) Version/5.0 Safari/533.16
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3; it-it) AppleWebKit/533.16 (KHTML, like Gecko) Version/5.0 Safari/533.16
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3; en-us) AppleWebKit/534.1+ (KHTML, like Gecko) Version/5.0 Safari/533.16
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3; en-au) AppleWebKit/533.16 (KHTML, like Gecko) Version/5.0 Safari/533.16
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3; el-gr) AppleWebKit/533.16 (KHTML, like Gecko) Version/5.0 Safari/533.16
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3; ca-es) AppleWebKit/533.16 (KHTML, like Gecko) Version/5.0 Safari/533.16
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; zh-tw) AppleWebKit/533.16 (KHTML, like Gecko) Version/5.0 Safari/533.16
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; ja-jp) AppleWebKit/533.16 (KHTML, like Gecko) Version/5.0 Safari/533.16
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; it-it) AppleWebKit/533.16 (KHTML, like Gecko) Version/5.0 Safari/533.16
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; fr-fr) AppleWebKit/533.16 (KHTML, like Gecko) Version/5.0 Safari/533.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.0; en-en) AppleWebKit/533.16 (KHTML, like Gecko) Version/4.1 Safari/533.16
|
||||||
|
Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10_4_11; nl-nl) AppleWebKit/533.16 (KHTML, like Gecko) Version/4.1 Safari/533.16
|
||||||
|
Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10_4_11; ja-jp) AppleWebKit/533.16 (KHTML, like Gecko) Version/4.1 Safari/533.16
|
||||||
|
Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10_4_11; de-de) AppleWebKit/533.16 (KHTML, like Gecko) Version/4.1 Safari/533.16
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_7; en-us) AppleWebKit/533.4 (KHTML, like Gecko) Version/4.1 Safari/533.4
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en) AppleWebKit/526.9 (KHTML, like Gecko) Version/4.0dp1 Safari/526.8
|
||||||
|
Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10_4_11; tr) AppleWebKit/528.4+ (KHTML, like Gecko) Version/4.0dp1 Safari/526.11.2
|
||||||
|
Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10_4_11; en) AppleWebKit/528.4+ (KHTML, like Gecko) Version/4.0dp1 Safari/526.11.2
|
||||||
|
Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10_4_11; de) AppleWebKit/528.4+ (KHTML, like Gecko) Version/4.0dp1 Safari/526.11.2
|
||||||
|
Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.5; en-US; rv:1.9.1b3pre) Gecko/20081212 Mozilla/5.0 (Windows; U; Windows NT 5.1; en) AppleWebKit/526.9 (KHTML, like Gecko) Version/4.0dp1 Safari/526.8
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_6; en-gb) AppleWebKit/528.10+ (KHTML, like Gecko) Version/4.0dp1 Safari/526.11.2
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_4; en-us) AppleWebKit/528.4+ (KHTML, like Gecko) Version/4.0dp1 Safari/526.11.2
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_4; en-gb) AppleWebKit/528.4+ (KHTML, like Gecko) Version/4.0dp1 Safari/526.11.2
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; es-ES) AppleWebKit/531.22.7 (KHTML, like Gecko) Version/4.0.5 Safari/531.22.7
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/533.18.1 (KHTML, like Gecko) Version/4.0.5 Safari/531.22.7
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/531.22.7 (KHTML, like Gecko) Version/4.0.5 Safari/531.22.7
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-gb) AppleWebKit/531.22.7 (KHTML, like Gecko) Version/4.0.5 Safari/531.22.7
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; cs-CZ) AppleWebKit/531.22.7 (KHTML, like Gecko) Version/4.0.5 Safari/531.22.7
|
||||||
|
Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10_5_8; en-us) AppleWebKit/531.22.7 (KHTML, like Gecko) Version/4.0.5 Safari/531.22.7
|
||||||
|
Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10_4_11; da-dk) AppleWebKit/531.22.7 (KHTML, like Gecko) Version/4.0.5 Safari/531.22.7
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3; ja-jp) AppleWebKit/531.22.7 (KHTML, like Gecko) Version/4.0.5 Safari/531.22.7
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3; en-us) AppleWebKit/533.4+ (KHTML, like Gecko) Version/4.0.5 Safari/531.22.7
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3; en-us) AppleWebKit/531.22.7 (KHTML, like Gecko) Version/4.0.5 Safari/531.22.7
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3; de-de) AppleWebKit/531.22.7 (KHTML, like Gecko) Version/4.0.5 Safari/531.22.7
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_2; ja-jp) AppleWebKit/531.22.7 (KHTML, like Gecko) Version/4.0.5 Safari/531.22.7
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; nl-nl) AppleWebKit/531.22.7 (KHTML, like Gecko) Version/4.0.5 Safari/531.22.7
|
||||||
|
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B5097d Safari/6531.22.7
|
||||||
|
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7
|
||||||
|
Mozilla/5.0(iPad; U; CPU iPhone OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B314 Safari/531.21.10gin_lib.cc
|
||||||
|
Mozilla/5.0(iPad; U; CPU iPhone OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B314 Safari/531.21.10
|
||||||
|
Mozilla/5.0(iPad; U; CPU iPhone OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B314 Safari/123
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; zh-TW) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; ko-KR) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/533.18.1 (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.2; en-US) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; de-DE) AppleWebKit/532+ (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10
|
||||||
|
Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10_4_11; hu-hu) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_3; en-us) AppleWebKit/531.21.11 (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_2; ru-ru) AppleWebKit/533.2+ (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_2; de-at) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10
|
||||||
|
Mozilla/5.0 (iPhone; U; CPU OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B334b Safari/531.21.10
|
||||||
|
Mozilla/5.0 (iPhone Simulator; U; CPU iPhone OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7D11 Safari/531.21.10
|
||||||
|
Mozilla/5.0 (iPad; U; CPU OS 3_2_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B500 Safari/53
|
||||||
|
Mozilla/5.0 (iPad; U; CPU OS 3_2 like Mac OS X; es-es) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B367 Safari/531.21.10
|
||||||
|
Mozilla/5.0 (iPad; U; CPU OS 3_2 like Mac OS X; es-es) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B360 Safari/531.21.10
|
||||||
|
Mozilla/5.0 (iPad; U; CPU OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B334b Safari/531.21.1021.10gin_lib.cc
|
||||||
|
Mozilla/5.0 (iPad; U; CPU iPhone OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B314
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-us) AppleWebKit/531.9 (KHTML, like Gecko) Version/4.0.3 Safari/531.9
|
||||||
|
Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10_5_8; en-us) AppleWebKit/532.0+ (KHTML, like Gecko) Version/4.0.3 Safari/531.9.2009
|
||||||
|
Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10_5_8; en-us) AppleWebKit/532.0+ (KHTML, like Gecko) Version/4.0.3 Safari/531.9
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_1; nl-nl) AppleWebKit/532.3+ (KHTML, like Gecko) Version/4.0.3 Safari/531.9
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; fi-fi) AppleWebKit/531.9 (KHTML, like Gecko) Version/4.0.3 Safari/531.9
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; en-us) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.3 Safari/531.21.10
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/532+ (KHTML, like Gecko) Version/4.0.2 Safari/530.19.1
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/530.19.2 (KHTML, like Gecko) Version/4.0.2 Safari/530.19.1
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; zh-TW) AppleWebKit/530.19.2 (KHTML, like Gecko) Version/4.0.2 Safari/530.19.1
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; pl-PL) AppleWebKit/530.19.2 (KHTML, like Gecko) Version/4.0.2 Safari/530.19.1
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; ja-JP) AppleWebKit/530.19.2 (KHTML, like Gecko) Version/4.0.2 Safari/530.19.1
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; fr-FR) AppleWebKit/530.19.2 (KHTML, like Gecko) Version/4.0.2 Safari/530.19.1
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/530.19.2 (KHTML, like Gecko) Version/4.0.2 Safari/530.19.1
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.2; de-DE) AppleWebKit/530.19.2 (KHTML, like Gecko) Version/4.0.2 Safari/530.19.1
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN) AppleWebKit/530.19.2 (KHTML, like Gecko) Version/4.0.2 Safari/530.19.1
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/530.19.2 (KHTML, like Gecko) Version/4.0.2 Safari/530.19.1
|
||||||
|
Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10_5_7; en-us) AppleWebKit/530.19.2 (KHTML, like Gecko) Version/4.0.2 Safari/530.19
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_7; en-us) AppleWebKit/530.19.2 (KHTML, like Gecko) Version/4.0.2 Safari/530.19
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_7; en-us) AppleWebKit/531.2+ (KHTML, like Gecko) Version/4.0.1 Safari/530.18
|
||||||
|
Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_7; en-us) AppleWebKit/530.19.2 (KHTML, like Gecko) Version/4.0.1 Safari/530.18
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; ru-RU) AppleWebKit/528.16 (KHTML, like Gecko) Version/4.0 Safari/528.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; ja-JP) AppleWebKit/528.16 (KHTML, like Gecko) Version/4.0 Safari/528.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; hu-HU) AppleWebKit/528.16 (KHTML, like Gecko) Version/4.0 Safari/528.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; he-IL) AppleWebKit/528.16 (KHTML, like Gecko) Version/4.0 Safari/528.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; he-IL) AppleWebKit/528+ (KHTML, like Gecko) Version/4.0 Safari/528.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; fr-FR) AppleWebKit/528.16 (KHTML, like Gecko) Version/4.0 Safari/528.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; es-es) AppleWebKit/528.16 (KHTML, like Gecko) Version/4.0 Safari/528.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/528.16 (KHTML, like Gecko) Version/4.0 Safari/528.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; en) AppleWebKit/528.16 (KHTML, like Gecko) Version/4.0 Safari/528.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.0; de-DE) AppleWebKit/528.16 (KHTML, like Gecko) Version/4.0 Safari/528.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-TW) AppleWebKit/528.16 (KHTML, like Gecko) Version/4.0 Safari/528.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN) AppleWebKit/528.16 (KHTML, like Gecko) Version/4.0 Safari/528.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; sv-SE) AppleWebKit/528.16 (KHTML, like Gecko) Version/4.0 Safari/528.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; ru-RU) AppleWebKit/528.16 (KHTML, like Gecko) Version/4.0 Safari/528.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; pt-PT) AppleWebKit/528.16 (KHTML, like Gecko) Version/4.0 Safari/528.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; pt-BR) AppleWebKit/528.16 (KHTML, like Gecko) Version/4.0 Safari/528.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; nb-NO) AppleWebKit/528.16 (KHTML, like Gecko) Version/4.0 Safari/528.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; hu-HU) AppleWebKit/528.16 (KHTML, like Gecko) Version/4.0 Safari/528.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; fr-FR) AppleWebKit/528.16 (KHTML, like Gecko) Version/4.0 Safari/528.16
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 5.1; fi-FI) AppleWebKit/528.16 (KHTML, like Gecko) Version/4.0 Safari/528.16
|
||||||
|
Opera/9.80 (Windows NT 6.0; U; en) Presto/2.8.99 Version/11.10
|
||||||
|
Opera/9.80 (Windows NT 5.1; U; zh-tw) Presto/2.8.131 Version/11.10
|
||||||
|
Opera/9.80 (X11; Linux x86_64; U; Ubuntu/10.10 (maverick); pl) Presto/2.7.62 Version/11.01
|
||||||
|
Opera/9.80 (X11; Linux i686; U; ja) Presto/2.7.62 Version/11.01
|
||||||
|
Opera/9.80 (X11; Linux i686; U; fr) Presto/2.7.62 Version/11.01
|
||||||
|
Opera/9.80 (Windows NT 6.1; U; zh-tw) Presto/2.7.62 Version/11.01
|
||||||
|
Opera/9.80 (Windows NT 6.1; U; zh-cn) Presto/2.7.62 Version/11.01
|
||||||
|
Opera/9.80 (Windows NT 6.1; U; sv) Presto/2.7.62 Version/11.01
|
||||||
|
Opera/9.80 (Windows NT 6.1; U; en-US) Presto/2.7.62 Version/11.01
|
||||||
|
Opera/9.80 (Windows NT 6.1; U; cs) Presto/2.7.62 Version/11.01
|
||||||
|
Opera/9.80 (Windows NT 6.0; U; pl) Presto/2.7.62 Version/11.01
|
||||||
|
Opera/9.80 (Windows NT 5.2; U; ru) Presto/2.7.62 Version/11.01
|
||||||
|
Opera/9.80 (Windows NT 5.1; U;) Presto/2.7.62 Version/11.01
|
||||||
|
Opera/9.80 (Windows NT 5.1; U; cs) Presto/2.7.62 Version/11.01
|
||||||
|
Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101213 Opera/9.80 (Windows NT 6.1; U; zh-tw) Presto/2.7.62 Version/11.01
|
||||||
|
Mozilla/5.0 (Windows NT 6.1; U; nl; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 Opera 11.01
|
||||||
|
Mozilla/5.0 (Windows NT 6.1; U; de; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 Opera 11.01
|
||||||
|
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; de) Opera 11.01
|
||||||
|
Opera/9.80 (X11; Linux x86_64; U; pl) Presto/2.7.62 Version/11.00
|
||||||
|
Opera/9.80 (X11; Linux i686; U; it) Presto/2.7.62 Version/11.00
|
||||||
|
Opera/9.80 (Windows NT 6.1; U; zh-cn) Presto/2.6.37 Version/11.00
|
||||||
|
Opera/9.80 (Windows NT 6.1; U; pl) Presto/2.7.62 Version/11.00
|
||||||
|
Opera/9.80 (Windows NT 6.1; U; ko) Presto/2.7.62 Version/11.00
|
||||||
|
Opera/9.80 (Windows NT 6.1; U; fi) Presto/2.7.62 Version/11.00
|
||||||
|
Opera/9.80 (Windows NT 6.1; U; en-GB) Presto/2.7.62 Version/11.00
|
||||||
|
Opera/9.80 (Windows NT 6.1 x64; U; en) Presto/2.7.62 Version/11.00
|
||||||
|
Opera/9.80 (Windows NT 6.0; U; en) Presto/2.7.39 Version/11.00
|
||||||
|
Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.7.39 Version/11.00
|
||||||
|
Opera/9.80 (Windows NT 5.1; U; MRA 5.5 (build 02842); ru) Presto/2.7.62 Version/11.00
|
||||||
|
Opera/9.80 (Windows NT 5.1; U; it) Presto/2.7.62 Version/11.00
|
||||||
|
Mozilla/5.0 (Windows NT 6.0; U; ja; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 Opera 11.00
|
||||||
|
Mozilla/5.0 (Windows NT 5.1; U; pl; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 Opera 11.00
|
||||||
|
Mozilla/5.0 (Windows NT 5.1; U; de; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 Opera 11.00
|
||||||
|
Mozilla/4.0 (compatible; MSIE 8.0; X11; Linux x86_64; pl) Opera 11.00
|
||||||
|
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; fr) Opera 11.00
|
||||||
|
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; ja) Opera 11.00
|
||||||
|
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; en) Opera 11.00
|
||||||
|
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; pl) Opera 11.00
|
||||||
|
Opera/9.80 (Windows NT 6.1; U; pl) Presto/2.6.31 Version/10.70
|
||||||
|
Mozilla/5.0 (Windows NT 5.2; U; ru; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 Opera 10.70
|
||||||
|
Mozilla/5.0 (Windows NT 5.1; U; zh-cn; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 Opera 10.70
|
||||||
|
Opera/9.80 (Windows NT 5.2; U; zh-cn) Presto/2.6.30 Version/10.63
|
||||||
|
Opera/9.80 (Windows NT 5.2; U; en) Presto/2.6.30 Version/10.63
|
||||||
|
Opera/9.80 (Windows NT 5.1; U; MRA 5.6 (build 03278); ru) Presto/2.6.30 Version/10.63
|
||||||
|
Opera/9.80 (Windows NT 5.1; U; pl) Presto/2.6.30 Version/10.62
|
||||||
|
Mozilla/5.0 (X11; Linux x86_64; U; de; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 Opera 10.62
|
||||||
|
Mozilla/4.0 (compatible; MSIE 8.0; X11; Linux x86_64; de) Opera 10.62
|
||||||
|
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; en) Opera 10.62
|
||||||
|
Opera/9.80 (X11; Linux i686; U; pl) Presto/2.6.30 Version/10.61
|
||||||
|
Opera/9.80 (X11; Linux i686; U; es-ES) Presto/2.6.30 Version/10.61
|
||||||
|
Opera/9.80 (Windows NT 6.1; U; zh-cn) Presto/2.6.30 Version/10.61
|
||||||
|
Opera/9.80 (Windows NT 6.1; U; en) Presto/2.6.30 Version/10.61
|
||||||
|
Opera/9.80 (Windows NT 6.0; U; it) Presto/2.6.30 Version/10.61
|
||||||
|
Opera/9.80 (Windows NT 5.2; U; ru) Presto/2.6.30 Version/10.61
|
||||||
|
Opera/9.80 (Windows 98; U; de) Presto/2.6.30 Version/10.61
|
||||||
|
Opera/9.80 (Macintosh; Intel Mac OS X; U; nl) Presto/2.6.30 Version/10.61
|
||||||
|
Opera/9.80 (X11; Linux i686; U; en) Presto/2.5.27 Version/10.60
|
||||||
|
Opera/9.80 (Windows NT 6.0; U; nl) Presto/2.6.30 Version/10.60
|
||||||
|
Opera/10.60 (Windows NT 5.1; U; zh-cn) Presto/2.6.30 Version/10.60
|
||||||
|
Opera/10.60 (Windows NT 5.1; U; en-US) Presto/2.6.30 Version/10.60
|
||||||
|
Opera/9.80 (X11; Linux i686; U; it) Presto/2.5.24 Version/10.54
|
||||||
|
Opera/9.80 (X11; Linux i686; U; en-GB) Presto/2.5.24 Version/10.53
|
||||||
|
Mozilla/5.0 (Windows NT 5.1; U; zh-cn; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 Opera 10.53
|
||||||
|
Mozilla/5.0 (Windows NT 5.1; U; Firefox/5.0; en; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 Opera 10.53
|
||||||
|
Mozilla/5.0 (Windows NT 5.1; U; Firefox/4.5; en; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 Opera 10.53
|
||||||
|
Mozilla/5.0 (Windows NT 5.1; U; Firefox/3.5; en; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 Opera 10.53
|
||||||
|
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; ko) Opera 10.53
|
||||||
|
Opera/9.80 (Windows NT 6.1; U; fr) Presto/2.5.24 Version/10.52
|
||||||
|
Opera/9.80 (Windows NT 6.1; U; en) Presto/2.5.22 Version/10.51
|
||||||
|
Opera/9.80 (Windows NT 6.0; U; cs) Presto/2.5.22 Version/10.51
|
||||||
|
Opera/9.80 (Windows NT 5.2; U; ru) Presto/2.5.22 Version/10.51
|
||||||
|
Opera/9.80 (Linux i686; U; en) Presto/2.5.22 Version/10.51
|
||||||
|
Mozilla/5.0 (Windows NT 6.1; U; en-GB; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 Opera 10.51
|
||||||
|
Mozilla/5.0 (Linux i686; U; en; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6 Opera 10.51
|
||||||
|
Mozilla/4.0 (compatible; MSIE 8.0; Linux i686; en) Opera 10.51
|
||||||
|
Opera/9.80 (Windows NT 6.1; U; zh-tw) Presto/2.5.22 Version/10.50
|
||||||
|
Opera/9.80 (Windows NT 6.1; U; zh-cn) Presto/2.5.22 Version/10.50
|
||||||
|
Opera/9.80 (Windows NT 6.1; U; sk) Presto/2.6.22 Version/10.50
|
||||||
|
Opera/9.80 (Windows NT 6.1; U; ja) Presto/2.5.22 Version/10.50
|
||||||
|
Opera/9.80 (Windows NT 6.0; U; zh-cn) Presto/2.5.22 Version/10.50
|
||||||
|
Opera/9.80 (Windows NT 5.1; U; sk) Presto/2.5.22 Version/10.50
|
||||||
|
Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.5.22 Version/10.50
|
||||||
|
Opera/10.50 (Windows NT 6.1; U; en-GB) Presto/2.2.2
|
||||||
|
Opera/9.80 (S60; SymbOS; Opera Tablet/9174; U; en) Presto/2.7.81 Version/10.5
|
||||||
|
Opera/9.80 (X11; U; Linux i686; en-US; rv:1.9.2.3) Presto/2.2.15 Version/10.10
|
||||||
|
Opera/9.80 (X11; Linux x86_64; U; it) Presto/2.2.15 Version/10.10
|
||||||
|
Opera/9.80 (Windows NT 6.1; U; de) Presto/2.2.15 Version/10.10
|
||||||
|
Opera/9.80 (Windows NT 6.0; U; Gecko/20100115; pl) Presto/2.2.15 Version/10.10
|
||||||
|
Opera/9.80 (Windows NT 6.0; U; en) Presto/2.2.15 Version/10.10
|
||||||
|
Opera/9.80 (Windows NT 5.1; U; de) Presto/2.2.15 Version/10.10
|
||||||
|
Opera/9.80 (Windows NT 5.1; U; cs) Presto/2.2.15 Version/10.10
|
||||||
|
Mozilla/5.0 (Windows NT 6.0; U; tr; rv:1.8.1) Gecko/20061208 Firefox/2.0.0 Opera 10.10
|
||||||
|
Mozilla/4.0 (compatible; MSIE 6.0; X11; Linux i686; de) Opera 10.10
|
||||||
|
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 6.0; tr) Opera 10.10
|
||||||
|
Opera/9.80 (X11; Linux x86_64; U; en-GB) Presto/2.2.15 Version/10.01
|
||||||
|
Opera/9.80 (X11; Linux x86_64; U; en) Presto/2.2.15 Version/10.00
|
||||||
|
Opera/9.80 (X11; Linux x86_64; U; de) Presto/2.2.15 Version/10.00
|
||||||
|
Opera/9.80 (X11; Linux i686; U; ru) Presto/2.2.15 Version/10.00
|
||||||
|
Opera/9.80 (X11; Linux i686; U; pt-BR) Presto/2.2.15 Version/10.00
|
||||||
|
Opera/9.80 (X11; Linux i686; U; pl) Presto/2.2.15 Version/10.00
|
||||||
|
Opera/9.80 (X11; Linux i686; U; nb) Presto/2.2.15 Version/10.00
|
||||||
|
Opera/9.80 (X11; Linux i686; U; en-GB) Presto/2.2.15 Version/10.00
|
||||||
|
Opera/9.80 (X11; Linux i686; U; en) Presto/2.2.15 Version/10.00
|
||||||
|
Opera/9.80 (X11; Linux i686; U; Debian; pl) Presto/2.2.15 Version/10.00
|
||||||
|
Opera/9.80 (X11; Linux i686; U; de) Presto/2.2.15 Version/10.00
|
||||||
|
Opera/9.80 (Windows NT 6.1; U; zh-cn) Presto/2.2.15 Version/10.00
|
||||||
|
Opera/9.80 (Windows NT 6.1; U; fi) Presto/2.2.15 Version/10.00
|
||||||
|
Opera/9.80 (Windows NT 6.1; U; en) Presto/2.2.15 Version/10.00
|
||||||
|
Opera/9.80 (Windows NT 6.1; U; de) Presto/2.2.15 Version/10.00
|
||||||
|
Opera/9.80 (Windows NT 6.1; U; cs) Presto/2.2.15 Version/10.00
|
||||||
|
Opera/9.80 (Windows NT 6.0; U; en) Presto/2.2.15 Version/10.00
|
||||||
|
Opera/9.80 (Windows NT 6.0; U; de) Presto/2.2.15 Version/10.00
|
||||||
|
Opera/9.80 (Windows NT 5.2; U; en) Presto/2.2.15 Version/10.00
|
||||||
|
Opera/9.80 (Windows NT 5.1; U; zh-cn) Presto/2.2.15 Version/10.00
|
||||||
|
Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.2.15 Version/10.00
|
||||||
0
soulbook/database/__init__.py
Normal file
0
soulbook/database/__init__.py
Normal file
1
soulbook/database/mongodb/__init__.py
Normal file
1
soulbook/database/mongodb/__init__.py
Normal file
@ -0,0 +1 @@
|
|||||||
|
from .motorbase import MotorBase, MotorBaseOld, PyMongoDb
|
||||||
140
soulbook/database/mongodb/motorbase.py
Normal file
140
soulbook/database/mongodb/motorbase.py
Normal file
@ -0,0 +1,140 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
import asyncio
|
||||||
|
|
||||||
|
from pymongo import MongoClient
|
||||||
|
from motor.motor_asyncio import AsyncIOMotorClient
|
||||||
|
|
||||||
|
from soulbook.config import CONFIG
|
||||||
|
from soulbook.utils.tools import singleton
|
||||||
|
|
||||||
|
|
||||||
|
@singleton
|
||||||
|
class MotorBase:
|
||||||
|
"""
|
||||||
|
更改mongodb连接方式 单例模式下支持多库操作
|
||||||
|
About motor's doc: https://github.com/mongodb/motor
|
||||||
|
"""
|
||||||
|
_db = {}
|
||||||
|
_collection = {}
|
||||||
|
MONGODB = CONFIG.MONGODB
|
||||||
|
|
||||||
|
def __init__(self, loop=None):
|
||||||
|
self.motor_uri = ''
|
||||||
|
self.loop = loop or asyncio.get_event_loop()
|
||||||
|
|
||||||
|
def client(self, db):
|
||||||
|
# motor
|
||||||
|
self.motor_uri = 'mongodb://{account}{host}:{port}/{database}'.format(
|
||||||
|
account='{username}:{password}@'.format(
|
||||||
|
username=self.MONGODB['MONGO_USERNAME'],
|
||||||
|
password=self.MONGODB['MONGO_PASSWORD']) if self.MONGODB['MONGO_USERNAME'] else '',
|
||||||
|
host=self.MONGODB['MONGO_HOST'] if self.MONGODB['MONGO_HOST'] else 'localhost',
|
||||||
|
port=self.MONGODB['MONGO_PORT'] if self.MONGODB['MONGO_PORT'] else 27017,
|
||||||
|
database=db)
|
||||||
|
return AsyncIOMotorClient(self.motor_uri, io_loop=self.loop)
|
||||||
|
|
||||||
|
def get_db(self, db=MONGODB['DATABASE']):
|
||||||
|
"""
|
||||||
|
Get a db instance
|
||||||
|
:param db: database name
|
||||||
|
:return: the motor db instance
|
||||||
|
"""
|
||||||
|
if db not in self._db:
|
||||||
|
self._db[db] = self.client(db)[db]
|
||||||
|
|
||||||
|
return self._db[db]
|
||||||
|
|
||||||
|
def get_collection(self, db_name, collection):
|
||||||
|
"""
|
||||||
|
Get a collection instance
|
||||||
|
:param db_name: database name
|
||||||
|
:param collection: collection name
|
||||||
|
:return: the motor collection instance
|
||||||
|
"""
|
||||||
|
collection_key = db_name + collection
|
||||||
|
if collection_key not in self._collection:
|
||||||
|
self._collection[collection_key] = self.get_db(db_name)[collection]
|
||||||
|
|
||||||
|
return self._collection[collection_key]
|
||||||
|
|
||||||
|
|
||||||
|
class MotorBaseOld:
|
||||||
|
"""
|
||||||
|
use motor to connect mongodb
|
||||||
|
2017-09-21 deleted
|
||||||
|
"""
|
||||||
|
_db = None
|
||||||
|
MONGODB = CONFIG.MONGODB
|
||||||
|
|
||||||
|
def client(self, db):
|
||||||
|
# motor
|
||||||
|
self.motor_uri = 'mongodb://{account}{host}:{port}/{database}'.format(
|
||||||
|
account='{username}:{password}@'.format(
|
||||||
|
username=self.MONGODB['MONGO_USERNAME'],
|
||||||
|
password=self.MONGODB['MONGO_PASSWORD']) if self.MONGODB['MONGO_USERNAME'] else '',
|
||||||
|
host=self.MONGODB['MONGO_HOST'] if self.MONGODB['MONGO_HOST'] else 'localhost',
|
||||||
|
port=self.MONGODB['MONGO_PORT'] if self.MONGODB['MONGO_PORT'] else 27017,
|
||||||
|
database=db)
|
||||||
|
return AsyncIOMotorClient(self.motor_uri)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def db(self):
|
||||||
|
if self._db is None:
|
||||||
|
self._db = self.client(self.MONGODB['DATABASE'])[self.MONGODB['DATABASE']]
|
||||||
|
|
||||||
|
return self._db
|
||||||
|
|
||||||
|
|
||||||
|
@singleton
|
||||||
|
class PyMongoDb:
|
||||||
|
_db = None
|
||||||
|
MONGODB = {
|
||||||
|
'MONGO_HOST': '127.0.0.1',
|
||||||
|
'MONGO_PORT': '',
|
||||||
|
'MONGO_USERNAME': '',
|
||||||
|
'MONGO_PASSWORD': '',
|
||||||
|
'DATABASE': 'soulbook'
|
||||||
|
}
|
||||||
|
|
||||||
|
def client(self):
|
||||||
|
# motor
|
||||||
|
self.mongo_uri = 'mongodb://{account}{host}:{port}/'.format(
|
||||||
|
account='{username}:{password}@'.format(
|
||||||
|
username=self.MONGODB['MONGO_USERNAME'],
|
||||||
|
password=self.MONGODB['MONGO_PASSWORD']) if self.MONGODB['MONGO_USERNAME'] else '',
|
||||||
|
host=self.MONGODB['MONGO_HOST'] if self.MONGODB['MONGO_HOST'] else 'localhost',
|
||||||
|
port=self.MONGODB['MONGO_PORT'] if self.MONGODB['MONGO_PORT'] else 27017)
|
||||||
|
return MongoClient(self.mongo_uri)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def db(self):
|
||||||
|
if self._db is None:
|
||||||
|
self._db = self.client()[self.MONGODB['DATABASE']]
|
||||||
|
|
||||||
|
return self._db
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
def async_callback(func, **kwargs):
|
||||||
|
"""
|
||||||
|
Call the asynchronous function
|
||||||
|
:param func: a async function
|
||||||
|
:param kwargs: params
|
||||||
|
:return: result
|
||||||
|
"""
|
||||||
|
loop = asyncio.get_event_loop()
|
||||||
|
task = asyncio.ensure_future(func(**kwargs))
|
||||||
|
loop.run_until_complete(task)
|
||||||
|
return task.result()
|
||||||
|
|
||||||
|
|
||||||
|
motor_base = MotorBase()
|
||||||
|
motor_db = motor_base.get_db()
|
||||||
|
|
||||||
|
|
||||||
|
async def insert(data):
|
||||||
|
print(data)
|
||||||
|
await motor_db.test.save(data)
|
||||||
|
|
||||||
|
|
||||||
|
async_callback(insert, data={'hi': 'soulbook'})
|
||||||
1
soulbook/database/redis/__init__.py
Normal file
1
soulbook/database/redis/__init__.py
Normal file
@ -0,0 +1 @@
|
|||||||
|
from .redisbase import *
|
||||||
24
soulbook/database/redis/redisbase.py
Normal file
24
soulbook/database/redis/redisbase.py
Normal file
@ -0,0 +1,24 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
import asyncio_redis
|
||||||
|
|
||||||
|
from soulbook.config import CONFIG
|
||||||
|
|
||||||
|
|
||||||
|
# Token from https://github.com/subyraman/sanic_session
|
||||||
|
class RedisSession:
|
||||||
|
"""
|
||||||
|
A simple wrapper class that allows you to share a connection
|
||||||
|
pool across your application.
|
||||||
|
"""
|
||||||
|
_pool = None
|
||||||
|
|
||||||
|
async def get_redis_pool(self):
|
||||||
|
if not self._pool:
|
||||||
|
REDIS_DICT = CONFIG.REDIS_DICT
|
||||||
|
self._pool = await asyncio_redis.Pool.create(
|
||||||
|
host=str(REDIS_DICT.get('REDIS_ENDPOINT', "localhost")), port=int(REDIS_DICT.get('REDIS_PORT', 6379)),
|
||||||
|
poolsize=int(REDIS_DICT.get('POOLSIZE', 10)), password=REDIS_DICT.get('REDIS_PASSWORD', None),
|
||||||
|
db=REDIS_DICT.get('SESSION_DB', None)
|
||||||
|
)
|
||||||
|
|
||||||
|
return self._pool
|
||||||
1
soulbook/fetcher/__init__.py
Normal file
1
soulbook/fetcher/__init__.py
Normal file
@ -0,0 +1 @@
|
|||||||
|
from .response_base import UniResponse, ResponseReply, ResponseCode, ResponseField
|
||||||
294
soulbook/fetcher/cache.py
Normal file
294
soulbook/fetcher/cache.py
Normal file
@ -0,0 +1,294 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""
|
||||||
|
Created by howie.hu.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import re
|
||||||
|
import aiohttp
|
||||||
|
import async_timeout
|
||||||
|
|
||||||
|
from bs4 import BeautifulSoup
|
||||||
|
from aiocache.serializers import PickleSerializer, JsonSerializer
|
||||||
|
|
||||||
|
from urllib.parse import urlparse, parse_qs, urljoin
|
||||||
|
|
||||||
|
from soulbook.database.mongodb import MotorBase
|
||||||
|
from soulbook.fetcher.decorators import cached
|
||||||
|
from soulbook.fetcher.function import target_fetch, get_time, get_html_by_requests, \
|
||||||
|
get_random_user_agent
|
||||||
|
from soulbook.fetcher.extract_novels import extract_pre_next_chapter
|
||||||
|
from soulbook.config import RULES, LATEST_RULES, LOGGER
|
||||||
|
|
||||||
|
list_class_content = ("content", "content_read", "txt")
|
||||||
|
list_id_content = ("chaptercontent", "content", "txt")
|
||||||
|
|
||||||
|
|
||||||
|
@cached(ttl=300, key_from_attr='url', serializer=PickleSerializer(), namespace="main")
|
||||||
|
async def cache_owllook_novels_content(url, chapter_url, netloc):
|
||||||
|
headers = {
|
||||||
|
'user-agent': await get_random_user_agent()
|
||||||
|
}
|
||||||
|
html = await target_fetch(headers=headers, url=url)
|
||||||
|
if not html:
|
||||||
|
html = get_html_by_requests(url=url, headers=headers)
|
||||||
|
if html:
|
||||||
|
soup = BeautifulSoup(html, 'html5lib')
|
||||||
|
# 章节内容
|
||||||
|
content = None
|
||||||
|
if RULES.get(netloc):
|
||||||
|
selector = RULES[netloc].content_selector
|
||||||
|
if selector.get('id', None):
|
||||||
|
content = soup.find_all(id_=selector['id'])
|
||||||
|
elif selector.get('class', None):
|
||||||
|
content = soup.find_all(class_=selector['class'])
|
||||||
|
else:
|
||||||
|
content = soup.find_all(selector.get('tag'))
|
||||||
|
if not content:
|
||||||
|
# print("novels_content: body")
|
||||||
|
# content = soup.find_all("body")
|
||||||
|
|
||||||
|
# for i in list_class_content:
|
||||||
|
# content = soup.find_all(class_=i)
|
||||||
|
# if content:
|
||||||
|
# break
|
||||||
|
# else:
|
||||||
|
# for i in list_id_content:
|
||||||
|
# content = soup.find_all(id=i)
|
||||||
|
# if content:
|
||||||
|
# break
|
||||||
|
|
||||||
|
for i in list_id_content:
|
||||||
|
content = soup.find_all(id=i)
|
||||||
|
if content:
|
||||||
|
break
|
||||||
|
else:
|
||||||
|
for i in list_class_content:
|
||||||
|
content = soup.find_all(class_=i)
|
||||||
|
if content:
|
||||||
|
break
|
||||||
|
|
||||||
|
if not content:
|
||||||
|
# 直接提取整个版块块
|
||||||
|
content = soup.find_all("body")
|
||||||
|
if content:
|
||||||
|
# 提取出真正的章节标题
|
||||||
|
title_reg = r'(第?\s*[一二两三四五六七八九十○零百千万亿0-9１２３４５６７８９０]{1,6}\s*[章回卷节折篇幕集]\s*.*?)[_,-]'
|
||||||
|
title = soup.title.string
|
||||||
|
extract_title = re.findall(title_reg, title, re.I)
|
||||||
|
if extract_title:
|
||||||
|
title = extract_title[0]
|
||||||
|
else:
|
||||||
|
try:
|
||||||
|
title = soup.select('h1')[0].get_text()
|
||||||
|
except Exception as e:
|
||||||
|
print(e)
|
||||||
|
if not title:
|
||||||
|
title = soup.title.string
|
||||||
|
# if "_" in title:
|
||||||
|
# title = title.split('_')[0]
|
||||||
|
# elif "-" in title:
|
||||||
|
# title = title.split('-')[0]
|
||||||
|
next_chapter = extract_pre_next_chapter(url=url, chapter_url=chapter_url,
|
||||||
|
html=str(soup))
|
||||||
|
content = [str(i) for i in content]
|
||||||
|
data = {
|
||||||
|
'content': str(''.join(content)),
|
||||||
|
'next_chapter': next_chapter,
|
||||||
|
'title': title
|
||||||
|
}
|
||||||
|
else:
|
||||||
|
data = None
|
||||||
|
return data
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
ListIdTab = ("box_con", 'box_chap', 'layout layout-col1', 'listmain', "content", "list-chapterAll")
|
||||||
|
ListClassTab = (
|
||||||
|
"box_con", 'box_chap', 'layout layout-col1', 'listmain', 'book chapterlist', 'book_list',
|
||||||
|
'TabCss', 'inner', 'mulu', 'bg')
|
||||||
|
|
||||||
|
|
||||||
|
@cached(ttl=300, key_from_attr='url', serializer=PickleSerializer(), namespace="main")
|
||||||
|
async def cache_owllook_novels_chapter(url, netloc):
|
||||||
|
headers = {
|
||||||
|
'user-agent': await get_random_user_agent()
|
||||||
|
}
|
||||||
|
html = await target_fetch(headers=headers, url=url)
|
||||||
|
if not html:
|
||||||
|
html = get_html_by_requests(url=url, headers=headers)
|
||||||
|
if html:
|
||||||
|
soup = BeautifulSoup(html, 'html5lib')
|
||||||
|
# 提取出的内容
|
||||||
|
content = None
|
||||||
|
# 存在解析规则
|
||||||
|
if RULES.get(netloc):
|
||||||
|
# 匹配解析规则
|
||||||
|
selector = RULES[netloc].chapter_selector
|
||||||
|
if selector.get('id', None):
|
||||||
|
content = soup.find_all(id=selector['id'])
|
||||||
|
elif selector.get('class', None):
|
||||||
|
content = soup.find_all(class_=selector['class'])
|
||||||
|
else:
|
||||||
|
content = soup.find_all(selector.get('tag'))
|
||||||
|
else:
|
||||||
|
data = str(html)
|
||||||
|
for i in ListIdTab:
|
||||||
|
if 'id="%s"' % i in data:
|
||||||
|
content = soup.find_all(id=i)
|
||||||
|
break
|
||||||
|
for i in ListClassTab:
|
||||||
|
if 'class="%s"' % i in data:
|
||||||
|
content = soup.find_all(class_=i)
|
||||||
|
break
|
||||||
|
if not content:
|
||||||
|
# 直接提取整个版块块
|
||||||
|
content = soup.find_all("body")
|
||||||
|
if content:
|
||||||
|
# 防止章节被display:none
|
||||||
|
return str(content).replace('style', '') if content else None
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
@cached(ttl=10800, key_from_attr='search_ranking', serializer=JsonSerializer(), namespace="ranking")
|
||||||
|
async def cache_owllook_search_ranking():
|
||||||
|
motor_db = MotorBase().get_db()
|
||||||
|
keyword_cursor = motor_db.search_records.find(
|
||||||
|
{'count': {'$gte': 50}},
|
||||||
|
{'keyword': 1, 'count': 1, '_id': 0}
|
||||||
|
).sort('count', -1).limit(35)
|
||||||
|
result = []
|
||||||
|
index = 1
|
||||||
|
async for document in keyword_cursor:
|
||||||
|
result.append({'keyword': document['keyword'], 'count': document['count'], 'index': index})
|
||||||
|
index += 1
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
@cached(ttl=3600, key_from_attr='search_ranking', serializer=JsonSerializer(), namespace="ranking")
|
||||||
|
async def cache_others_search_ranking(spider='qidian', novel_type='全部类别'):
|
||||||
|
motor_db = MotorBase().get_db()
|
||||||
|
item_data = await motor_db.novels_ranking.find_one({'spider': spider, 'type': novel_type},
|
||||||
|
{'data': 1, '_id': 0})
|
||||||
|
return item_data
|
||||||
|
|
||||||
|
|
||||||
|
async def get_the_latest_chapter(chapter_url, timeout=15):
|
||||||
|
try:
|
||||||
|
with async_timeout.timeout(timeout):
|
||||||
|
url = parse_qs(urlparse(chapter_url).query).get('url', '')
|
||||||
|
novels_name = parse_qs(urlparse(chapter_url).query).get('novels_name', '')
|
||||||
|
data = None
|
||||||
|
if url and novels_name:
|
||||||
|
url = url[0]
|
||||||
|
novels_name = novels_name[0]
|
||||||
|
netloc = urlparse(url).netloc
|
||||||
|
if netloc in LATEST_RULES.keys():
|
||||||
|
headers = {
|
||||||
|
'user-agent': await get_random_user_agent()
|
||||||
|
}
|
||||||
|
try:
|
||||||
|
html = await target_fetch(url=url, headers=headers, timeout=timeout)
|
||||||
|
if html is None:
|
||||||
|
html = get_html_by_requests(url=url, headers=headers, timeout=timeout)
|
||||||
|
except TypeError:
|
||||||
|
html = get_html_by_requests(url=url, headers=headers, timeout=timeout)
|
||||||
|
except Exception as e:
|
||||||
|
LOGGER.exception(e)
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
soup = BeautifulSoup(html, 'html5lib')
|
||||||
|
except Exception as e:
|
||||||
|
LOGGER.exception(e)
|
||||||
|
return None
|
||||||
|
latest_chapter_name, latest_chapter_url = None, None
|
||||||
|
if LATEST_RULES[netloc].plan:
|
||||||
|
meta_value = LATEST_RULES[netloc].meta_value
|
||||||
|
latest_chapter_name = soup.select(
|
||||||
|
'meta[property="{0}"]'.format(
|
||||||
|
meta_value["latest_chapter_name"])) or soup.select(
|
||||||
|
'meta[name="{0}"]'.format(meta_value["latest_chapter_name"]))
|
||||||
|
|
||||||
|
latest_chapter_name = latest_chapter_name[0].get('content',
|
||||||
|
None) if latest_chapter_name else None
|
||||||
|
latest_chapter_url = soup.select(
|
||||||
|
'meta[property="{0}"]'.format(
|
||||||
|
meta_value["latest_chapter_url"])) or soup.select(
|
||||||
|
'meta[name="{0}"]'.format(meta_value["latest_chapter_url"]))
|
||||||
|
latest_chapter_url = urljoin(chapter_url,
|
||||||
|
latest_chapter_url[0].get('content',
|
||||||
|
None)) if latest_chapter_url else None
|
||||||
|
else:
|
||||||
|
selector = LATEST_RULES[netloc].selector
|
||||||
|
content_url = selector.get('content_url')
|
||||||
|
if selector.get('id', None):
|
||||||
|
latest_chapter_soup = soup.find_all(id=selector['id'])
|
||||||
|
elif selector.get('class', None):
|
||||||
|
latest_chapter_soup = soup.find_all(class_=selector['class'])
|
||||||
|
else:
|
||||||
|
latest_chapter_soup = soup.select(selector.get('tag'))
|
||||||
|
if latest_chapter_soup:
|
||||||
|
if content_url == '1':
|
||||||
|
# TODO
|
||||||
|
pass
|
||||||
|
elif content_url == '0':
|
||||||
|
# TODO
|
||||||
|
pass
|
||||||
|
else:
|
||||||
|
latest_chapter_url = content_url + latest_chapter_soup[0].get(
|
||||||
|
'href', None)
|
||||||
|
latest_chapter_name = latest_chapter_soup[0].get('title', None)
|
||||||
|
if latest_chapter_name and latest_chapter_url:
|
||||||
|
time_current = get_time()
|
||||||
|
# print(latest_chapter_url)
|
||||||
|
data = {
|
||||||
|
"latest_chapter_name": latest_chapter_name,
|
||||||
|
"latest_chapter_url": latest_chapter_url,
|
||||||
|
"owllook_chapter_url": chapter_url,
|
||||||
|
"owllook_content_url": "/owllook_content?url={latest_chapter_url}&name={name}&chapter_url={chapter_url}&novels_name={novels_name}".format(
|
||||||
|
latest_chapter_url=latest_chapter_url,
|
||||||
|
name=latest_chapter_name,
|
||||||
|
chapter_url=url,
|
||||||
|
novels_name=novels_name,
|
||||||
|
),
|
||||||
|
}
|
||||||
|
# 存储最新章节
|
||||||
|
motor_db = MotorBase().get_db()
|
||||||
|
await motor_db.latest_chapter.update_one(
|
||||||
|
{"novels_name": novels_name, 'owllook_chapter_url': chapter_url},
|
||||||
|
{'$set': {'data': data, "finished_at": time_current}}, upsert=True)
|
||||||
|
return data
|
||||||
|
except Exception as e:
|
||||||
|
LOGGER.exception(e)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
async def update_all_books(loop, timeout=15):
|
||||||
|
try:
|
||||||
|
motor_db = MotorBase().get_db()
|
||||||
|
# 获取所有书架链接游标
|
||||||
|
books_url_cursor = motor_db.user_message.find({}, {'books_url.book_url': 1, '_id': 0})
|
||||||
|
book_urls = []
|
||||||
|
already_urls = set()
|
||||||
|
async for document in books_url_cursor:
|
||||||
|
if document:
|
||||||
|
books_url = document['books_url']
|
||||||
|
|
||||||
|
for book_url in books_url:
|
||||||
|
chapter_url = book_url['book_url']
|
||||||
|
if chapter_url not in already_urls:
|
||||||
|
try:
|
||||||
|
await get_the_latest_chapter(chapter_url, timeout)
|
||||||
|
except Exception as e:
|
||||||
|
LOGGER.exception(e)
|
||||||
|
already_urls.add(chapter_url)
|
||||||
|
# 一组书架链接列表数据
|
||||||
|
# book_urls += [book_url['book_url'] for book_url in books_url]
|
||||||
|
# url_tasks = [get_the_latest_chapter(each_url, loop) for each_url in set(book_urls)]
|
||||||
|
# tasks = [asyncio.ensure_future(i) for i in url_tasks]
|
||||||
|
# try:
|
||||||
|
# await asyncio.gather(*tasks)
|
||||||
|
# except asyncio.TimeoutError as e:
|
||||||
|
# pass
|
||||||
|
except Exception as e:
|
||||||
|
LOGGER.exception(e)
|
||||||
|
return False
|
||||||
153
soulbook/fetcher/decorators.py
Normal file
153
soulbook/fetcher/decorators.py
Normal file
@ -0,0 +1,153 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""
|
||||||
|
Created by howie.hu at 2018/5/28.
|
||||||
|
"""
|
||||||
|
from functools import wraps
|
||||||
|
|
||||||
|
from sanic import response
|
||||||
|
|
||||||
|
from aiocache.log import logger
|
||||||
|
from aiocache.utils import get_args_dict, get_cache
|
||||||
|
from sanic.request import Request
|
||||||
|
|
||||||
|
try:
|
||||||
|
from ujson import loads as json_loads
|
||||||
|
from ujson import dumps as json_dumps
|
||||||
|
except:
|
||||||
|
from json import loads as json_loads
|
||||||
|
from json import dumps as json_dumps
|
||||||
|
|
||||||
|
from soulbook.fetcher import UniResponse
|
||||||
|
from soulbook.config import CONFIG, LOGGER
|
||||||
|
|
||||||
|
|
||||||
|
def authenticator(key):
|
||||||
|
"""
|
||||||
|
|
||||||
|
:param keys: 验证方式 Owllook-Api-Key : Magic Key, Authorization : Token
|
||||||
|
:return: 返回值
|
||||||
|
"""
|
||||||
|
|
||||||
|
def wrapper(func):
|
||||||
|
@wraps(func)
|
||||||
|
async def authenticate(request, *args, **kwargs):
|
||||||
|
value = request.headers.get(key, None)
|
||||||
|
if value and CONFIG.AUTH[key] == value:
|
||||||
|
response = await func(request, *args, **kwargs)
|
||||||
|
return response
|
||||||
|
else:
|
||||||
|
return response_handle(request, UniResponse.NOT_AUTHORIZED, status=401)
|
||||||
|
|
||||||
|
return authenticate
|
||||||
|
|
||||||
|
return wrapper
|
||||||
|
|
||||||
|
|
||||||
|
def auth_params(*keys):
|
||||||
|
"""
|
||||||
|
|
||||||
|
:param keys: 判断必须要有的参数
|
||||||
|
:return: 返回值
|
||||||
|
"""
|
||||||
|
|
||||||
|
def wrapper(func):
|
||||||
|
@wraps(func)
|
||||||
|
async def auth_param(request, *args, **kwargs):
|
||||||
|
request_params = {}
|
||||||
|
# POST request
|
||||||
|
if request.method == 'POST' or request.method == 'DELETE':
|
||||||
|
try:
|
||||||
|
post_data = json_loads(str(request.body, encoding='utf-8'))
|
||||||
|
except Exception as e:
|
||||||
|
LOGGER.exception(e)
|
||||||
|
return response_handle(request, UniResponse.PARAM_PARSE_ERR, status=400)
|
||||||
|
else:
|
||||||
|
request_params.update(post_data)
|
||||||
|
params = [key for key, value in post_data.items() if value]
|
||||||
|
elif request.method == 'GET':
|
||||||
|
request_params.update(request.args)
|
||||||
|
params = [key for key, value in request.args.items() if value]
|
||||||
|
else:
|
||||||
|
# TODO
|
||||||
|
return response_handle(request, UniResponse.PARAM_UNKNOWN_ERR, status=400)
|
||||||
|
if set(keys).issubset(set(params)):
|
||||||
|
try:
|
||||||
|
kwargs['request_params'] = request_params
|
||||||
|
response = await func(request, *args, **kwargs)
|
||||||
|
return response
|
||||||
|
except Exception as e:
|
||||||
|
LOGGER.exception(e)
|
||||||
|
return response_handle(request, UniResponse.SERVER_UNKNOWN_ERR, 500)
|
||||||
|
else:
|
||||||
|
return response_handle(request, UniResponse.PARAM_ERR, status=400)
|
||||||
|
|
||||||
|
return auth_param
|
||||||
|
|
||||||
|
return wrapper
|
||||||
|
|
||||||
|
|
||||||
|
# Token from https://github.com/argaen/aiocache/blob/master/aiocache/decorators.py
|
||||||
|
def cached(
|
||||||
|
ttl=0, key=None, key_from_attr=None, cache=None, serializer=None, plugins=None, **kwargs):
|
||||||
|
"""
|
||||||
|
Caches the functions return value into a key generated with module_name, function_name and args.
|
||||||
|
|
||||||
|
In some cases you will need to send more args to configure the cache object.
|
||||||
|
An example would be endpoint and port for the RedisCache. You can send those args as
|
||||||
|
kwargs and they will be propagated accordingly.
|
||||||
|
|
||||||
|
:param ttl: int seconds to store the function call. Default is 0 which means no expiration.
|
||||||
|
:param key: str value to set as key for the function return. Takes precedence over
|
||||||
|
key_from_attr param. If key and key_from_attr are not passed, it will use module_name
|
||||||
|
+ function_name + args + kwargs
|
||||||
|
:param key_from_attr: arg or kwarg name from the function to use as a key.
|
||||||
|
:param cache: cache class to use when calling the ``set``/``get`` operations.
|
||||||
|
Default is the one configured in ``aiocache.settings.DEFAULT_CACHE``
|
||||||
|
:param serializer: serializer instance to use when calling the ``dumps``/``loads``.
|
||||||
|
Default is the one configured in ``aiocache.settings.DEFAULT_SERIALIZER``
|
||||||
|
:param plugins: plugins to use when calling the cmd hooks
|
||||||
|
Default is the one configured in ``aiocache.settings.DEFAULT_PLUGINS``
|
||||||
|
"""
|
||||||
|
cache_kwargs = kwargs
|
||||||
|
|
||||||
|
def cached_decorator(func):
|
||||||
|
async def wrapper(*args, **kwargs):
|
||||||
|
cache_instance = get_cache(
|
||||||
|
cache=cache, serializer=serializer, plugins=plugins, **cache_kwargs)
|
||||||
|
args_dict = get_args_dict(func, args, kwargs)
|
||||||
|
cache_key = key or args_dict.get(
|
||||||
|
key_from_attr,
|
||||||
|
(func.__module__ or 'stub') + func.__name__ + str(args) + str(kwargs))
|
||||||
|
|
||||||
|
try:
|
||||||
|
if await cache_instance.exists(cache_key):
|
||||||
|
return await cache_instance.get(cache_key)
|
||||||
|
|
||||||
|
except Exception:
|
||||||
|
logger.exception("Unexpected error with %s", cache_instance)
|
||||||
|
|
||||||
|
result = await func(*args, **kwargs)
|
||||||
|
if result:
|
||||||
|
try:
|
||||||
|
await cache_instance.set(cache_key, result, ttl=ttl)
|
||||||
|
except Exception:
|
||||||
|
logger.exception("Unexpected error with %s", cache_instance)
|
||||||
|
|
||||||
|
return result
|
||||||
|
|
||||||
|
return wrapper
|
||||||
|
|
||||||
|
return cached_decorator
|
||||||
|
|
||||||
|
|
||||||
|
def response_handle(request, dict_value, status=200):
|
||||||
|
"""
|
||||||
|
Return sanic.response or json depending on the request
|
||||||
|
:param request: sanic.request.Request or dict
|
||||||
|
:param dict_value:
|
||||||
|
:return:
|
||||||
|
"""
|
||||||
|
if isinstance(request, Request):
|
||||||
|
return response.json(dict_value, status=status)
|
||||||
|
else:
|
||||||
|
return json_dumps(dict_value, ensure_ascii=False)
|
||||||
85
soulbook/fetcher/extract_novels.py
Normal file
85
soulbook/fetcher/extract_novels.py
Normal file
@ -0,0 +1,85 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""
|
||||||
|
Created by howie.hu at 2018/5/28.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import re
|
||||||
|
|
||||||
|
from bs4 import BeautifulSoup
|
||||||
|
from collections import OrderedDict
|
||||||
|
from operator import itemgetter
|
||||||
|
from urllib.parse import urljoin, urlparse
|
||||||
|
|
||||||
|
from soulbook.config import LOGGER
|
||||||
|
|
||||||
|
|
||||||
|
def extract_chapters(chapters_url, html):
|
||||||
|
"""
|
||||||
|
通用解析小说目录
|
||||||
|
:param chapter_url: 小说目录页url
|
||||||
|
:param res: 当前页面html
|
||||||
|
:return:
|
||||||
|
"""
|
||||||
|
# 参考https://greasyfork.org/zh-CN/scripts/292-my-novel-reader
|
||||||
|
chapters_reg = r'(<a\s+.*?>.*第?\s*[一二两三四五六七八九十○零百千万亿0-9１２３４５６７８９０]{1,6}\s*[章回卷节折篇幕集].*?</a>)'
|
||||||
|
# 这里不能保证获取的章节分得很清楚，但能保证这一串str是章节目录。可以利用bs安心提取a
|
||||||
|
chapters_res = re.findall(chapters_reg, str(html), re.I)
|
||||||
|
str_chapters_res = '\n'.join(chapters_res)
|
||||||
|
chapters_res_soup = BeautifulSoup(str_chapters_res, 'html5lib')
|
||||||
|
all_chapters = []
|
||||||
|
for link in chapters_res_soup.find_all('a'):
|
||||||
|
each_data = {}
|
||||||
|
url = urljoin(chapters_url, link.get('href')) or ''
|
||||||
|
name = link.text or ''
|
||||||
|
each_data['chapter_url'] = url
|
||||||
|
each_data['chapter_name'] = name
|
||||||
|
each_data['index'] = int(urlparse(url).path.split('.')[0].split('/')[-1])
|
||||||
|
all_chapters.append(each_data)
|
||||||
|
chapters_sorted = sorted(all_chapters, reverse=True, key=itemgetter('index'))
|
||||||
|
return chapters_sorted
|
||||||
|
|
||||||
|
|
||||||
|
def extract_pre_next_chapter(url, chapter_url, html):
|
||||||
|
"""
|
||||||
|
获取单章节上一页下一页
|
||||||
|
:param chapter_url:
|
||||||
|
:param html:
|
||||||
|
:return:
|
||||||
|
"""
|
||||||
|
next_chapter = OrderedDict()
|
||||||
|
try:
|
||||||
|
# 参考https://greasyfork.org/zh-CN/scripts/292-my-novel-reader
|
||||||
|
next_reg = r'(<a\s+.*?>.*[第上前下后][一]?[0-9]{0,6}?[页张个篇章节步].*?</a>)'
|
||||||
|
judge_reg = r'[第上前下后][一]?[0-9]{0,6}?[页张个篇章节步]'
|
||||||
|
# 这里同样需要利用bs再次解析
|
||||||
|
next_res = re.findall(next_reg, html.replace('<<', '').replace('>>', ''), re.I)
|
||||||
|
str_next_res = '\n'.join(next_res)
|
||||||
|
next_res_soup = BeautifulSoup(str_next_res, 'html5lib')
|
||||||
|
for link in next_res_soup.find_all('a'):
|
||||||
|
text = link.text or ''
|
||||||
|
text = text.replace(' ', '')
|
||||||
|
if novels_list(text):
|
||||||
|
is_next = re.search(judge_reg, text)
|
||||||
|
# is_ok = is_chapter(text)
|
||||||
|
if is_next:
|
||||||
|
url = urljoin(chapter_url, link.get('href')) or ''
|
||||||
|
regex = re.compile("^http://|^https://")
|
||||||
|
if regex.sub('', chapter_url) == regex.sub('', url):
|
||||||
|
url = False
|
||||||
|
next_chapter[text[:5]] = url
|
||||||
|
|
||||||
|
# nextDic = [{v[0]: v[1]} for v in sorted(next_chapter.items(), key=lambda d: d[1])]
|
||||||
|
return next_chapter
|
||||||
|
except Exception as e:
|
||||||
|
LOGGER.exception(e)
|
||||||
|
return next_chapter
|
||||||
|
|
||||||
|
|
||||||
|
def novels_list(text):
|
||||||
|
rm_list = ['后一个', '天上掉下个']
|
||||||
|
for i in rm_list:
|
||||||
|
if i in text:
|
||||||
|
return False
|
||||||
|
else:
|
||||||
|
continue
|
||||||
|
return True
|
||||||
121
soulbook/fetcher/function.py
Normal file
121
soulbook/fetcher/function.py
Normal file
@ -0,0 +1,121 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""
|
||||||
|
Created by howie.hu at 2018/5/28.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import random
|
||||||
|
import time
|
||||||
|
|
||||||
|
import aiofiles
|
||||||
|
import aiohttp
|
||||||
|
import arrow
|
||||||
|
import async_timeout
|
||||||
|
import cchardet
|
||||||
|
import requests
|
||||||
|
|
||||||
|
from urllib.parse import urlparse
|
||||||
|
|
||||||
|
from soulbook.config import LOGGER, CONFIG
|
||||||
|
from soulbook.fetcher.novels_factory.ssl_factory import ssl_gen
|
||||||
|
|
||||||
|
|
||||||
|
async def _get_data(filename, default='') -> list:
|
||||||
|
"""
|
||||||
|
Get data from a file
|
||||||
|
:param filename: filename
|
||||||
|
:param default: default value
|
||||||
|
:return: data
|
||||||
|
"""
|
||||||
|
root_folder = os.path.dirname(os.path.dirname(__file__))
|
||||||
|
user_agents_file = os.path.join(
|
||||||
|
os.path.join(root_folder, 'data'), filename)
|
||||||
|
try:
|
||||||
|
async with aiofiles.open(user_agents_file, mode='r') as f:
|
||||||
|
data = [_.strip() for _ in await
|
||||||
|
f.readlines()]
|
||||||
|
except:
|
||||||
|
data = [default]
|
||||||
|
return data
|
||||||
|
|
||||||
|
|
||||||
|
async def get_random_user_agent() -> str:
|
||||||
|
"""
|
||||||
|
Get a random user agent string.
|
||||||
|
:return: Random user agent string.
|
||||||
|
"""
|
||||||
|
return random.choice(await _get_data('user_agents.txt', CONFIG.USER_AGENT))
|
||||||
|
|
||||||
|
|
||||||
|
def get_time() -> str:
|
||||||
|
utc = arrow.utcnow()
|
||||||
|
local = utc.to(CONFIG.TIMEZONE)
|
||||||
|
time = local.format("YYYY-MM-DD HH:mm:ss")
|
||||||
|
return time
|
||||||
|
|
||||||
|
|
||||||
|
def get_timestamp() -> float:
|
||||||
|
return time.time()
|
||||||
|
|
||||||
|
|
||||||
|
def get_netloc(url):
|
||||||
|
"""
|
||||||
|
获取netloc
|
||||||
|
:param url:
|
||||||
|
:return: netloc
|
||||||
|
"""
|
||||||
|
netloc = urlparse(url).netloc
|
||||||
|
return netloc or None
|
||||||
|
|
||||||
|
|
||||||
|
def get_scheme(url):
|
||||||
|
"""
|
||||||
|
获取scheme
|
||||||
|
:param url:
|
||||||
|
:return: scheme
|
||||||
|
"""
|
||||||
|
netloc = urlparse(url).scheme
|
||||||
|
return netloc or None
|
||||||
|
|
||||||
|
|
||||||
|
async def target_fetch(url, headers, timeout=15):
|
||||||
|
"""
|
||||||
|
:param url: target url
|
||||||
|
:return: text
|
||||||
|
"""
|
||||||
|
async with async_timeout.timeout(timeout):
|
||||||
|
try:
|
||||||
|
async with aiohttp.ClientSession() as client:
|
||||||
|
# async with client.get(url, headers=headers) as response:
|
||||||
|
async with client.get(url, headers=headers, ssl=ssl_gen()) as response:
|
||||||
|
assert response.status == 200
|
||||||
|
LOGGER.info('Task url: {}'.format(response.url))
|
||||||
|
try:
|
||||||
|
text = await response.text()
|
||||||
|
except:
|
||||||
|
try:
|
||||||
|
text = await response.read()
|
||||||
|
except aiohttp.ServerDisconnectedError as e:
|
||||||
|
LOGGER.exception(e)
|
||||||
|
text = None
|
||||||
|
return text
|
||||||
|
except Exception as e:
|
||||||
|
LOGGER.exception(str(e))
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def get_html_by_requests(url, headers, timeout=15):
|
||||||
|
"""
|
||||||
|
:param url:
|
||||||
|
:return:
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
response = requests.get(url=url, headers=headers, verify=False, timeout=timeout)
|
||||||
|
response.raise_for_status()
|
||||||
|
content = response.content
|
||||||
|
charset = cchardet.detect(content)
|
||||||
|
text = content.decode(charset['encoding'])
|
||||||
|
return text
|
||||||
|
except Exception as e:
|
||||||
|
LOGGER.exception(e)
|
||||||
|
return None
|
||||||
4
soulbook/fetcher/novels_factory/__init__.py
Normal file
4
soulbook/fetcher/novels_factory/__init__.py
Normal file
@ -0,0 +1,4 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""
|
||||||
|
Created by howie.hu at 2018/5/28.
|
||||||
|
"""
|
||||||
117
soulbook/fetcher/novels_factory/baidu_novels.py
Normal file
117
soulbook/fetcher/novels_factory/baidu_novels.py
Normal file
@ -0,0 +1,117 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""
|
||||||
|
Created by howie.hu at 2018/5/28.
|
||||||
|
"""
|
||||||
|
import aiohttp
|
||||||
|
import asyncio
|
||||||
|
import async_timeout
|
||||||
|
|
||||||
|
from aiocache.serializers import PickleSerializer
|
||||||
|
from bs4 import BeautifulSoup
|
||||||
|
from urllib.parse import urlparse
|
||||||
|
|
||||||
|
from soulbook.fetcher.decorators import cached
|
||||||
|
from soulbook.fetcher.function import get_random_user_agent
|
||||||
|
from soulbook.fetcher.novels_factory.base_novels import BaseNovels
|
||||||
|
|
||||||
|
|
||||||
|
class BaiduNovels(BaseNovels):
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
super(BaiduNovels, self).__init__()
|
||||||
|
|
||||||
|
async def data_extraction(self, html):
|
||||||
|
"""
|
||||||
|
小说信息抓取函数
|
||||||
|
:return:
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
url = html.select('h3.t a')[0].get('href', None)
|
||||||
|
real_url = await self.get_real_url(url=url) if url else None
|
||||||
|
if real_url:
|
||||||
|
real_str_url = str(real_url)
|
||||||
|
netloc = urlparse(real_str_url).netloc
|
||||||
|
if "http://" + netloc + "/" == real_str_url:
|
||||||
|
return None
|
||||||
|
if 'baidu' in real_str_url or netloc in self.black_domain:
|
||||||
|
return None
|
||||||
|
is_parse = 1 if netloc in self.rules.keys() else 0
|
||||||
|
title = html.select('h3.t a')[0].get_text()
|
||||||
|
is_recommend = 1 if netloc in self.latest_rules.keys() else 0
|
||||||
|
# time = re.findall(r'\d+-\d+-\d+', source)
|
||||||
|
# time = time[0] if time else None
|
||||||
|
timestamp = 0
|
||||||
|
time = ""
|
||||||
|
return {'title': title, 'url': real_str_url.replace('index.html', ''), 'time': time,
|
||||||
|
'is_parse': is_parse,
|
||||||
|
'is_recommend': is_recommend,
|
||||||
|
'timestamp': timestamp,
|
||||||
|
'netloc': netloc}
|
||||||
|
else:
|
||||||
|
return None
|
||||||
|
except Exception as e:
|
||||||
|
self.logger.exception(e)
|
||||||
|
return None
|
||||||
|
|
||||||
|
async def get_real_url(self, url):
|
||||||
|
"""
|
||||||
|
获取百度搜索结果真实url
|
||||||
|
:param url:
|
||||||
|
:return:
|
||||||
|
"""
|
||||||
|
with async_timeout.timeout(5):
|
||||||
|
try:
|
||||||
|
async with aiohttp.ClientSession() as client:
|
||||||
|
headers = {'user-agent': await get_random_user_agent()}
|
||||||
|
async with client.head(url, headers=headers, allow_redirects=True) as response:
|
||||||
|
self.logger.info('Parse url: {}'.format(response.url))
|
||||||
|
url = response.url if response.url else None
|
||||||
|
return url
|
||||||
|
except Exception as e:
|
||||||
|
self.logger.exception(e)
|
||||||
|
return None
|
||||||
|
|
||||||
|
async def novels_search(self, novels_name):
|
||||||
|
"""
|
||||||
|
小说搜索入口函数
|
||||||
|
:return:
|
||||||
|
"""
|
||||||
|
url = self.config.URL_PC
|
||||||
|
params = {'wd': novels_name, 'ie': 'utf-8', 'rn': self.config.BAIDU_RN, 'vf_bl': 1}
|
||||||
|
headers = {'user-agent': await get_random_user_agent()}
|
||||||
|
html = await self.fetch_url(url=url, params=params, headers=headers)
|
||||||
|
if html:
|
||||||
|
soup = BeautifulSoup(html, 'html5lib')
|
||||||
|
result = soup.find_all(class_='result')
|
||||||
|
extra_tasks = [self.data_extraction(html=i) for i in result]
|
||||||
|
tasks = [asyncio.ensure_future(i) for i in extra_tasks]
|
||||||
|
done_list, pending_list = await asyncio.wait(tasks)
|
||||||
|
res = [task.result() for task in done_list if task.result()]
|
||||||
|
return res
|
||||||
|
else:
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
@cached(ttl=259200, key_from_attr='novels_name', serializer=PickleSerializer(), namespace="novels_name")
|
||||||
|
async def start(novels_name):
|
||||||
|
"""
|
||||||
|
Start spider
|
||||||
|
:return:
|
||||||
|
"""
|
||||||
|
return await BaiduNovels.start(novels_name)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
# Start
|
||||||
|
import aiocache
|
||||||
|
|
||||||
|
REDIS_DICT = {}
|
||||||
|
aiocache.settings.set_defaults(
|
||||||
|
class_="aiocache.RedisCache",
|
||||||
|
endpoint=REDIS_DICT.get('REDIS_ENDPOINT', 'localhost'),
|
||||||
|
port=REDIS_DICT.get('REDIS_PORT', 6379),
|
||||||
|
db=REDIS_DICT.get('CACHE_DB', 0),
|
||||||
|
password=REDIS_DICT.get('REDIS_PASSWORD', None),
|
||||||
|
)
|
||||||
|
res = asyncio.get_event_loop().run_until_complete(start('intitle:雪中悍刀行 小说 阅读'))
|
||||||
|
print(res)
|
||||||
67
soulbook/fetcher/novels_factory/base_novels.py
Normal file
67
soulbook/fetcher/novels_factory/base_novels.py
Normal file
@ -0,0 +1,67 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""
|
||||||
|
Created by howie.hu at 2018/5/28.
|
||||||
|
"""
|
||||||
|
import aiohttp
|
||||||
|
import async_timeout
|
||||||
|
|
||||||
|
from soulbook.config import CONFIG, LOGGER, BLACK_DOMAIN, RULES, LATEST_RULES
|
||||||
|
from soulbook.fetcher.novels_factory.ssl_factory import ssl_gen
|
||||||
|
|
||||||
|
|
||||||
|
class BaseNovels:
|
||||||
|
"""
|
||||||
|
小说抓取父类
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, logger=None):
|
||||||
|
self.black_domain = BLACK_DOMAIN
|
||||||
|
self.config = CONFIG
|
||||||
|
self.latest_rules = LATEST_RULES
|
||||||
|
self.logger = logger if logger else LOGGER
|
||||||
|
self.rules = RULES
|
||||||
|
|
||||||
|
async def fetch_url(self, url, params, headers):
|
||||||
|
"""
|
||||||
|
公共抓取函数
|
||||||
|
:param client:
|
||||||
|
:param url:
|
||||||
|
:param params:
|
||||||
|
:return:
|
||||||
|
"""
|
||||||
|
async with async_timeout.timeout(15):
|
||||||
|
|
||||||
|
try:
|
||||||
|
async with aiohttp.ClientSession() as client:
|
||||||
|
print("headers: ", headers)
|
||||||
|
# async with client.get(url, params=params, headers=headers) as response:
|
||||||
|
async with client.get(url, params=params, headers=headers, ssl=ssl_gen()) as \
|
||||||
|
response:
|
||||||
|
assert response.status == 200
|
||||||
|
LOGGER.info('[fetch] Task url: {}'.format(response.url))
|
||||||
|
try:
|
||||||
|
text = await response.text()
|
||||||
|
except:
|
||||||
|
text = await response.read()
|
||||||
|
return text
|
||||||
|
except Exception as e:
|
||||||
|
LOGGER.exception(e)
|
||||||
|
return None
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
async def start(cls, novels_name):
|
||||||
|
return await cls().novels_search(novels_name)
|
||||||
|
|
||||||
|
async def data_extraction(self, html):
|
||||||
|
"""
|
||||||
|
小说信息抓取函数
|
||||||
|
:return:
|
||||||
|
"""
|
||||||
|
raise NotImplementedError
|
||||||
|
|
||||||
|
async def novels_search(self, novels_name):
|
||||||
|
"""
|
||||||
|
小说搜索入口函数
|
||||||
|
:return:
|
||||||
|
"""
|
||||||
|
raise NotImplementedError
|
||||||
95
soulbook/fetcher/novels_factory/bing_novels.py
Normal file
95
soulbook/fetcher/novels_factory/bing_novels.py
Normal file
@ -0,0 +1,95 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""
|
||||||
|
Created by howie.hu at 2018/5/28.
|
||||||
|
"""
|
||||||
|
import asyncio
|
||||||
|
|
||||||
|
from aiocache.serializers import PickleSerializer
|
||||||
|
from bs4 import BeautifulSoup
|
||||||
|
from urllib.parse import urlparse
|
||||||
|
|
||||||
|
from soulbook.fetcher.decorators import cached
|
||||||
|
from soulbook.fetcher.function import get_random_user_agent
|
||||||
|
from soulbook.fetcher.novels_factory.base_novels import BaseNovels
|
||||||
|
|
||||||
|
|
||||||
|
class BingNovels(BaseNovels):
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
super(BingNovels, self).__init__()
|
||||||
|
|
||||||
|
async def data_extraction(self, html):
|
||||||
|
"""
|
||||||
|
小说信息抓取函数
|
||||||
|
:return:
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
title = html.select('h2 a')[0].get_text()
|
||||||
|
url = html.select('h2 a')[0].get('href', None)
|
||||||
|
netloc = urlparse(url).netloc
|
||||||
|
url = url.replace('index.html', '').replace('Index.html', '')
|
||||||
|
# if not url or 'baidu' in url or 'baike.so.com' in url or netloc in self.black_domain or '.html' in url:
|
||||||
|
# return None
|
||||||
|
is_parse = 1 if netloc in self.rules.keys() else 0
|
||||||
|
is_recommend = 1 if netloc in self.latest_rules.keys() else 0
|
||||||
|
timestamp = 0
|
||||||
|
time = ''
|
||||||
|
return {'title': title,
|
||||||
|
'url': url,
|
||||||
|
'time': time,
|
||||||
|
'is_parse': is_parse,
|
||||||
|
'is_recommend': is_recommend,
|
||||||
|
'timestamp': timestamp,
|
||||||
|
'netloc': netloc}
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
self.logger.exception(e)
|
||||||
|
return None
|
||||||
|
|
||||||
|
async def novels_search(self, novels_name):
|
||||||
|
"""
|
||||||
|
小说搜索入口函数
|
||||||
|
:return:
|
||||||
|
"""
|
||||||
|
url = self.config.BY_URL
|
||||||
|
headers = {
|
||||||
|
'user-agent': await get_random_user_agent(),
|
||||||
|
'referer': "https://www.bing.com/"
|
||||||
|
}
|
||||||
|
params = {'q': novels_name, 'ensearch': 0, "sc": "1-20"}
|
||||||
|
html = await self.fetch_url(url=url, params=params, headers=headers)
|
||||||
|
if html:
|
||||||
|
soup = BeautifulSoup(html, 'html5lib')
|
||||||
|
result = soup.find_all(class_='b_algo')
|
||||||
|
extra_tasks = [self.data_extraction(html=i) for i in result]
|
||||||
|
tasks = [asyncio.ensure_future(i) for i in extra_tasks]
|
||||||
|
done_list, pending_list = await asyncio.wait(tasks)
|
||||||
|
res = [task.result() for task in done_list if task.result()]
|
||||||
|
return res
|
||||||
|
else:
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
@cached(ttl=259200, key_from_attr='novels_name', serializer=PickleSerializer(), namespace="novels_name")
|
||||||
|
async def start(novels_name):
|
||||||
|
"""
|
||||||
|
Start spider
|
||||||
|
:return:
|
||||||
|
"""
|
||||||
|
return await BingNovels.start(novels_name)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
# Start
|
||||||
|
import aiocache
|
||||||
|
|
||||||
|
REDIS_DICT = {}
|
||||||
|
aiocache.settings.set_defaults(
|
||||||
|
class_="aiocache.RedisCache",
|
||||||
|
endpoint=REDIS_DICT.get('REDIS_ENDPOINT', 'localhost'),
|
||||||
|
port=REDIS_DICT.get('REDIS_PORT', 6379),
|
||||||
|
db=REDIS_DICT.get('CACHE_DB', 0),
|
||||||
|
password=REDIS_DICT.get('REDIS_PASSWORD', None),
|
||||||
|
)
|
||||||
|
res = asyncio.get_event_loop().run_until_complete(start('雪中悍刀行 小说 阅读 最新章节'))
|
||||||
|
print(res)
|
||||||
89
soulbook/fetcher/novels_factory/duck_go_novels.py
Normal file
89
soulbook/fetcher/novels_factory/duck_go_novels.py
Normal file
@ -0,0 +1,89 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""
|
||||||
|
Created by howie.hu at 2018/5/28.
|
||||||
|
"""
|
||||||
|
import asyncio
|
||||||
|
|
||||||
|
from aiocache.serializers import PickleSerializer
|
||||||
|
from bs4 import BeautifulSoup
|
||||||
|
from urllib.parse import parse_qs, urlparse
|
||||||
|
|
||||||
|
from soulbook.fetcher.decorators import cached
|
||||||
|
from soulbook.fetcher.function import get_random_user_agent
|
||||||
|
from soulbook.fetcher.novels_factory.base_novels import BaseNovels
|
||||||
|
|
||||||
|
|
||||||
|
class DuckGoNovels(BaseNovels):
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
super(DuckGoNovels, self).__init__()
|
||||||
|
|
||||||
|
async def data_extraction(self, html):
|
||||||
|
"""
|
||||||
|
小说信息抓取函数
|
||||||
|
:return:
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
title = html.select('h2 a')[0].get_text()
|
||||||
|
url = html.select('h2 a')[0].get('href', None)
|
||||||
|
url = parse_qs(url).get('uddg', ['#'])[0]
|
||||||
|
netloc = urlparse(url).netloc
|
||||||
|
url = url.replace('index.html', '').replace('Index.html', '')
|
||||||
|
if not url or 'baidu' in url or 'baike.so.com' in url or netloc in self.black_domain or '.html' in url:
|
||||||
|
return None
|
||||||
|
is_parse = 1 if netloc in self.rules.keys() else 0
|
||||||
|
is_recommend = 1 if netloc in self.latest_rules.keys() else 0
|
||||||
|
# time = html.select('div.b_attribution')[0].get_text()
|
||||||
|
# time = re.findall(r'\d+-\d+-\d+', time)
|
||||||
|
# time = time[0] if time else ''
|
||||||
|
timestamp = 0
|
||||||
|
time = ''
|
||||||
|
return {'title': title,
|
||||||
|
'url': url,
|
||||||
|
'time': time,
|
||||||
|
'is_parse': is_parse,
|
||||||
|
'is_recommend': is_recommend,
|
||||||
|
'timestamp': timestamp,
|
||||||
|
'netloc': netloc}
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
self.logger.exception(e)
|
||||||
|
return None
|
||||||
|
|
||||||
|
async def novels_search(self, novels_name):
|
||||||
|
"""
|
||||||
|
小说搜索入口函数
|
||||||
|
:return:
|
||||||
|
"""
|
||||||
|
url = self.config.DUCKGO_URL
|
||||||
|
headers = {
|
||||||
|
'user-agent': await get_random_user_agent(),
|
||||||
|
'referer': "https://duckduckgo.com/"
|
||||||
|
}
|
||||||
|
params = {'q': novels_name}
|
||||||
|
html = await self.fetch_url(url=url, params=params, headers=headers)
|
||||||
|
if html:
|
||||||
|
soup = BeautifulSoup(html, 'html5lib')
|
||||||
|
result = soup.find_all(class_='result')
|
||||||
|
extra_tasks = [self.data_extraction(html=i) for i in result]
|
||||||
|
tasks = [asyncio.ensure_future(i) for i in extra_tasks]
|
||||||
|
done_list, pending_list = await asyncio.wait(tasks)
|
||||||
|
res = [task.result() for task in done_list if task.result()]
|
||||||
|
return res
|
||||||
|
else:
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
@cached(ttl=259200, key_from_attr='novels_name', serializer=PickleSerializer(), namespace="novels_name")
|
||||||
|
async def start(novels_name):
|
||||||
|
"""
|
||||||
|
Start spider
|
||||||
|
:return:
|
||||||
|
"""
|
||||||
|
return await DuckGoNovels.start(novels_name)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
# Start
|
||||||
|
res = asyncio.get_event_loop().run_until_complete(start('雪中悍刀行 小说 阅读 最新章节'))
|
||||||
|
print(res)
|
||||||
105
soulbook/fetcher/novels_factory/so_novels.py
Normal file
105
soulbook/fetcher/novels_factory/so_novels.py
Normal file
@ -0,0 +1,105 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""
|
||||||
|
Created by howie.hu at 2018/5/28.
|
||||||
|
"""
|
||||||
|
import asyncio
|
||||||
|
|
||||||
|
from aiocache.serializers import PickleSerializer
|
||||||
|
from bs4 import BeautifulSoup
|
||||||
|
from urllib.parse import parse_qs, urlparse
|
||||||
|
|
||||||
|
from soulbook.fetcher.decorators import cached
|
||||||
|
from soulbook.fetcher.function import get_random_user_agent
|
||||||
|
from soulbook.fetcher.novels_factory.base_novels import BaseNovels
|
||||||
|
|
||||||
|
|
||||||
|
class SoNovels(BaseNovels):
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
super(SoNovels, self).__init__()
|
||||||
|
|
||||||
|
async def data_extraction(self, html):
|
||||||
|
"""
|
||||||
|
小说信息抓取函数
|
||||||
|
:return:
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
# 2017.09.09 修改 更加全面地获取title && url
|
||||||
|
try:
|
||||||
|
title = html.select('h3 a')[0].get_text()
|
||||||
|
url = html.select('h3 a')[0].get('href', None)
|
||||||
|
except Exception as e:
|
||||||
|
self.logger.exception(e)
|
||||||
|
return None
|
||||||
|
|
||||||
|
# 针对不同的请进行url的提取
|
||||||
|
if "www.so.com/link?m=" in url:
|
||||||
|
url = html.select('h3 a')[0].get('data-mdurl', None)
|
||||||
|
if "www.so.com/link?url=" in url:
|
||||||
|
url = parse_qs(urlparse(url).query).get('url', None)
|
||||||
|
url = url[0] if url else None
|
||||||
|
|
||||||
|
netloc = urlparse(url).netloc
|
||||||
|
if not url or 'baidu' in url or 'baike.so.com' in url or netloc in self.black_domain:
|
||||||
|
return None
|
||||||
|
is_parse = 1 if netloc in self.rules.keys() else 0
|
||||||
|
is_recommend = 1 if netloc in self.latest_rules.keys() else 0
|
||||||
|
time = ''
|
||||||
|
timestamp = 0
|
||||||
|
return {'title': title, 'url': url.replace('index.html', '').replace('Index.html', ''), 'time': time,
|
||||||
|
'is_parse': is_parse,
|
||||||
|
'is_recommend': is_recommend,
|
||||||
|
'timestamp': timestamp,
|
||||||
|
'netloc': netloc}
|
||||||
|
except Exception as e:
|
||||||
|
self.logger.exception(e)
|
||||||
|
return None
|
||||||
|
|
||||||
|
async def novels_search(self, novels_name):
|
||||||
|
"""
|
||||||
|
小说搜索入口函数
|
||||||
|
:return:
|
||||||
|
"""
|
||||||
|
url = self.config.SO_URL
|
||||||
|
|
||||||
|
headers = {
|
||||||
|
'User-Agent': await get_random_user_agent(),
|
||||||
|
'Referer': "http://www.so.com/haosou.html?src=home"
|
||||||
|
}
|
||||||
|
params = {'ie': 'utf-8', 'src': 'noscript_home', 'shb': 1, 'q': novels_name, }
|
||||||
|
html = await self.fetch_url(url=url, params=params, headers=headers)
|
||||||
|
if html:
|
||||||
|
soup = BeautifulSoup(html, 'html5lib')
|
||||||
|
result = soup.find_all(class_='res-list')
|
||||||
|
extra_tasks = [self.data_extraction(html=i) for i in result]
|
||||||
|
tasks = [asyncio.ensure_future(i) for i in extra_tasks]
|
||||||
|
done_list, pending_list = await asyncio.wait(tasks)
|
||||||
|
res = [task.result() for task in done_list if task.result()]
|
||||||
|
return res
|
||||||
|
else:
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
@cached(ttl=259200, key_from_attr='novels_name', serializer=PickleSerializer(), namespace="novels_name")
|
||||||
|
async def start(novels_name):
|
||||||
|
"""
|
||||||
|
Start spider
|
||||||
|
:return:
|
||||||
|
"""
|
||||||
|
return await SoNovels.start(novels_name)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
# Start
|
||||||
|
import aiocache
|
||||||
|
|
||||||
|
REDIS_DICT = {}
|
||||||
|
aiocache.settings.set_defaults(
|
||||||
|
class_="aiocache.RedisCache",
|
||||||
|
endpoint=REDIS_DICT.get('REDIS_ENDPOINT', 'localhost'),
|
||||||
|
port=REDIS_DICT.get('REDIS_PORT', 6379),
|
||||||
|
db=REDIS_DICT.get('CACHE_DB', 0),
|
||||||
|
password=REDIS_DICT.get('REDIS_PASSWORD', None),
|
||||||
|
)
|
||||||
|
res = asyncio.get_event_loop().run_until_complete(start('雪中悍刀行 小说 最新章节'))
|
||||||
|
print(res)
|
||||||
31
soulbook/fetcher/novels_factory/ssl_factory.py
Normal file
31
soulbook/fetcher/novels_factory/ssl_factory.py
Normal file
@ -0,0 +1,31 @@
|
|||||||
|
# coding: utf-8
|
||||||
|
"""
|
||||||
|
绕过 JA3指纹反爬机制
|
||||||
|
参考：https://mp.weixin.qq.com/s/tuZScT0qVwPl6bQewqz0zg
|
||||||
|
"""
|
||||||
|
|
||||||
|
import random
|
||||||
|
import ssl
|
||||||
|
|
||||||
|
# ssl._create_default_https_context = ssl._create_unverified_context
|
||||||
|
|
||||||
|
|
||||||
|
ORIGIN_CIPHERS = ('ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:'
|
||||||
|
'DH+HIGH:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+HIGH:RSA+3DES')
|
||||||
|
|
||||||
|
|
||||||
|
class SSLFactory:
|
||||||
|
def __init__(self):
|
||||||
|
self.ciphers = ORIGIN_CIPHERS.split(":")
|
||||||
|
|
||||||
|
def __call__(self) -> ssl.SSLContext:
|
||||||
|
random.shuffle(self.ciphers)
|
||||||
|
ciphers = ":".join(self.ciphers)
|
||||||
|
ciphers = ciphers + ":!aNULL:!eNULL:!MD5"
|
||||||
|
|
||||||
|
context = ssl.create_default_context()
|
||||||
|
context.set_ciphers(ciphers)
|
||||||
|
return context
|
||||||
|
|
||||||
|
|
||||||
|
ssl_gen = SSLFactory()
|
||||||
33
soulbook/fetcher/novels_schedule.py
Normal file
33
soulbook/fetcher/novels_schedule.py
Normal file
@ -0,0 +1,33 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""
|
||||||
|
Created by howie.hu at 2018/5/28.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import os
|
||||||
|
import schedule
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
import uvloop
|
||||||
|
|
||||||
|
os.environ['MODE'] = 'PRO'
|
||||||
|
sys.path.append('../../')
|
||||||
|
|
||||||
|
from soulbook.fetcher.cache import update_all_books
|
||||||
|
|
||||||
|
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
|
||||||
|
loop = asyncio.get_event_loop()
|
||||||
|
|
||||||
|
|
||||||
|
def update_all_books_schedule():
|
||||||
|
task = asyncio.ensure_future(update_all_books(loop))
|
||||||
|
loop.run_until_complete(task)
|
||||||
|
return task.result() or None
|
||||||
|
|
||||||
|
|
||||||
|
# python novels_schedule.py
|
||||||
|
schedule.every(180).minutes.do(update_all_books_schedule)
|
||||||
|
|
||||||
|
while True:
|
||||||
|
schedule.run_pending()
|
||||||
|
time.sleep(1)
|
||||||
36
soulbook/fetcher/novels_tools.py
Normal file
36
soulbook/fetcher/novels_tools.py
Normal file
@ -0,0 +1,36 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""
|
||||||
|
Created by howie.hu at 2018/5/28.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from importlib import import_module
|
||||||
|
|
||||||
|
|
||||||
|
async def get_novels_info(class_name, novels_name):
|
||||||
|
novels_module = import_module(
|
||||||
|
"soulbook.fetcher.{}.{}_novels".format('novels_factory', class_name))
|
||||||
|
try:
|
||||||
|
# 获取对应渠道实例化对象
|
||||||
|
novels_info = await novels_module.start(novels_name)
|
||||||
|
except Exception as e:
|
||||||
|
print("get_novels_info: %s" % e)
|
||||||
|
novels_info = None
|
||||||
|
return novels_info
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
import asyncio
|
||||||
|
import aiocache
|
||||||
|
|
||||||
|
REDIS_DICT = {}
|
||||||
|
aiocache.settings.set_defaults(
|
||||||
|
class_="aiocache.RedisCache",
|
||||||
|
endpoint=REDIS_DICT.get('REDIS_ENDPOINT', 'localhost'),
|
||||||
|
port=REDIS_DICT.get('REDIS_PORT', 6379),
|
||||||
|
db=REDIS_DICT.get('CACHE_DB', 0),
|
||||||
|
password=REDIS_DICT.get('REDIS_PASSWORD', None),
|
||||||
|
)
|
||||||
|
|
||||||
|
res = asyncio.get_event_loop().run_until_complete(
|
||||||
|
get_novels_info(class_name='baidu', novels_name='intitle:雪中悍刀行 小说 阅读'))
|
||||||
|
print(res)
|
||||||
64
soulbook/fetcher/response_base.py
Normal file
64
soulbook/fetcher/response_base.py
Normal file
@ -0,0 +1,64 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""
|
||||||
|
Created by howie.hu at 17-9-22.
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
class ResponseField():
|
||||||
|
"""
|
||||||
|
Define the response field
|
||||||
|
"""
|
||||||
|
MESSAGE = 'info'
|
||||||
|
STATUS = 'state'
|
||||||
|
DATA = 'data'
|
||||||
|
FINISH_AT = 'finished_at'
|
||||||
|
|
||||||
|
|
||||||
|
class ResponseReply():
|
||||||
|
"""
|
||||||
|
Define field description
|
||||||
|
"""
|
||||||
|
# ERROR
|
||||||
|
UNKNOWN_ERR = '未知错误'
|
||||||
|
PARAM_ERR = '参数错误!'
|
||||||
|
PARAM_PARSE_ERR = "参数解析错误!"
|
||||||
|
DB_ERROR = "数据库操作错误"
|
||||||
|
# FORBIDDEN
|
||||||
|
IP_FORBIDDEN = "ip被禁"
|
||||||
|
# NOT AUTHORIZED
|
||||||
|
NOT_AUTHORIZED = "验证未通过"
|
||||||
|
# SUCCESS
|
||||||
|
SUCCESS = 'ok'
|
||||||
|
|
||||||
|
|
||||||
|
class ResponseCode():
|
||||||
|
"""
|
||||||
|
Define the response code
|
||||||
|
"""
|
||||||
|
SUCCESS = 200
|
||||||
|
CREATED = 201
|
||||||
|
NO_CONTENT = 204
|
||||||
|
BAD_REQUEST = 400
|
||||||
|
NOT_AUTHORIZED = 401
|
||||||
|
FORBIDDEN = 403
|
||||||
|
NOT_FOUND = 404
|
||||||
|
SERVER_ERR = 500
|
||||||
|
|
||||||
|
|
||||||
|
class UniResponse():
|
||||||
|
NOT_AUTHORIZED = {ResponseField.MESSAGE: ResponseReply.NOT_AUTHORIZED,
|
||||||
|
ResponseField.STATUS: ResponseCode.NOT_AUTHORIZED}
|
||||||
|
PARAM_PARSE_ERR = {ResponseField.MESSAGE: ResponseReply.PARAM_PARSE_ERR,
|
||||||
|
ResponseField.STATUS: ResponseCode.BAD_REQUEST}
|
||||||
|
PARAM_ERR = {ResponseField.MESSAGE: ResponseReply.PARAM_ERR,
|
||||||
|
ResponseField.STATUS: ResponseCode.BAD_REQUEST}
|
||||||
|
PARAM_UNKNOWN_ERR = {ResponseField.MESSAGE: ResponseReply.UNKNOWN_ERR,
|
||||||
|
ResponseField.STATUS: ResponseCode.BAD_REQUEST}
|
||||||
|
IP_FORBIDDEN = {ResponseField.MESSAGE: ResponseReply.IP_FORBIDDEN,
|
||||||
|
ResponseField.STATUS: ResponseCode.FORBIDDEN}
|
||||||
|
SERVER_DB_ERR = {ResponseField.MESSAGE: ResponseReply.DB_ERROR,
|
||||||
|
ResponseField.STATUS: ResponseCode.SERVER_ERR}
|
||||||
|
SERVER_UNKNOWN_ERR = {ResponseField.MESSAGE: ResponseReply.UNKNOWN_ERR,
|
||||||
|
ResponseField.STATUS: ResponseCode.SERVER_ERR}
|
||||||
|
SUCCESS = {ResponseField.MESSAGE: ResponseReply.SUCCESS,
|
||||||
|
ResponseField.STATUS: ResponseCode.SUCCESS}
|
||||||
1
soulbook/recommend/__init__.py
Normal file
1
soulbook/recommend/__init__.py
Normal file
@ -0,0 +1 @@
|
|||||||
|
from .cosinesimilarity import CosineSimilarity
|
||||||
58
soulbook/recommend/cosinesimilarity.py
Normal file
58
soulbook/recommend/cosinesimilarity.py
Normal file
@ -0,0 +1,58 @@
|
|||||||
|
import numpy as np
|
||||||
|
|
||||||
|
from functools import reduce
|
||||||
|
from math import sqrt
|
||||||
|
|
||||||
|
|
||||||
|
class CosineSimilarity(object):
|
||||||
|
"""
|
||||||
|
余弦相似性计算相似度
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, initQuery, userData):
|
||||||
|
self.title = initQuery
|
||||||
|
self.data = userData
|
||||||
|
|
||||||
|
def create_vector(self):
|
||||||
|
"""
|
||||||
|
创建兴趣向量
|
||||||
|
:return: wordVector = {} 目标用户以及各个兴趣对应的向量
|
||||||
|
"""
|
||||||
|
wordVector = {}
|
||||||
|
for web, value in self.data.items():
|
||||||
|
wordVector[web] = []
|
||||||
|
titleVector, valueVector = [], []
|
||||||
|
allWord = set(self.title + value)
|
||||||
|
for eachWord in allWord:
|
||||||
|
titleNum = self.title.count(eachWord)
|
||||||
|
valueNum = value.count(eachWord)
|
||||||
|
titleVector.append(titleNum)
|
||||||
|
valueVector.append(valueNum)
|
||||||
|
wordVector[web].append(titleVector)
|
||||||
|
wordVector[web].append(valueVector)
|
||||||
|
return wordVector
|
||||||
|
|
||||||
|
def calculate(self, wordVector):
|
||||||
|
"""
|
||||||
|
计算余弦相似度
|
||||||
|
:param wordVector: wordVector = {} 目标用户以及各个兴趣对应的向量
|
||||||
|
:return: 返回各个用户相似度值
|
||||||
|
"""
|
||||||
|
resultDic = {}
|
||||||
|
for web, value in wordVector.items():
|
||||||
|
valueArr = np.array(value)
|
||||||
|
# 余弦相似性
|
||||||
|
squares = []
|
||||||
|
numerator = reduce(lambda x, y: x + y, valueArr[0] * valueArr[1])
|
||||||
|
square_title, square_data = 0.0, 0.0
|
||||||
|
for num in range(len(valueArr[0])):
|
||||||
|
square_title += pow(valueArr[0][num], 2)
|
||||||
|
square_data += pow(valueArr[1][num], 2)
|
||||||
|
squares.append(sqrt(square_title))
|
||||||
|
squares.append(sqrt(square_data))
|
||||||
|
mul_of_squares = reduce(lambda x, y: x * y, squares)
|
||||||
|
value = float(('%.5f' % (numerator / mul_of_squares)))
|
||||||
|
if value > 0:
|
||||||
|
resultDic[web] = value
|
||||||
|
resultDic = [{v[0]: v[1]} for v in sorted(resultDic.items(), key=lambda d: d[1], reverse=True)]
|
||||||
|
return resultDic
|
||||||
1
soulbook/recommend/owl_ranking.py
Normal file
1
soulbook/recommend/owl_ranking.py
Normal file
@ -0,0 +1 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
22
soulbook/run.py
Normal file
22
soulbook/run.py
Normal file
@ -0,0 +1,22 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""
|
||||||
|
Created by howie.hu at 2018/8/13.
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
import subprocess
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
os.environ['MODE'] = 'PRO'
|
||||||
|
servers = [
|
||||||
|
["pipenv", "run", "gunicorn", "-c", "config/gunicorn.py", "--worker-class", "sanic.worker.GunicornWorker",
|
||||||
|
"server:app"],
|
||||||
|
["pipenv", "run", "python", "scheduled_task.py"]
|
||||||
|
]
|
||||||
|
procs = []
|
||||||
|
for server in servers:
|
||||||
|
proc = subprocess.Popen(server)
|
||||||
|
procs.append(proc)
|
||||||
|
for proc in procs:
|
||||||
|
proc.wait()
|
||||||
|
if proc.poll():
|
||||||
|
exit(0)
|
||||||
47
soulbook/scheduled_task.py
Normal file
47
soulbook/scheduled_task.py
Normal file
@ -0,0 +1,47 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""
|
||||||
|
Created by howie.hu at 2018/8/13.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import asyncio
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
|
||||||
|
import schedule
|
||||||
|
import uvloop
|
||||||
|
|
||||||
|
os.environ['MODE'] = 'PRO'
|
||||||
|
sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||||
|
|
||||||
|
from soulbook.config import CONFIG
|
||||||
|
from soulbook.spiders import QidianRankingSpider, ZHRankingSpider
|
||||||
|
from soulbook.fetcher.cache import update_all_books
|
||||||
|
|
||||||
|
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
|
||||||
|
loop = asyncio.get_event_loop()
|
||||||
|
|
||||||
|
|
||||||
|
def start_spider():
|
||||||
|
QidianRankingSpider.start()
|
||||||
|
ZHRankingSpider.start()
|
||||||
|
|
||||||
|
|
||||||
|
def update_all_books_schedule():
|
||||||
|
task = asyncio.ensure_future(update_all_books(loop))
|
||||||
|
loop.run_until_complete(task)
|
||||||
|
return task.result() or None
|
||||||
|
|
||||||
|
|
||||||
|
def refresh_task():
|
||||||
|
schedule.every(CONFIG.SCHEDULED_DICT['SPIDER_INTERVAL']).minutes.do(start_spider)
|
||||||
|
# schedule.every(CONFIG.SCHEDULED_DICT['NOVELS_INTERVAL']).minutes.do(update_all_books_schedule)
|
||||||
|
|
||||||
|
while True:
|
||||||
|
schedule.run_pending()
|
||||||
|
time.sleep(1)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
start_spider()
|
||||||
|
refresh_task()
|
||||||
88
soulbook/server.py
Normal file
88
soulbook/server.py
Normal file
@ -0,0 +1,88 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
import aiocache
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
|
||||||
|
from sanic import Sanic
|
||||||
|
from sanic.response import html, redirect
|
||||||
|
from sanic_session import RedisSessionInterface
|
||||||
|
|
||||||
|
sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||||
|
|
||||||
|
from soulbook.views import admin_bp, api_bp, except_bp, md_bp, novels_bp, operate_bp
|
||||||
|
from soulbook.database.redis import RedisSession
|
||||||
|
from soulbook.config import LOGGER, CONFIG
|
||||||
|
|
||||||
|
app = Sanic(__name__)
|
||||||
|
app.blueprint(md_bp)
|
||||||
|
app.blueprint(novels_bp)
|
||||||
|
app.blueprint(operate_bp)
|
||||||
|
app.blueprint(except_bp)
|
||||||
|
app.blueprint(admin_bp)
|
||||||
|
app.blueprint(api_bp)
|
||||||
|
|
||||||
|
|
||||||
|
@app.listener('before_server_start')
|
||||||
|
def init_cache(app, loop):
|
||||||
|
LOGGER.info("Starting aiocache")
|
||||||
|
app.config.from_object(CONFIG)
|
||||||
|
REDIS_DICT = CONFIG.REDIS_DICT
|
||||||
|
aiocache.settings.set_defaults(
|
||||||
|
class_="aiocache.RedisCache",
|
||||||
|
endpoint=REDIS_DICT.get('REDIS_ENDPOINT', 'localhost'),
|
||||||
|
port=REDIS_DICT.get('REDIS_PORT', 6379),
|
||||||
|
db=REDIS_DICT.get('CACHE_DB', 0),
|
||||||
|
password=REDIS_DICT.get('REDIS_PASSWORD', None),
|
||||||
|
loop=loop,
|
||||||
|
)
|
||||||
|
LOGGER.info("Starting redis pool")
|
||||||
|
redis_session = RedisSession()
|
||||||
|
# redis instance for app
|
||||||
|
app.get_redis_pool = redis_session.get_redis_pool
|
||||||
|
# pass the getter method for the connection pool into the session
|
||||||
|
app.session_interface = RedisSessionInterface(
|
||||||
|
app.get_redis_pool, cookie_name="owl_sid", expiry=30 * 24 * 60 * 60)
|
||||||
|
|
||||||
|
|
||||||
|
@app.middleware('request')
|
||||||
|
async def add_session_to_request(request):
|
||||||
|
# before each request initialize a session
|
||||||
|
# using the client's request
|
||||||
|
host = request.headers.get('host', None)
|
||||||
|
user_agent = request.headers.get('user-agent', None)
|
||||||
|
if user_agent:
|
||||||
|
user_ip = request.headers.get('X-Forwarded-For')
|
||||||
|
LOGGER.info('user ip is: {}'.format(user_ip))
|
||||||
|
if user_ip in CONFIG.FORBIDDEN:
|
||||||
|
return html("<h3>网站正在维护...</h3>")
|
||||||
|
if CONFIG.VAL_HOST == 'true':
|
||||||
|
if not host or host not in CONFIG.HOST:
|
||||||
|
return redirect('http://www.owllook.net')
|
||||||
|
if CONFIG.WEBSITE['IS_RUNNING']:
|
||||||
|
await app.session_interface.open(request)
|
||||||
|
else:
|
||||||
|
return html("<h3>网站正在维护...</h3>")
|
||||||
|
else:
|
||||||
|
return html("<h3>网站正在维护...</h3>")
|
||||||
|
|
||||||
|
|
||||||
|
@app.middleware('response')
|
||||||
|
async def save_session(request, response):
|
||||||
|
# after each request save the session,
|
||||||
|
# pass the response to set client cookies
|
||||||
|
# await app.session_interface.save(request, response)
|
||||||
|
if request.path == '/operate/login' and request['session'].get('user', None):
|
||||||
|
await app.session_interface.save(request, response)
|
||||||
|
import datetime
|
||||||
|
response.cookies['owl_sid']['expires'] = datetime.datetime.now(
|
||||||
|
) + datetime.timedelta(days=30)
|
||||||
|
elif request.path == '/register':
|
||||||
|
try:
|
||||||
|
response.cookies['reg_index'] = str(request['session']['index'][0])
|
||||||
|
except KeyError as e:
|
||||||
|
LOGGER.error(e)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
workers = 1 if os.name == 'nt' else 2
|
||||||
|
app.run(host="0.0.0.0", workers=workers, port=8001, debug=CONFIG.DEBUG)
|
||||||
24
soulbook/spiders/README.md
Normal file
24
soulbook/spiders/README.md
Normal file
@ -0,0 +1,24 @@
|
|||||||
|
## 小说榜单爬虫
|
||||||
|
|
||||||
|
> 获取各大小说站的小说榜单排名，使用自己写的简单爬虫框架[talospider](https://github.com/howie6879/talospider)来进行脚本的编写
|
||||||
|
|
||||||
|
### 1.起点榜单
|
||||||
|
|
||||||
|
> 网址：http://r.qidian.com/?chn=-1
|
||||||
|
|
||||||
|
| 参数 | 说明 |
|
||||||
|
| :-------------------------: | :--: |
|
||||||
|
| http://r.qidian.com/?chn=-1 | 全部分类 |
|
||||||
|
| http://r.qidian.com/?chn=21 | 玄幻 |
|
||||||
|
| http://r.qidian.com/?chn=1 | 奇幻 |
|
||||||
|
| http://r.qidian.com/?chn=2 | 武侠 |
|
||||||
|
| http://r.qidian.com/?chn=22 | 仙侠 |
|
||||||
|
| http://r.qidian.com/?chn=4 | 都市 |
|
||||||
|
| http://r.qidian.com/?chn=15 | 职场 |
|
||||||
|
| http://r.qidian.com/?chn=6 | 军事 |
|
||||||
|
| http://r.qidian.com/?chn=5 | 历史 |
|
||||||
|
| http://r.qidian.com/?chn=7 | 游戏 |
|
||||||
|
| http://r.qidian.com/?chn=8 | 体育 |
|
||||||
|
| http://r.qidian.com/?chn=9 | 科幻 |
|
||||||
|
| http://r.qidian.com/?chn=10 | 灵异 |
|
||||||
|
| http://r.qidian.com/?chn=12 | 二次元 |
|
||||||
2
soulbook/spiders/__init__.py
Normal file
2
soulbook/spiders/__init__.py
Normal file
@ -0,0 +1,2 @@
|
|||||||
|
from .qidian_ranking import QidianRankingSpider
|
||||||
|
from .zh_ranking import ZHRankingSpider
|
||||||
102
soulbook/spiders/heiyan_novel_info.py
Normal file
102
soulbook/spiders/heiyan_novel_info.py
Normal file
@ -0,0 +1,102 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""
|
||||||
|
Created by howie.hu at 30/03/2018.
|
||||||
|
"""
|
||||||
|
import time
|
||||||
|
|
||||||
|
from pprint import pprint
|
||||||
|
|
||||||
|
from ruia import Spider, Item, TextField, AttrField
|
||||||
|
from ruia_ua import middleware as ua_middleware
|
||||||
|
|
||||||
|
from soulbook.database.mongodb import MotorBase
|
||||||
|
|
||||||
|
|
||||||
|
class HYNovelInfoItem(Item):
|
||||||
|
"""
|
||||||
|
定义继承自item的Item类
|
||||||
|
"""
|
||||||
|
novel_name = AttrField(css_select="meta[property='og:title']", attr='content')
|
||||||
|
author = AttrField(css_select="meta[property='og:novel:author']", attr='content')
|
||||||
|
cover = AttrField(css_select="meta[property='og:image']", attr='content')
|
||||||
|
abstract = AttrField(css_select="meta[property='og:description']", attr='content')
|
||||||
|
status = AttrField(css_select="meta[property='og:novel:status']", attr='content')
|
||||||
|
novels_type = AttrField(css_select="meta[property='og:novel:category']", attr='content')
|
||||||
|
novel_chapter_url = AttrField(css_select='div#voteList a.index', attr='href')
|
||||||
|
latest_chapter = AttrField(css_select="meta[property='og:novel:latest_chapter_name']", attr='content')
|
||||||
|
latest_chapter_url = AttrField(css_select="meta[property='og:novel:latest_chapter_url']", attr='content')
|
||||||
|
latest_chapter_time = AttrField(css_select="meta[property='og:novel:update_time']", attr='content')
|
||||||
|
|
||||||
|
# novel_name = TextField(css_select='div.c-left>div.mod>div.hd>h2')
|
||||||
|
# author = TextField(css_select='div.author-zone div.right a.name strong')
|
||||||
|
# cover = AttrField(css_select='img.book-cover', attr='src')
|
||||||
|
# abstract = TextField(css_select='pre.note')
|
||||||
|
# status = ''
|
||||||
|
# novels_type = TextField(css_select='div.c-left>div.mod>div.hd>p.infos>span.cate>a')
|
||||||
|
# latest_chapter = ''
|
||||||
|
# novel_chapter_url = AttrField(css_select='div#voteList a.index', attr='href')
|
||||||
|
|
||||||
|
async def clean_cover(self, cover):
|
||||||
|
if 'https' in cover:
|
||||||
|
return cover
|
||||||
|
else:
|
||||||
|
return cover.replace('http', 'https')
|
||||||
|
|
||||||
|
async def clean_novels_type(self, novels_type):
|
||||||
|
types_dict = {
|
||||||
|
'社会': '都市'
|
||||||
|
}
|
||||||
|
print(types_dict.get(str(novels_type).strip(), novels_type))
|
||||||
|
return types_dict.get(str(novels_type).strip(), novels_type)
|
||||||
|
|
||||||
|
async def clean_latest_chapter_time(self, latest_chapter_time):
|
||||||
|
return latest_chapter_time.replace(u'今天', str(time.strftime("%Y-%m-%d ", time.localtime()))).replace(u'昨日', str(
|
||||||
|
time.strftime("%Y-%m-%d ", time.localtime(time.time() - 24 * 60 * 60))))
|
||||||
|
|
||||||
|
|
||||||
|
class HYNovelInfoSpider(Spider):
|
||||||
|
start_urls = []
|
||||||
|
request_config = {
|
||||||
|
'RETRIES': 3,
|
||||||
|
'TIMEOUT': 10
|
||||||
|
}
|
||||||
|
|
||||||
|
async def parse(self, res):
|
||||||
|
self.motor_db = MotorBase(loop=self.loop).get_db()
|
||||||
|
item = await HYNovelInfoItem.get_item(html=res.html)
|
||||||
|
|
||||||
|
item_data = {
|
||||||
|
'novel_name': item.novel_name,
|
||||||
|
'author': item.author,
|
||||||
|
'cover': item.cover,
|
||||||
|
'abstract': item.abstract,
|
||||||
|
'status': item.status,
|
||||||
|
'novels_type': item.novels_type,
|
||||||
|
'novel_chapter_url': item.novel_chapter_url,
|
||||||
|
'latest_chapter': item.latest_chapter,
|
||||||
|
'latest_chapter_time': item.latest_chapter_time,
|
||||||
|
'spider': 'heiyan',
|
||||||
|
'target_url': res.url,
|
||||||
|
'updated_at': time.strftime("%Y-%m-%d %X", time.localtime())
|
||||||
|
}
|
||||||
|
|
||||||
|
print('获取 {} 小说信息成功'.format(item_data['novel_name']))
|
||||||
|
await self.save(res_dic=item_data)
|
||||||
|
|
||||||
|
async def save(self, res_dic):
|
||||||
|
# 存进数据库
|
||||||
|
try:
|
||||||
|
motor_db = MotorBase().get_db()
|
||||||
|
await motor_db.all_novels_info.update_one({
|
||||||
|
'novel_name': res_dic['novel_name'], 'spider': 'heiyan'},
|
||||||
|
{'$set': res_dic},
|
||||||
|
upsert=True)
|
||||||
|
except Exception as e:
|
||||||
|
self.logger.exception(e)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
HYNovelInfoSpider.start_urls = ['http://www.heiyan.com/book/62599']
|
||||||
|
# HYNovelInfoSpider.start_urls = [each.get('novel_url', '') for each in search_author('火星引力', 'qidian')]
|
||||||
|
print(HYNovelInfoSpider.start_urls)
|
||||||
|
HYNovelInfoSpider.start(middleware=ua_middleware)
|
||||||
31
soulbook/spiders/middlewares.py
Normal file
31
soulbook/spiders/middlewares.py
Normal file
@ -0,0 +1,31 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""
|
||||||
|
Created by howie.hu at 2018/10/17.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from ruia import Middleware
|
||||||
|
|
||||||
|
from soulbook.spiders.spider_tools import get_proxy_ip
|
||||||
|
|
||||||
|
owl_middleware = Middleware()
|
||||||
|
|
||||||
|
|
||||||
|
@owl_middleware.request
|
||||||
|
async def add_random_proxy(request):
|
||||||
|
request.kwargs.update({'proxy': await update_proxy()})
|
||||||
|
request.request_config.update({'RETRY_FUNC': retry_func})
|
||||||
|
|
||||||
|
|
||||||
|
async def update_proxy():
|
||||||
|
proxy = await get_proxy_ip()
|
||||||
|
if proxy:
|
||||||
|
proxy = 'http://' + proxy
|
||||||
|
else:
|
||||||
|
proxy = None
|
||||||
|
return proxy
|
||||||
|
|
||||||
|
|
||||||
|
async def retry_func(request):
|
||||||
|
proxy = await update_proxy()
|
||||||
|
request.kwargs.update({'proxy': proxy})
|
||||||
|
return request
|
||||||
120
soulbook/spiders/qidian_all_novels.py
Normal file
120
soulbook/spiders/qidian_all_novels.py
Normal file
@ -0,0 +1,120 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""
|
||||||
|
Created by howie.hu at 25/02/2018.
|
||||||
|
Target URI: https://www.qidian.com/all
|
||||||
|
Param:?page=1
|
||||||
|
"""
|
||||||
|
import asyncio
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
|
||||||
|
from ruia import Spider, Item, TextField, AttrField
|
||||||
|
from ruia_ua import middleware as ua_middleware
|
||||||
|
|
||||||
|
# os.environ['MODE'] = 'PRO'
|
||||||
|
|
||||||
|
from soulbook.database.mongodb import MotorBase
|
||||||
|
from soulbook.spiders.middlewares import owl_middleware
|
||||||
|
|
||||||
|
try:
|
||||||
|
import uvloop
|
||||||
|
|
||||||
|
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
loop = asyncio.get_event_loop()
|
||||||
|
asyncio.set_event_loop(loop)
|
||||||
|
|
||||||
|
|
||||||
|
class QidianNovelsItem(Item):
|
||||||
|
target_item = TextField(css_select='ul.all-img-list>li')
|
||||||
|
novel_url = AttrField(css_select='div.book-img-box>a', attr='href')
|
||||||
|
novel_name = TextField(css_select='div.book-mid-info>h4')
|
||||||
|
novel_author = TextField(css_select='div.book-mid-info>p.author>a.name')
|
||||||
|
novel_author_home_url = AttrField(css_select='div.book-mid-info>p.author>a.name', attr='href')
|
||||||
|
novel_type = TextField(css_select='div.book-mid-info > p.author > a:nth-child(4)')
|
||||||
|
novel_cover = AttrField(css_select='div.book-img-box img', attr='src')
|
||||||
|
novel_abstract = TextField(css_select='div.book-mid-info p.intro')
|
||||||
|
|
||||||
|
# novel_latest_chapter = TextField(css_select='div.bookupdate a')
|
||||||
|
|
||||||
|
async def clean_novel_url(self, novel_url):
|
||||||
|
return 'https:' + novel_url
|
||||||
|
|
||||||
|
async def clean_novel_author(self, novel_author):
|
||||||
|
if isinstance(novel_author, list):
|
||||||
|
novel_author = novel_author[0].text
|
||||||
|
return novel_author
|
||||||
|
|
||||||
|
async def clean_novel_author_home_url(self, novel_author_home_url):
|
||||||
|
if isinstance(novel_author_home_url, list):
|
||||||
|
novel_author_home_url = novel_author_home_url[0].get('href').strip()
|
||||||
|
return 'https:' + novel_author_home_url
|
||||||
|
|
||||||
|
async def clean_novel_cover(self, novel_cover):
|
||||||
|
return 'https:' + novel_cover
|
||||||
|
|
||||||
|
|
||||||
|
class QidianNovelsSpider(Spider):
|
||||||
|
# start_urls = ['https://www.qidian.com/all?page=1']
|
||||||
|
|
||||||
|
request_config = {
|
||||||
|
'RETRIES': 15,
|
||||||
|
'DELAY': 0,
|
||||||
|
'TIMEOUT': 3
|
||||||
|
}
|
||||||
|
concurrency = 20
|
||||||
|
motor_db = MotorBase(loop=loop).get_db()
|
||||||
|
|
||||||
|
async def parse(self, res):
|
||||||
|
items_data = await QidianNovelsItem.get_items(html=res.html)
|
||||||
|
tasks = []
|
||||||
|
for item in items_data:
|
||||||
|
res_dic = {
|
||||||
|
'novel_url': item.novel_url,
|
||||||
|
'novel_name': item.novel_name,
|
||||||
|
'novel_author': item.novel_author,
|
||||||
|
'novel_author_home_url': item.novel_author_home_url,
|
||||||
|
'novel_type': item.novel_type,
|
||||||
|
'novel_cover': item.novel_cover,
|
||||||
|
'novel_abstract': item.novel_abstract,
|
||||||
|
'spider': 'qidian',
|
||||||
|
'updated_at': time.strftime("%Y-%m-%d %X", time.localtime()),
|
||||||
|
}
|
||||||
|
tasks.append(asyncio.ensure_future(self.save(res_dic)))
|
||||||
|
|
||||||
|
good_nums = 0
|
||||||
|
if tasks:
|
||||||
|
done_list, pending_list = await asyncio.wait(tasks)
|
||||||
|
for task in done_list:
|
||||||
|
if task.result():
|
||||||
|
good_nums += 1
|
||||||
|
print(f"共{len(tasks)}本小说，抓取成功{good_nums}本")
|
||||||
|
|
||||||
|
async def save(self, res_dic):
|
||||||
|
# 存进数据库
|
||||||
|
try:
|
||||||
|
await self.motor_db.all_novels.update_one(
|
||||||
|
{'novel_url': res_dic['novel_url'], 'novel_name': res_dic['novel_name']},
|
||||||
|
{'$set': res_dic},
|
||||||
|
upsert=True)
|
||||||
|
print(res_dic['novel_name'] + ' - 抓取成功')
|
||||||
|
return True
|
||||||
|
except Exception as e:
|
||||||
|
self.logger.exception(e)
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
# 51793
|
||||||
|
for page in range(248, 519):
|
||||||
|
print(f"正在爬取第{page}页")
|
||||||
|
start_page = page * 100
|
||||||
|
end_page = start_page + 100
|
||||||
|
if end_page > 51793:
|
||||||
|
end_page = 51793
|
||||||
|
QidianNovelsSpider.start_urls = ['https://www.qidian.com/all?page={i}'.format(i=i) for i in
|
||||||
|
range(start_page, end_page)]
|
||||||
|
# 其他多item示例：https://gist.github.com/howie6879/3ef4168159e5047d42d86cb7fb706a2f
|
||||||
|
QidianNovelsSpider.start(loop=loop, middleware=[ua_middleware, owl_middleware], close_event_loop=False)
|
||||||
64
soulbook/spiders/qidian_honor_spider.py
Normal file
64
soulbook/spiders/qidian_honor_spider.py
Normal file
@ -0,0 +1,64 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""
|
||||||
|
Created by howie.hu at 11/03/2018.
|
||||||
|
获取起点荣誉数据，如：https://book.qidian.com/honor/1009704712
|
||||||
|
荣誉类型：
|
||||||
|
- 推荐票
|
||||||
|
- 收藏
|
||||||
|
- 点击
|
||||||
|
"""
|
||||||
|
|
||||||
|
from ruia import Spider, Item, TextField
|
||||||
|
from ruia_ua import middleware
|
||||||
|
|
||||||
|
|
||||||
|
class QidianHonorItem(Item):
|
||||||
|
target_item = TextField(css_select='li.cf')
|
||||||
|
honor_text = TextField(css_select='span.decs')
|
||||||
|
honor_time = TextField(css_select='span.time')
|
||||||
|
|
||||||
|
|
||||||
|
class QidianHonorSpider(Spider):
|
||||||
|
start_urls = ['https://book.qidian.com/honor/1009531496']
|
||||||
|
|
||||||
|
request_config = {
|
||||||
|
'RETRIES': 3,
|
||||||
|
'DELAY': 0,
|
||||||
|
'TIMEOUT': 10
|
||||||
|
}
|
||||||
|
|
||||||
|
async def parse(self, res):
|
||||||
|
items_data = await QidianHonorItem.get_items(html=res.html)
|
||||||
|
click_list, col_list, rec_list, other_list = [], [], [], []
|
||||||
|
for item in items_data:
|
||||||
|
data = {
|
||||||
|
'honor_text': item.honor_text,
|
||||||
|
'honor_time': item.honor_time,
|
||||||
|
}
|
||||||
|
if "点击" in data['honor_text'] and '月点击' not in data['honor_text']:
|
||||||
|
click_list.append(data)
|
||||||
|
elif "收藏" in data['honor_text']:
|
||||||
|
col_list.append(data)
|
||||||
|
elif "推荐票" in data['honor_text']:
|
||||||
|
rec_list.append(data)
|
||||||
|
else:
|
||||||
|
other_list.append(data)
|
||||||
|
print('点击荣誉\n')
|
||||||
|
for i in click_list:
|
||||||
|
print(str(i.get('honor_time')) + " - " + str(i.get('honor_text')))
|
||||||
|
|
||||||
|
print('收藏荣誉\n')
|
||||||
|
for i in col_list:
|
||||||
|
print(str(i.get('honor_time')) + " - " + str(i.get('honor_text')))
|
||||||
|
|
||||||
|
print('推荐票荣誉\n')
|
||||||
|
for i in rec_list:
|
||||||
|
print(str(i.get('honor_time')) + " - " + str(i.get('honor_text')))
|
||||||
|
|
||||||
|
print('强推荣誉\n')
|
||||||
|
for i in other_list:
|
||||||
|
print(str(i.get('honor_time')) + " - " + str(i.get('honor_text')))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
QidianHonorSpider.start(middleware=middleware)
|
||||||
85
soulbook/spiders/qidian_novel_info.py
Normal file
85
soulbook/spiders/qidian_novel_info.py
Normal file
@ -0,0 +1,85 @@
|
|||||||
|
# -*- coding:utf-8 -*-
|
||||||
|
# !/usr/bin/env python
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
|
||||||
|
from ruia import Spider, Item, TextField, AttrField
|
||||||
|
from ruia_ua import middleware
|
||||||
|
|
||||||
|
os.environ['MODE'] = 'PRO'
|
||||||
|
|
||||||
|
from soulbook.database.mongodb import MotorBaseOld
|
||||||
|
|
||||||
|
|
||||||
|
class QidianNovelInfoItem(Item):
|
||||||
|
"""
|
||||||
|
定义继承自item的Item类
|
||||||
|
"""
|
||||||
|
novel_name = TextField(css_select='.book-info>h1>em')
|
||||||
|
author = TextField(css_select='a.writer')
|
||||||
|
# 当提取的值是属性的时候，要定义AttrField
|
||||||
|
cover = AttrField(css_select='a#bookImg>img', attr='src')
|
||||||
|
abstract = TextField(css_select='div.book-intro>p')
|
||||||
|
status = TextField(css_select='p.tag>span.blue')
|
||||||
|
novels_type = TextField(css_select='p.tag>a.red')
|
||||||
|
latest_chapter = TextField(css_select='li.update>div.detail>p.cf>a')
|
||||||
|
latest_chapter_time = TextField(css_select='div.detail>p.cf>em')
|
||||||
|
|
||||||
|
async def clean_cover(self, cover):
|
||||||
|
return 'http:' + cover
|
||||||
|
|
||||||
|
async def clean_status(self, status):
|
||||||
|
"""
|
||||||
|
当目标值的对象只有一个，默认将值提取出来，否则返回list，可以在这里定义一个函数进行循环提取
|
||||||
|
:param ele_tag:
|
||||||
|
:return:
|
||||||
|
"""
|
||||||
|
return '#'.join([i.text for i in status])
|
||||||
|
|
||||||
|
async def clean_novels_type(self, novels_type):
|
||||||
|
return '#'.join([i.text for i in novels_type])
|
||||||
|
|
||||||
|
async def clean_latest_chapter_time(self, latest_chapter_time):
|
||||||
|
return latest_chapter_time.replace(u'今天', str(time.strftime("%Y-%m-%d ", time.localtime()))).replace(u'昨日', str(
|
||||||
|
time.strftime("%Y-%m-%d ", time.localtime(time.time() - 24 * 60 * 60))))
|
||||||
|
|
||||||
|
|
||||||
|
class QidianNovelInfoSpider(Spider):
|
||||||
|
request_config = {
|
||||||
|
'RETRIES': 3,
|
||||||
|
'TIMEOUT': 10
|
||||||
|
}
|
||||||
|
|
||||||
|
async def parse(self, res):
|
||||||
|
motor_db = MotorBaseOld().db
|
||||||
|
item = await QidianNovelInfoItem.get_item(html=res.html)
|
||||||
|
item_data = {
|
||||||
|
'novel_name': item.novel_name,
|
||||||
|
'author': item.author,
|
||||||
|
'cover': item.cover,
|
||||||
|
'abstract': item.abstract,
|
||||||
|
'status': item.status,
|
||||||
|
'novels_type': item.novels_type,
|
||||||
|
'latest_chapter': item.latest_chapter,
|
||||||
|
'latest_chapter_time': item.latest_chapter_time,
|
||||||
|
'spider': 'qidian',
|
||||||
|
'target_url': res.url,
|
||||||
|
'updated_at': time.strftime("%Y-%m-%d %X", time.localtime())
|
||||||
|
}
|
||||||
|
print('获取 {} 小说信息成功'.format(item.novel_name))
|
||||||
|
await motor_db.all_novels_info.update_one(
|
||||||
|
{'novel_name': item_data['novel_name'], 'spider': item_data['spider']},
|
||||||
|
{'$set': item_data},
|
||||||
|
upsert=True)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
import random
|
||||||
|
|
||||||
|
# 其他多item示例：https://gist.github.com/howie6879/3ef4168159e5047d42d86cb7fb706a2f
|
||||||
|
QidianNovelInfoSpider.start_urls = ['https://book.qidian.com/info/1004608738',
|
||||||
|
'https://book.qidian.com/info/3602691',
|
||||||
|
'https://book.qidian.com/info/3347595', 'https://book.qidian.com/info/1887208']
|
||||||
|
|
||||||
|
# QidianNovelInfoSpider.start_urls = all_urls
|
||||||
|
QidianNovelInfoSpider.start(middleware=middleware)
|
||||||
98
soulbook/spiders/qidian_ranking.py
Normal file
98
soulbook/spiders/qidian_ranking.py
Normal file
@ -0,0 +1,98 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
import asyncio
|
||||||
|
import time
|
||||||
|
|
||||||
|
from ruia import Spider, Item, AttrField, HtmlField, TextField
|
||||||
|
from ruia_ua import middleware
|
||||||
|
|
||||||
|
from soulbook.database.mongodb import MotorBaseOld
|
||||||
|
|
||||||
|
|
||||||
|
class RankingItem(Item):
|
||||||
|
target_item = TextField(css_select='.rank-list')
|
||||||
|
ranking_title = TextField(css_select='h3.wrap-title')
|
||||||
|
more = AttrField(css_select='h3>a.more', attr='href')
|
||||||
|
book_list = HtmlField(css_select='div.book-list>ul>li', many=True)
|
||||||
|
|
||||||
|
async def clean_ranking_title(self, ranking_title):
|
||||||
|
if isinstance(ranking_title, list):
|
||||||
|
return ranking_title[0].text
|
||||||
|
else:
|
||||||
|
return str(ranking_title).split('榜')[0] + '榜'
|
||||||
|
|
||||||
|
async def clean_more(self, more):
|
||||||
|
return "https:" + more
|
||||||
|
|
||||||
|
|
||||||
|
class NameItem(Item):
|
||||||
|
top_name = TextField(css_select='h4', default='')
|
||||||
|
other_name = TextField(css_select='a.name', default='')
|
||||||
|
|
||||||
|
|
||||||
|
class QidianRankingSpider(Spider):
|
||||||
|
start_urls = [f"https://www.qidian.com/rank/?chn={key}" for key in
|
||||||
|
[-1, 21, 1, 2, 22, 4, 15, 6, 5, 7, 8, 9, 10, 12]]
|
||||||
|
|
||||||
|
concurrency = 3
|
||||||
|
qidian_type = {
|
||||||
|
'-1': '全部类别',
|
||||||
|
'21': '玄幻',
|
||||||
|
'1': '奇幻',
|
||||||
|
'2': '武侠',
|
||||||
|
'22': '仙侠',
|
||||||
|
'4': '都市',
|
||||||
|
'15': '职场',
|
||||||
|
'6': '军事',
|
||||||
|
'5': '历史',
|
||||||
|
'7': '游戏',
|
||||||
|
'8': '体育',
|
||||||
|
'9': '科幻',
|
||||||
|
'10': '灵异',
|
||||||
|
'12': '二次元',
|
||||||
|
}
|
||||||
|
|
||||||
|
async def parse(self, res):
|
||||||
|
result = []
|
||||||
|
res_dic = {}
|
||||||
|
async for item in RankingItem.get_items(html=res.html):
|
||||||
|
each_book_list = []
|
||||||
|
# 只取排名前十的书籍数据
|
||||||
|
for index, value in enumerate(item.book_list[:10]):
|
||||||
|
item_data = await NameItem.get_item(html=value)
|
||||||
|
name = item_data.top_name or item_data.other_name
|
||||||
|
each_book_list.append({
|
||||||
|
'num': index + 1,
|
||||||
|
'name': name
|
||||||
|
})
|
||||||
|
data = {
|
||||||
|
'title': item.ranking_title,
|
||||||
|
'more': item.more,
|
||||||
|
'book_list': each_book_list,
|
||||||
|
'updated_at': time.strftime("%Y-%m-%d %X", time.localtime()),
|
||||||
|
}
|
||||||
|
result.append(data)
|
||||||
|
res_dic['data'] = result
|
||||||
|
res_dic['target_url'] = res.url
|
||||||
|
res_dic['type'] = self.qidian_type.get(res.url.split('=')[-1])
|
||||||
|
res_dic['spider'] = "qidian"
|
||||||
|
await self.save(res_dic=res_dic)
|
||||||
|
|
||||||
|
async def save(self, res_dic):
|
||||||
|
# 存进数据库
|
||||||
|
try:
|
||||||
|
motor_db = MotorBaseOld().db
|
||||||
|
await motor_db.novels_ranking.update_one({
|
||||||
|
'target_url': res_dic['target_url']},
|
||||||
|
{'$set': {
|
||||||
|
'data': res_dic['data'],
|
||||||
|
'spider': res_dic['spider'],
|
||||||
|
'type': res_dic['type'],
|
||||||
|
'finished_at': time.strftime("%Y-%m-%d %X", time.localtime())
|
||||||
|
}},
|
||||||
|
upsert=True)
|
||||||
|
except Exception as e:
|
||||||
|
self.logger.exception(e)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
QidianRankingSpider.start(middleware=middleware)
|
||||||
23
soulbook/spiders/spider_console.py
Normal file
23
soulbook/spiders/spider_console.py
Normal file
@ -0,0 +1,23 @@
|
|||||||
|
# !/usr/bin/env python
|
||||||
|
import os
|
||||||
|
import schedule
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
|
||||||
|
os.environ['MODE'] = 'PRO'
|
||||||
|
sys.path.append('../../')
|
||||||
|
|
||||||
|
from soulbook.spiders import QidianRankingSpider, ZHRankingSpider
|
||||||
|
|
||||||
|
|
||||||
|
def start_spider():
|
||||||
|
QidianRankingSpider.start()
|
||||||
|
ZHRankingSpider.start()
|
||||||
|
|
||||||
|
|
||||||
|
# python novels_schedule.py
|
||||||
|
schedule.every(60).minutes.do(start_spider)
|
||||||
|
|
||||||
|
while True:
|
||||||
|
schedule.run_pending()
|
||||||
|
time.sleep(1)
|
||||||
35
soulbook/spiders/spider_tools.py
Normal file
35
soulbook/spiders/spider_tools.py
Normal file
@ -0,0 +1,35 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""
|
||||||
|
Created by howie.hu at 2018/10/17.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import random
|
||||||
|
|
||||||
|
from ruia import Request
|
||||||
|
|
||||||
|
from soulbook.config import CONFIG
|
||||||
|
|
||||||
|
|
||||||
|
async def get_proxy_ip(valid: int = 1) -> str:
|
||||||
|
# random_server = ['http://0.0.0.0:8662/']
|
||||||
|
proxy_server = CONFIG.REMOTE_SERVER.get('proxy_server')
|
||||||
|
# proxy_server = random_server[random.randint(0, 1)]
|
||||||
|
kwargs = {
|
||||||
|
'json': {
|
||||||
|
"act_id": 1704,
|
||||||
|
"version": "1.0",
|
||||||
|
"data": {
|
||||||
|
"valid": 1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
res = await Request(url=proxy_server, method='POST', res_type='json', **kwargs).fetch()
|
||||||
|
proxy = ''
|
||||||
|
if res.status == 200:
|
||||||
|
proxy = res.html.get('info').get('proxy')
|
||||||
|
return proxy
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
print(asyncio.get_event_loop().run_until_complete(get_proxy_ip()))
|
||||||
75
soulbook/spiders/zh_ranking.py
Normal file
75
soulbook/spiders/zh_ranking.py
Normal file
@ -0,0 +1,75 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""
|
||||||
|
Created by howie.hu at 29/11/2017.
|
||||||
|
"""
|
||||||
|
import time
|
||||||
|
|
||||||
|
from ruia import Spider, Item, AttrField, HtmlField, TextField
|
||||||
|
from ruia_ua import middleware
|
||||||
|
|
||||||
|
from soulbook.database.mongodb import MotorBaseOld
|
||||||
|
|
||||||
|
|
||||||
|
class RankingItem(Item):
|
||||||
|
target_item = TextField(css_select='div.rank_i_p_list')
|
||||||
|
ranking_title = TextField(css_select='div.rank_i_p_tit')
|
||||||
|
more = AttrField(css_select='div.rank_i_more a', attr='href')
|
||||||
|
book_list = HtmlField(css_select='div.rank_i_p_list>div.rank_i_li', many=True)
|
||||||
|
|
||||||
|
|
||||||
|
class NameItem(Item):
|
||||||
|
top_name = TextField(css_select='div.rank_i_bname a.rank_i_l_a_book', default='')
|
||||||
|
other_name = TextField(css_select='div.rank_i_bname a', default='')
|
||||||
|
|
||||||
|
|
||||||
|
class ZHRankingSpider(Spider):
|
||||||
|
start_urls = ['http://book.zongheng.com/rank.html']
|
||||||
|
|
||||||
|
concurrency = 3
|
||||||
|
|
||||||
|
async def parse(self, res):
|
||||||
|
result = []
|
||||||
|
res_dic = {}
|
||||||
|
|
||||||
|
async for item in RankingItem.get_items(html=res.html):
|
||||||
|
each_book_list = []
|
||||||
|
# 只取排名前十的书籍数据
|
||||||
|
for index, value in enumerate(item.book_list[:10]):
|
||||||
|
item_data = await NameItem.get_item(html=value)
|
||||||
|
name = item_data.top_name or item_data.other_name
|
||||||
|
each_book_list.append({
|
||||||
|
'num': index + 1,
|
||||||
|
'name': name
|
||||||
|
})
|
||||||
|
data = {
|
||||||
|
'title': item.ranking_title,
|
||||||
|
'more': item.more,
|
||||||
|
'book_list': each_book_list,
|
||||||
|
'updated_at': time.strftime("%Y-%m-%d %X", time.localtime()),
|
||||||
|
}
|
||||||
|
result.append(data)
|
||||||
|
|
||||||
|
res_dic['data'] = result
|
||||||
|
res_dic['target_url'] = res.url
|
||||||
|
res_dic['type'] = "人气榜单"
|
||||||
|
res_dic['spider'] = "zongheng"
|
||||||
|
await self.save(res_dic)
|
||||||
|
|
||||||
|
async def save(self, res_dic):
|
||||||
|
try:
|
||||||
|
motor_db = MotorBaseOld().db
|
||||||
|
await motor_db.novels_ranking.update_one({
|
||||||
|
'target_url': res_dic['target_url']},
|
||||||
|
{'$set': {
|
||||||
|
'data': res_dic['data'],
|
||||||
|
'spider': res_dic['spider'],
|
||||||
|
'type': res_dic['type'],
|
||||||
|
'finished_at': time.strftime("%Y-%m-%d %X", time.localtime())
|
||||||
|
}},
|
||||||
|
upsert=True)
|
||||||
|
except Exception as e:
|
||||||
|
self.logger.exception(e)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
ZHRankingSpider.start(middleware=middleware)
|
||||||
127
soulbook/spiders/zongheng_all_novels.py
Normal file
127
soulbook/spiders/zongheng_all_novels.py
Normal file
@ -0,0 +1,127 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
"""
|
||||||
|
Created by howie.hu at 14/03/2018.
|
||||||
|
纵横小说信息提取：http://book.zongheng.com/store/c0/c0/b9/u0/p1/v9/s9/t0/ALL.html
|
||||||
|
"""
|
||||||
|
import asyncio
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
|
||||||
|
from ruia import Spider, Item, TextField, AttrField, Request
|
||||||
|
from ruia_ua import middleware as ua_middleware
|
||||||
|
|
||||||
|
# os.environ['MODE'] = 'PRO'
|
||||||
|
from soulbook.database.mongodb import MotorBase
|
||||||
|
from soulbook.spiders.middlewares import owl_middleware
|
||||||
|
|
||||||
|
try:
|
||||||
|
import uvloop
|
||||||
|
|
||||||
|
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
loop = asyncio.get_event_loop()
|
||||||
|
asyncio.set_event_loop(loop)
|
||||||
|
|
||||||
|
|
||||||
|
class ZHNovelsItem(Item):
|
||||||
|
target_item = TextField(css_select='div.store_collist div.bookbox')
|
||||||
|
novel_url = AttrField(css_select='div.bookinfo div.bookname a', attr='href')
|
||||||
|
novel_name = TextField(css_select='div.bookinfo div.bookname a')
|
||||||
|
novel_author = TextField(css_select='div.bookilnk a:nth-child(1)')
|
||||||
|
novel_author_home_url = AttrField(css_select='div.bookilnk a:nth-child(1)', attr='href')
|
||||||
|
novel_type = TextField(css_select='div.bookilnk a:nth-child(2)')
|
||||||
|
novel_cover = AttrField(css_select='div.bookimg img', attr='src')
|
||||||
|
novel_abstract = TextField(css_select='div.bookintro')
|
||||||
|
novel_latest_chapter = TextField(css_select='div.bookupdate a')
|
||||||
|
|
||||||
|
# def tal_novel_url(self, novel_url):
|
||||||
|
# return 'http:' + novel_url
|
||||||
|
|
||||||
|
async def clean_novel_author(self, novel_author):
|
||||||
|
if novel_author:
|
||||||
|
if isinstance(novel_author, list):
|
||||||
|
novel_author = novel_author[0].text
|
||||||
|
return novel_author
|
||||||
|
else:
|
||||||
|
return ''
|
||||||
|
|
||||||
|
# def tal_novel_author_home_url(self, novel_author_home_url):
|
||||||
|
# if isinstance(novel_author_home_url, list):
|
||||||
|
# novel_author_home_url = novel_author_home_url[0].get('href').strip()
|
||||||
|
# return 'http:' + novel_author_home_url
|
||||||
|
|
||||||
|
|
||||||
|
class ZHNovelsSpider(Spider):
|
||||||
|
start_urls = ['http://book.zongheng.com/store/c0/c0/b9/u0/p1/v9/s9/t0/ALL.html']
|
||||||
|
|
||||||
|
request_config = {
|
||||||
|
'RETRIES': 8,
|
||||||
|
'DELAY': 0,
|
||||||
|
'TIMEOUT': 3
|
||||||
|
}
|
||||||
|
concurrency = 60
|
||||||
|
motor_db = MotorBase(loop=loop).get_db()
|
||||||
|
|
||||||
|
async def parse(self, res):
|
||||||
|
items_data = await ZHNovelsItem.get_items(html=res.html)
|
||||||
|
tasks = []
|
||||||
|
for item in items_data:
|
||||||
|
if item.novel_url:
|
||||||
|
res_dic = {
|
||||||
|
'novel_url': item.novel_url,
|
||||||
|
'novel_name': item.novel_name,
|
||||||
|
'novel_author': item.novel_author,
|
||||||
|
'novel_author_home_url': item.novel_author_home_url,
|
||||||
|
'novel_type': item.novel_type,
|
||||||
|
'novel_cover': item.novel_cover,
|
||||||
|
'novel_abstract': item.novel_abstract,
|
||||||
|
'novel_latest_chapter': item.novel_latest_chapter,
|
||||||
|
'spider': 'zongheng',
|
||||||
|
'updated_at': time.strftime("%Y-%m-%d %X", time.localtime()),
|
||||||
|
}
|
||||||
|
tasks.append(asyncio.ensure_future(self.save(res_dic)))
|
||||||
|
# if self.all_novels_col.find_one(
|
||||||
|
# {"novel_name": item.novel_name, 'novel_author': item.novel_author}) is None:
|
||||||
|
# self.all_novels_col.insert_one(res_dic)
|
||||||
|
# # async_callback(self.save, res_dic=res_dic)
|
||||||
|
# print(item.novel_name + ' - 抓取成功')
|
||||||
|
good_nums = 0
|
||||||
|
if tasks:
|
||||||
|
done_list, pending_list = await asyncio.wait(tasks)
|
||||||
|
for task in done_list:
|
||||||
|
if task.result():
|
||||||
|
good_nums += 1
|
||||||
|
print(f"共{len(tasks)}本小说，抓取成功{good_nums}本")
|
||||||
|
|
||||||
|
async def save(self, res_dic):
|
||||||
|
# 存进数据库
|
||||||
|
res_dic = res_dic
|
||||||
|
try:
|
||||||
|
|
||||||
|
await self.motor_db.all_novels.update_one({
|
||||||
|
'novel_url': res_dic['novel_url'], 'novel_name': res_dic['novel_name']},
|
||||||
|
{'$set': res_dic},
|
||||||
|
upsert=True)
|
||||||
|
print(res_dic['novel_name'] + ' - 抓取成功')
|
||||||
|
return True
|
||||||
|
except Exception as e:
|
||||||
|
self.logger.exception(e)
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
# 其他多item示例：https://gist.github.com/howie6879/3ef4168159e5047d42d86cb7fb706a2f
|
||||||
|
# 51793
|
||||||
|
for page in range(0, 10):
|
||||||
|
print(f"正在爬取第{page}页")
|
||||||
|
start_page = page * 100
|
||||||
|
end_page = start_page + 100
|
||||||
|
if end_page > 999:
|
||||||
|
end_page = 999
|
||||||
|
ZHNovelsSpider.start_urls = ['http://book.zongheng.com/store/c0/c0/b9/u0/p{i}/v9/s9/t0/ALL.html'.format(i=i) for
|
||||||
|
i in
|
||||||
|
range(start_page, end_page)]
|
||||||
|
# 其他多item示例：https://gist.github.com/howie6879/3ef4168159e5047d42d86cb7fb706a2f
|
||||||
|
ZHNovelsSpider.start(loop=loop, middleware=[ua_middleware, owl_middleware], close_event_loop=False)
|
||||||
118
soulbook/spiders/zongheng_novel_info.py
Normal file
118
soulbook/spiders/zongheng_novel_info.py
Normal file
@ -0,0 +1,118 @@
|
|||||||
|
# -*- coding:utf-8 -*-
|
||||||
|
# !/usr/bin/env python
|
||||||
|
import asyncio
|
||||||
|
import time
|
||||||
|
|
||||||
|
from pprint import pprint
|
||||||
|
|
||||||
|
from ruia import Spider, Item, TextField, AttrField
|
||||||
|
|
||||||
|
from ruia_ua import middleware as ua_middleware
|
||||||
|
|
||||||
|
from soulbook.database.mongodb import MotorBase
|
||||||
|
from soulbook.spiders.middlewares import owl_middleware
|
||||||
|
|
||||||
|
try:
|
||||||
|
import uvloop
|
||||||
|
|
||||||
|
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
loop = asyncio.get_event_loop()
|
||||||
|
asyncio.set_event_loop(loop)
|
||||||
|
|
||||||
|
|
||||||
|
class ZHNovelInfoItem(Item):
|
||||||
|
"""
|
||||||
|
定义继承自item的Item类
|
||||||
|
"""
|
||||||
|
novel_name = TextField(css_select='div.main div.status h1 a')
|
||||||
|
author = TextField(css_select='div.main div.status div.booksub a')
|
||||||
|
# 当提取的值是属性的时候，要定义AttrField
|
||||||
|
cover = AttrField(css_select='div.main div.book_cover img', attr='src')
|
||||||
|
abstract = TextField(css_select='div.main div.status div.info_con p')
|
||||||
|
status = AttrField(css_select='div.main div.status h1 em', attr='title')
|
||||||
|
novels_type = TextField(css_select='div.main div.status div.booksub a')
|
||||||
|
novel_chapter_url = AttrField(css_select='div.main div.status div.book_btn span.list a', attr='href')
|
||||||
|
|
||||||
|
async def clean_author(self, author):
|
||||||
|
if isinstance(author, list):
|
||||||
|
return author[0].text
|
||||||
|
else:
|
||||||
|
return author
|
||||||
|
|
||||||
|
async def clean_status(self, status):
|
||||||
|
"""
|
||||||
|
当目标值的对象只有一个，默认将值提取出来，否则返回list，可以在这里定义一个函数进行循环提取
|
||||||
|
:param ele_tag:
|
||||||
|
:return:
|
||||||
|
"""
|
||||||
|
if isinstance(status, list):
|
||||||
|
return '#'.join([i.get('title').strip().replace('作品', '') for i in status])
|
||||||
|
else:
|
||||||
|
return status
|
||||||
|
|
||||||
|
async def clean_novels_type(self, novels_type):
|
||||||
|
if isinstance(novels_type, list):
|
||||||
|
try:
|
||||||
|
return novels_type[1].text
|
||||||
|
except:
|
||||||
|
return ''
|
||||||
|
else:
|
||||||
|
return ''
|
||||||
|
|
||||||
|
|
||||||
|
class ZHNovelInfoSpider(Spider):
|
||||||
|
start_urls = []
|
||||||
|
request_config = {
|
||||||
|
'RETRIES': 3,
|
||||||
|
'DELAY': 2,
|
||||||
|
'TIMEOUT': 10
|
||||||
|
}
|
||||||
|
motor_db = MotorBase(loop=loop).get_db()
|
||||||
|
|
||||||
|
async def parse(self, res):
|
||||||
|
item = await ZHNovelInfoItem.get_item(html=res.html)
|
||||||
|
|
||||||
|
item_data = {
|
||||||
|
'novel_name': item.novel_name,
|
||||||
|
'author': item.author,
|
||||||
|
'cover': item.cover,
|
||||||
|
'abstract': item.abstract,
|
||||||
|
'status': item.status,
|
||||||
|
'novels_type': item.novels_type,
|
||||||
|
'novel_chapter_url': item.novel_chapter_url,
|
||||||
|
'target_url': res.url,
|
||||||
|
'spider': 'zongheng',
|
||||||
|
'updated_at': time.strftime("%Y-%m-%d %X", time.localtime()),
|
||||||
|
}
|
||||||
|
|
||||||
|
print('获取 {} 小说信息成功'.format(item_data['novel_name']))
|
||||||
|
print(item_data)
|
||||||
|
await self.motor_db.all_novels_info.update_one(
|
||||||
|
{'novel_name': item_data['novel_name'], 'spider': 'zongheng'},
|
||||||
|
{'$set': item_data},
|
||||||
|
upsert=True)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
import random
|
||||||
|
|
||||||
|
# 其他多item示例：https://gist.github.com/howie6879/3ef4168159e5047d42d86cb7fb706a2f
|
||||||
|
ZHNovelInfoSpider.start_urls = ['http://book.zongheng.com/book/672340.html']
|
||||||
|
ZHNovelInfoSpider.start(middleware=[ua_middleware, owl_middleware])
|
||||||
|
|
||||||
|
# def all_novels_info():
|
||||||
|
# all_urls = []
|
||||||
|
#
|
||||||
|
# for each in ZHNovelInfoSpider.all_novels_col.find({'spider': 'zongheng'}):
|
||||||
|
# if 'zongheng' in each['novel_url']:
|
||||||
|
# all_urls.append(each['novel_url'])
|
||||||
|
# random.shuffle(all_urls)
|
||||||
|
#
|
||||||
|
# ZHNovelInfoSpider.start_urls = all_urls
|
||||||
|
# ZHNovelInfoSpider.start()
|
||||||
|
#
|
||||||
|
#
|
||||||
|
# all_novels_info()
|
||||||
3
soulbook/start.sh
Normal file
3
soulbook/start.sh
Normal file
@ -0,0 +1,3 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
ps -ef | grep "gunicorn -c config/gunicorn.py --worker-class sanic.worker.GunicornWorker server:app" | grep -v grep | awk '{print $2}' | xargs kill -9
|
||||||
|
nohup gunicorn -c config/gunicorn.py --worker-class sanic.worker.GunicornWorker server:app >/dev/null &2>1 &
|
||||||
41
soulbook/static/md/css/md.css
Normal file
41
soulbook/static/md/css/md.css
Normal file
@ -0,0 +1,41 @@
|
|||||||
|
th, td {
|
||||||
|
text-align: center !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
table a {
|
||||||
|
color: #337ab7 !important;
|
||||||
|
text-decoration: none !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
table a:hover {
|
||||||
|
color: black !important;
|
||||||
|
}
|
||||||
|
|
||||||
|
.mdui-panel-fluid {
|
||||||
|
margin-top: 2%;
|
||||||
|
width: 80%;
|
||||||
|
margin-left: 10%
|
||||||
|
}
|
||||||
|
|
||||||
|
.email-error, .pass-error {
|
||||||
|
height: 12px;
|
||||||
|
margin-top: 8px;
|
||||||
|
overflow: hidden;
|
||||||
|
font-size: 12px;
|
||||||
|
line-height: 12px;
|
||||||
|
color: #f44336;
|
||||||
|
text-overflow: ellipsis;
|
||||||
|
white-space: nowrap;
|
||||||
|
visibility: hidden;
|
||||||
|
}
|
||||||
|
|
||||||
|
.owl-ranking {
|
||||||
|
max-width: 300px;
|
||||||
|
height: 500px;
|
||||||
|
margin: 5px;
|
||||||
|
}
|
||||||
|
|
||||||
|
.author-name-tag{
|
||||||
|
margin-left: 10%;
|
||||||
|
margin-top: 13px;
|
||||||
|
}
|
||||||
6
soulbook/static/md/css/mdui.min0.2.1.css
Normal file
6
soulbook/static/md/css/mdui.min0.2.1.css
Normal file
File diff suppressed because one or more lines are too long
201
soulbook/static/md/fonts/roboto/LICENSE.txt
Normal file
201
soulbook/static/md/fonts/roboto/LICENSE.txt
Normal file
@ -0,0 +1,201 @@
|
|||||||
|
Apache License
|
||||||
|
Version 2.0, January 2004
|
||||||
|
http://www.apache.org/licenses/
|
||||||
|
|
||||||
|
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||||
|
|
||||||
|
1. Definitions.
|
||||||
|
|
||||||
|
"License" shall mean the terms and conditions for use, reproduction,
|
||||||
|
and distribution as defined by Sections 1 through 9 of this document.
|
||||||
|
|
||||||
|
"Licensor" shall mean the copyright owner or entity authorized by
|
||||||
|
the copyright owner that is granting the License.
|
||||||
|
|
||||||
|
"Legal Entity" shall mean the union of the acting entity and all
|
||||||
|
other entities that control, are controlled by, or are under common
|
||||||
|
control with that entity. For the purposes of this definition,
|
||||||
|
"control" means (i) the power, direct or indirect, to cause the
|
||||||
|
direction or management of such entity, whether by contract or
|
||||||
|
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||||
|
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||||
|
|
||||||
|
"You" (or "Your") shall mean an individual or Legal Entity
|
||||||
|
exercising permissions granted by this License.
|
||||||
|
|
||||||
|
"Source" form shall mean the preferred form for making modifications,
|
||||||
|
including but not limited to software source code, documentation
|
||||||
|
source, and configuration files.
|
||||||
|
|
||||||
|
"Object" form shall mean any form resulting from mechanical
|
||||||
|
transformation or translation of a Source form, including but
|
||||||
|
not limited to compiled object code, generated documentation,
|
||||||
|
and conversions to other media types.
|
||||||
|
|
||||||
|
"Work" shall mean the work of authorship, whether in Source or
|
||||||
|
Object form, made available under the License, as indicated by a
|
||||||
|
copyright notice that is included in or attached to the work
|
||||||
|
(an example is provided in the Appendix below).
|
||||||
|
|
||||||
|
"Derivative Works" shall mean any work, whether in Source or Object
|
||||||
|
form, that is based on (or derived from) the Work and for which the
|
||||||
|
editorial revisions, annotations, elaborations, or other modifications
|
||||||
|
represent, as a whole, an original work of authorship. For the purposes
|
||||||
|
of this License, Derivative Works shall not include works that remain
|
||||||
|
separable from, or merely link (or bind by name) to the interfaces of,
|
||||||
|
the Work and Derivative Works thereof.
|
||||||
|
|
||||||
|
"Contribution" shall mean any work of authorship, including
|
||||||
|
the original version of the Work and any modifications or additions
|
||||||
|
to that Work or Derivative Works thereof, that is intentionally
|
||||||
|
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||||
|
or by an individual or Legal Entity authorized to submit on behalf of
|
||||||
|
the copyright owner. For the purposes of this definition, "submitted"
|
||||||
|
means any form of electronic, verbal, or written communication sent
|
||||||
|
to the Licensor or its representatives, including but not limited to
|
||||||
|
communication on electronic mailing lists, source code control systems,
|
||||||
|
and issue tracking systems that are managed by, or on behalf of, the
|
||||||
|
Licensor for the purpose of discussing and improving the Work, but
|
||||||
|
excluding communication that is conspicuously marked or otherwise
|
||||||
|
designated in writing by the copyright owner as "Not a Contribution."
|
||||||
|
|
||||||
|
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||||
|
on behalf of whom a Contribution has been received by Licensor and
|
||||||
|
subsequently incorporated within the Work.
|
||||||
|
|
||||||
|
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||||
|
this License, each Contributor hereby grants to You a perpetual,
|
||||||
|
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||||
|
copyright license to reproduce, prepare Derivative Works of,
|
||||||
|
publicly display, publicly perform, sublicense, and distribute the
|
||||||
|
Work and such Derivative Works in Source or Object form.
|
||||||
|
|
||||||
|
3. Grant of Patent License. Subject to the terms and conditions of
|
||||||
|
this License, each Contributor hereby grants to You a perpetual,
|
||||||
|
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||||
|
(except as stated in this section) patent license to make, have made,
|
||||||
|
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||||
|
where such license applies only to those patent claims licensable
|
||||||
|
by such Contributor that are necessarily infringed by their
|
||||||
|
Contribution(s) alone or by combination of their Contribution(s)
|
||||||
|
with the Work to which such Contribution(s) was submitted. If You
|
||||||
|
institute patent litigation against any entity (including a
|
||||||
|
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||||
|
or a Contribution incorporated within the Work constitutes direct
|
||||||
|
or contributory patent infringement, then any patent licenses
|
||||||
|
granted to You under this License for that Work shall terminate
|
||||||
|
as of the date such litigation is filed.
|
||||||
|
|
||||||
|
4. Redistribution. You may reproduce and distribute copies of the
|
||||||
|
Work or Derivative Works thereof in any medium, with or without
|
||||||
|
modifications, and in Source or Object form, provided that You
|
||||||
|
meet the following conditions:
|
||||||
|
|
||||||
|
(a) You must give any other recipients of the Work or
|
||||||
|
Derivative Works a copy of this License; and
|
||||||
|
|
||||||
|
(b) You must cause any modified files to carry prominent notices
|
||||||
|
stating that You changed the files; and
|
||||||
|
|
||||||
|
(c) You must retain, in the Source form of any Derivative Works
|
||||||
|
that You distribute, all copyright, patent, trademark, and
|
||||||
|
attribution notices from the Source form of the Work,
|
||||||
|
excluding those notices that do not pertain to any part of
|
||||||
|
the Derivative Works; and
|
||||||
|
|
||||||
|
(d) If the Work includes a "NOTICE" text file as part of its
|
||||||
|
distribution, then any Derivative Works that You distribute must
|
||||||
|
include a readable copy of the attribution notices contained
|
||||||
|
within such NOTICE file, excluding those notices that do not
|
||||||
|
pertain to any part of the Derivative Works, in at least one
|
||||||
|
of the following places: within a NOTICE text file distributed
|
||||||
|
as part of the Derivative Works; within the Source form or
|
||||||
|
documentation, if provided along with the Derivative Works; or,
|
||||||
|
within a display generated by the Derivative Works, if and
|
||||||
|
wherever such third-party notices normally appear. The contents
|
||||||
|
of the NOTICE file are for informational purposes only and
|
||||||
|
do not modify the License. You may add Your own attribution
|
||||||
|
notices within Derivative Works that You distribute, alongside
|
||||||
|
or as an addendum to the NOTICE text from the Work, provided
|
||||||
|
that such additional attribution notices cannot be construed
|
||||||
|
as modifying the License.
|
||||||
|
|
||||||
|
You may add Your own copyright statement to Your modifications and
|
||||||
|
may provide additional or different license terms and conditions
|
||||||
|
for use, reproduction, or distribution of Your modifications, or
|
||||||
|
for any such Derivative Works as a whole, provided Your use,
|
||||||
|
reproduction, and distribution of the Work otherwise complies with
|
||||||
|
the conditions stated in this License.
|
||||||
|
|
||||||
|
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||||
|
any Contribution intentionally submitted for inclusion in the Work
|
||||||
|
by You to the Licensor shall be under the terms and conditions of
|
||||||
|
this License, without any additional terms or conditions.
|
||||||
|
Notwithstanding the above, nothing herein shall supersede or modify
|
||||||
|
the terms of any separate license agreement you may have executed
|
||||||
|
with Licensor regarding such Contributions.
|
||||||
|
|
||||||
|
6. Trademarks. This License does not grant permission to use the trade
|
||||||
|
names, trademarks, service marks, or product names of the Licensor,
|
||||||
|
except as required for reasonable and customary use in describing the
|
||||||
|
origin of the Work and reproducing the content of the NOTICE file.
|
||||||
|
|
||||||
|
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||||
|
agreed to in writing, Licensor provides the Work (and each
|
||||||
|
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||||
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||||
|
implied, including, without limitation, any warranties or conditions
|
||||||
|
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||||
|
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||||
|
appropriateness of using or redistributing the Work and assume any
|
||||||
|
risks associated with Your exercise of permissions under this License.
|
||||||
|
|
||||||
|
8. Limitation of Liability. In no event and under no legal theory,
|
||||||
|
whether in tort (including negligence), contract, or otherwise,
|
||||||
|
unless required by applicable law (such as deliberate and grossly
|
||||||
|
negligent acts) or agreed to in writing, shall any Contributor be
|
||||||
|
liable to You for damages, including any direct, indirect, special,
|
||||||
|
incidental, or consequential damages of any character arising as a
|
||||||
|
result of this License or out of the use or inability to use the
|
||||||
|
Work (including but not limited to damages for loss of goodwill,
|
||||||
|
work stoppage, computer failure or malfunction, or any and all
|
||||||
|
other commercial damages or losses), even if such Contributor
|
||||||
|
has been advised of the possibility of such damages.
|
||||||
|
|
||||||
|
9. Accepting Warranty or Additional Liability. While redistributing
|
||||||
|
the Work or Derivative Works thereof, You may choose to offer,
|
||||||
|
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||||
|
or other liability obligations and/or rights consistent with this
|
||||||
|
License. However, in accepting such obligations, You may act only
|
||||||
|
on Your own behalf and on Your sole responsibility, not on behalf
|
||||||
|
of any other Contributor, and only if You agree to indemnify,
|
||||||
|
defend, and hold each Contributor harmless for any liability
|
||||||
|
incurred by, or claims asserted against, such Contributor by reason
|
||||||
|
of your accepting any such warranty or additional liability.
|
||||||
|
|
||||||
|
END OF TERMS AND CONDITIONS
|
||||||
|
|
||||||
|
APPENDIX: How to apply the Apache License to your work.
|
||||||
|
|
||||||
|
To apply the Apache License to your work, attach the following
|
||||||
|
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||||
|
replaced with your own identifying information. (Don't include
|
||||||
|
the brackets!) The text should be enclosed in the appropriate
|
||||||
|
comment syntax for the file format. We also recommend that a
|
||||||
|
file or class name and description of purpose be included on the
|
||||||
|
same "printed page" as the copyright notice for easier
|
||||||
|
identification within third-party archives.
|
||||||
|
|
||||||
|
Copyright [yyyy] [name of copyright owner]
|
||||||
|
|
||||||
|
Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
you may not use this file except in compliance with the License.
|
||||||
|
You may obtain a copy of the License at
|
||||||
|
|
||||||
|
http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
|
||||||
|
Unless required by applicable law or agreed to in writing, software
|
||||||
|
distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
See the License for the specific language governing permissions and
|
||||||
|
limitations under the License.
|
||||||
BIN
soulbook/static/md/fonts/roboto/Roboto-Black.woff
Normal file
BIN
soulbook/static/md/fonts/roboto/Roboto-Black.woff
Normal file
Binary file not shown.
BIN
soulbook/static/md/fonts/roboto/Roboto-Black.woff2
Normal file
BIN
soulbook/static/md/fonts/roboto/Roboto-Black.woff2
Normal file
Binary file not shown.
BIN
soulbook/static/md/fonts/roboto/Roboto-BlackItalic.woff
Normal file
BIN
soulbook/static/md/fonts/roboto/Roboto-BlackItalic.woff
Normal file
Binary file not shown.
BIN
soulbook/static/md/fonts/roboto/Roboto-BlackItalic.woff2
Normal file
BIN
soulbook/static/md/fonts/roboto/Roboto-BlackItalic.woff2
Normal file
Binary file not shown.
BIN
soulbook/static/md/fonts/roboto/Roboto-Bold.woff
Normal file
BIN
soulbook/static/md/fonts/roboto/Roboto-Bold.woff
Normal file
Binary file not shown.
BIN
soulbook/static/md/fonts/roboto/Roboto-Bold.woff2
Normal file
BIN
soulbook/static/md/fonts/roboto/Roboto-Bold.woff2
Normal file
Binary file not shown.
BIN
soulbook/static/md/fonts/roboto/Roboto-BoldItalic.woff
Normal file
BIN
soulbook/static/md/fonts/roboto/Roboto-BoldItalic.woff
Normal file
Binary file not shown.
BIN
soulbook/static/md/fonts/roboto/Roboto-BoldItalic.woff2
Normal file
BIN
soulbook/static/md/fonts/roboto/Roboto-BoldItalic.woff2
Normal file
Binary file not shown.
BIN
soulbook/static/md/fonts/roboto/Roboto-Light.woff
Normal file
BIN
soulbook/static/md/fonts/roboto/Roboto-Light.woff
Normal file
Binary file not shown.
BIN
soulbook/static/md/fonts/roboto/Roboto-Light.woff2
Normal file
BIN
soulbook/static/md/fonts/roboto/Roboto-Light.woff2
Normal file
Binary file not shown.
BIN
soulbook/static/md/fonts/roboto/Roboto-LightItalic.woff
Normal file
BIN
soulbook/static/md/fonts/roboto/Roboto-LightItalic.woff
Normal file
Binary file not shown.
BIN
soulbook/static/md/fonts/roboto/Roboto-LightItalic.woff2
Normal file
BIN
soulbook/static/md/fonts/roboto/Roboto-LightItalic.woff2
Normal file
Binary file not shown.
BIN
soulbook/static/md/fonts/roboto/Roboto-Medium.woff
Normal file
BIN
soulbook/static/md/fonts/roboto/Roboto-Medium.woff
Normal file
Binary file not shown.
BIN
soulbook/static/md/fonts/roboto/Roboto-Medium.woff2
Normal file
BIN
soulbook/static/md/fonts/roboto/Roboto-Medium.woff2
Normal file
Binary file not shown.
BIN
soulbook/static/md/fonts/roboto/Roboto-MediumItalic.woff
Normal file
BIN
soulbook/static/md/fonts/roboto/Roboto-MediumItalic.woff
Normal file
Binary file not shown.
BIN
soulbook/static/md/fonts/roboto/Roboto-MediumItalic.woff2
Normal file
BIN
soulbook/static/md/fonts/roboto/Roboto-MediumItalic.woff2
Normal file
Binary file not shown.
BIN
soulbook/static/md/fonts/roboto/Roboto-Regular.woff
Normal file
BIN
soulbook/static/md/fonts/roboto/Roboto-Regular.woff
Normal file
Binary file not shown.
BIN
soulbook/static/md/fonts/roboto/Roboto-Regular.woff2
Normal file
BIN
soulbook/static/md/fonts/roboto/Roboto-Regular.woff2
Normal file
Binary file not shown.
BIN
soulbook/static/md/fonts/roboto/Roboto-RegularItalic.woff
Normal file
BIN
soulbook/static/md/fonts/roboto/Roboto-RegularItalic.woff
Normal file
Binary file not shown.
BIN
soulbook/static/md/fonts/roboto/Roboto-RegularItalic.woff2
Normal file
BIN
soulbook/static/md/fonts/roboto/Roboto-RegularItalic.woff2
Normal file
Binary file not shown.
BIN
soulbook/static/md/fonts/roboto/Roboto-Thin.woff
Normal file
BIN
soulbook/static/md/fonts/roboto/Roboto-Thin.woff
Normal file
Binary file not shown.
BIN
soulbook/static/md/fonts/roboto/Roboto-Thin.woff2
Normal file
BIN
soulbook/static/md/fonts/roboto/Roboto-Thin.woff2
Normal file
Binary file not shown.
BIN
soulbook/static/md/fonts/roboto/Roboto-ThinItalic.woff
Normal file
BIN
soulbook/static/md/fonts/roboto/Roboto-ThinItalic.woff
Normal file
Binary file not shown.
BIN
soulbook/static/md/fonts/roboto/Roboto-ThinItalic.woff2
Normal file
BIN
soulbook/static/md/fonts/roboto/Roboto-ThinItalic.woff2
Normal file
Binary file not shown.
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue
Block a user