Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(route): 新增 bing 搜索; 迁移 bing 每日图片到 v2; 新增 百度搜索; 迁移 搜狗特色LOGO 到 v2 规范;添加 搜狗搜索;添加 Google Search #13936

Merged
merged 32 commits into from
Dec 25, 2023
Merged
Show file tree
Hide file tree
Changes from 27 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
83a7846
fix(route): 修复 米游社 公告栏 template 错误
CaoMeiYouRen Sep 5, 2023
7b65948
Merge branch 'DIYgod:master' into master
CaoMeiYouRen Dec 1, 2023
2ea08ff
feat(route): 新增 bing 搜索
CaoMeiYouRen Dec 1, 2023
de072b3
docs: Update other.mdx
CaoMeiYouRen Dec 1, 2023
98aee17
docs: fix docs
TonyRL Dec 1, 2023
13b91e1
feat(route): 新增 百度搜索
CaoMeiYouRen Dec 1, 2023
32d028a
Merge branch 'master' of github.com:CaoMeiYouRen/RSSHub
CaoMeiYouRen Dec 1, 2023
dd283a2
fix(route): 修复 pubDate 解析错误
CaoMeiYouRen Dec 1, 2023
288b606
Merge branch 'master' of github.com:DIYgod/RSSHub
CaoMeiYouRen Dec 1, 2023
b0b6b4d
fix(route): 优化 百度搜索的缓存,减轻反爬问题
CaoMeiYouRen Dec 1, 2023
e9afda3
feat(route): 新增 360 搜索
CaoMeiYouRen Dec 4, 2023
3171faa
Merge branch 'master' of github.com:DIYgod/RSSHub
CaoMeiYouRen Dec 5, 2023
7abe59c
feat(route): 迁移 搜狗特色LOGO 到 v2 规范;添加 搜狗搜索
CaoMeiYouRen Dec 5, 2023
4751557
Merge branch 'master' of github.com:DIYgod/RSSHub
CaoMeiYouRen Dec 10, 2023
7f68ac2
fix(route): 百度搜索增加图片
CaoMeiYouRen Dec 10, 2023
12d0477
feat(route): 新增 Google Search
CaoMeiYouRen Dec 10, 2023
f81821d
chore: merge branch 'master' of github.com:DIYgod/RSSHub
CaoMeiYouRen Dec 16, 2023
05ce45c
Merge branch 'master' of github.com:DIYgod/RSSHub
CaoMeiYouRen Dec 19, 2023
1185a64
fix(route): 修复 百度搜索相关问题
CaoMeiYouRen Dec 19, 2023
58c722d
fix(route): 修复 Google 相关问题
CaoMeiYouRen Dec 19, 2023
1b00eb6
fix(route): 修复 360 搜索
CaoMeiYouRen Dec 19, 2023
08dca4d
fix(route): 修复 搜狗搜索
CaoMeiYouRen Dec 19, 2023
90b1aa7
fix(route): 修复 await 问题
CaoMeiYouRen Dec 19, 2023
da49d03
Merge branch 'master' of github.com:DIYgod/RSSHub
CaoMeiYouRen Dec 19, 2023
931b769
fix: 移除 google sites
CaoMeiYouRen Dec 19, 2023
a997c01
Merge branch 'master' of github.com:DIYgod/RSSHub
CaoMeiYouRen Dec 23, 2023
22923a5
fix(route): 修复 缓存和过滤逻辑问题
CaoMeiYouRen Dec 23, 2023
90ebcd6
Merge branch 'master' of github.com:DIYgod/RSSHub
CaoMeiYouRen Dec 25, 2023
cd5d832
fix(route): 修复 360 搜索缺少 cookie 的问题
CaoMeiYouRen Dec 25, 2023
31118c2
fix(route): 修复 360 搜索 cookie 的问题
CaoMeiYouRen Dec 25, 2023
219f9d8
feat(route): 移除 so.com 路由
CaoMeiYouRen Dec 25, 2023
c548af5
fix: merge conflict
TonyRL Dec 25, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions lib/router.js
Original file line number Diff line number Diff line change
Expand Up @@ -376,7 +376,7 @@ router.get('/autotrader/:query', lazyloadRouteHandler('./routes/autotrader'));
router.get('/geekpark/breakingnews', lazyloadRouteHandler('./routes/geekpark/breakingnews'));

// 搜狗
router.get('/sogou/doodles', lazyloadRouteHandler('./routes/sogou/doodles'));
// router.get('/sogou/doodles', lazyloadRouteHandler('./routes/sogou/doodles'));

// 香港天文台
router.get('/hko/weather', lazyloadRouteHandler('./routes/hko/weather'));
Expand Down Expand Up @@ -690,7 +690,7 @@ router.get('/digitaling/articles/:category/:subcate?', lazyloadRouteHandler('./r
router.get('/digitaling/projects/:category', lazyloadRouteHandler('./routes/digitaling/project'));

// Bing壁纸
router.get('/bing', lazyloadRouteHandler('./routes/bing/index'));
// router.get('/bing', lazyloadRouteHandler('./routes/bing/index'));

// AlgoCasts
router.get('/algocasts', lazyloadRouteHandler('./routes/algocasts/all'));
Expand Down
1 change: 1 addition & 0 deletions lib/v2/baidu/maintainer.js
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
module.exports = {
'/gushitong/index': ['CaoMeiYouRen'],
'/search/:keyword': ['CaoMeiYouRen'],
'/tieba/forum/good/:kw/:cid?/:sortBy?': ['u3u'],
'/tieba/forum/:kw/:sortBy?': ['u3u'],
'/tieba/post/:id': ['u3u'],
Expand Down
13 changes: 12 additions & 1 deletion lib/v2/baidu/radar.js
Original file line number Diff line number Diff line change
@@ -1,6 +1,17 @@
module.exports = {
'baidu.com': {
_name: '百度',
www: [
{
title: '搜索',
docs: 'https://docs.rsshub.app/routes/other#bai-du-sou-suo',
source: ['/'],
target: (params, url) => {
const keyword = new URL(url).searchParams.get('wd');
return `/baidu/search/${keyword}`;
},
},
],
gushitong: [
{
title: '首页指数',
Expand Down Expand Up @@ -73,7 +84,7 @@ module.exports = {
top: [
{
title: '热搜榜单',
docs: 'https://docs.rsshub.app/routes/other#bai-du-re-sou',
docs: 'https://docs.rsshub.app/routes/other#bai-du-re-sou-bang-dan',
source: ['/board'],
target: (_, url) => `/baidu/top/${new URL(url).searchParams.get('tab')}`,
},
Expand Down
1 change: 1 addition & 0 deletions lib/v2/baidu/router.js
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
module.exports = (router) => {
router.get('/gushitong/index', require('./gushitong'));
router.get('/search/:keyword', require('./search'));
router.get('/tieba/forum/good/:kw/:cid?/:sortBy?', require('./tieba/forum'));
router.get('/tieba/forum/:kw/:sortBy?', require('./tieba/forum'));
router.get('/tieba/post/:id', require('./tieba/post'));
Expand Down
54 changes: 54 additions & 0 deletions lib/v2/baidu/search.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
const got = require('@/utils/got');
const cheerio = require('cheerio');
const { art } = require('@/utils/render');
const path = require('path');
const renderDescription = (description, images) => art(path.join(__dirname, './templates/description.art'), { description, images });
const config = require('@/config').value;

module.exports = async (ctx) => {
const { keyword } = ctx.params;
const url = `https://www.baidu.com/s?wd=${encodeURIComponent(keyword)}`;
const key = `baidu-search:${url}`;

const items = await ctx.cache.tryGet(
key,
async () => {
const response = (await got(url)).data;
const visitedLinks = new Set();
const $ = cheerio.load(response);
const contentLeft = $('#content_left');
const containers = contentLeft.find('.c-container');
return containers
.map((i, el) => {
const element = $(el);
const link = element.find('h3 a').first().attr('href');
if (link && !visitedLinks.has(link)) {
visitedLinks.add(link);
const imgs = element
.find('img')
.map((_j, _el) => $(_el).attr('src'))
.toArray();
const description = element.find('.c-gap-top-small [class^="content-right_"]').first().text() || element.find('.c-row').first().text() || element.find('.cos-row').first().text();
return {
title: element.find('h3').first().text(),
description: renderDescription(description, imgs),
link: element.find('h3 a').first().attr('href'),
author: element.find('.c-row .c-color-gray').first().text() || '',
};
}
return null;
})
.toArray()
.filter((e) => e?.link);
},
config.cache.routeExpire,
false
);

ctx.state.data = {
title: `${keyword} - 百度搜索`,
description: `${keyword} - 百度搜索`,
link: url,
item: items,
};
};
6 changes: 6 additions & 0 deletions lib/v2/baidu/templates/description.art
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{{@ description }}
{{if images}}
{{each images}}
<img src="{{ $value }}">
{{/each}}
{{/if}}
9 changes: 4 additions & 5 deletions lib/routes/bing/index.js → lib/v2/bing/daily-wallpaper.js
Original file line number Diff line number Diff line change
@@ -1,22 +1,21 @@
const got = require('@/utils/got');
const queryString = require('query-string');

module.exports = async (ctx) => {
const response = await got({
method: 'get',
prefixUrl: 'https://cn.bing.com',
url: 'HPImageArchive.aspx',
searchParams: queryString.stringify({
searchParams: {
format: 'js',
idx: 0,
n: 7,
n: ctx.query.limit ? parseInt(ctx.query.limit, 10) : 7,
mkt: 'zh-CN',
}),
},
});
const data = response.data;
ctx.state.data = {
title: 'Bing每日壁纸',
link: `https://cn.bing.com/`,
link: 'https://cn.bing.com/',
item: data.images.map((item) => ({
title: item.copyright,
description: `<img src="https://cn.bing.com${item.url}">`,
Expand Down
4 changes: 4 additions & 0 deletions lib/v2/bing/maintainer.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
module.exports = {
'/': ['FHYunCai'],
'/search/:keyword': ['CaoMeiYouRen'],
};
22 changes: 22 additions & 0 deletions lib/v2/bing/radar.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
module.exports = {
'bing.com': {
_name: 'Bing',
cn: [
{
title: '每日壁纸',
docs: 'https://docs.rsshub.app/routes/picture#bing',
source: '/',
target: '/bing',
},
{
title: '搜索',
docs: 'https://docs.rsshub.app/routes/other#bing',
source: '/',
target: (params, url) => {
const q = new URL(url).searchParams.get('q');
return `/bing/search/${q}`;
},
},
],
},
};
4 changes: 4 additions & 0 deletions lib/v2/bing/router.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
module.exports = function (router) {
router.get('/', require('./daily-wallpaper'));
router.get('/search/:keyword', require('./search'));
};
28 changes: 28 additions & 0 deletions lib/v2/bing/search.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
const parser = require('@/utils/rss-parser');
const { parseDate } = require('@/utils/parse-date');
const dayjs = require('dayjs');
const localizedFormat = require('dayjs/plugin/localizedFormat');
require('dayjs/locale/zh-cn');
dayjs.extend(localizedFormat);

module.exports = async (ctx) => {
const q = ctx.params.keyword;
const searchParams = new URLSearchParams({
format: 'rss',
q,
});
const url = new URL('https://cn.bing.com/search');
url.search = searchParams.toString();
const data = await parser.parseURL(url.toString());
ctx.state.data = {
title: data.title,
link: data.link,
description: data.description + ' - ' + data.copyright,
image: data.image.url,
item: data.items.map((e) => ({
...e,
description: e.content,
pubDate: parseDate(e.pubDate, 'dddd, DD MMM YYYY HH:mm:ss [GMT]', 'zh-cn'),
})),
};
};
1 change: 1 addition & 0 deletions lib/v2/google/maintainer.js
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ module.exports = {
'/fonts/:sort?': ['Fatpandac'],
'/news/:category/:locale': ['zoenglinghou'],
'/scholar/:query': ['HenryQW'],
'/search/:keyword/:language?': ['CaoMeiYouRen'],
'/sites/:id': ['hoilc'],
'/sites/recentChanges/:id': ['nczitzk'],
};
12 changes: 12 additions & 0 deletions lib/v2/google/radar.js
Original file line number Diff line number Diff line change
@@ -1,6 +1,18 @@
module.exports = {
'google.com': {
_name: '谷歌',
www: [
{
title: '搜索',
docs: 'https://docs.rsshub.app/routes/other#google',
source: '/',
target: (params, url, document) => {
const q = new URL(url).searchParams.get('q');
const lang = document.documentElement.lang;
return `/google/search/${q}/${lang}`;
},
},
],
chrome: [
{
title: '插件更新',
Expand Down
1 change: 1 addition & 0 deletions lib/v2/google/router.js
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,5 @@ module.exports = function (router) {
router.get('/fonts/:sort?', require('./fonts'));
router.get('/news/:category/:locale', require('./news'));
router.get('/scholar/:query', require('./scholar'));
router.get('/search/:keyword/:language?', require('./search'));
};
62 changes: 62 additions & 0 deletions lib/v2/google/search.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
const got = require('@/utils/got');
const cheerio = require('cheerio');
const { art } = require('@/utils/render');
const path = require('path');
const config = require('@/config').value;

const renderDescription = (description, images) => art(path.join(__dirname, './templates/description.art'), { description, images });

module.exports = async (ctx) => {
const { keyword, language } = ctx.params;
const searchParams = new URLSearchParams({
q: keyword,
});
const tempUrl = new URL('https://www.google.com/search');
tempUrl.search = searchParams.toString();
const url = tempUrl.toString();
const key = `google-search:${language}:${url}`;
const items = await ctx.cache.tryGet(
key,
async () => {
const response = (
await got(url, {
headers: {
'Accept-Language': language,
},
})
).data;
const $ = cheerio.load(response);
const content = $('#rso');
return content
.find('> div')
.map((i, el) => {
const element = $(el);
const link = element.find('div > div > div > div > div > span > a').first().attr('href');
const title = element.find('div > div > div> div > div > span > a > h3').first().text();
const imgs = element
.find('img')
.map((_j, _el) => $(_el).attr('src'))
.toArray();
const description = element.find('div[style="-webkit-line-clamp:2"]').first().text() || element.find('div[role="heading"]').first().text();
const author = element.find('div > div > div > div > div > span > a > div > div > span').first().text() || '';
return {
link,
title,
description: renderDescription(description, imgs),
author,
};
})
.toArray()
.filter((e) => e?.link);
},
config.cache.routeExpire,
false
);

ctx.state.data = {
title: `${keyword} - Google Search`,
description: `${keyword} - Google Search`,
link: url,
item: items,
};
};
6 changes: 6 additions & 0 deletions lib/v2/google/templates/description.art
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{{@ description }}
{{if images}}
{{each images}}
<img src="{{ $value }}">
{{/each}}
{{/if}}
3 changes: 3 additions & 0 deletions lib/v2/so/maintainer.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
module.exports = {
'/search/:keyword': ['CaoMeiYouRen'],
};
16 changes: 16 additions & 0 deletions lib/v2/so/radar.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
module.exports = {
'so.com': {
_name: '360 公司',
www: [
{
title: '搜索',
docs: 'https://docs.rsshub.app/routes/other#360-sou-suo-sou-suo',
source: '/',
target: (params, url) => {
const q = new URL(url).searchParams.get('q');
return `/so/search/${q}`;
},
},
],
},
};
3 changes: 3 additions & 0 deletions lib/v2/so/router.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
module.exports = function (router) {
router.get('/search/:keyword', require('./search'));
};
48 changes: 48 additions & 0 deletions lib/v2/so/search.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
const got = require('@/utils/got');
const cheerio = require('cheerio');
const { art } = require('@/utils/render');
const path = require('path');
const config = require('@/config').value;

const renderDescription = (description, images) => art(path.join(__dirname, './templates/description.art'), { description, images });

module.exports = async (ctx) => {
const { keyword } = ctx.params;
const url = `https://www.so.com/s?q=${encodeURIComponent(keyword)}`;
const key = `360-so-search:${url}`;
const items = await ctx.cache.tryGet(
key,
async () => {
const response = (await got(url)).data;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This request requires cookie. https://github.com/DIYgod/RSSHub/actions/runs/7309647602/job/19917459502#step:12:35

Thank you, I will fix this issue as soon as possible

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cookie.

Hello,
I have tried several ways to solve the issue, but I still cannot pass the GitHub Action request.
I tested it locally and did not encounter the MaxRedirectsError or cookie problem. Even when accessing it through a proxy, I did not encounter this issue.
Therefore, I suspect it is an IP issue with GitHub Action, and there may be some restrictions on so.com.
If it doesn't work, I will temporarily remove the route to so.com to ensure that other changes can be merged in.

const $ = cheerio.load(response);
const result = $('#container .result');
const resList = result.find('.res-list');
return resList
.map((i, el) => {
const element = $(el);
const imgs = element
.find('img')
.map((j, el2) => $(el2).attr('src'))
.toArray();
const description = element.find('.res-desc').first().text() || element.find('.mh-content-desc-info').first().text() || element.find('.res-comm-con').first().text();
return {
link: element.find('h3 a').first().attr('href'),
title: element.find('h3').first().text(),
description: renderDescription(description, imgs),
author: element.find('.g-linkinfo cite').first().text() || '',
};
})
.toArray()
.filter((e) => e?.link);
},
config.cache.routeExpire,
false
);

ctx.state.data = {
title: `${keyword} - 360 搜索`,
description: `${keyword} - 360 搜索`,
link: url,
item: items,
};
};
6 changes: 6 additions & 0 deletions lib/v2/so/templates/description.art
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{{@ description }}
{{if images}}
{{each images}}
<img src="{{ $value }}">
{{/each}}
{{/if}}
File renamed without changes.
4 changes: 4 additions & 0 deletions lib/v2/sogou/maintainer.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
module.exports = {
'/doodles': ['xyqfer'],
'/search/:keyword': ['CaoMeiYouRen'],
};
Loading