Skip to content

Commit

Permalink
step
Browse files Browse the repository at this point in the history
  • Loading branch information
Miniast committed May 14, 2024
1 parent 0d8f6cd commit 834058e
Show file tree
Hide file tree
Showing 21 changed files with 677 additions and 116 deletions.
556 changes: 554 additions & 2 deletions README.md

Large diffs are not rendered by default.

18 changes: 9 additions & 9 deletions dist/crawler.d.ts
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
/// <reference types="node" />
import { EventEmitter } from "events";
import type { crawlerOptions, requestOptions } from "./types/crawler.js";
import type { CrawlerOptions, RequestConfig, CrawlerResponse } from "./types/crawler.js";
declare class Crawler extends EventEmitter {
private _limiters;
private _UAIndex;
private _proxyIndex;
options: crawlerOptions;
options: CrawlerOptions;
globalOnlyOptions: string[];
seen: any;
constructor(options?: crawlerOptions);
private _checkHtml;
constructor(options?: CrawlerOptions);
private _detectHtmlOnHeaders;
private _schedule;
private _execute;
private _handler;
Expand All @@ -27,7 +27,7 @@ declare class Crawler extends EventEmitter {
* crawler.setLimiter(0, "rateLimit", 1000);
* ```
*/
setLimiter(rateLimiterId: number, property: string, value: any): void;
setLimiter(rateLimiterId: number, property: string, value: unknown): void;
/**
*
* @param options
Expand All @@ -45,13 +45,13 @@ declare class Crawler extends EventEmitter {
* await crawler.send("https://example.com");
* ```
*/
send: (options: string | requestOptions) => Promise<any>;
send: (options: RequestConfig) => Promise<CrawlerResponse>;
/**
* @deprecated
* @description Old interface version. It is recommended to use `Crawler.send()` instead.
* @see Crawler.send
*/
direct: (options: string | requestOptions) => Promise<any>;
direct: (options: RequestConfig) => Promise<CrawlerResponse>;
/**
*
* @param options
Expand All @@ -65,13 +65,13 @@ declare class Crawler extends EventEmitter {
* });
* ```
*/
add: (options: string | requestOptions | requestOptions[]) => void;
add: (options: RequestConfig) => void;
/**
* @deprecated
* @description Old interface version. It is recommended to use `Crawler.add()` instead.
* @see Crawler.add
*/
queue: (options: string | requestOptions | requestOptions[]) => void;
queue: (options: RequestConfig) => void;
}
export default Crawler;
//# sourceMappingURL=crawler.d.ts.map
2 changes: 1 addition & 1 deletion dist/crawler.d.ts.map

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 5 additions & 5 deletions dist/crawler.js

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion dist/crawler.js.map

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion dist/logger.d.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
export declare const logOptions: {
type: any;
type: "hidden" | "json" | "pretty";
name: string;
hideLogPositionForProduction: boolean;
prettyLogTemplate: string;
Expand Down
2 changes: 1 addition & 1 deletion dist/logger.js.map

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 4 additions & 3 deletions dist/options.d.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
export declare const getCharset: (headers: Record<string, string>) => null | string;
export declare const getValidOptions: (options: unknown) => Object;
export declare const alignOptions: (options: any) => any;
import { RequestConfig, RequestOptions } from "./types/crawler.js";
export declare const getCharset: (headers: Record<string, unknown>) => null | string;
export declare const getValidOptions: (options: RequestConfig) => RequestOptions;
export declare const alignOptions: (options: RequestOptions) => any;
//# sourceMappingURL=options.d.ts.map
2 changes: 1 addition & 1 deletion dist/options.d.ts.map

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 3 additions & 2 deletions dist/options.js

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion dist/options.js.map

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit 834058e

Please sign in to comment.