keywords:spider - npm search

robots-parser

A specification compliant robots.txt parser with wildcard (*) matching support.

published version 3.0.1, 2 years ago85 dependents licensed under $MIT

5,475,967

simplecrawler

Very straightforward, event driven web crawler. Features a flexible queue interface and a basic cache mechanism with extensible backend.

kbychkov

published version 1.1.9, 5 years ago77 dependents licensed under $BSD-2-Clause

194,182

shift-parser

ECMAScript parser that produces a Shift format AST

shapesecurity

published version 8.0.0, 3 years ago30 dependents licensed under $Apache-2.0

123,865

isbot-fast

JavaScript module detecting bots/crawlers/spiders via user-agent

mahovich

published version 1.2.0, 6 years ago11 dependents licensed under $MIT

88,712

spider-detector

A tiny node module to detect spiders/crawlers quickly and comes with optional middleware for ExpressJS

michael.heuberger

published version 2.1.0, 10 months ago6 dependents licensed under $MIT

20,108

es6-crawler-detect

This is an ES6 adaptation of the original PHP library CrawlerDetect, this library will help you detect bots/crawlers/spiders vie the useragent.

jefferyhus

published version 4.0.2, 2 months ago7 dependents licensed under $MIT

28,380

crawler

Crawler is a ready-to-use web spider that works with proxies, asynchrony, rate limit, configurable request pools, jQuery, and HTTP/2 support.

mike442144

published version 2.0.2, a year ago123 dependents licensed under $MIT

17,765

@zachleat/spider-pig

Get a list of local URL links from a root URL.

zachleat

published version 3.0.0, 4 years ago1 dependents licensed under $MIT

11,444

robots-txt-parser

A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.

cakroyd

published version 2.0.3, 3 years ago5 dependents licensed under $MIT

8,025

@spider-cloud/spider-client

Isomorphic Javascript SDK for Spider Cloud services

jeffmendez

published version 0.1.38, 6 days ago0 dependents licensed under $MIT

6,558

@nodebb/spider-detector

A tiny node module to detect spiders/crawlers quickly and comes with optional middleware for ExpressJS

baris

published version 2.0.3, a year ago0 dependents licensed under $MIT

5,483

crawlab-sdk

Node.js SDK for Crawlab

crawlab

published version 0.6.0-12, 3 years ago0 dependents licensed under $BSD-3-Clause

6,251

wget-parser

Parses the wget spider output into an object

muji

published version 2.0.0, 9 years ago0 dependents licensed under $MIT

4,023

@lightningchart/lcjs

A high-performance charting library.

niilo.keinanen

published version 7.1.1, 2 months ago2 dependents licensed under $SEE LICENSE IN LICENSE

3,971

node-spider

Generic web crawler powered by Node.js

flesler

published version 1.4.1, 9 years ago5 dependents licensed under $BSD-2-Clause

868

node-html-crawler

Crawler (spider) of site web pages by domain name

safonovpro

published version 1.2.3, 4 years ago0 dependents licensed under $MIT

772

roboto

A web crawler for Nodejs.

jculvey

published version 0.8.2, 11 years ago1 dependents licensed under $MIT

826

ananse

Ananse is a lightweight NodeJs framework with batteries included for building efficient, scalable and maintainable USSD applications.

ephrimlawrence

published version 1.9.11, a month ago0 dependents licensed under $MIT

654

node-webcrawler

Crawler is a web spider written with Nodejs. It gives you the full power of jQuery on the server to parse a big number of pages as they are downloaded, asynchronously