arsam
062656038f
Import the existing local project snapshot. This commit organizes existing files; it does not represent newly developed functionality.
57 lines
2.5 KiB
Markdown
57 lines
2.5 KiB
Markdown
Reactor Crawler
|
|
==========
|
|
[](https://github.com/reactor-joy/reactor-crw/actions/workflows/go.yml)
|
|
[](https://goreportcard.com/report/github.com/reactor-joy/reactor-crw)
|
|
|
|
Simple CLI content crawler for [Joyreactor](http://joyreactor.cc). He'll find all media content on the
|
|
page you've provided and save it. If there will be any kind of pagination... he'll go through all pages
|
|
as well unless you'll tell him to not.
|
|
|
|
<p>
|
|
<img src="https://i.imgur.com/DjD6cW0.gif" width="800" alt="reactor_crawler_example">
|
|
</p>
|
|
|
|
## Quick start
|
|
|
|
Here's the quickest way to download something and test the crawler:
|
|
* Download a build according to your OS.
|
|
* Pick some URL from [Joyreactor](http://joyreactor.cc).
|
|
* Run the crawler `$ reactor-crw -p "http://joyreactor.cc/tag/digital+art"`
|
|
|
|
## What else
|
|
|
|
There's a list of optional flags that adds a little more control over the crawler.
|
|
|
|
```
|
|
$ reactor-crw --help
|
|
|
|
Allows to quickly download all content by its direct url or entire tag or fandom from joyreactor.cc.
|
|
Example: reactor-crw -d "." -p "http://joyreactor.cc/tag/someTag/all" -w 2 -c "cookie-string"
|
|
|
|
Usage:
|
|
reactor-crw [flags]
|
|
|
|
Flags:
|
|
-c, --cookie string User's cookie. Some content may be unavailable without it
|
|
-d, --destination string Save path for content. Default value is a user's home folder
|
|
(example C:\Users\username for Windows) (default "/home/avpretty")
|
|
-h, --help help for reactor-crw
|
|
-p, --path string Provide a full page URL
|
|
-s, --search string A comma separated list of content types that should be downloaded.
|
|
Possible values: image,gif,webm,mp4. Example: -s "image,webm" (default "image,gif")
|
|
-o, --single-page Crawl only one page
|
|
-w, --workers int Amount of workers (default 1)
|
|
```
|
|
|
|
From all flags only `-p --path` is required. All other flags can be omitted and default values will be used.
|
|
|
|
Here's another example:
|
|
|
|
```
|
|
$ reactor-crw -p "http://joyreactor.cc/post/000000" -d "." -s "mp4" -o -c "cookies from joyreactor"
|
|
```
|
|
This one will download only `mp4` content from the post and will save it to the current directory.
|
|
`-o` means that only the current page will be parsed, and the user's cookie `-s` will be used by the crawler.
|
|
|
|
**Note**: some content may be parsed only with user's cookie.
|