diff --git a/README.MD b/README.MD new file mode 100644 index 0000000..637b0be --- /dev/null +++ b/README.MD @@ -0,0 +1,56 @@ +Reactor Crawler +========== +[![reactor-crw](https://github.com/reactor-joy/reactor-crw/actions/workflows/go.yml/badge.svg)](https://github.com/reactor-joy/reactor-crw/actions/workflows/go.yml) +[![Go Report Card](https://goreportcard.com/badge/github.com/reactor-joy/reactor-crw)](https://goreportcard.com/report/github.com/reactor-joy/reactor-crw) + +Simple CLI content crawler for [Joyreactor](http://joyreactor.cc). He'll find all media content on the +page you've provided and save it. If there will be any kind of pagination... he'll go through all pages +as well unless you'll tell him to not. + +

+ reactor_crawler_example +

+ +## Quick start + +Here's the quickest way to download something and test the crawler: +* Download a build according to your OS. +* Pick some URL from [Joyreactor](http://joyreactor.cc). +* Run the crawler `$ reactor-crw -p "http://joyreactor.cc/tag/digital+art"` + +## What else + +There's a list of optional flags that adds a little more control over the crawler. + +``` +$ reactor-crw --help + +Allows to quickly download all content by its direct url or entire tag or fandom from joyreactor.cc. +Example: reactor-crw -d "." -p "http://joyreactor.cc/tag/someTag/all" -w 2 -c "cookie-string" + +Usage: + reactor-crw [flags] + +Flags: + -c, --cookie string User's cookie. Some content may be unavailable without it + -d, --destination string Save path for content. Default value is a user's home folder + (example C:\Users\username for Windows) (default "/home/avpretty") + -h, --help help for reactor-crw + -p, --path string Provide a full page URL + -s, --search string A comma separated list of content types that should be downloaded. + Possible values: image,gif,webm,mp4. Example: -s "image,webm" (default "image,gif") + -o, --single-page Crawl only one page + -w, --workers int Amount of workers (default 1) +``` + +From all flags only `-p --path` is required. All other flags can be omitted and default values will be used. + +Here's another example: + +``` +$ reactor-crw -p "http://joyreactor.cc/post/000000" -d "." -s "mp4" -o -c "cookies from joyreactor" +``` +This one will download only `mp4` content from the post and will save it to the current directory. +`-o` means that only the current page will be parsed, and the user's cookie `-s` will be used by the crawler. + +**Note**: some content may be parsed only with user's cookie.