| layout |
|---|
default |
The email crawler is a modern designed GUI crawler for batch fetching useful infomation on the website. Not only for emails but other keywords are also supported.
This project is developed in java, which need jvm installed. JRE Download here and select the right package.
Note that unzip is always the first step after the zip package has been downloaded.
Double click the file named start_from_here.bat, It will call a black window out and a GUI software will be opened then.
Double click the jar file named crawler_main.jar(might vary in different version).
It contains 3 sections of function zone.
- Type in the root url or it's links which contains the email info mation.
- The url is 2 types: normal type and indexing. normal one is normal as it is. The indexing url is like
http://abc.com/web?studentpage=1. The number1could be replaced by other number and then we can see the other pages there. the max number is always shown on the page. - Crawl speed is the thread number of crawling. Under 100 is safe, while over that value might lead to lagging.
- The inner hyper link will never be fetched so that
strict modeis very useful for speed up crawling. But if you need to crawl links in the web page, the crawling mode should be closed. - prefix and sufix space and enter is recommonded commonly. This will avoid some common problems which might lead to crawling failure.
- The replacement area is the key feature in the software. You can customize keyword to crawl revealed great extention ablity. In some web page
@is replaced by[@],[#], etc to avoid crawling, but here we can use this feature to replace the@. Just type in the keywords even@image:[@],[#],<image='./at.jpeg'>seperated by comma, all these email types can be recorded well. - Debug mode is for mac, windows can see the debug infomation on the black concle.
- Press the start button to start. It will automatically stop when finished.
This feature is more valuable than the others. Using this feature in proper way can reduce your work load greatly. It can search all relavent urls containing the keywords, helping you locate target painlessly.
- Select the place to save ur file!
- Read me contains VERY IMPORTANT information, please go through it.
- Don't forget to press Save.
The email(or anything you like if keyword is customized) will be generated on the Desktop(as default, you can change the directory in setting) with a the website’s url and a path extention .csv. e.g. baidu.com.csv.
The cells is seperated by comma and could be opened by excel(or some other supported office software).
How to read the file.csv with column and lines on excel in Windows
- Could it run on my mac, windows or Linux? Yes. It is based on Java cross platform. Running on Android is also supported, but never try it on iOS devices.
- I click the
.batfile, then the black window said some error. Most likely you forget install the java JRE. - Will this software charge? NO. It is for free at all. No lock feature.
- Developing Plan? There is no developing plan. All version is build for specific demand.
- How could I reset the settings?
All settings is stored in the software folder. Deleting the
.configfile shall work. - How to report bugs? Open an issue for me. I will fix it ASAP.
See the open issues for a list of proposed features (and known issues).
email crawler is intended to make those people who need to find information's life easier. Thanks a lot to my friend Li xu, who strongly supported and supporting on this project. It should look great, but that goes without saying.
Interested in contributing? We'd love your help. Cayman is an open source project, built one contribution at a time by users like you. See the CONTRIBUTING file for instructions on how to contribute.
If you have any further question, please feel free to mail me or add my wechat Zhou_zhengchang.