Skip to content

Commit

Permalink
Merge pull request #6 from scrapinghub/edit_readme
Browse files Browse the repository at this point in the history
slightly better temporary readme
  • Loading branch information
scharfmn authored May 1, 2020
2 parents a472d72 + a19aa71 commit 9bcbc5d
Showing 1 changed file with 44 additions and 21 deletions.
65 changes: 44 additions & 21 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
Varanus
*******

*Major miner monitoring*
This tool wraps an API POST request to scrapinghub's job and other data storage API.

============
Requirements
Expand All @@ -14,20 +14,41 @@ Requirements
===============
Developer Notes
===============
::

$ poetry new varanus
For those unfamiliar with poetry, it's a virtualenv + package manager.

Created package varanus in varanus
The project originally was built to use it, but instead of beginning the ``poetry new`` command, we do a hybrid clone + poetry install

$ cd varanus
(Since the project uses a ``.lock`` file, using pipenv plus some virtualenv manager should also work, but these instructions use poetry.)

$ poetry install
To install:

``$ pip install poetry``

``$ git clone https://github.com/scrapinghub/varanus.git``

``$ cd varanus``


When you install an application using poetry, a virtualenv is created automagically:

``$ poetry install``

Creating virtualenv varanus-mrejzrgU-py3.8 in /home/mns/.cache/pypoetry/virtualenvs
Installing dependencies from lock file

Package operations: 47 installs, 0 updates, 0 removals

- Installing decorator (4.4.0)
- Installing ipython-genutils (0.2.0)
- Installing six (1.12.0)
- Installing attrs (19.1.0)
- Installing certifi (2019.6.16)
- Installing chardet (3.0.4)
- Installing idna (2.8)

[ . . . snip . . . ]

Updating dependencies
Resolving dependencies... (1.2s)
Writing lock file
Package operations: 7 installs, 0 updates, 0 removals
- Installing zipp (0.5.2)
- Installing importlib-metadata (0.19)
- Installing atomicwrites (1.3.0)
Expand All @@ -37,14 +58,16 @@ Developer Notes
- Installing pytest (3.10.1)
- Installing varanus (0.1.0)

$ poetry show

atomicwrites 1.3.0 Atomic file writes.
attrs 19.1.0 Classes Without Boilerplate
importlib-metadata 0.19 Read metadata from Python packages
more-itertools 7.2.0 More routines for operating on iterables, beyond itertools
pluggy 0.12.0 plugin and hook calling mechanisms for python
py 1.8.0 library with cross-python path, ini-parsing, io, code, log facilities
pytest 3.10.1 pytest: simple powerful testing with Python
six 1.12.0 Python 2 and 3 compatibility utilities
zipp 0.5.2 Backport of pathlib-compatible object wrapper for zip files
To invoke the virtualenv created by poetry, run using poetry:

``$ poetry run varanus jobs -p 376566 -s dod_953_tripadvisor``

●▬▬▬▬▬▬▬▬▬● <Response [200]> https://storage.scrapinghub.com/jobq/376566/list?content=results&fit_width=False&formatter=table&max_width=0&noindent=False&print_empty=False&project=376566&quote_mode=nonnumeric&start=0&jobmeta=project&jobmeta=spider&jobmeta=spider_args&jobmeta=job_cmd&jobmeta=tags&jobmeta=scrapystats&jobmeta=units&jobmeta=version&jobmeta=priority&jobmeta=pending_time&jobmeta=running_time&jobmeta=finished_time&jobmeta=scheduled_by&jobmeta=state&jobmeta=close_reason&state=finished&spider=dod_953_tripadvisor&count=10 ● varanus.__patch__:scrapinghub.client.HubstorageClient.request

+----------------+---------------------+----------+----------+------------------+------------------+-----+-------+-------+--------+----------+----------+-----------------+
| Key | Spider | Pnd mins | Run mins | Start | Finish | Err | Warn | Items | Pages | State | Reason | Version |
+================+=====================+==========+==========+==================+==================+=====+=======+=======+========+==========+==========+=================+
| 376566/418/805 | dod_953_tripadvisor | 0 | 9 | 2020/04/19 19:10 | 2020/04/19 19:19 | 0 | 41 | 73 | 567 | finished | finished | 2233af50-master |
+----------------+---------------------+----------+----------+------------------+------------------+-----+-------+-------+--------+----------+----------+-----------------+

To see the rest of the possible command line arguments, take a look at the ``add_argument`` calls in `The varanus CLI folder <https://github.com/scrapinghub/varanus/tree/master/src/varanus/cli>`_

0 comments on commit 9bcbc5d

Please sign in to comment.