Skip to content

Commit 39576b4

Browse files
chore: update README (vaexio#761)
* chore: update README Co-authored-by: Jovan Veljanoski <[email protected]>
1 parent 25860ac commit 39576b4

File tree

6 files changed

+156
-487
lines changed

6 files changed

+156
-487
lines changed

README.md

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
2+
[![Documentation](https://readthedocs.org/projects/vaex/badge/?version=latest)](https://docs.vaex.io)
3+
4+
# What is Vaex?
5+
6+
Vaex is a high performance Python library for lazy **Out-of-Core DataFrames**
7+
(similar to Pandas), to visualize and explore big tabular datasets. It
8+
calculates *statistics* such as mean, sum, count, standard deviation etc, on an
9+
*N-dimensional grid* for more than **a billion** (`10^9`) samples/rows **per
10+
second**. Visualization is done using **histograms**, **density plots** and **3d
11+
volume rendering**, allowing interactive exploration of big data. Vaex uses
12+
memory mapping, zero memory copy policy and lazy computations for best
13+
performance (no memory wasted).
14+
15+
# Key features
16+
## Instant opening of Huge data files (memory mapping)
17+
[HDF5](https://en.wikipedia.org/wiki/Hierarchical_Data_Format) and [Apache Arrow](https://arrow.apache.org/) supported.
18+
19+
![opening1a](https://user-images.githubusercontent.com/1765949/82818563-31c1e200-9e9f-11ea-9ee0-0a8c1994cdc9.png)
20+
21+
22+
![opening1b](https://user-images.githubusercontent.com/1765949/82820352-49e73080-9ea2-11ea-9153-d73aa399d329.png)
23+
24+
[Read the documentation on how to efficiently convert your data](https://docs.vaex.io/en/latest/example_io.html) from CSV files, Pandas DataFrames, or other sources.
25+
26+
27+
Lazy streaming from S3 supported in combination with memory mapping.
28+
29+
![opening1c](https://user-images.githubusercontent.com/1765949/82820516-a21e3280-9ea2-11ea-948b-07df26c4b5d3.png)
30+
31+
32+
## Expression system
33+
Don't waste memory or time with feature engineering, we (lazily) transform your data when needed.
34+
35+
36+
![expression](https://user-images.githubusercontent.com/1765949/82818733-70f03300-9e9f-11ea-80b0-ab28e7950b5c.png)
37+
38+
39+
40+
## Out-of-core DataFrame
41+
Filtering and evaluating expressions will not waste memory by making copies; the data is kept untouched on disk, and will be streamed only when needed. Delay the time before you need a cluster.
42+
43+
44+
![occ-animated](https://user-images.githubusercontent.com/1765949/82821111-c6c6da00-9ea3-11ea-9f9e-498de8133cc2.gif)
45+
46+
## Fast groupby / aggregations
47+
Vaex implements parallelized, highly performant `groupby` operations, especially when using categories (>1 billion/second).
48+
49+
50+
![groupby](https://user-images.githubusercontent.com/1765949/82818807-97ae6980-9e9f-11ea-8820-41dd4441057a.png)
51+
52+
53+
## Fast and efficient join
54+
Vaex doesn't copy/materialize the 'right' table when joining, saving gigabytes of memory. With subsecond joining on a billion rows, it's pretty fast!
55+
56+
![join](https://user-images.githubusercontent.com/1765949/82818840-a268fe80-9e9f-11ea-8ba2-6a6d52c4af88.png)
57+
58+
## More features
59+
60+
* Remote DataFrames (documentation coming soon)
61+
* Integration into [Jupyter and Voila for interactive notebooks and dashboards](https://vaex.readthedocs.io/en/latest/tutorial_jupyter.html)
62+
* [Machine Learning without (explicit) pipelines](https://vaex.readthedocs.io/en/latest/tutorial_ml.html)
63+
64+
65+
# Learn how to use Vaex efficiently
66+
* [Follow our tutorials](https://docs.vaex.io/en/latest/tutorials.html)
67+
* Watch our more recent talks:
68+
* [PyData London 2019](https://www.youtube.com/watch?v=2Tt0i823-ec)
69+
* [SciPy 2019](https://www.youtube.com/watch?v=ELtjRdPT8is)
70+
* Contact us for training or enterprise support at https://vaex.io/

README.rst

Lines changed: 0 additions & 261 deletions
This file was deleted.

dodo.py

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44
import pkg_resources
55
import vaex.meta._version
66
import re
7+
import shutil
78

89

910
def task_mybinder():
@@ -24,3 +25,16 @@ def action(targets):
2425
'targets': ["binder/requirements.txt"],
2526
'file_dep': ['packages/vaex-meta/vaex/meta/_version.py']
2627
}
28+
29+
30+
def task_sync_readme():
31+
"""Make the README for veax-meta up to date"""
32+
33+
def action(targets):
34+
shutil.copy('README.md', targets[0])
35+
36+
return {
37+
'actions': [action],
38+
'targets': ["packages/vaex-meta/README.md"],
39+
'file_dep': ['README.md']
40+
}

0 commit comments

Comments
 (0)