SparseConnection support

[n-shevko]

No description provided.

[Hananel-Hazan]

@C-Earl what do you think?

[C-Earl]

Seems like a reasonable start. One thing that might be worth testing is running a model with a profiler and seeing how expensive update.to_sparse is to run. Not sure if it's possible to avoid having to call to_sparse during the update though.

[n-shevko]

@C-Earl

Not sure if it's possible to avoid having to call to_sparse during the update though

Unfortunately, for now pytorch support only addition of the following types of tensors :

1. dense + sparse = dense
2. sparse + sparse = sparse

I decided to use #2
In case if we would use #1 we also can't avoid .to_sparse call for the result tensor (w in our case)

Seems like a reasonable start. One thing that might be worth testing is running a model with a profiler and seeing how expensive update.to_sparse is to run.

I run profiler on my desktop(I don't have nvidia). Screenshot is attached.
For example total _connection_update method took 5.2 seconds (total time).
to_sparse (4rth row), total time 0.2591 sec (5% of 5.2 seconds).
I can rerun it on googlecolab if needed.

[C-Earl]

Great, thanks for the data!

In the future we should test some comparisons between sparse and non-sparse models as well. It'd be interesting to see how much performance increase we get with this new format.