Using SSHash submodule for graph representation #481

mmarzett · 2024-01-26T10:15:26Z

Added the SSHash data structure as an option for the DBG representation, implemented using SSHash as a submodule.

adamant-pwn · 2024-01-29T12:03:33Z


[  FAILED  ] 10 tests, listed below:
[  FAILED  ] DeBruijnGraphTest/4.CallStartNodes, where TypeParam = mtg::graph::DBGSSHash
[  FAILED  ] DeBruijnGraphTest/4.CallUnitigsWithoutTips, where TypeParam = mtg::graph::DBGSSHash
[  FAILED  ] DeBruijnGraphTest/4.CallUnitigsWithoutTips2, where TypeParam = mtg::graph::DBGSSHash
[  FAILED  ] DeBruijnGraphTest/4.CallUnitigsCheckDegree, where TypeParam = mtg::graph::DBGSSHash
[  FAILED  ] DeBruijnGraphTest/4.CallUnitigsCross, where TypeParam = mtg::graph::DBGSSHash
[  FAILED  ] DeBruijnGraphTest/4.get_maximum_indegree, where TypeParam = mtg::graph::DBGSSHash
[  FAILED  ] DeBruijnGraphTest/4.get_degree2, where TypeParam = mtg::graph::DBGSSHash
[  FAILED  ] DeBruijnGraphTest/4.indegree_identity_incoming_indegree, where TypeParam = mtg::graph::DBGSSHash
[  FAILED  ] DeBruijnGraphTest/4.indegree_identity_traverse_back_incoming, where TypeParam = mtg::graph::DBGSSHash
[  FAILED  ] DeBruijnGraphTest/4.CallOutgoingEdges, where TypeParam = mtg::graph::DBGSSHash

10 FAILED TESTS

Are these the same tests that were disabled, or is it something new?

metagraph/src/cli/build.cpp

metagraph/tests/data/sshash_sequences/pthash.tmp.run1700659576911610385.free_slots.bin

metagraph/tests/data/sshash_sequences/pthash.tmp.run1700659576911610385.pilots.bin

metagraph/tests/data/sshash_sequences/sshash.tmp.run_1700659576859653407.minimizers.0.bin

metagraph/src/graph/representation/hash/dbg_sshash.cpp

hmusta · 2024-02-19T10:24:43Z


[  FAILED  ] 10 tests, listed below:
[  FAILED  ] DeBruijnGraphTest/4.CallStartNodes, where TypeParam = mtg::graph::DBGSSHash
[  FAILED  ] DeBruijnGraphTest/4.CallUnitigsWithoutTips, where TypeParam = mtg::graph::DBGSSHash
[  FAILED  ] DeBruijnGraphTest/4.CallUnitigsWithoutTips2, where TypeParam = mtg::graph::DBGSSHash
[  FAILED  ] DeBruijnGraphTest/4.CallUnitigsCheckDegree, where TypeParam = mtg::graph::DBGSSHash
[  FAILED  ] DeBruijnGraphTest/4.CallUnitigsCross, where TypeParam = mtg::graph::DBGSSHash
[  FAILED  ] DeBruijnGraphTest/4.get_maximum_indegree, where TypeParam = mtg::graph::DBGSSHash
[  FAILED  ] DeBruijnGraphTest/4.get_degree2, where TypeParam = mtg::graph::DBGSSHash
[  FAILED  ] DeBruijnGraphTest/4.indegree_identity_incoming_indegree, where TypeParam = mtg::graph::DBGSSHash
[  FAILED  ] DeBruijnGraphTest/4.indegree_identity_traverse_back_incoming, where TypeParam = mtg::graph::DBGSSHash
[  FAILED  ] DeBruijnGraphTest/4.CallOutgoingEdges, where TypeParam = mtg::graph::DBGSSHash

10 FAILED TESTS

Are these the same tests that were disabled, or is it something new?

DeBruijnGraphTest/4.CallStartNodes seems to SegFault because of a multi-threading issue. This goes away if I change its hard-coding from 8 threads to 1

metagraph/tests/graph/all/test_dbg_helpers.cpp

metagraph/src/graph/representation/hash/dbg_sshash.cpp

Add PTHash submodule, create boilerplate code for DBGSSHash class, add it to test_dbg_helpers.

and list of contigs

…hash

This reverts commit 07b4653.

… a custom exception

metagraph/src/graph/representation/hash/dbg_sshash.cpp

metagraph/src/graph/representation/base/sequence_graph.cpp

metagraph/src/graph/alignment/aligner_labeled.cpp

metagraph/src/graph/representation/hash/dbg_sshash.cpp

* install libdeflate-dev * Link against libdeflate

adamant-pwn

There is still one open comment related to jermp/pthash#63. I think otherwise the current state is good? We should probably open a separate issue to track jermp/pthash#63 if we merge this PR before it is fixed upstream.

mmarzett marked this pull request as draft January 26, 2024 10:19

adamant-pwn force-pushed the sshash_module branch from 7b242cf to d57beb7 Compare January 29, 2024 12:02

adamant-pwn requested changes Feb 15, 2024

View reviewed changes

hmusta requested changes Feb 20, 2024

View reviewed changes

metagraph/tests/graph/all/test_dbg_helpers.cpp Show resolved Hide resolved

metagraph/src/graph/representation/hash/dbg_sshash.cpp Outdated Show resolved Hide resolved

metagraph/src/graph/representation/hash/dbg_sshash.cpp Outdated Show resolved Hide resolved

adamant-pwn mentioned this pull request Mar 27, 2024

Add SSHash-based representation #479

Closed

adamant-pwn marked this pull request as ready for review March 27, 2024 20:06

adamant-pwn force-pushed the sshash_module branch from 44af9e1 to 8a0d415 Compare March 28, 2024 01:44

hmusta requested changes Mar 28, 2024

View reviewed changes

metagraph/src/graph/representation/hash/dbg_sshash.cpp Outdated Show resolved Hide resolved

adamant-pwn force-pushed the sshash_module branch from cd959a9 to a37e341 Compare May 2, 2024 11:49

adamant-pwn and others added 5 commits May 2, 2024 13:56

Fix CI failures for MacOS

bf6911c

update simde-no-tests

a412023

Switch to latest compiling simde-no-tests

2e70949

replace restrict_to_mask_epi64 with fully scalar verison

033bff0

Update CMakeLists.txt

27f2ce1

adamant-pwn force-pushed the sshash_module branch from a37e341 to ecce9eb Compare May 23, 2024 13:17

adamant-pwn and others added 13 commits May 23, 2024 15:26

Initial DBGSSHash commit

dce107e

Add PTHash submodule, create boilerplate code for DBGSSHash class, add it to test_dbg_helpers.

Use SSHash submodule instead of PTHash

9a68837

trying to use sshash as a submodule

e49815f

fixed index conversion between ssh and metagraph

d934254

and list of contigs

some changes to sshash, first try at avoiding the compile error in ss…

3afa7f2

…hash

changes for unit_test usage

6ce2c71

new sshash version

5f5eccf

ignore strict aliasing warning

d54f216

using sshash from cli

2071688

general path

381927a

fixed serialization and loading

d724d20

no longer need k.txt file + clean-up

76f8a72

disabled tests that are incompatible with sshash

4dca728

hmusta added 11 commits June 20, 2024 09:33

Revert "this works"

a01e9b5

This reverts commit 07b4653.

corner case

faeb485

minor

25aaf02

added stop_early to call_kmers. replace bad_function_call throws with…

147edb4

… a custom exception

addressed reviewer comment

fe6a5d6

minor

5d12145

make sure to properly override

e0cb5d5

make load and serialize work for general istream and ostream

8e48e58

Add generic implementation of adjacent_*_nodes

ff7d906

fix

45c9ae6

update sshash

72cfae2

adamant-pwn reviewed Jun 20, 2024

View reviewed changes

metagraph/src/graph/representation/hash/dbg_sshash.cpp Outdated Show resolved Hide resolved

num_nodes -> max_index, remove_const_ref -> decay_t

6c24e8e

adamant-pwn reviewed Jun 20, 2024

View reviewed changes

metagraph/src/graph/representation/base/sequence_graph.cpp Show resolved Hide resolved

adamant-pwn reviewed Jun 20, 2024

View reviewed changes

metagraph/src/graph/alignment/aligner_labeled.cpp Outdated Show resolved Hide resolved

adamant-pwn reviewed Jun 20, 2024

View reviewed changes

metagraph/src/graph/representation/hash/dbg_sshash.cpp Outdated Show resolved Hide resolved

adamant-pwn and others added 6 commits June 20, 2024 23:03

Update sshash, use essentials generic visitors

1460c6b

Update sshash

6bb3cb2

Update htslib (#490) (#491)

abf27f6

* install libdeflate-dev * Link against libdeflate

Install libdeflate on clang

ef644e4

Merge branch 'master' into sshash_module

3c94fed

Put early_term in sequence_graph, silence sshash with rdbuf swap

261d608

adamant-pwn approved these changes Jul 2, 2024

View reviewed changes

adamant-pwn added 2 commits July 2, 2024 23:55

Preserve failbit state

da9ec07

only use DBGSSHash in build_graph<DBGSSHash>

61c75b4

hmusta self-requested a review July 25, 2024 06:54

hmusta approved these changes Jul 25, 2024

View reviewed changes

hmusta merged commit 163b644 into master Jul 25, 2024
25 checks passed

hmusta deleted the sshash_module branch July 25, 2024 07:16

adamant-pwn mentioned this pull request Oct 8, 2024

Sshash module superkmer anno #497

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Using SSHash submodule for graph representation #481

Using SSHash submodule for graph representation #481

mmarzett commented Jan 26, 2024

adamant-pwn commented Jan 29, 2024

hmusta commented Feb 19, 2024

adamant-pwn left a comment

Using SSHash submodule for graph representation #481

Using SSHash submodule for graph representation #481

Conversation

mmarzett commented Jan 26, 2024

adamant-pwn commented Jan 29, 2024

hmusta commented Feb 19, 2024

adamant-pwn left a comment

Choose a reason for hiding this comment