Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

StreamDM-100 #101

Open
wants to merge 13 commits into
base: master
Choose a base branch
from
Open

StreamDM-100 #101

wants to merge 13 commits into from

Conversation

ioanna-ki
Copy link
Contributor

Bug report #100 - part of the solution

1. remove duplicates of the output coreset

2. fixed the break statement

still needs some fixes in StreamKM

Summary of the changes

BucketManager's getCoreset returned a lot of duplicate Examples in the resulted coreset. Almost over 50% of the examples was duplicates. With these changes this percentage is around 10% but not zero.

in splitCoresetTreeLeaf sometimes the one child had all the points and the other was empty. Now it rechooses a center and splits the points again.

in splitCoresetTree changed the way of choosing the which node to split. Similar to moa's treeCoreset
in BucketManager the break statement in getCoreset does not work properly. Now there is a flag that stops the process when a full bucket is found.

Tests

need to print the resulted coreset in StreamKM be sure getCoreset is invoked at least once

ioanna-ki added 13 commits July 25, 2018 00:25
/*
 *  Cache class keeps a hashmap where it 
 *  stores the number of the bucket that holds
 *  the major coreset and the coreset itself
 *  r is the merge threshold in this case
 *  we use r = 2 since we implemented an 2-way 
 *  coreset tree 
 */
/**
  * Implements the CachedKM++ algorithm for data streams. CachedKM++ computes a
  * small (weighted) sample of the stream by using <i>coresets</i> from cache
  * to reduce the number of coresets needed for a merge step, and then uses
  * it as an input to a k-means++ algorithm. It uses a data structure called
  * <tt>BucketManager</tt> to handle the coresets.
  *
  * <p>It uses the following options:
  * <ul>
  *  <li> Number of microclusters (<b>-m</b>)
  *  <li> Initial buffer size (<b>-b</b>)
  *  <li> Size of coresets (<b>-s</b>)
  *  <li> Learning window (<b>-w</b>) * </ul>
  */
prevent sum from NaN when funcost is zero.  splitCoresetTree needed more cases when leaf has no elements
@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants