Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support of bulk API #16

Open
v-kolesnikov opened this issue Feb 9, 2019 · 3 comments
Open

Add support of bulk API #16

v-kolesnikov opened this issue Feb 9, 2019 · 3 comments
Assignees
Labels

Comments

@v-kolesnikov
Copy link
Collaborator

Bulk API is useful in cases of create or update multiple documents. Current API has some problems with performance in cases of inserting thousands of documents.

Bulk API

@v-kolesnikov v-kolesnikov self-assigned this Feb 9, 2019
@v-kolesnikov
Copy link
Collaborator Author

v-kolesnikov commented Feb 10, 2019

I use the following code in my project:

    class Bulk < ROM::Commands::Create
      relation :<my_relation_name>
      register_as :bulk

      # @api private
      def execute(tuples)
        body = tuples.flat_map { |tuple| to_body(input[tuple]) }
        dataset.client.bulk(body: body)
      end

      private

      def to_body(tuple)
        # [metadata(tuple), tuple]
        params = dataset.params
        [
          {
            index: {
              _index: params[:index],
              _type: params[:type],
              **({ _id: tuple.fetch(_id) } if _id).to_h
            }
          },
          tuple
        ]
      end

      # @api private
      def dataset
        relation.dataset
      end

      def _id
        relation.schema.primary_key_name
      end
    end

But I can't add it to the core, because if I add this to rom/elasticsearch/commands.rb it just doesn't work and I haven't found any places in the code where commands being configured.

@pasivuorio
Copy link

Thanks for this code, managed to implement my bulk insert command using that. Probably you have already managed to configure this, I'll just leave the config needed here in case someone else is looking for same answer:
config = ROM::Configuration.new(:elasticsearch, "http://#{host}:#{port}")
config.register_command(Commands:: Bulk) #my commands are in commands namespace
rom_es = ROM.container(config)
...

@v-kolesnikov
Copy link
Collaborator Author

@pasivuorio You are welcome!
I'm glad to see probably third person who is interested in rom-elasticsearch! 😅

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants