'Repo.insert' throw 'IndexError' sometimes

[metacortex]

Sometimes simple insert query result is empty array [],
though all data were inserted to table correctly.

Index out of bounds (IndexError)
0x10a325535: *CallStack::new:CallStack at ??
0x10a32848e: *raise<IndexError>:NoReturn at ??

https://github.com/Crecto/crecto/blob/master/src/crecto/repo.cr#L184

The error is coming from first.

query = config.adapter.run_on_instance(tx || config.get_connection, :insert, changeset)
...
new_instance = changeset.instance.class.from_rs(query.as(DB::ResultSet)).first

I couldn't find any pattern, it looks like randomly occur.

---

mysql-5.6

[metacortex]

#96

[fridgerator]

fixed by #96

[onemanstartup]

@fridgerator I'm still experience IndexError on master. And "Unexpected byte 0xff in UTF-8 byte sequence" as well. I don't know how to debug this, I'm new in Crystal.

[fridgerator]

You get the unexpected byte sequence only on insert statements with mysql?

[onemanstartup]

@fridgerator I'm experience this with postgresql. I'm parsing json and then insert changeset.
Only 3 records will be saved out of 6, it will stuck after Unexpected byte error.
And DB log will be only the first record.

 1.77ms INSERT INTO tickers (market, currency, ticker, price, bid, ask, low_24h, high_24h, avg_24h, volume_24h, volume_30d) VALUES ('coinone', 'bch', 'KRW', '360500.0', '', '', '350100.0', '369500.0', '', '34075.8113', '') RETURNING *

Very confusing. I'm already successfully record other records.

With insert

#<Crecto::Changeset::Changeset(Ticker):0x10c569000 @action=nil, @source=nil, @class_key="Ticker", @instance_hash={:market => "coinone", :currency => "bch", :ticker => "KRW", :price => 360500.0, :bid => nil, :ask => nil, :low_24h => 350100.0, :high_24h => 369500.0, :avg_24h => nil, :volume_24h => 34075.1407, :volume_30d => nil}, @errors=[], @changes=[{:market => "coinone"}, {:currency => "bch"}, {:ticker => "KRW"}, {:price => 360500.0}, {:low_24h => 350100.0}, {:high_24h => 369500.0}, {:volume_24h => 34075.1407}], @valid=true, @instance=#<Ticker:0x10ee11f20 @initial_values=nil, @market="coinone", @currency="bch", @ticker="KRW", @price=360500.0, @bid=nil, @ask=nil, @low_24h=350100.0, @high_24h=369500.0, @avg_24h=nil, @volume_24h=34075.1407, @volume_30d=nil>, @initial_values={}>
insert error - Index out of bounds
#<Crecto::Changeset::Changeset(Ticker):0x10ee23900 @action=nil, @source=nil, @class_key="Ticker", @instance_hash={:market => "coinone", :currency => "qtum", :ticker => "KRW", :price => 12490.0, :bid => nil, :ask => nil, :low_24h => 11990.0, :high_24h => 13440.0, :avg_24h => nil, :volume_24h => 1920171.575, :volume_30d => nil}, @errors=[], @changes=[{:market => "coinone"}, {:currency => "qtum"}, {:ticker => "KRW"}, {:price => 12490.0}, {:low_24h => 11990.0}, {:high_24h => 13440.0}, {:volume_24h => 1920171.575}], @valid=true, @instance=#<Ticker:0x10ee11dc0 @initial_values=nil, @market="coinone", @currency="qtum", @ticker="KRW", @price=12490.0, @bid=nil, @ask=nil, @low_24h=11990.0, @high_24h=13440.0, @avg_24h=nil, @volume_24h=1920171.575, @volume_30d=nil>, @initial_values={}>
insert error - Unexpected byte 0xff in UTF-8 byte sequence
#<Crecto::Changeset::Changeset(Ticker):0x10ee294e0 @action=nil, @source=nil, @class_key="Ticker", @instance_hash={:market => "coinone", :currency => "etc", :ticker => "KRW", :price => 13910.0, :bid => nil, :ask => nil, :low_24h => 13260.0, :high_24h => 14170.0, :avg_24h => nil, :volume_24h => 1039445.4814, :volume_30d => nil}, @errors=[], @changes=[{:market => "coinone"}, {:currency => "etc"}, {:ticker => "KRW"}, {:price => 13910.0}, {:low_24h => 13260.0}, {:high_24h => 14170.0}, {:volume_24h => 1039445.4814}], @valid=true, @instance=#<Ticker:0x10ee11d10 @initial_values=nil, @market="coinone", @currency="etc", @ticker="KRW", @price=13910.0, @bid=nil, @ask=nil, @low_24h=13260.0, @high_24h=14170.0, @avg_24h=nil, @volume_24h=1039445.4814, @volume_30d=nil>, @initial_values={}>

Without insert

#<Crecto::Changeset::Changeset(Ticker):0x10e76b000 @action=nil, @source=nil, @class_key="Ticker", @instance_hash={:market => "coinone", :currency => "bch", :ticker => "KRW", :price => 360500.0, :bid => nil, :ask => nil, :low_24h => 350100.0, :high_24h => 369500.0, :avg_24h => nil, :volume_24h => 34075.1407, :volume_30d => nil}, @errors=[], @changes=[{:market => "coinone"}, {:currency => "bch"}, {:ticker => "KRW"}, {:price => 360500.0}, {:low_24h => 350100.0}, {:high_24h => 369500.0}, {:volume_24h => 34075.1407}], @valid=true, @instance=#<Ticker:0x111013f20 @initial_values=nil, @market="coinone", @currency="bch", @ticker="KRW", @price=360500.0, @bid=nil, @ask=nil, @low_24h=350100.0, @high_24h=369500.0, @avg_24h=nil, @volume_24h=34075.1407, @volume_30d=nil>, @initial_values={}>
#<Crecto::Changeset::Changeset(Ticker):0x111014900 @action=nil, @source=nil, @class_key="Ticker", @instance_hash={:market => "coinone", :currency => "qtum", :ticker => "KRW", :price => 12490.0, :bid => nil, :ask => nil, :low_24h => 11990.0, :high_24h => 13440.0, :avg_24h => nil, :volume_24h => 1920056.2903, :volume_30d => nil}, @errors=[], @changes=[{:market => "coinone"}, {:currency => "qtum"}, {:ticker => "KRW"}, {:price => 12490.0}, {:low_24h => 11990.0}, {:high_24h => 13440.0}, {:volume_24h => 1920056.2903}], @valid=true, @instance=#<Ticker:0x111013e70 @initial_values=nil, @market="coinone", @currency="qtum", @ticker="KRW", @price=12490.0, @bid=nil, @ask=nil, @low_24h=11990.0, @high_24h=13440.0, @avg_24h=nil, @volume_24h=1920056.2903, @volume_30d=nil>, @initial_values={}>
#<Crecto::Changeset::Changeset(Ticker):0x1110142a0 @action=nil, @source=nil, @class_key="Ticker", @instance_hash={:market => "coinone", :currency => "etc", :ticker => "KRW", :price => 13910.0, :bid => nil, :ask => nil, :low_24h => 13260.0, :high_24h => 14170.0, :avg_24h => nil, :volume_24h => 1039445.4814, :volume_30d => nil}, @errors=[], @changes=[{:market => "coinone"}, {:currency => "etc"}, {:ticker => "KRW"}, {:price => 13910.0}, {:low_24h => 13260.0}, {:high_24h => 14170.0}, {:volume_24h => 1039445.4814}], @valid=true, @instance=#<Ticker:0x111013dc0 @initial_values=nil, @market="coinone", @currency="etc", @ticker="KRW", @price=13910.0, @bid=nil, @ask=nil, @low_24h=13260.0, @high_24h=14170.0, @avg_24h=nil, @volume_24h=1039445.4814, @volume_30d=nil>, @initial_values={}>
#<Crecto::Changeset::Changeset(Ticker):0x111018c00 @action=nil, @source=nil, @class_key="Ticker", @instance_hash={:market => "coinone", :currency => "btc", :ticker => "KRW", :price => 6445000.0, :bid => nil, :ask => nil, :low_24h => 6080500.0, :high_24h => 6989500.0, :avg_24h => nil, :volume_24h => 18770.8274, :volume_30d => nil}, @errors=[], @changes=[{:market => "coinone"}, {:currency => "btc"}, {:ticker => "KRW"}, {:price => 6445000.0}, {:low_24h => 6080500.0}, {:high_24h => 6989500.0}, {:volume_24h => 18770.8274}], @valid=true, @instance=#<Ticker:0x111013d10 @initial_values=nil, @market="coinone", @currency="btc", @ticker="KRW", @price=6445000.0, @bid=nil, @ask=nil, @low_24h=6080500.0, @high_24h=6989500.0, @avg_24h=nil, @volume_24h=18770.8274, @volume_30d=nil>, @initial_values={}>
#<Crecto::Changeset::Changeset(Ticker):0x1110185a0 @action=nil, @source=nil, @class_key="Ticker", @instance_hash={:market => "coinone", :currency => "eth", :ticker => "KRW", :price => 386950.0, :bid => nil, :ask => nil, :low_24h => 349250.0, :high_24h => 395900.0, :avg_24h => nil, :volume_24h => 282213.7917, :volume_30d => nil}, @errors=[], @changes=[{:market => "coinone"}, {:currency => "eth"}, {:ticker => "KRW"}, {:price => 386950.0}, {:low_24h => 349250.0}, {:high_24h => 395900.0}, {:volume_24h => 282213.7917}], @valid=true, @instance=#<Ticker:0x111013c60 @initial_values=nil, @market="coinone", @currency="eth", @ticker="KRW", @price=386950.0, @bid=nil, @ask=nil, @low_24h=349250.0, @high_24h=395900.0, @avg_24h=nil, @volume_24h=282213.7917, @volume_30d=nil>, @initial_values={}>
#<Crecto::Changeset::Changeset(Ticker):0x11101cf00 @action=nil, @source=nil, @class_key="Ticker", @instance_hash={:market => "coinone", :currency => "xrp", :ticker => "KRW", :price => 294.0, :bid => nil, :ask => nil, :low_24h => 271.0, :high_24h => 298.0, :avg_24h => nil, :volume_24h => 170112858.829, :volume_30d => nil}, @errors=[], @changes=[{:market => "coinone"}, {:currency => "xrp"}, {:ticker => "KRW"}, {:price => 294.0}, {:low_24h => 271.0}, {:high_24h => 298.0}, {:volume_24h => 170112858.829}], @valid=true, @instance=#<Ticker:0x111013bb0 @initial_values=nil, @market="coinone", @currency="xrp", @ticker="KRW", @price=294.0, @bid=nil, @ask=nil, @low_24h=271.0, @high_24h=298.0, @avg_24h=nil, @volume_24h=170112858.829, @volume_30d=nil>, @initial_values={}>

I found that if I replace nil values with values then all 6 will be recorded. But rescue will print out
PG::ResultSet#read returned a PG::Numeric. A (Float64 | Nil) was expected.

                begin
                  Repo.insert(changeset)
                rescue ex
                  puts "insert error - #{ex}"
                end

[fridgerator]

@onemanstartup are you able to share your schema and json that you are parsing? I tried replicating locally and couldn't

[onemanstartup]

@fridgerator This is the code. I don't understand why records persists but throw errors.. strange..

class Ticker < Crecto::Model
  schema "tickers", primary_key: false do
    field :market, String
    field :currency, String
    field :ticker, String
    field :price, Float64
    field :bid, Float64
    field :ask, Float64
    field :low_24h, Float64
    field :high_24h, Float64
    field :avg_24h, Float64
    field :volume_24h, Float64
    field :volume_30d, Float64

    set_created_at_field nil
    set_updated_at_field nil
  end

  validate_required [:market]
  validate_format :market, /^[a-zA-Z]*$/
end

CREATE TABLE tickers (
  time       TIMESTAMPTZ       NOT NULL DEFAULT now(),
  market     TEXT              NOT NULL,
  currency   TEXT              NOT NULL,
  ticker     TEXT              NOT NULL,
  price      numeric           NOT NULL,
  bid        numeric           NULL,
  ask        numeric           NULL,
  low_24h    numeric           NULL,
  high_24h   numeric           NULL,
  avg_24h    numeric           NULL,
  volume_24h numeric           NULL,
  volume_30d numeric           NULL
);

{"bch":{"volume":"214659.4839","last":"418000","yesterday_last":"352400","yesterday_low":"348000","high":"451500","currency":"bch","low":"349700","yesterday_first":"353500","yesterday_volume":"12689.9568","yesterday_high":"360300","first":"351400"},"qtum":{"volume":"2417040.9742","last":"12830","yesterday_last":"11960","yesterday_low":"11790","high":"13200","currency":"qtum","low":"11950","yesterday_first":"12060","yesterday_volume":"657970.4149","yesterday_high":"12360","first":"11980"},"errorCode":"0","etc":{"volume":"983336.4399","last":"13700","yesterday_last":"13430","yesterday_low":"13230","high":"14050","currency":"etc","low":"13260","yesterday_first":"13470","yesterday_volume":"477616.7548","yesterday_high":"13690","first":"13440"},"result":"success","btc":{"volume":"4240.6907","last":"6372000","yesterday_last":"6347000","yesterday_low":"6200000","high":"6486500","currency":"btc","low":"6300000","yesterday_first":"6332500","yesterday_volume":"5541.9419","yesterday_high":"6499500","first":"6375000"},"timestamp":"1508249874","eth":{"volume":"83368.7347","last":"367100","yesterday_last":"380400","yesterday_low":"368500","high":"383000","currency":"eth","low":"365000","yesterday_first":"375000","yesterday_volume":"165312.9340","yesterday_high":"394900","first":"381400"},"xrp":{"volume":"306820375.5870","last":"288","yesterday_last":"319","yesterday_low":"293","high":"323","currency":"xrp","low":"278","yesterday_first":"296","yesterday_volume":"334760612.5759","yesterday_high":"327","first":"323"}}

            JSON.parse(response.body).each do |k, v|
              next if k == "errorCode" || k == "result" || k == "timestamp"
              ticker = Mapping::Ticker.from_json(v.to_json)
              new_ticker            = Ticker.new
              new_ticker.market     = @market
              new_ticker.currency   = k.to_s
              new_ticker.ticker     = "KRW"
              new_ticker.price      = ticker.last.to_f
              new_ticker.low_24h    = ticker.low.to_f
              new_ticker.high_24h   = ticker.high.to_f
              new_ticker.volume_24h = ticker.volume.to_f
              new_ticker.avg_24h = 0.to_f
              new_ticker.volume_30d = 0.to_f
              new_ticker.bid = 0.to_f
              new_ticker.ask = 0.to_f
              changeset = Ticker.changeset(new_ticker)
              if changeset.valid?
                begin
                  Repo.insert(changeset)
                rescue ex
                  puts "insert error - #{ex}"
                end
              else
                puts changeset.errors
              end
            end

[fridgerator]

If you change the numeric fields to float it works

[onemanstartup]

@fridgerator I tried set types to PG::Numeric in Crecto::Model, but it's doesn't work. It seems either I don't understand how casting works in pg shard or crystal lacks decimal field leads to this. Anyway thanks for helping out, will save in floats for now, until figure out how this works.

[fridgerator]

yeah you'll have to use float right now, PG::Numeric wont work in the model right now. I'll add an issue to support this type. Thanks!

[timkendall]

I'm still getting this error with Crecto 0.8.4. Here's my Postgres schema:

CREATE TABLE foods(
  id SERIAL PRIMARY KEY,
  upc VARCHAR,
  usda_db_id VARCHAR,
  name VARCHAR NOT NULL,
  
  description VARCHAR,
  
  protein  FLOAT,
  fat  FLOAT,
  carbohydrates FLOAT,

  serving_unit VARCHAR,
  serving_label VARCHAR,
  serving_value INTEGER,

  created_at TIMESTAMP,
  updated_at TIMESTAMP
);

[timkendall]

Looks like it was bad permissions settings on my part. Solved by making my dev postgres user a superuser with:

ALTER USER postgres WITH SUPERUSER;

[timkendall]

Nevermind, still get the IndexError when using Repo.get_by/2 or Repo.insert/1

[fridgerator]

@timkendall Does this happen all the time or randomly? It would be great to get a log of the sql generated

[timkendall]

All of the time - I'll add that when I get back to my personal computer.

[fridgerator]

@timkendall Creating a table with withe sql you posted, the following is working for me (crystal 0.24.2, crecto master branch)

require "pg"
require "./src/crecto"

Crecto::DbLogger.set_handler(STDOUT)

module Repo
  extend Crecto::Repo

  config do |conf|
    conf.adapter = Crecto::Adapters::Postgres
    conf.database = "issue_95"
    conf.hostname = "localhost"
    conf.port = 5432
  end
end

class Food < Crecto::Model
  schema "foods" do
    field :upc, String
    field :usda_db_id, String
    field :name, String
    field :description, String
    field :protein, Float64
    field :fat, Float64
    field :carbohydrates, Float64
    field :serving_unit, String
    field :serving_label, String
    field :serving_value, Int32
  end
end

food = Food.new
food.upc = "123"
food.usda_db_id = "456"
food.name = "hey"
food.description = "description"
food.protein = 342.222
food.fat = 123.234
food.carbohydrates = 54.234
food.serving_unit = "g"
food.serving_label = "good food"
food.serving_value = 123

changeset = Repo.insert(food)
puts Repo.get_by(Food, id: changeset.instance.id)

[watzon]

@timkendall did that fix end up working for you? Or can you let me know if you're still getting this error? I know this is an old issue.