(URGENT) XML not being read in from custom model

[opoudjis]

Another blocker.

Consider the following code, which uses an externally defined class for content:

require "lutaml/model"
require "lutaml/model/xml_adapter/nokogiri_adapter"
require "relaton"
require "relaton/cli"

Lutaml::Model::Config.configure do |config|
  config.xml_adapter = Lutaml::Model::XmlAdapter::NokogiriAdapter
  config.yaml_adapter = Lutaml::Model::YamlAdapter::StandardYamlAdapter
  config.json_adapter = Lutaml::Model::JsonAdapter::StandardJsonAdapter
end

module SimpleModel
  class Bibdata < Lutaml::Model::Serializable
    # attribute :code, :string
    model ::RelatonBib::BibliographicItem
  end

  class Address < Lutaml::Model::Serializable
    attribute :street, :string, raw: true
    attribute :city, :string, raw: true
    attribute :bibdata, Bibdata

    yaml do
      map "street", to: :street
      map "city", to: :city
      map "bibdata", to: :bibdata, with: { from: :bibdata_from_yaml,
                                           to: :bibdata_to_yaml }
    end

    xml do
      root "address"
      map_element "street", to: :street # ,  with: { from: :street_from_xml, to: :street_to_xml }
      map_element "city", to: :city # , with: { from: :city_from_xml, to: :city_to_xml }
      map_element "bibdata", to: :bibdata, with: { from: :bibdata_from_xml,
                                                   to: :bibdata_to_xml }
    end

    def bibdata_from_yaml(model, value)
      value and !value.empty? or return
      model.bibdata = ::Relaton::Cli::YAMLConvertor.convert_single_file(value)
    end 

    def bibdata_to_yaml(model, doc)
      doc["bibdata"] = model.bibdata&.to_hash
    end 

    def bibdata_from_xml(model, node)
      model.bibdata = node # Relaton::Cli.parse_xml(node.content)
    end 

    def bibdata_to_xml(model, parent, doc)
      b = model.bibdata or return
      elem = b.to_xml(bibdata: true, date_format: :full)
      doc.add_element(parent, elem)
    end 

    def city_from_xml(model, node)
      model.city = node
    end

    def city_to_xml(model, parent, doc)
      doc.add_element(parent, "<city>#{model.city}</city>")
    end

    def street_from_xml(model, node)
      model.street = node
    end

    def street_to_xml(model, parent, doc)
      doc.add_element(parent, "<street>#{model.street}</street>")
    end
  end


  xml = <<~XML
      <address>
      <street>
          A <p>b</p> B <p>c</p> C
      </street>
      <bibdata type="collection" schema-version="v1.2.8">  <title type="title-main" format="text/plain" language="en">JCGM Collection 1</title>
      <docidentifier type="iso">JCGM 12345</docidentifier>
      <date type="created">
        <on>2020-01-01</on>
      </date>
      <date type="issued">
        <on>2020-01-01</on>
      </date>
      <edition>1</edition>
      <copyright>
        <from>2020</from>
        <owner>
          <organization>
            <name>International Organization for Standardization</name>
            <abbreviation>JCMG</abbreviation>
          </organization>
        </owner>
      </copyright>
    </bibdata>
      </address>
  XML

  yaml = <<~YAML
    street: A <a>N</a> B
    city: B <a>N</a> B
    bibdata:
      title:
        type: title-main
        language: en
        content: JCGM Collection 1
      type: collection
      docid:
        type: iso
        id: JCGM 12345
      edition: 1
      date:
        - type: created
          value: "2020"
        - type: issued
          value: "2020"
      copyright:
        owner:
          name: International Organization for Standardization
          abbreviation: JCMG
        from: "2020"
  YAML

  puts "YAML > YAML"
  sample = SimpleModel::Address.from_yaml(yaml).to_yaml
  puts sample
  puts "XML > YAML"
  sample = SimpleModel::Address.from_xml(xml).to_yaml
  puts sample
  puts "YAML > XML"
  sample = SimpleModel::Address.from_yaml(yaml).to_xml
  puts sample
  puts "XML > XML"
  sample = SimpleModel::Address.from_xml(xml).to_xml
  puts sample

And I am not going to come up with a simple test class instead of ::RelatonBib::BibliographicItem, because after three months of this, I'm not going to spend an extra couple of hours doing mockups.

The custom model is working fine for reading YAML, via ::Relaton::Cli::YAMLConvertor.convert_single_file(value). Both the YAML and the XML outputs work.

The custom model fails to read the XML, other than the schema-version attribute: all it outputs is:

<bibdata schema-version="v1.2.9"/>

bibdata:
  schema-version: v1.2.9

In Lutaml::Model::Serialize::ClassMethods#apply_mappings, doc.to_xml has bibdata in it intact:

"<metanorma-collection>\n  <bibdata type=\"collection\">\n    <title type=\"title-main\" format=\"text/plain\" language=\"en\">ISO Collection 1</title>\n    <docidentifier type=\"iso\">ISO 12345</docidentifier>\n    <date type=\"created\">\n      <on>2020-01-01</on>\n    </date>\n    <date type=\"issued\">\n      <on>2020-01-01</on>\n    </date>\n    <edition>1</edition>\n    <copyright>\n      <from>2020</from>\n      <owner>\n        <organization>\n          <name>International Organization for Standardization</name>\n          <abbreviation>ISO</abbreviation>\n        </organization>\n      </owner>\n    </copyright>\n  </bibdata>\n  <directive>\n    <key>documents-external</key>\n....

And the conversion to mapping_hash in

doc_hash = doc.parse_element(doc.root, self, :xml)

preserves the content:

{"text"=>["\n  ", "\n  ", "\n  "],
 "street"=>"\n      A <p>b</p> B <p>c</p> C\n  ",
 "bibdata"=>
  {"text"=>["  ", "\n  ", "\n  ", "\n  ", "\n  ", "\n  ", "\n"],
   "title"=>{"text"=>"JCGM Collection 1", "type"=>"title-main", "format"=>"text/plain", "language"=>"en"},
   "docidentifier"=>{"text"=>"JCGM 12345", "type"=>"iso"},
   "date"=>
    [{"text"=>["\n    ", "\n  "], "on"=>{"text"=>"2020-01-01"}, "type"=>"created"},
     {"text"=>["\n    ", "\n  "], "on"=>{"text"=>"2020-01-01"}, "type"=>"issued"}],
   "edition"=>{"text"=>"1"},
   "copyright"=>
    {"text"=>["\n    ", "\n    ", "\n  "],
     "from"=>{"text"=>"2020"},
     "owner"=>
      {"text"=>["\n      ", "\n    "],
       "organization"=>
        {"text"=>["\n        ", "\n        ", "\n      "],
         "name"=>{"text"=>"International Organization for Standardization"},
         "abbreviation"=>{"text"=>"JCMG"}}}},
   "type"=>"collection",
   "schema-version"=>"v1.2.8"}}

I sure this is going wrong because of the foundational problem with mapping_hash which I have pointed out, that it does not cope with mixed content, or with leading whitespace. The complication here is that this is not happening when writing XML, but when reading XML: mapping_hash preserves the information ostensibly from the foregoing, but just doesn't pass it on to serialisation though apply_mappings(doc_hash, format).

In fact, when I make

    xml do
      root "address", mixed: true

which should be coping with mixed content, not only does XML > YAML fail to output the bibdata, but XML > XML crashes:

/Users/nickn/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/lutaml-model-0.3.14/lib/lutaml/model/xml_adapter/xml_document.rb:142:in `add_to_xml': undefined method `new' for nil (NoMethodError)

            options[:mapper_class].new.send(rule.custom_methods[:to], element, xml.parent, xml)
                                  ^^^^
	from /Users/nickn/.rbenv/versions/3.3.4/lib/ruby/gems/3.3.0/gems/lutaml-model-0.3.14/lib/lutaml/model/xml_adapter/nokogiri_adapter.rb:83:in `block (2 levels) in build_ordered_element'

[ronaldtse]

@HassanAkbar @suleman-uzair has this PR fixed this issue?

#119

[HassanAkbar]

@ronaldtse

@HassanAkbar @suleman-uzair has this PR fixed this issue?

#119

#119 will not fix this issue; it was for a separate issue that occurred when using custom models with the render_default check.

[ronaldtse]

Got it.

[ronaldtse]

@HassanAkbar so #126 fixes part of this problem but not fully?

[HassanAkbar]

@ronaldtse It fixes the crash and adds an internal representation of the node to the mapping hash. So I think this problem is fully solved by #126.

@opoudjis Is there something else needed for this ticket?

[ronaldtse]

The latest gem is released for this.

@opoudjis could you please help try if this works? Thanks.