Facing issues while parsing More then 300 MB XML file.

[kanad-smartbear]

We are trying to parse an XML file [more than 300 MB]

We have a working code where XML parsing is a working file with small data sets like 20,30,50 up to 250 MB file size.
Were when we try with more than 300MB file it's not working.

class ExportParser < Nokogiri::XML::SAX::Document
  attr_accessor :project, :import_data

  include MethodMissing

  def initialize(context = nil, keep_hierarchy: false, fix_duplicated_actionwords: false, only_aw: false)
    @import_data = Importers::ProjectExport::ImportData.new
    @import_data.fix_duplicated_actionwords = fix_duplicated_actionwords
    @project_parser = Importers::ProjectExport::ProjectParser.new(import_data, context)
    @project_parser.keep_hierarchy = keep_hierarchy
    @project_parser.only_aw = only_aw
    @project = @project_parser.context
  end

  def imported
    @project
  end

  def start_element(name, attrs = [])
    @project_parser.start_element(name, attrs)
  end

  def end_element(name)
    @project_parser.end_element(name)
  end

  def characters(value)
    @project_parser.characters(value)
  end
end>

Staging/Production

This is how we are calling the parsing method

     export_parser_instance = Importers::ProjectExport::ExportParser.new(@context, keep_hierarchy: @keep_hierarchy, only_aw: @only_aw)
      parser = Nokogiri::XML::SAX::Parser.new(export_parser_instance, "UTF-8")
      xml_parser = parser.parse(@xml)

Any suggestion on this, how we can fix this issue?

[flavorjones]

Can you explain more about what "not working" means?

If you add an error handler to your sax document handler, you should get a report of any parsing errors that are encountered along the way.

Some questions for you:

• what's the output from nokogiri -v, escaped by triple-backtick
• can you help us reproduce what you're seeing
• what errors are you seeing during sax parsing (see comment above)

[kanad-smartbear]

nokogiri -v

Nokogiri (1.15.5)
---
warnings: []
nokogiri:
version: 1.15.5
cppflags:
- "-I/home/kanadkumar/.asdf/installs/ruby/2.7.6/lib/ruby/gems/2.7.0/gems/nokogiri-1.15.5-x86_64-linux/ext/nokogiri"
- "-I/home/kanadkumar/.asdf/installs/ruby/2.7.6/lib/ruby/gems/2.7.0/gems/nokogiri-1.15.5-x86_64-linux/ext/nokogiri/include"
- "-I/home/kanadkumar/.asdf/installs/ruby/2.7.6/lib/ruby/gems/2.7.0/gems/nokogiri-1.15.5-x86_64-linux/ext/nokogiri/include/libxml2"
ldflags: []
ruby:
version: 2.7.6
platform: x86_64-linux
gem_platform: x86_64-linux
description: ruby 2.7.6p219 (2022-04-12 revision c9c2245c0a) [x86_64-linux]
engine: ruby
libxml:
source: packaged
precompiled: true
patches:
- 0001-Remove-script-macro-support.patch
- 0002-Update-entities-to-remove-handling-of-ssi.patch
- 0003-libxml2.la-is-in-top_builddir.patch
- '0009-allow-wildcard-namespaces.patch'
- 0010-update-config.guess-and-config.sub-for-libxml2.patch
- 0011-rip-out-libxml2-s-libc_single_threaded-support.patch
libxml2_path: "/home/kanadkumar/.asdf/installs/ruby/2.7.6/lib/ruby/gems/2.7.0/gems/nokogiri-1.15.5-x86_64-linux/ext/nokogiri"
memory_management: ruby
iconv_enabled: true
compiled: 2.11.6
loaded: 2.11.6
libxslt:
source: packaged
precompiled: true
patches:
- 0001-update-config.guess-and-config.sub-for-libxslt.patch
datetime_enabled: true
compiled: 1.1.39
loaded: 1.1.39
other_libraries:
zlib: 1.2.13
libgumbo: 1.0.0-nokogiri

[kanad-smartbear]

xml_parser = parser.parse(@xml)

at this process got stuck and we are not getting errors as well.
earlier for a smaller file it was consuming more memory so we increased the RAM size form 3 GB to 5GB, and then its started working for up to 200 MB files.

[flavorjones]

Unfortunately I can't help you much if I can't reproduce what you're seeing. Can you try to write a reproduction script that doesn't include non-Nokogiri code and that uses a file you can either share or that can be generated dynamically?