Looks like greenbone/vulnerability-tests version 5958053f7549 is corrupted

Looks like greenbone/vulnerability-tests image version 5958053f7549 is corrupted.

I trying to run communirty version, but in the gvmd container logs I see the repeated error message:

md manage:   INFO:2023-05-19 10h48.26 UTC:86: SCAP database does not exist (yet), skipping CERT severity score update
md manage:   INFO:2023-05-19 10h48.26 UTC:86: sync_cert: Updating CERT info succeeded.
noname.xml:670532: parser error : internal error: Huge input lookup
src=g&hsa_net=adwords&hsa_mt=&hsa_ver=3&hsa_ad=111046699381&
                                                                               ^
noname.xml:670532: parser error : attributes construct error

^
noname.xml:670532: parser error : Couldn't find end of Start Tag reference line 670532

^
noname.xml:670532: parser error : EndTag: '</' not found

^
md manage:WARNING:2023-05-19 10h49.42 UTC:84: update_scap_cpes_from_file: Failed to parse element
md manage:WARNING:2023-05-19 10h49.52 UTC:132: update_scap: No SCAP db present, rebuilding SCAP db from scratch
md manage:   INFO:2023-05-19 10h49.52 UTC:134: OSP service has different VT status (version 202305191005) from database (version (null), 0 VTs). Starting update ...
md manage:   INFO:2023-05-19 10h49.52 UTC:132: update_scap: Updating data from feed
md manage:   INFO:2023-05-19 10h49.52 UTC:132: Updating CPEs
noname.xml:670532: parser error : internal error: Huge input lookup
src=g&amp;hsa_net=adwords&amp;hsa_mt=&amp;hsa_ver=3&amp;hsa_ad=111046699381&amp;
                                                                               ^
noname.xml:670532: parser error : attributes construct error

^
noname.xml:670532: parser error : Couldn't find end of Start Tag reference line 670532

^
noname.xml:670532: parser error : EndTag: '</' not found

^
md manage:WARNING:2023-05-19 10h51.53 UTC:132: update_scap_cpes_from_file: Failed to parse element
md manage:   INFO:2023-05-19 11h02.00 UTC:134: Updating VTs in database ... 128431 new VTs, 0 changed VTs
md manage:   INFO:2023-05-19 11h02.03 UTC:134: Updating VTs in database ... done (128431 VTs).
md manage:WARNING:2023-05-19 11h02.05 UTC:406: update_scap: No SCAP db present, rebuilding SCAP db from scratch
md manage:   INFO:2023-05-19 11h02.06 UTC:406: update_scap: Updating data from feed
md manage:   INFO:2023-05-19 11h02.06 UTC:406: Updating CPEs
noname.xml:670532: parser error : internal error: Huge input lookup
src=g&amp;hsa_net=adwords&amp;hsa_mt=&amp;hsa_ver=3&amp;hsa_ad=111046699381&amp;
                                                                               ^
noname.xml:670532: parser error : attributes construct error

^
noname.xml:670532: parser error : Couldn't find end of Start Tag reference line 670532

^
noname.xml:670532: parser error : EndTag: '</' not found

^
md manage:WARNING:2023-05-19 11h04.07 UTC:406: update_scap_cpes_from_file: Failed to parse element
md manage:WARNING:2023-05-19 11h04.16 UTC:454: update_scap: No SCAP db present, rebuilding SCAP db from scratch
md manage:   INFO:2023-05-19 11h04.17 UTC:454: update_scap: Updating data from feed
md manage:   INFO:2023-05-19 11h04.17 UTC:454: Updating CPEs
noname.xml:670532: parser error : internal error: Huge input lookup
src=g&amp;hsa_net=adwords&amp;hsa_mt=&amp;hsa_ver=3&amp;hsa_ad=111046699381&amp;
                                                                               ^
noname.xml:670532: parser error : attributes construct error

^
noname.xml:670532: parser error : Couldn't find end of Start Tag reference line 670532

^
noname.xml:670532: parser error : EndTag: '</' not found

Could you please check if automation is build such image are working corretly?

3 Likes

Hell Rakanskyi, and welcome to the Greenbone community.

Thank you for the bug report, I was able to reproduce the problem. I have opened an internal issue to investigate this, we’ll get back to you here once we have more information.

I’m also experiencing the same issue. If this test fails, do the remaining tests continue to complete?

Increasing LIBXML2’s default limit (which is 10M 10000000) for nodeset sizes seem to have solved it for for me:

export LIBXML_MAX_NODESET_LENGTH=40000000

Edit: 40MB was selected by random, IIRC it work even with 20M.

1 Like

Just for the record I had to install a newer version of libxml2 for the above to work. Have a look at the “install_libxml2” function in install-gse.sh here: https://github.com/martinboller/gse

Is there a way to rollback to pull from a previous feed version while this is being investigated?

Pure guesswork, however I’d expect the dataset to keep growing, so the workaround with increasing the nodeset length and/or Greenbone Devs to use XML_PARSE_HUGE in gvmd would likely be a better way forward. The data isn’t corrupt, but merely has outgrown the (artificial?) limit of libxml2

3 Likes

Thank you for the analysis and workaround(s) martinboller.

We are currently working on an official solution for this problem. Unfortunately, we cannot make use of a newer libxml2 easily, so this will take a bit more time.

We may also try a workaround on the feed side, but this is still to be determined.

We will update anyone involved here when we have new information!

6 Likes

To follow-up / as a reference / for tracking purposes:

1 Like

I updated the GVM feed successfully and rebooted my box, but I keep seeing the XML parse error on gvmd status and logs. Any ideas? Thanks.

Same for me, without the reboot. and since then (yesterday) one CPU-core is at 100% with postgres and xml_split in change. UI says “Aktualisierung läuft…” (Refreshing) for everything except NVTs, which are two days old now.

The solution sounds like the problem should be gone after a feed refresh, or do we still need the new library?

1 Like

I am going to update to the latest version of GVM on Kali, and see if that fixes the issue… Thanks!

I’m running OpenVAS on a VM and experiencing the same issue. I’ve tried restarting my VM and also tried re-running the feed updates, my feed status is still “stuck” in an updating state:

Output of /var/log/gvm/gvmd.log

@bricks Do you maybe know if there is a way to manually force all feed updates to stop? I would like to try stopping the feeds that are currently in an “updating” state and then re-run the feed update scripts individually to see if that works.

I managed to resolve the issue by creating a new VM from a backup image that I had that was taken before the xml feed update became too large as described here, but I would like to try and fix the issue on my original VM that is stuck in feed update mode if possible because I have a lot (a few hundred) of scans and targets set up that I would like to preserve.

1 Like