Problems with greenbone-feed-sync / gvmd service

hi there, i got a problem with the greenbone-feed-sync / gvmd service.
syncing the feeds works well, tried it in two different ways, and it worked without problems / errors.

greenbone-feed-sync --type GVMD_DATA
greenbone-feed-sync --type SCAP
greenbone-feed-sync --type CERT

and

greenbone-nvt-sync
greenbone-certdata-sync
greenbone-scapdata-sync

BUT after the sync i’m gettings massiv errors / warning in the gvmd log (/opt/gvm/var/log/gvm/gvmd.log) file.

md   main:MESSAGE:2020-12-06 08h43.27 utc:46982:    Greenbone Vulnerability Manager version 20.08.0~git-3a1f0705-gvmd-20.08 (GIT revision 3a1f0705-gvmd-20.08) (DB revision 233)
util gpgme:MESSAGE:2020-12-06 08h43.36 utc:46989: Setting GnuPG dir to '/opt/gvm/var/lib/gvm/gvmd/gnupg'
util gpgme:MESSAGE:2020-12-06 08h43.36 utc:46989: Using OpenPGP engine version '2.2.19'
md manage:   INFO:2020-12-06 08h44.02 utc:47106: update_scap: Updating data from feed
md manage:   INFO:2020-12-06 08h44.02 utc:47106: Updating CPEs
md manage:WARNING:2020-12-06 08h44.03 utc:47106: split_xml_file: system failed with ret 32512, 1 (127), xml_split -s40Mb split.xml && head -n 2 split-00.xml > head.xml && echo '</cpe-list>' > tail.xml && for F in split-*.xml; do    awk 'NR>3 {print last} {last=$0}' $F > body.xml    && cat head.xml body.xml tail.xml > $F;    done
md manage:WARNING:2020-12-06 08h44.03 utc:47106: split_xml_file: and failed to chdir back
md manage:WARNING:2020-12-06 08h44.03 utc:47106: update_scap_cpes: Failed to split CPEs, attempting with full file

this goes on and on and is always showing the same error, additionaly it also fills up the /tmp directory with specific tmp files from the gvmd service. the directory getting really big >30GB and at some point the hard disk of the virtual machine is full and the gvmd logs is producing even more errors.

it looks like this:

root@scan:/tmp# du -h
230M	./systemd-private-0abe87b366e74665b9e9481a70ddd0f3-gvmd.service-jeP1Rg/tmp/gvmd-split-xml-file-tPxI8L
230M	./systemd-private-0abe87b366e74665b9e9481a70ddd0f3-gvmd.service-jeP1Rg/tmp/gvmd-split-xml-file-im9Vv3
230M	./systemd-private-0abe87b366e74665b9e9481a70ddd0f3-gvmd.service-jeP1Rg/tmp/gvmd-split-xml-file-pK8sof
230M	./systemd-private-0abe87b366e74665b9e9481a70ddd0f3-gvmd.service-jeP1Rg/tmp/gvmd-split-xml-file-lOYHQw
230M	./systemd-private-0abe87b366e74665b9e9481a70ddd0f3-gvmd.service-jeP1Rg/tmp/gvmd-split-xml-file-4mEwsZ
230M	./systemd-private-0abe87b366e74665b9e9481a70ddd0f3-gvmd.service-jeP1Rg/tmp/gvmd-split-xml-file-6o8aZH
230M	./systemd-private-0abe87b366e74665b9e9481a70ddd0f3-gvmd.service-jeP1Rg/tmp/gvmd-split-xml-file-H7r9s6
230M	./systemd-private-0abe87b366e74665b9e9481a70ddd0f3-gvmd.service-jeP1Rg/tmp/gvmd-split-xml-file-skgqvi
230M	./systemd-private-0abe87b366e74665b9e9481a70ddd0f3-gvmd.service-jeP1Rg/tmp/gvmd-split-xml-file-6bEXD7
2.1G	./systemd-private-0abe87b366e74665b9e9481a70ddd0f3-gvmd.service-jeP1Rg/tmp
2.1G	./systemd-private-0abe87b366e74665b9e9481a70ddd0f3-gvmd.service-jeP1Rg

what i tried is to restart all related services, resyncing the feeds but without success. beside the errors in the gvmd services all services running without problems / errors.

systemctl restart gvmd
systemctl restart gsad
systemctl restart ospd-openvas
systemctl restart nginx
systemctl restart postgresql
systemctl restart redis-server
  • is there a method to reset the postgresql feed database?
  • how i can find the cause of the problem?

GVM versions

gsad: Greenbone Security Assistant 20.08.0~git-17a736a39-gsa-20.08
gvmd: Greenbone Vulnerability Manager 20.08.0~git-3a1f0705-gvmd-20.08
GIT revision 3a1f0705-gvmd-20.08
Manager DB revision 233
openvas-scanner: OpenVAS 20.8.0 gvm-libs 20.8.0~git-3597093-gvm-libs-20.08
gvm-libs:

Environment

Operating system: Ubuntu 20.04.1 LTS
Kernel: 5.4.0-56-generic #62-Ubuntu SMP Mon Nov 23 19:20:19 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Installation method / source: from source

Seems your startup / systemd scripts are using the PrivateTmp=true. Not sure but it could be the source of your problem.

Thanks for the info but the service config file include the parameter PrivateTmp=true per default.
If you change this paramter to PrivateTmp=false the services have even more problems.

/etc/systemd/system/gvmd.service
/etc/systemd/system/gsad.service
/etc/systemd/system/ospd-openvas.service

If I start the gvmd service I constantly getting the following errors again and again.

md manage: INFO:2020-12-18 16h21.25 utc:14370: Updating CPEs
md manage:WARNING:2020-12-18 16h21.26 utc:14370: split_xml_file: system failed with ret 32512, 1 (127), xml_split -s40Mb split.xml && head -n 2 split-00.xml > head.xml && echo ‘’ > tail.xml && for F in split-*.xml; do awk ‘NR>3 {print last} {last=$0}’ $F > body.xml && cat head.xml body.xml tail.xml > $F; done
md manage:WARNING:2020-12-18 16h21.26 utc:14370: split_xml_file: and failed to chdir back
md manage:WARNING:2020-12-18 16h21.26 utc:14370: update_scap_cpes: Failed to split CPEs, attempting with full file
md manage: INFO:2020-12-18 16h21.40 utc:14413: update_scap: Updating data from feed

Seems the sync-feed-software have problems extracting the data.
I resized the disk of my VM but it doesn’t help, it just getting more and more temp files.

root@scan:/tmp# du -hs systemd-private-4fe16d73d78c4268a643fa19a2190b37-gvmd.service-m5hUXf/tmp/*
234M systemd-private-4fe16d73d78c4268a643fa19a2190b37-gvmd.service-m5hUXf/tmp/gvmd-split-xml-file-0SY95T
234M systemd-private-4fe16d73d78c4268a643fa19a2190b37-gvmd.service-m5hUXf/tmp/gvmd-split-xml-file-0g9yFK
234M systemd-private-4fe16d73d78c4268a643fa19a2190b37-gvmd.service-m5hUXf/tmp/gvmd-split-xml-file-0zXrg6
234M systemd-private-4fe16d73d78c4268a643fa19a2190b37-gvmd.service-m5hUXf/tmp/gvmd-split-xml-file-1DBdRe
234M systemd-private-4fe16d73d78c4268a643fa19a2190b37-gvmd.service-m5hUXf/tmp/gvmd-split-xml-file-1KCa7K
234M systemd-private-4fe16d73d78c4268a643fa19a2190b37-gvmd.service-m5hUXf/tmp/gvmd-split-xml-file-1OVcUp
234M systemd-private-4fe16d73d78c4268a643fa19a2190b37-gvmd.service-m5hUXf/tmp/gvmd-split-xml-file-1Qdw6O
230M systemd-private-4fe16d73d78c4268a643fa19a2190b37-gvmd.service-m5hUXf/tmp/gvmd-split-xml-file-1YCLnv
234M systemd-private-4fe16d73d78c4268a643fa19a2190b37-gvmd.service-m5hUXf/tmp/gvmd-split-xml-file-1eyTCA
234M systemd-private-4fe16d73d78c4268a643fa19a2190b37-gvmd.service-m5hUXf/tmp/gvmd-split-xml-file-1xng6a
234M systemd-private-4fe16d73d78c4268a643fa19a2190b37-gvmd.service-m5hUXf/tmp/gvmd-split-xml-file-2D6ey2
234M systemd-private-4fe16d73d78c4268a643fa19a2190b37-gvmd.service-m5hUXf/tmp/gvmd-split-xml-file-2EGqlS
234M systemd-private-4fe16d73d78c4268a643fa19a2190b37-gvmd.service-m5hUXf/tmp/gvmd-split-xml-file-2IGVP8
234M systemd-private-4fe16d73d78c4268a643fa19a2190b37-gvmd.service-m5hUXf/tmp/gvmd-split-xml-file-2bFeJw
234M systemd-private-4fe16d73d78c4268a643fa19a2190b37-gvmd.service-m5hUXf/tmp/gvmd-

76G systemd-private-4fe16d73d78c4268a643fa19a2190b37-gvmd.service-m5hUXf/tmp

Please anyone, I need help. Does anyone see this behavior of the gvmd service?

Is there a method to reset/delete all the greenbone feed data, that I can try to download and update it again?

I’m updating the feeds the following way:

su - gvm

greenbone-nvt-sync
greenbone-feed-sync --type GVMD_DATA
greenbone-feed-sync --type SCAP
greenbone-feed-sync --type CERT

This seems to be something your package maintainer seems to be enforcing because the gvmd.service shipped in gvmd doesn’t use that line.

Have you tried to run that command manually to see what is going wrong with it on your system? Is xml_split installed as mentioned in the install instructions of gvmd?

FWIW, this is basically the same issue I reported here: Feed update and SQL failure

I’m really interested in whatever solution you may find.

I’m pretty certain that the issue is not the absence of xml_split. The failure is due to the filesystem being full. I have observed the same behavior as the OP, and only get this error after multiple feed update attempts have filled up /tmp.

xml_split is present on my system.

In my case, those scripts work just fine. Can you confirm/deny whether that is the case for you?

What I observe is a failure of gvmd to take that feed data and sync it with its data in postgresql.

Thanks @cfi
I just did this apt install xml-twig-tools and it solved my problem!
Strange, that the feed update worked without errors in the first days of the installation.

Now the logs looks quit normal and the /tmp is not getting filled up.

md manage: INFO:2020-12-29 20h48.26 utc:1325835: Updating CPEs
md manage: INFO:2020-12-29 20h50.24 utc:1325835: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2020.xml
md manage: INFO:2020-12-29 20h50.49 utc:1325835: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2002.xml
md manage: INFO:2020-12-29 20h50.57 utc:1325835: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2003.xml
md manage: INFO:2020-12-29 20h50.59 utc:1325835: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2005.xml
md manage: INFO:2020-12-29 20h51.07 utc:1325835: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2016.xml
md manage: INFO:2020-12-29 20h51.24 utc:1325835: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2014.xml
md manage: INFO:2020-12-29 20h51.33 utc:1325835: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2008.xml
md manage: INFO:2020-12-29 20h51.47 utc:1325835: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2006.xml
md manage: INFO:2020-12-29 20h52.01 utc:1325835: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2007.xml
md manage: INFO:2020-12-29 20h52.08 utc:1325835: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2013.xml
md manage: INFO:2020-12-29 20h52.19 utc:1325835: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2019.xml
md manage: INFO:2020-12-29 20h53.04 utc:1325835: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2017.xml
md manage: INFO:2020-12-29 20h53.31 utc:1325835: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2012.xml
md manage: INFO:2020-12-29 20h53.43 utc:1325835: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2010.xml
md manage: INFO:2020-12-29 20h54.04 utc:1325835: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2004.xml
md manage: INFO:2020-12-29 20h54.18 utc:1325835: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2015.xml
md manage: INFO:2020-12-29 20h54.38 utc:1325835: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2018.xml
md manage: INFO:2020-12-29 20h55.40 utc:1325835: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2011.xml
md manage: INFO:2020-12-29 20h56.10 utc:1325835: Updating /opt/gvm/var/lib/gvm/scap-data/nvdcve-2.0-2009.xml
md manage: INFO:2020-12-29 20h56.34 utc:1325835: Updating OVAL data
md manage: INFO:2020-12-29 20h56.43 utc:1325835: Updating /opt/gvm/var/lib/gvm/scap-data/oval/5.10/org.mitre.oval/c/oval.xml
md manage: INFO:2020-12-29 20h56.43 utc:1325835: Updating /opt/gvm/var/lib/gvm/scap-data/oval/5.10/org.mitre.oval/m/oval.xml
md manage: INFO:2020-12-29 20h56.43 utc:1325835: Updating /opt/gvm/var/lib/gvm/scap-data/oval/5.10/org.mitre.oval/v/family/ios.xml
md manage: INFO:2020-12-29 20h56.44 utc:1325835: Updating /opt/gvm/var/lib/gvm/scap-data/oval/5.10/org.mitre.oval/v/family/pixos.xml
md manage: INFO:2020-12-29 20h56.44 utc:1325835: Updating /opt/gvm/var/lib/gvm/scap-data/oval/5.10/org.mitre.oval/p/oval.xml
md manage: INFO:2020-12-29 20h57.03 utc:1325835: Updating /opt/gvm/var/lib/gvm/scap-data/oval/5.10/org.mitre.oval/i/oval.xml
md manage: INFO:2020-12-29 20h57.05 utc:1325835: Updating /opt/gvm/var/lib/gvm/scap-data/oval/5.10/org.mitre.oval/v/family/macos.xml
md manage: INFO:2020-12-29 20h57.05 utc:1325835: Updating /opt/gvm/var/lib/gvm/scap-data/oval/5.10/org.mitre.oval/v/family/unix.xml
md manage: INFO:2020-12-29 20h57.08 utc:1325835: Updating /opt/gvm/var/lib/gvm/scap-data/oval/5.10/org.mitre.oval/v/family/windows.xml
md manage: INFO:2020-12-29 20h57.15 utc:1325835: Updating user OVAL definitions.
md manage: INFO:2020-12-29 20h57.15 utc:1325835: Updating CVSS scores and CVE counts for CPEs
md manage: INFO:2020-12-29 21h01.11 utc:1325835: Updating CVSS scores for OVAL definitions
md manage: INFO:2020-12-29 21h01.20 utc:1325835: Updating placeholder CPEs
md manage: INFO:2020-12-29 21h02.47 utc:1325835: Updating Max CVSS for DFN-CERT
md manage: INFO:2020-12-29 21h04.00 utc:1325835: Updating DFN-CERT CVSS max succeeded.
md manage: INFO:2020-12-29 21h04.00 utc:1325835: Updating Max CVSS for CERT-Bund
md manage: INFO:2020-12-29 21h04.10 utc:1325835: Updating CERT-Bund CVSS max succeeded.
md manage: INFO:2020-12-29 21h04.21 utc:1325835: update_scap_end: Updating SCAP info succeeded

2 Likes