Scan interrupts randomly / problem with XML file split

Scanning / scan configuration
1

Continuing the discussion from Does an nvt feed update interrupt a current running scan:

GVM versions

gsad: Greenbone Security Assistant 22.05.1
gvmd: Greenbone Vulnerability Manager 22.5.4
openvas-scanner: OpenVAS 22.7.2
gvm-libs: gvm-libs 22.6.3

Environment

Operating system: Debian GNU/Linux 12
Kernel: 6.1.37-1
Installation method / source: GVM Installed from source using the official guide

I have the same issue. Due to my investigation, it happens when NVT update consumes a lot of CPU and RAM. There are 4 2.4Ghz CPU and 8 GB of RAM.

2024-02-02_20-49
2024-02-02_20-491903×440 101 KB

An odd thing here: you can see openvas: testing in the process list but the scan has interrupted state in the WebUI and writes in logs:

event task:MESSAGE:2024-02-02 15h47.42 utc:1385303: Status of task task_20240129 (ee1b2831-71fe-4f8d-b0c2-595d71667efb) has changed to Interrupted

Also, it seems like the feed update stuck with the same repeated messages in the logs:

md manage: INFO:2024-02-02 15h55.09 UTC:1385990: OSP service has different VT status (version 202402020650) from database (version 202402010857, 135427 VTs). Starting update …

It was started with greenbone-feed-sync from cron. But now ps aux|grep doesn’t find this process, so I even don’t know how to stop it.

Should I update something? Or maybe tune some settings to prevent scan interruptions? If I miss something, please let me know how to investigate further.

2

I think the inner-workings of this process is worth looking into the source code to determine whether greenbone-feed-sync places some sort of lock which logically interupts the scan task, and releases it when the feed data has finished populating the database. However, other than a source code review, I don’t know how else you can determine the answer.

3

Something definitely wrong with the update mechanism. It exhausted all free space in the weekend creating /tmp/gvmd-split-xml-file-... files. I fixed it by:

systemctl stop gsad gvmd notus-scanner ospd-openvas
rm -rf /tmp/*
/usr/local/bin/greenbone-feed-sync
systemctl start gsad gvmd notus-scanner ospd-openvas

After that feed statuses changed from Update in progress... to Current and the new scan started successfully. But now it is interrupted again, Feeds are in Update in progress... and the previous solution doesn’t work anymore. At least new files in /tmp aren’t created near each other. It stuck with creating a temp file, filling it with 1,1 GB of data, deleting it and so on.

4

These could be relevant:

5

8 GB of RAM is also on the low side for running our stack, especially when a feed update and vulnerability scans run in parallel, both of which can be very resource-intensive. See this post for further information. Is it possible to assign more RAM to the VM and see if the problem persists?

Since we also have multipe, if rare, reports of this problem still happening, I will open an internal issue to look into it. I cannot make any promises regarding a solution or a time frame however. If you want to support this process, please open a new issue at Issues · greenbone/gvmd · GitHub including full log outtakes, similar to the old issue GVM 20.08 crashes after disk (/tmp directory) runs full because of cpe xml split · Issue #1433 · greenbone/gvmd · GitHub.

Last but not least we recommend to update the whole stack to the current versions.

3 Likes
6

I thought that the low side is 4GB as documentation claims. But anyway we doubled RAM. And it looks like 16GB is enough for now to run a scan and update feeds simultaneously. I’ll take a look at it for several days and report back here. Thank you for the help!

1 Like
7

Thanks for pointing that out, we’ll look into updating the docs!

1 Like
8

It seems to be fixed with RAM increasing: there are no issues with the feed updates and no scan interruptions now. Thank you for your assistance!

3 Likes