Export all scan results (from a single report or multiple) when then are more than 1000 results

GVM versions

gsad: 21.4.4
gvmd: 21.4.5
openvas-scanner: 21.4.4
gvm-libs: 21.4.4

Environment

Operating system: Linux
Kernel: 5.16.0-amd64
Installation method / source: debian package

The problem

I’d like to know how to get ALL the results from the already-complete scan with more than 1000 results in xml format.
At this point any way will do - using GSA, or the python-gvm.

I understand that 1000 is a hard-limit for any entities, but getting scan results is a pretty common task,
so WTF - why is there no sane documentation on how to accomplish this?! Moreover - GSA doesn’t notify in any way that not all the results have been included. This is outrageous.

Things tried so far

  • ignore_pagination=1 levels=hmlg min_qod=0 filter
  • using first=1000 to get the rest of the results in filter
  • rows=-1 filter

@Yurii does --max-rows 0 fix the issue? or --max-rows 3000 or similar?

sounds like hard coded value and cannot find it from sourcecodes

Eero

@Eero
Could you please specify the script/tool that is supposed to take the key (–max-rows)?

@Yurii sounds like value is hardcoded into sourcecode. so, it means that you can only get 1000 results for one host. that is “enought for most of people”…

Eero

@Eero For one host - sure. My problem is that I cannot get more than 1000 results from the scan, which was performed on more than 300 hosts. Whatever I’ve tried the export yields 1000 results, which is about 60% of all results, available in the report.

@Yurii that is feature, not a bug :slight_smile:

Eero

@Yurii maybe it’s possible to modify source to skip this hardcoded max value. this means recompiling some parts of software.

Eero

Hi,

to get more then 1000 results you need to request a report with setting ignore_pagination to 1. For example : <get_reports report_id="..." ignore_pagination="1" details="1" filter="levels=hmlg min_qod=0" />. It’s not possible to request more then 1000 items with <get_results/>.

2 Likes

@bricks This sounds like great news. Could you please be more specific:

  • is the example you’ve provided meant to be used as a reference for raw api request?
  • can the same be done in gvm-python - I’ve tried get_reports(), but it doesn’t return the expected ‘report/results’ tag
  • can the same be done in gsa gui?

To allow more then 1000 items, you can run

gvmd --modify-setting 76374a7a-0569-11e6-b6da-28d24461215b --value 10000 as your user that is run gvmd like gvm

ref: https://github.com/greenbone/gvmd/blob/7862bd54a0c7524524e6460dad70f155dd30510d/src/manage_sql.h#L113

but this will affect also some other things and makes it slower.

@bricks example was for the gvm-cli raw xml request.

get_report() should return what you try with get_reports()?

yes, you can export the xml.

1 Like

Yes

Yes of course. See the get_report API docs.

gmp.get_report(report_id, filter_string="...", ignore_pagination=True, details=True)

The UI is doing exactly this. You can take a look at the network requests in your browser’s console.

1 Like

@Dexus @bricks
Thank you. Got it - gvm-cli xml req works nicely.

1 Like

Please share your command that it’s worked. @Yurii

Hi @Paiii3 did you try the commands mentioned by @bricks in Export all scan results (from a single report or multiple) when then are more than 1000 results - #8 by bricks - Greenbone Community Edition - Greenbone Community Portal?

If you want to use python-gvm, you will need to set the same variables to the same values. See API.

E.g.:

result = gmp.get_reports("1234-5678-90ab-cdef1234", details=True, ignore_pagination=True)

Thanks for your help.

I tried to scrape my Greenbone Community site using developer tools. Not sure if this might work.

https://<yourGreenbone?>/gmp?token=???&cmd=get_results&details=1&filter=sort-reverse%3Dseverity%20rows%3D100000000000000000000000

image

The output is XML so you could pipe into something else for parsing?

If it’s still relevant:

get_report_cmd="<get_reports report_id=\"$report_id\" ignore_pagination=\"1\" details=\"1\" filter=\"$filter\" />"
gvm-cli \
    --gmp-username "$user" \
    --gmp-password "$pass" \
    socket \
    --xml "$get_report_cmd"

mind you will need to include yout preferred authentication, user/pass is just as an example

2 Likes

Hello, how to export to csv with 15000resulsts?Or someone created a converter from xml to csv?
I can upload xml like this
gvm-cli --gmp-username user --gmp-password password --timeout 900 socket --socketpath /var/run/gvmd/gvmd.sock --xml “<get_reports report_id="ID" details="1" ignore_pagination="1" format_id="a994b278-1f62-11e1-96ac-406186ea4fc5"/>” >test.xml
The data is all there, but the rows are divided into sections according to an xml scheme unknown to me and I cannot match the necessary columns, as in csv. I only need
IP,Hostname,CVSS,Severity,NVT Name, Summary, Specific Result, NVT OID, CVEs, Task Name, Timestamp, Impact, Solution, how to do this?
sorry for the my english, i use google translate…

Hi. Gsa (web-interface) has the option to export directly to csv. Alternatively you can export a small set, take a look at the xml scheme and convert to scv using python - this would be a really small and straightforward script.