Feed is Syncing Issue

Community Containers
1

I am facing a serious issues since last 2 days and unable to find a solution. I am using community containers to build my OpenVAS image. All works fine and I am able to access the web panel. Now the Feed is syncing which roughly takes 30-40 minutes

Now moving on, I am using the same VM in an offline environment, I use the same command sudo docker compose up -d; however, when the UI starts ir says Feed is Syncing, whereas I don’t have internet on the system

Therefore, I cannot initiate any scans.

Is there any global variable to disable the syncing, once it has fetched info once?

The vulnerability test conainer takes most of the time (roughly 3 mintes, when I do start up again)

/bin # cat init.sh #!/bin/sh set -e rm -f "${STATE_FILE}" license_file=${STORAGE_PATH}/LICENSE if [ -n "${FEED_RELEASE}" ]; then echo "Using feed release ${FEED_RELEASE}" STORAGE_PATH="${STORAGE_PATH}/${FEED_RELEASE}/vt-data/nasl" fi if [ ! -d "${STORAGE_PATH}" ]; then echo "Vulnerability tests not found in ${STORAGE_PATH}" exit 1 fi if [ -e "${license_file}" ]; then cat "${license_file}" fi echo -e "\nCopying vulnerability tests data... " if [ -d "${MOUNT_PATH}" ]; then [ -z "$KEEP_DATA" ] && rm -rf "${MOUNT_PATH}/"* cp -r "${STORAGE_PATH}/"* "${MOUNT_PATH}" state_dir=$(dirname ${STATE_FILE}) mkdir -p "${state_dir}" touch "${STATE_FILE}" if [ "$(id -u)" = "0" ]; then if [ -n "${USER_ID}" ]; then chown -R "${USER_ID}" "${MOUNT_PATH}" echo "changed user permissions to ${USER_ID}" fi if [ -n "${GROUP_ID}" ]; then chgrp -R "${GROUP_ID}" "${MOUNT_PATH}" echo "changed group permissions to ${GROUP_ID}" fi fi echo "files copied." else echo "nothing to do." fi if [ -n "${KEEP_ALIVE}" ]; then sleep infinity fi /bin #

Is there any way to disable syncing once it is synced once?
2

Offline Support for the Community Container is not supported, some commercial appliances does support Airgap and fully offline operations but the container needs at least feed access. You can try to add the feed.community.greenbone.net mapped to localhost to the local host file to get a faster timeout :wink:

1 Like
3

I’ve mentioned this several times and it also listed at the community docs. The feed sync consists of two parts. First part is downloading the new feed content. Second part is loading the new content into databases. For the community containers part one is done by pulling new images and (re-)starting the data containers (see Workflows - Greenbone Community Documentation). If you don’t (re-)start the data containers part two will not be run.

2 Likes
4

This is the output, when I will restart the system

greenbone-community-container$ sudo docker ps
CONTAINER ID   IMAGE                                                               COMMAND                  CREATED        STATUS        PORTS                    NAMES`
716a21e1487b   registry.community.greenbone.net/community/gsa:stable               “/usr/local/bin/entr…”   21 hours ago   Up 21 hours   127.0.0.1:9392->80/tcp   greenbone-community-edition-gsa-1
83470a2782e5   registry.community.greenbone.net/community/gvmd:stable              “/usr/local/bin/entr…”   21 hours ago   Up 21 hours                            greenbone-community-edition-gvmd-1`
2b5c46ac9c65   registry.community.greenbone.net/community/openvas-scanner:stable   “/bin/sh -c /usr/loc…”   21 hours ago   Up 21 hours                            greenbone-community-edition-openvasd-1
9ba4b72701a4   registry.community.greenbone.net/community/ospd-openvas:stable      “/usr/bin/tini – /u…”    21 hours ago   Up 21 hours                            greenbone-community-edition-ospd-openvas-1
924a39b3aff1   registry.community.greenbone.net/community/openvas-scanner:stable   “/bin/sh -c 'cat /et…”   21 hours ago   Up 21 hours                            greenbone-community-edition-openvas-1`
27c8664daf98   registry.community.greenbone.net/community/redis-server             “docker-entrypoint.s…”   21 hours ago   Up 21 hours   6379/tcp                 greenbone-community-edition-redis-server-1
8b2e9dc1698f   registry.community.greenbone.net/community/pg-gvm:stable            “/usr/local/bin/entr…”   21 hours ago   Up 21 hours                            greenbone-community-edition-pg-gvm-1`

When I log in, the system shows the above containers. When I log in, it says Feed is syncing even on the offline system. What should I change so that it should JUST NOT do syncing on start? do I need to run any other data container or change any state flag?

I really need this please. The feeds are already downloaded. I tried exploring the /init.sh scripts but no help.

5

Once the containers are running
I am using the the updated yml

name: greenbone-community-edition

services:

############################################

FEED SERVICES (DISABLED FOR OFFLINE MODE)

############################################

vulnerability-tests:

image: registry.community.greenbone.net/community/vulnerability-tests

environment:

FEED_RELEASE: “24.10”

volumes:

- vt_data_vol:/mnt

notus-data:

image: registry.community.greenbone.net/community/notus-data

volumes:

- notus_data_vol:/mnt

scap-data:

image: registry.community.greenbone.net/community/scap-data

volumes:

- scap_data_vol:/mnt

cert-bund-data:

image: registry.community.greenbone.net/community/cert-bund-data

volumes:

- cert_data_vol:/mnt

dfn-cert-data:

image: registry.community.greenbone.net/community/dfn-cert-data

volumes:

- cert_data_vol:/mnt

depends_on:

- cert-bund-data

data-objects:

image: registry.community.greenbone.net/community/data-objects

environment:

FEED_RELEASE: “24.10”

volumes:

- data_objects_vol:/mnt

report-formats:

image: registry.community.greenbone.net/community/report-formats

environment:

FEED_RELEASE: “24.10”

volumes:

- data_objects_vol:/mnt

depends_on:

- data-objects

gpg-data:

image: registry.community.greenbone.net/community/gpg-data

volumes:

- gpg_data_vol:/mnt

############################################

CORE SERVICES ONLY (OFFLINE MODE)

############################################

redis-server:
image: registry.community.greenbone.net/community/redis-server
restart: on-failure
volumes:

redis_socket_vol:/run/redis/

pg-gvm:
image: registry.community.greenbone.net/community/pg-gvm:stable
restart: on-failure
volumes:

psql_data_vol:/var/lib/postgresql

psql_socket_vol:/var/run/postgresql

gvmd:
image: registry.community.greenbone.net/community/gvmd:stable
restart: on-failure
volumes:

gvmd_data_vol:/var/lib/gvm

scap_data_vol:/var/lib/gvm/scap-data/

cert_data_vol:/var/lib/gvm/cert-data

data_objects_vol:/var/lib/gvm/data-objects/gvmd

vt_data_vol:/var/lib/openvas/plugins

psql_data_vol:/var/lib/postgresql

gvmd_socket_vol:/run/gvmd

ospd_openvas_socket_vol:/run/ospd

psql_socket_vol:/var/run/postgresql
depends_on:
pg-gvm:
condition: service_started
ospd-openvas:
condition: service_started

gsa:
image: registry.community.greenbone.net/community/gsa:stable
restart: on-failure
ports:

127.0.0.1:9392:80
volumes:

gvmd_socket_vol:/run/gvmd
depends_on:

gvmd

configure-openvas:
image: registry.community.greenbone.net/community/openvas-scanner:stable
volumes:

openvas_data_vol:/mnt

openvas_log_data_vol:/var/log/openvas
command:

/bin/sh

-c

|
printf “table_driven_lsc = yes\nopenvasd_server = http://openvasd:80\n” > /mnt/openvas.conf
sed “s/127/128/” /etc/openvas/openvas_log.conf | sed ‘s/gvm/openvas/’ > /mnt/openvas_log.conf
chmod 644 /mnt/openvas.conf
chmod 644 /mnt/openvas_log.conf
touch /var/log/openvas/openvas.log
chmod 666 /var/log/openvas/openvas.log

openvas:
image: registry.community.greenbone.net/community/openvas-scanner:stable
restart: on-failure
volumes:

openvas_data_vol:/etc/openvas

openvas_log_data_vol:/var/log/openvas
command:

/bin/sh

-c

|
cat /etc/openvas/openvas.conf
tail -f /var/log/openvas/openvas.log
depends_on:
configure-openvas:
condition: service_completed_successfully

openvasd:
image: registry.community.greenbone.net/community/openvas-scanner:stable
restart: on-failure
environment:
OPENVASD_MODE: service_notus
GNUPGHOME: /etc/openvas/gnupg
LISTENING: 0.0.0.0:80
volumes:

openvas_data_vol:/etc/openvas

openvas_log_data_vol:/var/log/openvas

gpg_data_vol:/etc/openvas/gnupg

notus_data_vol:/var/lib/notus
depends_on:
configure-openvas:
condition: service_completed_successfully
networks:
default:
aliases:

openvasd

ospd-openvas:
image: registry.community.greenbone.net/community/ospd-openvas:stable
restart: on-failure
hostname: ospd-openvas.local
cap_add:

NET_ADMIN

NET_RAW
security_opt:

seccomp=unconfined

apparmor=unconfined
command:
[
“ospd-openvas”,
“-f”,
“–config”,
“/etc/gvm/ospd-openvas.conf”,
“–notus-feed-dir”,
“/var/lib/notus/advisories”,
“-m”,
“666”
]
volumes:

gpg_data_vol:/etc/openvas/gnupg

vt_data_vol:/var/lib/openvas/plugins

notus_data_vol:/var/lib/notus

ospd_openvas_socket_vol:/run/ospd

redis_socket_vol:/run/redis/

openvas_data_vol:/etc/openvas/

openvas_log_data_vol:/var/log/openvas
depends_on:
redis-server:
condition: service_started
configure-openvas:
condition: service_completed_successfully

gvm-tools:
image: registry.community.greenbone.net/community/gvm-tools
volumes:

gvmd_socket_vol:/run/gvmd

ospd_openvas_socket_vol:/run/ospd
depends_on:

gvmd

ospd-openvas

############################################

VOLUMES (FEED DATA STILL STORED HERE)

############################################

volumes:
gpg_data_vol:
scap_data_vol:
cert_data_vol:
data_objects_vol:
gvmd_data_vol:
psql_data_vol:
vt_data_vol:
notus_data_vol:
psql_socket_vol:
gvmd_socket_vol:
ospd_openvas_socket_vol:
redis_socket_vol:
openvas_data_vol:
openvas_log_data_vol:

so that the data container won’t start; however, it is still showing Feed is currently syncing.

6

As I wrote, if you restart the data containers or the gvmd or openvas containers, a feed sync will happen. The daemons will read the feed content (again) and load it into the databases. Even in offline cases, because downloading (the online case) is only the first part of the feed sync.

7

And please format your output with three backticks for better readability. See Start Here/Read Me First - #3 by bricks for a detailed explanation and examples.

8

Thank you bricks.

Is there any way to disable this feed sync like Immaus has disabled feed sync in his code GitHub - immauss/openvas: Containers for running the Greenbone Vulnerability Manager. Run as a single container with all services or separate single applications containers via docker-compose.

I mean is there any way to disable the sync at all once the feeds are downloaded by maybe tweaking any code or disabling a service?

9

I don’t know what @immauss did with his containers. But again if the feed content files are updated in some kind, even just overridden with the same content, the daemons need to read them and need to update the databases.

If you don’t update the data and don’t restart the daemons no feed sync is run. If the message still shows up it is considered to be a bug.

1 Like
10

@bricks I just have an environment variable that causes the sync not to run on startup. It’s primarily just for this type of scenario. Thought process being, if the feeds are in the image, and the image is updated, then there should be no need to sync. I’m also bundling a database, so if you start without the sync, the container can be up and ready to scan in less than 15 minutes when starting from Zero.

-Scott

11

Thank you @immauss How easy is it possible to tweak the branding. I see you have done some image changes through branding.sh, but how can I restore the login page back to the original one? I want to give proper attribution to you through a footer credits though

12

@immauss I am using your container with the option
sudo docker run -d -p 9392:9392 -e PASSWORD=“admin” -e SKIPSYNC=true --restart=always -v openvas:/data --name openvas immauss/openvas

the aim is to skip syncing, it works fine and skip syncing. but when I restart the system and the container starts automatically on startup, it shows feeds are syncing which is weird. I want to foce skipsnycing.

13

I am still not sure what you are doing actually. As I wrote several time, the feed sync consists of two phases. The downloading of the feed content and the loading of the feed content into the databases. Did you patch gvmd and openvas-scanner to not apply the second phase?

14

I am using @immauss custom openvas version, which allows skipping the feed sync, once the feed is synced, if we start the container without SKIPSYNC value, it just skip syncing and the machine is available more fastly

the aim is to skip syncing, it works fine and skip syncing. but when I restart the system and the container starts automatically on startup, it shows feeds are syncing which is weird. I want to foce skipsnycing.

15

My last message was for @immauss

BUT again YOU CAN’T SKIP THE SECOND PHASE FOR THE FEED SYNC without patching gvmd and openvas-scanner.

16

yes this is what I was looking for, I want to patch the gvmd and make some updates to skip the scanning (@immauss is already doiing this but it is not persistent on restart)

17

I’m not skipping anything. I’m doing it in advance. The image ships with a backup of the database that is loaded before gvmd starts on initial image startup. I rebuild that database from the current feeds regularly. So, when you start with SKIPSYNC=true, it is as if you had started, run through both phases @bricks described, then restarted. No need to patch gvmd or openvas. Even if you do run the feedsync, it takes less time as it only needs to pull and update from the changes since the image was built.
@sagox23662
As for rebranding … I think the branding.sh is a legacy …. The branding changes are mostly done in the build scripts for the image now. I’d have to look to be certain, but not easily reversed without rebuilding the image.

-Scott

18

I’d have to look to be certain, but not easily reversed without rebuilding the image.

Thank you, it would be a great favor if you can redirect me to right scripts

So, when you start with SKIPSYNC=true, it is as if you had started

I am encountering a weird scenario, where if you start it with SKIPSYNC=true and restart your system, next time the container starts automatically but this time the feeds start syncing again. The SKIPSYNC is not persistent.

19

@sagox23662
That sounds odd.
Could you please open an issue here:

Please make sure to include all info requested in the template (start method, logs , etc)

-Scott