Update Community Container Feed

I have not tested this, but AFAIK you should not experience any data loss if you just replace the docker-compose file and do an docker compose up
The container names have not been changed and the mount points for the volumes have not, too, thus all your data should be available independent from where you load the images from …!

The project name has changed from greenbone-community-edition to greenbone-community-container. Thus the mountpoint has also changed (gvmd data volume as example) from /var/lib/docker/volumes/greenbone-community-edition_gvmd_data_vol/_data to /var/lib/docker/volumes/greenbone-community-container_gvmd_data_vol/_data.

user@host:~# docker volume inspect greenbone-community-edition_gvmd_data_vol
[
    {
        "CreatedAt": "2023-09-29T15:24:17+02:00",
        "Driver": "local",
        "Labels": {
            "com.docker.compose.project": "greenbone-community-edition",
            "com.docker.compose.version": "2.22.0",
            "com.docker.compose.volume": "gvmd_data_vol"
        },
        "Mountpoint": "/var/lib/docker/volumes/greenbone-community-edition_gvmd_data_vol/_data",
        "Name": "greenbone-community-edition_gvmd_data_vol",
        "Options": null,
        "Scope": "local"
    }
]
user@host:~# docker volume inspect greenbone-community-container_gvmd_data_vol
[
    {
        "CreatedAt": "2024-10-01T09:41:43+02:00",
        "Driver": "local",
        "Labels": {
            "com.docker.compose.project": "greenbone-community-container",
            "com.docker.compose.version": "2.22.0",
            "com.docker.compose.volume": "gvmd_data_vol"
        },
        "Mountpoint": "/var/lib/docker/volumes/greenbone-community-container_gvmd_data_vol/_data",
        "Name": "greenbone-community-container_gvmd_data_vol",
        "Options": null,
        "Scope": "local"
    }
]

I’m not really experienced with Docker, so I’m looking for a way to migrate currently.

— EDIT —
That’s what happens when you don’t pay attention and just run docker compose down, docker compose pull and docker compose up.

Short version: Make sure to run docker compose -f $DOWNLOAD_DIR/docker-compose.yml -p greenbone-community-edition pull and docker compose -f $DOWNLOAD_DIR/docker-compose.yml -p greenbone-community-edition up -d instead of docker compose pull and docker compose up.

Long version: Per the Greenbone Community Containers documentation, the $DOWNLOAD_DIR is defined as greenbone-community-container. According to the Docker documentation, “the default project name is derived from the base name of the project directory”, which also takes effect on the Docker volume names and locations. All commands in the documentation are using the -p switch to explicitly set the project name to greenbone-community-edition. So since the project names don’t match any longer, fresh volumes are created.

In my opinion, this could be mitigated by the $DOWNLOAD_DIR having the same name as the project name that is set explicitly each time, but in the end it’s just a layer 8 problem.

2 Likes

You are correct.
We have not specified a name: attribute in the docker-compose.yml file, thus I thought, that there was no problem in migration. But I just saw, that we switched from greenbone-community-edition to greenbone-community-container for the $DOWNLOAD_DIR as stated by you already! (Second) but we did not do this change recently, I will need some more time to identify when we changed this!

Nevertheless greenbone-community-container is the correct name and maybe your compose-file download has been made some time ago?

1 Like

The $DOWNLOAD_DIR hasn’t changed as far as I can see looking at the WayBackMachine. At least in a snapshot from the 9th of August 2022 it’s already greenbone-community-container.

The issue was wholly on my part: I omitted the -p greenbone-community-edition in my docker pull command, so the project name was automatically set according to the directory name greenbone-community-container, making it a separate project. Check the edit of my previous reply for more in-depth info.

2 Likes

3 posts were split to a new topic: Number of NVTs in Feed

Yes - In my case we were using a very old compose-file.
So I have ensured that my DOWNLOAD_DIR variable is set and have run:

docker compose -f $DOWNLOAD_DIR/docker-compose.yml -p greenbone-community-edition pull
and
docker compose -f $DOWNLOAD_DIR/docker-compose.yml -p greenbone-community-edition up -d

I’ve updated to docker-composev2 and all these commands run successfully.
Now I’m still seeing that my feeds are all Too old (37+ days on each).
What command should I run to change from the old docker.com registry to the new feed location without losing my existing data?
I have the following files in my /greenbone-community-container folder:

docker-compose-22.4.yml Feb 20 2023
docker-compose.yml Feb 20 2023

I really appreciate your help!

As mentioned by @TreAtW you probably want to use -p greenbone-community-edition in your docker commands … (also see documentation).

As you can see in my post above, I already do use those commands with -p greenbone-community-edition in them.
What is the command I need to use replace the docker compose file without blowing up my existing containers?
I’m thinking it is a new compose file: cd $DOWNLOAD_DIR && curl -f -L https://greenbone.github.io/docs/latest/_static/docker-compose-22.4.yml -o docker-compose.yml
But what can I do to be sure that the new compose file will not replace my containers with data in them?

Right now I’m seeing this error in my logs:
gvmd-1 | md manage:WARNING:2024-10-17 16h37.17 utc:5285: osp_scanner_feed_version: failed to connect to /run/ospd/ospd-openvas.sock

I ran a diff on the new vs old docker-compose files and see these changes:

$ diff docker-compose.yml docker-compose-new.yml 
3c3
<     image: greenbone/vulnerability-tests
---
>     image: registry.community.greenbone.net/community/vulnerability-tests
10c10
<     image: greenbone/notus-data
---
>     image: registry.community.greenbone.net/community/notus-data
15c15
<     image: greenbone/scap-data
---
>     image: registry.community.greenbone.net/community/scap-data
20c20
<     image: greenbone/cert-bund-data
---
>     image: registry.community.greenbone.net/community/cert-bund-data
25c25
<     image: greenbone/dfn-cert-data
---
>     image: registry.community.greenbone.net/community/dfn-cert-data
32c32
<     image: greenbone/data-objects
---
>     image: registry.community.greenbone.net/community/data-objects
37c37
<     image: greenbone/report-formats
---
>     image: registry.community.greenbone.net/community/report-formats
44c44
<     image: greenbone/gpg-data
---
>     image: registry.community.greenbone.net/community/gpg-data
49c49
<     image: greenbone/redis-server
---
>     image: registry.community.greenbone.net/community/redis-server
55c55
<     image: greenbone/pg-gvm:stable
---
>     image: registry.community.greenbone.net/community/pg-gvm:stable
62c62
<     image: greenbone/gvmd:stable
---
>     image: registry.community.greenbone.net/community/gvmd:stable
89c89
<     image: greenbone/gsa:stable
---
>     image: registry.community.greenbone.net/community/gsa:stable
92c92
<       - 9392:80
---
>       - 127.0.0.1:9392:80
96a97,160
>   # Sets log level of openvas to the set LOG_LEVEL within the env
>   # and changes log output to /var/log/openvas instead /var/log/gvm
>   # to reduce likelyhood of unwanted log interferences
>   configure-openvas:
>     image: registry.community.greenbone.net/community/openvas-scanner:stable
>     volumes:
>       - openvas_data_vol:/mnt
>       - openvas_log_data_vol:/var/log/openvas
>     command:
>       - /bin/sh
>       - -c
>       - |
>         printf "table_driven_lsc = yes\nopenvasd_server = http://openvasd:80\n" > /mnt/openvas.conf
>         sed "s/127/128/" /etc/openvas/openvas_log.conf | sed 's/gvm/openvas/' > /mnt/openvas_log.conf
>         chmod 644 /mnt/openvas.conf
>         chmod 644 /mnt/openvas_log.conf
>         touch /var/log/openvas/openvas.log
>         chmod 666 /var/log/openvas/openvas.log
> 
>   # shows logs of openvas
>   openvas:
>     image: registry.community.greenbone.net/community/openvas-scanner:stable
>     restart: on-failure
>     volumes:
>       - openvas_data_vol:/etc/openvas
>       - openvas_log_data_vol:/var/log/openvas
>     command:
>       - /bin/sh
>       - -c
>       - |
>         cat /etc/openvas/openvas.conf
>         tail -f /var/log/openvas/openvas.log
>     depends_on:
>       configure-openvas:
>         condition: service_completed_successfully
> 
>   openvasd:
>     image: registry.community.greenbone.net/community/openvas-scanner:stable
>     restart: on-failure
>     environment:
>       # `service_notus` is set to disable everything but notus,
>       # if you want to utilize openvasd directly removed `OPENVASD_MODE`
>       OPENVASD_MODE: service_notus
>       GNUPGHOME: /etc/openvas/gnupg
>       LISTENING: 0.0.0.0:80
>     volumes:
>       - openvas_data_vol:/etc/openvas
>       - openvas_log_data_vol:/var/log/openvas
>       - gpg_data_vol:/etc/openvas/gnupg
>       - notus_data_vol:/var/lib/notus
>     # enable port forwarding when you want to use the http api from your host machine
>     # ports:
>     #   - 127.0.0.1:3000:80
>     depends_on:
>       vulnerability-tests:
>         condition: service_completed_successfully
>       configure-openvas:
>         condition: service_completed_successfully
>       gpg-data:
>         condition: service_completed_successfully
>     networks:
>       default:
>         aliases:
>           - openvasd
99c163
<     image: greenbone/ospd-openvas:stable
---
>     image: registry.community.greenbone.net/community/ospd-openvas:stable
101d164
<     init: true
115,116d177
<         "--mqtt-broker-address",
<         "mqtt-broker",
127a189,190
>       - openvas_data_vol:/etc/openvas/
>       - openvas_log_data_vol:/var/log/openvas
135,159c198,199
< 
<   mqtt-broker:
<     restart: on-failure
<     image: greenbone/mqtt-broker
<     ports:
<       - 1883:1883
<     networks:
<       default:
<         aliases:
<           - mqtt-broker
<           - broker
< 
<   notus-scanner:
<     restart: on-failure
<     image: greenbone/notus-scanner:stable
<     volumes:
<       - notus_data_vol:/var/lib/notus
<       - gpg_data_vol:/etc/openvas/gnupg
<     environment:
<       NOTUS_SCANNER_MQTT_BROKER_ADDRESS: mqtt-broker
<       NOTUS_SCANNER_PRODUCTS_DIRECTORY: /var/lib/notus/products
<     depends_on:
<       - mqtt-broker
<       - gpg-data
<       - vulnerability-tests
---
>       configure-openvas:
>         condition: service_completed_successfully
162c202
<     image: greenbone/gvm-tools
---
>     image: registry.community.greenbone.net/community/gvm-tools
182a223,224
>   openvas_data_vol:
>   openvas_log_data_vol:

It seems that I need to apply this new compose file but I want to keep my old data / config. Did I miss some discussion around the transition of this old compose file structure to the new?

Maybe the issue revolves arround the addition of the openvas_data_vol configuration.

Hi!

I’m facing the same issue @mark2tech , I run the feed sync and it`showing the old feed, follows more informations:
Openvas Community Edition 22.9.1 (Docker)

Sync comand:

docker compose -f /root/docker-compose.yml -p greenbone-community-edition pull notus-data vulnerability-tests scap-data dfn-cert-data cert-bund-data report-formats data-objects
docker compose -f /root/docker-compose.yml -p greenbone-community-edition up -d notus-data vulnerability-tests scap-data dfn-cert-data cert-bund-data report-formats data-objects

Sync Results:

root@ :~# docker compose -f /root/docker-compose.yml -p greenbone-community-edition pull notus-data vulnerability-tests scap-data dfn-cert-data cert-bund-data report-formats data-objects
[+] Pulling 7/7
 ✔ scap-data Pulled                                                                                                                                 1.6s 
 ✔ notus-data Pulled                                                                                                                                1.6s 
 ✔ dfn-cert-data Pulled                                                                                                                             1.5s 
 ✔ cert-bund-data Pulled                                                                                                                            1.6s 
 ✔ report-formats Pulled                                                                                                                            1.6s 
 ✔ vulnerability-tests Pulled                                                                                                                       1.6s 
 ✔ data-objects Pulled                                                                                                                              1.6s 
root@ :~# 
root@ :~# 
root@ :~# 
root@ :~# 
root@ :~# docker compose -f /root/docker-compose.yml -p greenbone-community-edition up -d notus-data vulnerability-tests scap-data dfn-cert-data cert-bund-data report-formats data-objects
[+] Running 7/7
 ✔ Container greenbone-community-edition-notus-data-1           Started                                                                             1.1s 
 ✔ Container greenbone-community-edition-data-objects-1         Started                                                                             0.9s 
 ✔ Container greenbone-community-edition-report-formats-1       Started                                                                             2.0s 
 ✔ Container greenbone-community-edition-scap-data-1            Started                                                                             1.1s 
 ✔ Container greenbone-community-edition-vulnerability-tests-1  Started                                                                             1.2s 
 ✔ Container greenbone-community-edition-cert-bund-data-1       Started                                                                             0.5s 
 ✔ Container greenbone-community-edition-dfn-cert-data-1        Started                                                                             1.5s 
root@ :~# ls -ld

After a update feed status:

After a while (10 minutes) feed status shows that the NVts are 36 days old too.

I try to make a new installation of Openvas, I followed the oficial documentation (Workflows - Greenbone Community Documentation).

Somebody can help, try everything I found.

@rpradela If this is a new installation, sometimes the feed updates can take a while.

I usually monitor with docker stats to see if the container CPU is ‘working hard’. Once it completes, then the interface will update.

If you do have the same issue with an old .yml file, then let me know if you’ve found the proper way to migrate from 2023 to current .yml file without losing data/config.
Thanks!

Hi everyone,
I have a problem when using Openvas, according to the workflow, I can use the following 2 commands to update everything:
docker compose -f $DOWNLOAD_DIR/docker-compose.yml -p greenbone-community-edition pull docker compose -f $DOWNLOAD_DIR/docker-compose.yml -p greenbone-community-edition up -d
OR some commands to update feed only
docker compose -f $DOWNLOAD_DIR/docker-compose.yml -p greenbone-community-edition pull notus-data vulnerability-tests scap-data dfn-cert-data cert-bund-data report-formats data-objects
docker compose -f $DOWNLOAD_DIR/docker-compose.yml -p greenbone-community-edition up -d notus-data vulnerability-tests scap-data dfn-cert-data cert-bund-data report-formats data-objects

But the only thing I get updated is the interface. Then I tried to get the new version on Docker Hub to put in, but even though the feed status changed to current, the list related to it was not updated. Typically, my Scap data has been updated to October 1, 2024, but the CVE list is still stuck in August. I use the docker version installed at Greenbone Community Containers - Greenbone Community Documentation

Hope to receive help from everyone


Hey, @iamz24 please see in this topic on how you can fix your problem.
TL;DR: Check the documentation, we have switched to a new self-hosted registry and do not support our images on docker.com anymore!

Yeah @y0urself , when i realize my fault, i try to use new docker-compose.yaml, but it told that timeout, i’ve connected to internet.

I just tried pulling the images, and everything seems to be fine, you may want to try again, and check the connection on your side?

Yeah thanks for your help, in the first time and second time i pull, it is error, but when i pull up to fourth time, it’s done

I have read the documentation multiple times. There is no reference to how to safely move from the old .yml to the new .yml that has changed the names of the containers and prevent from overwriting my old data storage. @iamz24 were you able to simply replace your .yml with the new version? Any Data loss? I’d really appreciate someone helping my to run the right commands, as I’ve been stuck here for some time. My posts above outline clearly my problem, but no one eseems to want to take on the reality that the new compose files have different data storage locations and I just need help determining how to migrate.
Thanks in advance!

Update: I replaced my docker-compose.yml and ran only the feed sync commands:

docker compose -f $DOWNLOAD_DIR/docker-compose.yml -p greenbone-community-edition pull notus-data vulnerability-tests scap-data dfn-cert-data cert-bund-data report-formats data-objects

docker compose -f $DOWNLOAD_DIR/docker-compose.yml -p greenbone-community-edition up -d notus-data vulnerability-tests scap-data dfn-cert-data cert-bund-data report-formats data-objects

The interface now shows all feeds up to date except for NVT.
Logs show:

gvmd-1                 | md manage:WARNING:2024-11-07 17h58.12 utc:47190: osp_scanner_feed_version: failed to connect to /run/ospd/ospd-openvas.sock
gvmd-1                 | md manage:WARNING:2024-11-07 17h58.17 utc:47191: nvts_feed_info_internal: failed to connect to /run/ospd/ospd-openvas.sock

If I run the standard update procedure:

I can no longer open the management interface.
docker ps results in:

I would really appreciate some help to get past this update issue that seems to be related to the renaming of containers or changing of the volumes for storage.

1 Like

I am also facing the same problem and very distressed

I also don’t know why my Openvas run, when i compose it for four times, it pulled successfully, and i went to my web, at the first 3 minutes i can’t login, but after that i success, and everything is ok. Maybe i’m lucky, i just replace my old yaml with new and replace the first line 127.0.0.1:9392:80 i seen to 0.0.0.0:9392:80 because i want to access my web from another VM. That’s all my experience

I think you should run the command compose up with --remove-orphans, let’s test it. It’s may be because of your old image



That’s all command i use, i build it from the first step, i have snapped shot because i scared of lossing data, but it’s not :laughing: