Greenbone-feed-sync error

Building from Source and Advanced Topics
1

Hello everyone, I tried installing from source the Greenbone community edition. I followed step by step the installation process on the official documentation. When it’s time to sync the vuln feeds i get an error:

UnicodeEncodeError: ‘latin-1’ codec can’t encode character ‘\u2827’ in position 10: ordinal not in range(256)
*** You may need to add PYTHONIOENCODING=utf-8 to your environment ***

I tried launching the command with -vvv options to get a better understanding of it, but it only showed the vuln getting downloaded.

Then i tried accessing the webserver and everything was working fine. I checked the SecInfo section and it populated correctly the NVTs and CVEs:

image
image1415×340 237 KB

When i tried to do some scanning, no vulnerability is showing up. I also followed some advices found here: Openvas does not detect new vulnerabilities - #13 by cgqc

and tried scanning a freshly installed metasploitable VM. No vulnerabilities found.

Apart from the aforementioned problem, nothing else seems to work. Has anybody an idea on how to solve this problem?

Thanks in advance and have a nice day

p.s. i can provide any other important detail, just ask me an i will get them,

2

The error message suggests adding the environment variable PYTHONIOENCODING=utf-8 to your environment. This environment variable sets the default encoding for Python to ‘utf-8’, which is a much broader character encoding that can represent a wide range of characters, including those outside the Latin script. Using ‘utf-8’ is the defacto encoding standard for handling text that includes characters from various scripts and languages.

Setting PYTHONIOENCODING=utf-8 ensures that Python uses the UTF-8 encoding by default and can handle characters like ‘\u2827’ without encoding errors.

export PYTHONIOENCODING=utf-8

In Python 3.7 and later versions, you can use the PYTHONUTF8 environment variable to set the default encoding to UTF-8.

export PYTHONUTF8=1
1 Like
3

Hi it would also be nice if you could add the full traceback. In that case I might be able to debug and fix the issue

2 Likes
4

Hi, you mean the steps i took and full details of the process? I tried also the command suggested by rippledj export PYTHONIOENCODING=utf-8 , but it didn’t work. Today i will try the other one and get back at you. Thanks in advance for the help!

5

The traceback is the full error message with the stack trace. It contains the code lines where the error has occurred and has propagated through the function calls.

Additionally which distro in which version do you use (just copy the output of cat /etc/os-release) ? What’s your systems encoding (output of locale)?

2 Likes
6

Sorry for my ignorance, here is what you asked me for:

Locale

LANG=en_US
LANGUAGE=en_US:
LC_CTYPE=“en_US”
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE=“en_US”
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES=“en_US”
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

System version: Ubuntu 22.04.3 LTS

Traceback (most recent call last):
File “/usr/local/lib/python3.10/dist-packages/greenbone/feed/sync/helper.py”, line 107, in flock_wait
yield
File “/usr/local/lib/python3.10/dist-packages/greenbone/feed/sync/main.py”, line 204, in feed_sync
with Spinner(
File “/usr/local/lib/python3.10/dist-packages/greenbone/feed/sync/helper.py”, line 141, in exit
self._live.stop()
File “/usr/local/lib/python3.10/dist-packages/rich/live.py”, line 147, in stop
with self.console:
File “/usr/local/lib/python3.10/dist-packages/rich/console.py”, line 865, in exit
self._exit_buffer()
File “/usr/local/lib/python3.10/dist-packages/rich/console.py”, line 823, in _exit_buffer
self._check_buffer()
File “/usr/local/lib/python3.10/dist-packages/rich/console.py”, line 2060, in _check_buffer
self.file.write(text)
UnicodeEncodeError: ‘latin-1’ codec can’t encode character ‘\u2827’ in position 10: ordinal not in range(256)
*** You may need to add PYTHONIOENCODING=utf-8 to your environment ***

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/bin/greenbone-feed-sync”, line 8, in
sys.exit(main())
File “/usr/local/lib/python3.10/dist-packages/greenbone/feed/sync/main.py”, line 233, in main
sys.exit(asyncio.run(feed_sync(console, error_console)))
File “/usr/lib/python3.10/asyncio/runners.py”, line 44, in run
return loop.run_until_complete(main)
File “/usr/lib/python3.10/asyncio/base_events.py”, line 649, in run_until_complete
return future.result()
File “/usr/local/lib/python3.10/dist-packages/greenbone/feed/sync/main.py”, line 185, in feed_sync
async with flock_wait(
File “/usr/lib/python3.10/contextlib.py”, line 217, in aexit
await self.gen.athrow(typ, value, traceback)
File “/usr/local/lib/python3.10/dist-packages/greenbone/feed/sync/helper.py”, line 112, in flock_wait
console.print(f"Releasing lock on {path.absolute()}")
File “/usr/local/lib/python3.10/dist-packages/rich/console.py”, line 1673, in print
with self:
File “/usr/local/lib/python3.10/dist-packages/rich/console.py”, line 865, in exit
self._exit_buffer()
File “/usr/local/lib/python3.10/dist-packages/rich/console.py”, line 823, in _exit_buffer
self._check_buffer()
File “/usr/local/lib/python3.10/dist-packages/rich/console.py”, line 2060, in _check_buffer
self.file.write(text)
UnicodeEncodeError: ‘latin-1’ codec can’t encode character ‘\u2827’ in position 10: ordinal not in range(256)
*** You may need to add PYTHONIOENCODING=utf-8 to your environment ***

Thanks in advance for the support!

7

Update: i forced LC_CTYPE=en_US.UTF-8 in the locale file. Now the error doesnt show up. But i still can’t get any result from the scanning. I will try again later since I know it takes some time to download all the feeds.

I’m trying to scan some websites (i have the authorization to do so), and also I’m scanning a local metasploitable VM.

I’ll keep you updated on the matter. Thanks again!

8

Could you paste the output of python3 -c "import sys; print(sys.getfilesystemencoding()); print(sys.getdefaultencoding())" too?

We need to find out why the code tries to convert the output into latin1 despite your locale is using utf8.

1 Like
9

gift@Ubuntu2:~$ python3 -c “import sys; print(sys.getfilesystemencoding()); print(sys.getdefaultencoding())”
utf-8
utf-8

10

You need to be patient until all feed are downloaded and loaded by the services. This can take up to several hours. Check the Feed Status page for details.

2 Likes
11

Yes i will try again in a few hours. Thank you so much for your help!

12

Ah overlooked that your mixed different encoding not only languages. You MUST change your locales to fix this issue. You shouldn’t mix utf8 and latin1 which can result in strange behavior as in this thread. Therefore please use en_US.UTF-8 instead of en_US.

2 Likes
13

Ok i changed them, i’ll keep you updated

1 Like