Failed to verify local repo integrity because one or more files are missing or have incorrect checksums.

Symptoms

When trying to upgrade to a newer version, the downloading of packages fails.

In the awingu-worker-smc.service log file, similar entries are visible:

2023-04-24 09:07:26.233572 AWINGU-01 awingu-worker-smc.service[python:1199]: sha256sum: WARNING: 34 computed checksums did NOT match
2023-04-24 09:07:26.443026 AWINGU-01 awingu-worker-smc.service[manage.py:6224]: Task update.tasks.download_packages failed
Traceback (most recent call last):
  File "/opt/awingu/awingu-core/virtualenv/lib/python3.6/site-packages/awingucore/update/tasks.py", line 790, in verify_local_repo_integrity
    cwd=local_repo_dir)
  File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sha256sum', '--quiet', '--strict', '-c', 'checksums_files.txt']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/awingu/awingu-core/virtualenv/lib/python3.6/site-packages/django_q/cluster.py", line 448, in worker
    res = f(*task["args"], **task["kwargs"])
  File "/opt/awingu/awingu-core/virtualenv/lib/python3.6/site-packages/awingucore/update/tasks.py", line 767, in download_packages
    _download_packages(version_number)
  File "/usr/lib/python3.6/contextlib.py", line 52, in inner
    return func(*args, **kwds)
  File "/opt/awingu/awingu-core/virtualenv/lib/python3.6/site-packages/awingucore/update/tasks.py", line 725, in _download_packages
    verify_local_repo_integrity(tmp)
  File "/opt/awingu/awingu-core/virtualenv/lib/python3.6/site-packages/awingucore/update/tasks.py", line 793, in verify_local_repo_integrity
    'Failed to verify local repo integrity because one or more '
update.tasks.CorruptRepoException: Failed to verify local repo integrity because one or more files are missing or have incorrect checksums.
2023-04-24 09:07:26.456573 AWINGU-01 awingu-worker-smc.service[manage.py:2092]: Failed [snake-mirror-montana-lion] - Failed to verify local repo integrity because one or more files are missing or have incorrect checksums.

Cause

When upgrading from a version to a newer (but old) Awingu version, many individual packages need to be downloaded from our repository servers.

This fails for some customers due to some form of connectivity issue.

A list of causes (in order of likelihood):

• Some form of rate limiting or throttling (number of requests) at a firewall level or HTTP proxy server level makes the actual network request (download) fail.
• Too much of a packet inspection going on at a firewall.
• Some form of rate limiting or throttling at the DNS server. As a result, the appliance is unable to resolve the repository hostname for some requests.
• Bad connectivity.

Resolution

The actual resolution is to address whatever network issue may be occurring. As this is not related to our product, this is out of scope for the support team.

As a workaround: In a remote SSH intervention, the support team can try to force downloading the packages at a much slower rate. For most customers, this seems to avoid the issue. Mind that this action needs to be manually performed for each version upgrade.

From version 5.4.0 onwards, the mechanism has been improved. Only two files are downloaded now: a checksum file and one big package.