When using combined tag@digest references (e.g., nginx:1.21@sha256:abc...),
Docker does NOT store the tag in RepoTags. It only stores the digest in
RepoDigests. The previous implementation required BOTH to match, which
always failed because RepoTags was empty.
This caused docker_container to pull the image on every run even when
the image with the correct digest already existed locally, breaking
idempotency.
The fix: When a digest is specified, match by digest only since it's the
authoritative identifier. The tag is informational for human readability.
Real-world example from docker image inspect:
"RepoTags": [], # Empty when pulled by digest\!
"RepoDigests": ["portainer/portainer-ee@sha256:7ecf2008..."]
Updated tests to reflect the correct behavior:
- test_empty_repo_tags_matches_by_digest (the critical fix case)
- test_combined_tag_digest_matches_even_if_tag_differs
- test_multiple_tags_irrelevant_for_combined
Extends the tag@digest fix to also cover push operations in docker_image.py.
The push_image() method was passing combined tag@digest format directly to
the Docker API's /images/{name}/push endpoint, which fails with
"invalid tag format" errors.
This fix:
1. Imports build_pull_arguments() into docker_image.py
2. Uses the helper in push_image() before calling the API
3. When tag contains @ (but isn't a pure digest), passes the full
reference as the image name and omits the tag parameter
This complements the previous fix to pull_image() methods, ensuring
both pull and push operations handle tag@digest correctly.
When using combined image:tag@digest references, parse_repository_tag
returns (repo, "tag@digest"). The Docker SDK and API don't accept
tag@digest in the tag parameter, causing "invalid tag format" errors.
This fix:
1. Adds build_pull_arguments() helper function to _util.py
2. Uses the helper in both pull_image implementations
3. When tag contains @ (but isn't a pure digest), passes the full
reference as the repository/fromImage parameter instead of splitting
Tested formats:
- portainer/portainer-ee:2.35.0-alpine@sha256:abc...
- ghcr.io/gethomepage/homepage:v1.7@sha256:abc...
- localhost:5000/myapp:v2.0@sha256:abc...
The existing filter_images_by_tag already handles tag@digest for
lookups, so find_image continues to work correctly.
Includes comprehensive unit tests for build_pull_arguments().
The parse_repository_tag fix alone is not sufficient because Docker stores
RepoTags and RepoDigests separately. When looking up an image with combined
tag@digest (e.g., nginx:1.21@sha256:abc...), the _image_lookup function must
split the combined format and match BOTH RepoTags (for the tag) AND
RepoDigests (for the digest).
Docker stores:
- RepoTags: ["nginx:1.21"]
- RepoDigests: ["nginx@sha256:abc..."]
But NEVER stores the combined format. The previous code would construct:
- lookup = "nginx:1.21@sha256:abc..." (never matches RepoTags)
- lookup_digest = "nginx@1.21@sha256:abc..." (never matches RepoDigests)
This fix:
1. Adds filter_images_by_tag() helper function to _util.py to avoid code
duplication between _common.py and _common_api.py
2. Detects combined tag@digest format in the tag parameter
3. Splits into tag_part and digest_part
4. Constructs proper lookups for both RepoTags and RepoDigests
5. Requires BOTH to match for successful image lookup
Without this fix, image_label_mismatch: ignore fails because the image
cannot be found, resulting in no image labels being included in expected
labels comparison.
Includes comprehensive unit tests in test__util.py covering all scenarios
including edge cases for multiple @ symbols and empty tag parts.
The parse_repository_tag() function was incorrectly parsing Docker image
references that contained both a tag and a digest (e.g., nginx:1.0@sha256:abc).
Previously, when splitting by '@' first, the tag would be included in the
repository name, resulting in incorrect parsing:
- Input: "nginx:1.0@sha256:abc123"
- Old output: ("nginx:1.0", "sha256:abc123")
- Expected: ("nginx", "1.0@sha256:abc123")
The fix now:
1. Checks for digest (@) separator first
2. Examines the part before the digest for a tag (:) separator
3. Combines tag and digest as "tag@digest" when both are present
Added test cases:
- test_index_image_tag_and_sha
- test_index_user_image_tag_and_sha
- test_private_reg_image_tag_and_sha
* Re-enable typing and improve config.
* Make mypy pass.
* Improve settings.
* First batch of types.
* Add more type hints.
* Fixes.
* Format.
* Fix split_port() without returning to previous type chaos.
* Continue with type hints (and ignores).
* Remove __metaclass__ = type.
for i in $(grep -REl '__metaclass__ = type' plugins/ tests/); do
sed -e '/^__metaclass__ = type/d' -i $i;
done
* Remove super arguments, and stop inheriting from object.
* Adjust all __future__ imports:
for i in $(grep -REl "__future__.*absolute_import" plugins/ tests/); do
sed -e 's/from __future__ import .*/from __future__ import annotations/g' -i $i;
done
* Remove all UTF-8 encoding specifications for Python source files:
for i in $(grep -REl '[-][*]- coding: utf-8 -[*]-' plugins/ tests/); do
sed -e '/^# -\*- coding: utf-8 -\*-/d' -i $i;
done
* Reformat.
* Make all doc fragments, module utils, and plugin utils private.
* Remove some unused and no longer needed imports.
This hopefully also fixes the CI issues, which do not happen locally for me...
* Fix formatting.
* Try to make CI happy, again.
* Fix imports.
* Lint.
* Remove unicode text prefixes.
* Replace str.format() uses with f-strings.
* Replace % with f-strings, and do some cleanup.
* Fix wrong variable.
* Avoid unnecessary string conversion.
* Vendor parts of the Docker SDK for Python
This is a combination of the latest git version
(db7f8b8bb6)
with some fixes to make it compatible with Python 2.7
and adjusting some imports.
* Polishing.
* Fix bug that prevents contexts to be found when no Docker config file is present.
Ref: https://github.com/docker/docker-py/issues/3190
* Linting.
* Fix typos.
* Adjust more to behavior of Docker CLI.
* Add first iteration of docker_context_info module.
* Improvements.
* Add basic CI.
* Add caveat on contexts[].config result.
* Add inventory filter capability.
* Use community.library_inventory_filtering_v1 collection.
* Bump dependency to 1.0.0.
* Mention the new dependency in the changelog.
* vendored Docker SDK for Python code: volume: added support for bind propagation
https://docs.docker.com/storage/bind-mounts/#configure-bind-propagation
Cherry-picked from bea63224e0
Co-authored-by: Janne Jakob Fleischer <janne.fleischer@ils-forschung.de>
Co-authored-by: Milas Bowman <milas.bowman@docker.com>
* vendored Docker SDK for Python code: fix: eventlet compatibility
Check if poll attribute exists on select module instead of win32 platform check
The implementation done in #2865 is breaking usage of docker-py library within eventlet.
As per the Python `select.poll` documentation (https://docs.python.org/3/library/select.html#select.poll) and eventlet select removal advice (eventlet/eventlet#608 (comment)), it is preferable to use an implementation based on the availability of the `poll()` method that trying to check if the platform is `win32`.
Fixes https://github.com/docker/docker-py/issues/3131
Cherry-picked from 78439ebbe1
Co-authored-by: Mathieu Virbel <mat@meltingrocks.com>
* vendored Docker SDK for Python code: fix: use response.text to get string rather than bytes
Adjusted from 0618951093
Co-authored-by: Mehmet Nuri Deveci <5735811+mndeveci@users.noreply.github.com>
Co-authored-by: Milas Bowman <milas.bowman@docker.com>
* vendored Docker SDK for Python code: Fix missing asserts or assignments
Cherry-picked from 0566f1260c
Co-authored-by: Aarni Koskela <akx@iki.fi>
---------
Co-authored-by: Janne Jakob Fleischer <janne.fleischer@ils-forschung.de>
Co-authored-by: Milas Bowman <milas.bowman@docker.com>
Co-authored-by: Mathieu Virbel <mat@meltingrocks.com>
Co-authored-by: Mehmet Nuri Deveci <5735811+mndeveci@users.noreply.github.com>
Co-authored-by: Aarni Koskela <akx@iki.fi>
* Move copying functionality to module_utils.
* Add docker_container_copy_into module.
* Use new module in other tests.
* Fix copyright and attributes.
* Improve idempotency, improve stat code.
* Document and test when a stopped container works.
* Improve owner/group detection error handling when container is stopped.
* Fix formulation.
Co-authored-by: Brian Scholer <1260690+briantist@users.noreply.github.com>
* Improve file comparison.
* Avoid reading whole file at once.
* Stream when fetching files from daemon.
* Fix comment.
* Use read() instead of read1().
* Stream files when copying into container.
* Linting.
* Add force parameter.
* Simplify library code.
* Linting.
* Add content and content_is_b64 options.
* Make force=false work as for copy module: only copy if the destination does not exist.
* Improve docs.
* content should be no_log.
* Implement diff mode.
* Improve error handling.
* Lint and improve.
* Set owner/group ID to avoid ID lookup (which fails in paused containers).
* Apply suggestions from code review
Co-authored-by: Brian Scholer <1260690+briantist@users.noreply.github.com>
Co-authored-by: Brian Scholer <1260690+briantist@users.noreply.github.com>
* utils: fix IPv6 address w/ port parsing
This was using a deprecated function (`urllib.splitnport`),
ostensibly to work around issues with brackets on IPv6 addresses.
Ironically, its usage was broken, and would result in mangled IPv6
addresses if they had a port specified in some instances.
Usage of the deprecated function has been eliminated and extra test
cases added where missing. All existing cases pass as-is. (The only
other change to the test was to improve assertion messages.)
Cherry-picked from
f16c4e1147
Co-authored-by: Milas Bowman <milas.bowman@docker.com>
* client: fix exception semantics in _raise_for_status
We want "The above exception was the direct cause of the following exception:" instead of "During handling of the above exception, another exception occurred:"
Cherry-picked from
bb11197ee3
Co-authored-by: Maor Kleinberger <kmaork@gmail.com>
* tls: use auto-negotiated highest version
Specific TLS versions are deprecated in latest Python, which
causes test failures due to treating deprecation errors as
warnings.
Luckily, the fix here is straightforward: we can eliminate some
custom version selection logic by using `PROTOCOL_TLS_CLIENT`,
which is the recommended method and will select the highest TLS
version supported by both client and server.
Cherry-picked from
56dd6de7df
Co-authored-by: Milas Bowman <milas.bowman@docker.com>
* transport: fix ProxyCommand for SSH conn
Cherry-picked from
4e19cc48df
Co-authored-by: Guy Lichtman <glicht@users.noreply.github.com>
* ssh: do not create unnecessary subshell on exec
Cherry-picked from
bb40ba051f
Co-authored-by: liubo <liubo@uniontech.com>
* ssh: reject unknown host keys when using Python SSH impl
In the Secure Shell (SSH) protocol, host keys are used to verify the identity of remote hosts. Accepting unknown host keys may leave the connection open to man-in-the-middle attacks.
Do not accept unknown host keys. In particular, do not set the default missing host key policy for the Paramiko library to either AutoAddPolicy or WarningPolicy. Both of these policies continue even when the host key is unknown. The default setting of RejectPolicy is secure because it throws an exception when it encounters an unknown host key.
Reference: https://cwe.mitre.org/data/definitions/295.html
NOTE: This only affects SSH connections using the native Python SSH implementation (Paramiko), when `use_ssh_client=False` (default). If using the system SSH client (`use_ssh_client=True`), the host configuration
(e.g. `~/.ssh/config`) will apply.
Cherry-picked from
d9298647d9
Co-authored-by: Audun Nes <audun.nes@gmail.com>
* lint: fix deprecation warnings from threading package
Set `daemon` attribute instead of using `setDaemon` method that
was deprecated in Python 3.10.
Cherry-picked from
adf5a97b12
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
* api: preserve cause when re-raising error
Use `from e` to ensure that the error context is propagated
correctly.
Cherry-picked from
05e143429e
Co-authored-by: Milas Bowman <milas.bowman@docker.com>
* build: trim trailing whitespace from dockerignore entries
Cherry-picked from
3ee3a2486f
Co-authored-by: Clément Loiselet <clement.loiselet@capgemini.com>
* Improve formulation, also mention the security change as a breaking change.
Co-authored-by: Milas Bowman <milas.bowman@docker.com>
Co-authored-by: Maor Kleinberger <kmaork@gmail.com>
Co-authored-by: Guy Lichtman <glicht@users.noreply.github.com>
Co-authored-by: liubo <liubo@uniontech.com>
Co-authored-by: Audun Nes <audun.nes@gmail.com>
Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com>
Co-authored-by: Clément Loiselet <clement.loiselet@capgemini.com>
* Prefer unitest.mock by using compat.mock
`mock` is a backport of the `unittest.mock` module from the stdlib, and
there's no reason to use it on newer Python versions. `mock` is deprecated
in Fedora, so I figured I'd propose this here before downstream patching
our ansible-collection-community-docker package.
* Remove compat.mock code for older Python 3 versions
This removes compatibility for older versions of Python 3 that are no
longer supported.
