The attribute 'remote_user' of the object 'connection._play_context' contains the variable name instead of its value.

[macksyma]

I've discovered the incorrect transmission of the remote_user variable's value when using delegate_to with Mitogen 3.4. For example, let's take a simple task:

    - name: "test_task"
      shell: "ls -lah"
      delegate_to: "{{ test_jump_box }}"
      remote_user: "{{ jumper_test_user }}"

Here is the error I've received executing this task:

fatal: [localhost -> XXXXXXXXXX]: UNREACHABLE! => {
    "changed": false,
    "msg": "EOF on stream; last 100 lines received:\nremote username contains invalid characters\r",
    "unreachable": true
}

In the debug log, I see that the ssh command line contains the name of the variable transmitted into the remote_user - {{ jumper_test_user }}, instead of the actual value of this variable:

[mux  2197866] 12:52:15.298866 D mitogen.parent: command line for Connection(None): ssh -o "LogLevel ERROR" -l "{{ jumper_test_user }}" -o "Compression yes" -o "ServerAliveInterval 30" -o "ServerAliveCountMax 10" -o "BatchMode yes" -o "StrictHostKeyChecking no" -o "UserKnownHostsFile /dev/null" -o "GlobalKnownHostsFile /dev/null" -F ssh.cfg ...

After further investigation, I've discovered that Mitogen takes the username for ssh from object _connection._play_context, transmitted to Mitogen's ActionModuleMixin from Ansible's TaskExecutor. Inside the ActionModuleMixin object, I've found that _self._play_context.remote_user contains the correct value of the variable jumper_test_user, but the connection object self._connection._play_context.remote_user contains the variable name, including brackets, just like I saw it in the ssh command line.

I've checked how the method _executeof Ansible's class TaskExecutor updates attributes of its own self._play_context and self._connection._play_context, and found that at some point, self._play_context of TaskExecutor also contains the name of variables instead of their value. But then, it executes the method post_validate() for its self._play_context (PlayContext inherits it from its parent classes):

self._play_context.post_validate(templar=templar)

that replaces variable names with their values. Unfortunately, it doesn't execute this method for _connection._play_context, and as a result, we have variable names instead of values. I'm not sure whether it is a bug or there is some reason to avoid execution of this method for _connection._play_context inside method _execute. As a temporary workaround, I've appended the constructor of ActionModuleMixin with the execution of self._connection._play_context.post_validate(templar=connection.templar):

class ActionModuleMixin(ansible.plugins.action.ActionBase):
    """
    ...
    """
    def __init__(self, task, connection, *args, **kwargs):
        """
        ...
        """
        super(ActionModuleMixin, self).__init__(task, connection, *args, **kwargs)
        if not isinstance(connection, ansible_mitogen.connection.Connection):
            _, self.__class__ = type(self).__bases__

        # required for python interpreter discovery
        connection.templar = self._templar
        self._finding_python_interpreter = False
        self._rediscovered_python = False
        # redeclaring interpreter discovery vars here in case running ansible < 2.8.0
        self._discovered_interpreter_key = None
        self._discovered_interpreter = False
        self._discovery_deprecation_warnings = []
        self._discovery_warnings = []

        self._connection._play_context.post_validate(templar=connection.templar)

and after that got the correct value of remote_user when using delegate_to in my tasks.

Probably you will find better approach solving this issue.

Thank you in advance.

• Which version of Ansible are you running?
  2.13 ansible core 2.14.0 and core 2.13.13 too
• Is your version of Ansible patched in any way?
  No
• Have you tried the latest master version from Git?
  Yes
• Mention your host and target OS and versions
  Host: Ubuntu 22.04.3 LTS (target host doesn't matter for this issue)
• Mention your host and target Python versions
  Host: python3.10 (target host doesn't matter for this issue)

[moreati]

Thank you, this may be closely related to #1022 or a case of it.

[rawat-he]

Hi,

I have the same issue.

mitogen_get_stack print without the changes mentioned by @macksyma

  discovered_interpreter: false
  result:
  - kwargs:
      check_host_keys: ignore
      compression: true
      connect_timeout: 10
      hostname: '{{ my_ip_address }}'
      identities_only: false
      identity_file: null
      keepalive_count: 10
      keepalive_interval: 30
      password: null
      port: null
      python_path:
      - /usr/bin/python3
      remote_name: null
      ssh_args:
      - -F
      - /data/my-work/my_custom_ssh_config
      - -o
      - UserKnownHostsFile=/dev/null
      - -o
      - ConnectTimeout=20
      - -o
      - ControlMaster=auto
      - -o
      - ControlPersist=60s
      ssh_debug_level: null
      ssh_path: ssh
      username: '{{ lookup(''env'', ''USERNAME'') | default(my.username, true) }}'
    method: ssh

mitogen_get_stack print after the changes mentioned by @macksyma

  discovered_interpreter: false
  result:
  - kwargs:
      check_host_keys: ignore
      compression: true
      connect_timeout: 10
      hostname: 10.31.103.20
      identities_only: false
      identity_file: null
      keepalive_count: 10
      keepalive_interval: 30
      password: null
      port: null
      python_path:
      - /usr/bin/python3
      remote_name: null
      ssh_args:
      - -F
      - /data/my-work/my_custom_ssh_config
      - -o
      - UserKnownHostsFile=/dev/null
      - -o
      - ConnectTimeout=20
      - -o
      - ControlMaster=auto
      - -o
      - ControlPersist=60s
      ssh_debug_level: null
      ssh_path: ssh
      username: hrawat
    method: ssh

Ansible: 2.14.0
Target OS: Alpine Linux v3.17
Python: 3.10.13
mitogen: 0.3.6

[moreati]

I've discovered the incorrect transmission of the remote_user variable's value when using delegate_to with Mitogen 3.4. For example, let's take a simple task:

    - name: "test_task"
      shell: "ls -lah"
      delegate_to: "{{ test_jump_box }}"
      remote_user: "{{ jumper_test_user }}"

Notes

• remote_user here is a keyword, not a variable.
• The SSH plugin option is also called remote_user. The keyword is listed as a source for the option
  https://docs.ansible.com/ansible/latest/collections/ansible/builtin/ssh_connection.html#parameter-remote_user
• The corresponding variables are ansible_user, ansible_ssh_user.
• The precendence is (in increasing order): config, CLI, keywords, variables, direct assignment
  https://docs.ansible.com/ansible/latest/reference_appendices/general_precedence.html#precedence-categories

[moreati]

@macksyma believe your example will now work, with Mitogen 0.3.13. Sorry for the long wait.

[maronovych-godaddy]

@moreati It seems that another bug came up after this fix. Now, variables defined in set_fact and used as remote_user seem not to reach the right context, and then the do_template method of the class Templar (ansible/template/__init__.py) raises AnsibleUndefinedVariable. As a result the task fails because of an undefined variable, which is actually defined. However variables defined before tasks:

---
- hosts: localhost
  gather_facts: no
  vars:
    my_user: "my_user"

  tasks:

or as extra vars:

ansible-playbook -vv  test_playbooks/test_mitogen.yml -e "my_user=my_user"

seem to hit the right context and everything works fine.

Here is the task list used for the test:

---
- hosts: localhost
  gather_facts: no

  tasks:

    - name: "define user and host"
      set_fact:
        my_user: "my_user"
        my_host: "10.192.65.127"
        exec_command: "uname -a"

    - debug:
        var: my_user

    - name: "test"
      shell: "{{ exec_command }}"
      delegate_to: "{{ my_host }}"
      remote_user: "{{ my_user }}"

execution result:

PLAY [localhost] *****************************************************************************************************************************************************************************************
Wednesday 16 October 2024  17:51:01 +0000 (0:00:01.307)       0:00:01.307 *****

TASK [define user and host] ******************************************************************************************************************************************************************************
task path: /home/test/test_playbooks/test_mitogen.yml:7
ok: [localhost] => {"ansible_facts": {"exec_command": "uname -a", "my_host": "10.192.65.127", "my_user": "my_user"}, "changed": false}
Wednesday 16 October 2024  17:51:01 +0000 (0:00:00.064)       0:00:01.371 *****

TASK [debug] *********************************************************************************************************************************************************************************************
task path: /home/test/test_playbooks/test_mitogen.yml:13
ok: [localhost] => {
    "my_user": "my_user"
}
Wednesday 16 October 2024  17:51:01 +0000 (0:00:00.065)       0:00:01.437 *****

TASK [test] **********************************************************************************************************************************************************************************************
task path: /home/test/test_playbooks/test_mitogen.yml:16
fatal: [localhost -> 10.192.65.127]: FAILED! => {"msg": "'my_user' is undefined. 'my_user' is undefined"}

PLAY RECAP ***********************************************************************************************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

• Which version of Ansible are you running?
  2.15.6
• Is your version of Ansible patched in any way?
  No
• Have you tried the latest master version from Git?
  Yes, I've tried 0.3.13 and 0.3.14
• Mention your host and target OS and versions
  Host: Ubuntu 22.04.3 LTS
• Mention your host and target Python versions
  Host: 3.10.12