<p dir="auto">Observing a regression with a playbook that runs on a daily basis.  The playbook relies on a custom ec2 module that has idempotency changes not yet accepted upstream.</p>
<p dir="auto">The problem appears to be triggered by code the following method call:</p>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="module.from_json(instance_tags)"><pre class="notranslate"><span class="pl-s1">module</span>.<span class="pl-en">from_json</span>(<span class="pl-s1">instance_tags</span>)</pre></div>
<p dir="auto">The task calling the module is as follows:</p>
<div class="highlight highlight-source-yaml notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="- name: create/locate ec2 instance(s)
  ec2:
    state: present
    idempotency_attribute: 'tags'
    aws_access_key: '{{ aws_access_key }}'
    aws_secret_key: '{{ aws_secret_key }}'
    region: '{{ aws_region }}'
    instance_type: '{{ aws_instance_type }}'
    instance_tags: '{&quot;Name&quot;: &quot;{{aws_name_prefix}}-{{item.name}}&quot;}'
    group: '{{ aws_security_group }}'
    key_name: '{{ aws_key_name }}'
    image: '{{ item.id }}'
    wait: yes
  with_items: aws_images
  register: aws"><pre class="notranslate">- <span class="pl-ent">name</span>: <span class="pl-s">create/locate ec2 instance(s)</span>
  <span class="pl-ent">ec2</span>:
    <span class="pl-ent">state</span>: <span class="pl-s">present</span>
    <span class="pl-ent">idempotency_attribute</span>: <span class="pl-s"><span class="pl-pds">'</span>tags<span class="pl-pds">'</span></span>
    <span class="pl-ent">aws_access_key</span>: <span class="pl-s"><span class="pl-pds">'</span>{{ aws_access_key }}<span class="pl-pds">'</span></span>
    <span class="pl-ent">aws_secret_key</span>: <span class="pl-s"><span class="pl-pds">'</span>{{ aws_secret_key }}<span class="pl-pds">'</span></span>
    <span class="pl-ent">region</span>: <span class="pl-s"><span class="pl-pds">'</span>{{ aws_region }}<span class="pl-pds">'</span></span>
    <span class="pl-ent">instance_type</span>: <span class="pl-s"><span class="pl-pds">'</span>{{ aws_instance_type }}<span class="pl-pds">'</span></span>
    <span class="pl-ent">instance_tags</span>: <span class="pl-s"><span class="pl-pds">'</span>{"Name": "{{aws_name_prefix}}-{{item.name}}"}<span class="pl-pds">'</span></span>
    <span class="pl-ent">group</span>: <span class="pl-s"><span class="pl-pds">'</span>{{ aws_security_group }}<span class="pl-pds">'</span></span>
    <span class="pl-ent">key_name</span>: <span class="pl-s"><span class="pl-pds">'</span>{{ aws_key_name }}<span class="pl-pds">'</span></span>
    <span class="pl-ent">image</span>: <span class="pl-s"><span class="pl-pds">'</span>{{ item.id }}<span class="pl-pds">'</span></span>
    <span class="pl-ent">wait</span>: <span class="pl-s">yes</span>
  <span class="pl-ent">with_items</span>: <span class="pl-s">aws_images</span>
  <span class="pl-ent">register</span>: <span class="pl-s">aws</span></pre></div>
<div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="TASK: [create_ec2 | create/locate ec2 instance(s)] **************************** 
failed: [127.0.0.1] =&gt; (item={'id': 'ami-eb6b0182', 'user': 'root', 'name': 'centos-6.4-x86_64'}) =&gt; {&quot;failed&quot;: true, &quot;item&quot;: {&quot;id&quot;: &quot;ami-eb6b0182&quot;, &quot;name&quot;: &quot;centos-6.4-x86_64&quot;, &quot;user&quot;: &quot;root&quot;}, &quot;parsed&quot;: false}
invalid output was: Traceback (most recent call last):
  File &quot;/Users/jlaska/.ansible/tmp/ansible-1383223541.39-224032155676733/ec2&quot;, line 1677, in &lt;module&gt;
    main()
  File &quot;/Users/jlaska/.ansible/tmp/ansible-1383223541.39-224032155676733/ec2&quot;, line 735, in main
    (instance_dict_array, new_instance_ids, changed) = create_instances(module, ec2)
  File &quot;/Users/jlaska/.ansible/tmp/ansible-1383223541.39-224032155676733/ec2&quot;, line 484, in create_instances
    previous_reservations = get_instances(module, ec2, **dict(state='running', **filter_args))
  File &quot;/Users/jlaska/.ansible/tmp/ansible-1383223541.39-224032155676733/ec2&quot;, line 390, in get_instances
    filters.update(dict((&quot;tag:&quot; + tn, tv) for tn, tv in module.from_json(instance_tags).items()))
  File &quot;/Users/jlaska/.ansible/tmp/ansible-1383223541.39-224032155676733/ec2&quot;, line 1492, in from_json
    return json.loads(data)
  File &quot;/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py&quot;, line 338, in loads
    return _default_decoder.decode(s)
  File &quot;/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py&quot;, line 365, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: expected string or buffer"><pre class="notranslate"><span class="pl-v">TASK</span>: [<span class="pl-s1">create_ec2</span> <span class="pl-c1">|</span> <span class="pl-s1">create</span><span class="pl-c1">/</span><span class="pl-s1">locate</span> <span class="pl-s1">ec2</span> <span class="pl-en">instance</span>(<span class="pl-s1">s</span>)] <span class="pl-c1">**</span><span class="pl-c1">**</span><span class="pl-c1">**</span><span class="pl-c1">**</span><span class="pl-c1">**</span><span class="pl-c1">**</span><span class="pl-c1">**</span><span class="pl-c1">**</span><span class="pl-c1">**</span><span class="pl-c1">**</span><span class="pl-c1">**</span><span class="pl-c1">**</span><span class="pl-c1">**</span><span class="pl-c1">**</span> 
<span class="pl-s1">failed</span>: [<span class="pl-c1">127.0</span>.<span class="pl-c1">0.1</span>] <span class="pl-c1">=</span><span class="pl-c1">&gt;</span> (<span class="pl-s1">item</span><span class="pl-c1">=</span>{<span class="pl-s">'id'</span>: <span class="pl-s">'ami-eb6b0182'</span>, <span class="pl-s">'user'</span>: <span class="pl-s">'root'</span>, <span class="pl-s">'name'</span>: <span class="pl-s">'centos-6.4-x86_64'</span>}) <span class="pl-c1">=</span><span class="pl-c1">&gt;</span> {<span class="pl-s">"failed"</span>: <span class="pl-s1">true</span>, <span class="pl-s">"item"</span>: {<span class="pl-s">"id"</span>: <span class="pl-s">"ami-eb6b0182"</span>, <span class="pl-s">"name"</span>: <span class="pl-s">"centos-6.4-x86_64"</span>, <span class="pl-s">"user"</span>: <span class="pl-s">"root"</span>}, <span class="pl-s">"parsed"</span>: <span class="pl-s1">false</span>}
<span class="pl-s1">invalid</span> <span class="pl-s1">output</span> <span class="pl-s1">was</span>: <span class="pl-v">Traceback</span> (<span class="pl-s1">most</span> <span class="pl-s1">recent</span> <span class="pl-s1">call</span> <span class="pl-s1">last</span>):
  <span class="pl-v">File</span> <span class="pl-s">"/Users/jlaska/.ansible/tmp/ansible-1383223541.39-224032155676733/ec2"</span>, <span class="pl-s1">line</span> <span class="pl-c1">1677</span>, <span class="pl-c1">in</span> <span class="pl-c1">&lt;</span><span class="pl-s1">module</span><span class="pl-c1">&gt;</span>
    <span class="pl-en">main</span>()
  <span class="pl-v">File</span> <span class="pl-s">"/Users/jlaska/.ansible/tmp/ansible-1383223541.39-224032155676733/ec2"</span>, <span class="pl-s1">line</span> <span class="pl-c1">735</span>, <span class="pl-s1">in</span> <span class="pl-s1">main</span>
    (<span class="pl-s1">instance_dict_array</span>, <span class="pl-s1">new_instance_ids</span>, <span class="pl-s1">changed</span>) <span class="pl-c1">=</span> <span class="pl-en">create_instances</span>(<span class="pl-s1">module</span>, <span class="pl-s1">ec2</span>)
  <span class="pl-v">File</span> <span class="pl-s">"/Users/jlaska/.ansible/tmp/ansible-1383223541.39-224032155676733/ec2"</span>, <span class="pl-s1">line</span> <span class="pl-c1">484</span>, <span class="pl-s1">in</span> <span class="pl-s1">create_instances</span>
    <span class="pl-s1">previous_reservations</span> <span class="pl-c1">=</span> <span class="pl-en">get_instances</span>(<span class="pl-s1">module</span>, <span class="pl-s1">ec2</span>, <span class="pl-c1">**</span><span class="pl-en">dict</span>(<span class="pl-s1">state</span><span class="pl-c1">=</span><span class="pl-s">'running'</span>, <span class="pl-c1">**</span><span class="pl-s1">filter_args</span>))
  <span class="pl-v">File</span> <span class="pl-s">"/Users/jlaska/.ansible/tmp/ansible-1383223541.39-224032155676733/ec2"</span>, <span class="pl-s1">line</span> <span class="pl-c1">390</span>, <span class="pl-s1">in</span> <span class="pl-s1">get_instances</span>
    <span class="pl-s1">filters</span>.<span class="pl-en">update</span>(<span class="pl-en">dict</span>((<span class="pl-s">"tag:"</span> <span class="pl-c1">+</span> <span class="pl-s1">tn</span>, <span class="pl-s1">tv</span>) <span class="pl-k">for</span> <span class="pl-s1">tn</span>, <span class="pl-s1">tv</span> <span class="pl-c1">in</span> <span class="pl-s1">module</span>.<span class="pl-en">from_json</span>(<span class="pl-s1">instance_tags</span>).<span class="pl-en">items</span>()))
  <span class="pl-v">File</span> <span class="pl-s">"/Users/jlaska/.ansible/tmp/ansible-1383223541.39-224032155676733/ec2"</span>, <span class="pl-s1">line</span> <span class="pl-c1">1492</span>, <span class="pl-s1">in</span> <span class="pl-s1">from_json</span>
    <span class="pl-k">return</span> <span class="pl-s1">json</span>.<span class="pl-en">loads</span>(<span class="pl-s1">data</span>)
  <span class="pl-v">File</span> <span class="pl-s">"/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">338</span>, <span class="pl-s1">in</span> <span class="pl-s1">loads</span>
    <span class="pl-k">return</span> <span class="pl-s1">_default_decoder</span>.<span class="pl-en">decode</span>(<span class="pl-s1">s</span>)
  <span class="pl-v">File</span> <span class="pl-s">"/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py"</span>, <span class="pl-s1">line</span> <span class="pl-c1">365</span>, <span class="pl-s1">in</span> <span class="pl-s1">decode</span>
    <span class="pl-s1">obj</span>, <span class="pl-s1">end</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">raw_decode</span>(<span class="pl-s1">s</span>, <span class="pl-s1">idx</span><span class="pl-c1">=</span><span class="pl-en">_w</span>(<span class="pl-s1">s</span>, <span class="pl-c1">0</span>).<span class="pl-en">end</span>())
<span class="pl-v">TypeError</span>: <span class="pl-s1">expected</span> <span class="pl-s1">string</span> <span class="pl-c1">or</span> <span class="pl-s1">buffer</span></pre></div>
<p dir="auto">Using <code class="notranslate">git bisect</code>, I've determined that the commit causing the problem is commit <a class="commit-link" data-hovercard-type="commit" data-hovercard-url="https://github.com/ansible/ansible/commit/b5c9d5a59b32b25ff1d0bc9448a92b0bf0656848/hovercard" href="https://github.com/ansible/ansible/commit/b5c9d5a59b32b25ff1d0bc9448a92b0bf0656848"><tt>b5c9d5a</tt></a>.  I'm unclear why this would cause issues.  I'm happy to share the custom <code class="notranslate">ec2</code> module if that will aid in resolution.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="b5c9d5a59b32b25ff1d0bc9448a92b0bf0656848 is the first bad commit
commit b5c9d5a59b32b25ff1d0bc9448a92b0bf0656848
Author: Michael DeHaan &lt;michael@ansibleworks.com&gt;
Date:   Wed Oct 30 20:48:53 2013 -0400

    Inventory directory should always be absolute.  Fixes #4535."><pre class="notranslate"><code class="notranslate">b5c9d5a59b32b25ff1d0bc9448a92b0bf0656848 is the first bad commit
commit b5c9d5a59b32b25ff1d0bc9448a92b0bf0656848
Author: Michael DeHaan &lt;michael@ansibleworks.com&gt;
Date:   Wed Oct 30 20:48:53 2013 -0400

    Inventory directory should always be absolute.  Fixes #4535.
</code></pre></div>
<p dir="auto">According to <a href="http://www.ansibleworks.com/docs/intro_configuration.html#the-ansible-configuration-file" rel="nofollow">http://www.ansibleworks.com/docs/intro_configuration.html#the-ansible-configuration-file</a>, <code class="notranslate">ansible.cfg</code> in the playbook directory should override any other file, but that doesn't happen, because instead of playbook's directory, <code class="notranslate">ansible.cfg</code> is read from current dir.</p>
<p dir="auto">It happens, because <code class="notranslate">ansible.cfg</code> lookup is made during constants import [1], before command line arguments are processed [2] and playbook location is detected. The solution is to reread configuration explicitly once playbook location is known.</p>
<ol dir="auto">
<li><a href="https://github.com/ansible/ansible/blob/devel/lib/ansible/constants.py#L80">https://github.com/ansible/ansible/blob/devel/lib/ansible/constants.py#L80</a></li>
<li><a href="https://github.com/ansible/ansible/blob/devel/bin/ansible-playbook#L26">https://github.com/ansible/ansible/blob/devel/bin/ansible-playbook#L26</a></li>
</ol>