<p dir="auto">I haven't found a way to create a custom datatype to reuse in mapping definition.</p>
<p dir="auto">I would like to have a way to define custom_datatype in the mapping like that :</p>
<div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
     &quot;settings&quot; : {
        &quot;number_of_shards&quot; : 1,
        &quot;custom_datatypes&quot;: [{
            &quot;string_with_raw&quot;: {
              &quot;mapping&quot;: {
                &quot;index&quot;: &quot;analyzed&quot;,
                &quot;omit_norms&quot;: true,
                &quot;type&quot;: &quot;string&quot;,
                &quot;fields&quot;: {
                  &quot;raw&quot;: {
                    &quot;ignore_above&quot;: 256,
                    &quot;index&quot;: &quot;not_analyzed&quot;,
                    &quot;type&quot;: &quot;string&quot;
                  }
                }
              }
            }
         }],
    },
    &quot;mappings&quot; : {
      &quot;my_type&quot;: {
        &quot;properties&quot;: {
          &quot;field1&quot;: {
                &quot;type&quot;: &quot;string_with_raw&quot;             
            },
           &quot;field2&quot;: {
                &quot;type&quot;: &quot;string_with_raw&quot;             
            },
            &quot;field3&quot;: {
                &quot;type&quot;: &quot;string&quot;             
            }

        }
      }
   }
}"><pre class="notranslate">{
     <span class="pl-ent">"settings"</span> : {
        <span class="pl-ent">"number_of_shards"</span> : <span class="pl-c1">1</span>,
        <span class="pl-ent">"custom_datatypes"</span>: [{
            <span class="pl-ent">"string_with_raw"</span>: {
              <span class="pl-ent">"mapping"</span>: {
                <span class="pl-ent">"index"</span>: <span class="pl-s"><span class="pl-pds">"</span>analyzed<span class="pl-pds">"</span></span>,
                <span class="pl-ent">"omit_norms"</span>: <span class="pl-c1">true</span>,
                <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>string<span class="pl-pds">"</span></span>,
                <span class="pl-ent">"fields"</span>: {
                  <span class="pl-ent">"raw"</span>: {
                    <span class="pl-ent">"ignore_above"</span>: <span class="pl-c1">256</span>,
                    <span class="pl-ent">"index"</span>: <span class="pl-s"><span class="pl-pds">"</span>not_analyzed<span class="pl-pds">"</span></span>,
                    <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>string<span class="pl-pds">"</span></span>
                  }
                }
              }
            }
         }],
    },
    <span class="pl-ent">"mappings"</span> : {
      <span class="pl-ent">"my_type"</span>: {
        <span class="pl-ent">"properties"</span>: {
          <span class="pl-ent">"field1"</span>: {
                <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>string_with_raw<span class="pl-pds">"</span></span>             
            },
           <span class="pl-ent">"field2"</span>: {
                <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>string_with_raw<span class="pl-pds">"</span></span>             
            },
            <span class="pl-ent">"field3"</span>: {
                <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>string<span class="pl-pds">"</span></span>             
            }

        }
      }
   }
}</pre></div>
<p dir="auto">For now I have to duplicate for every field :</p>
<div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content=" &quot;mappings&quot;: {
      &quot;my_type&quot;: {      
        &quot;properties&quot;: {
          &quot;message&quot;: {
               &quot;index&quot;: &quot;analyzed&quot;,
                &quot;omit_norms&quot;: true,
                &quot;type&quot;: &quot;string&quot;,
                &quot;fields&quot;: {
                  &quot;raw&quot;: {
                    &quot;ignore_above&quot;: 256,
                    &quot;index&quot;: &quot;not_analyzed&quot;,
                    &quot;type&quot;: &quot;string&quot;
                  }
                }     
            },
           &quot;field2&quot;: {
                &quot;index&quot;: &quot;analyzed&quot;,
                &quot;omit_norms&quot;: true,
                &quot;type&quot;: &quot;string&quot;,
                &quot;fields&quot;: {
                  &quot;raw&quot;: {
                    &quot;ignore_above&quot;: 256,
                    &quot;index&quot;: &quot;not_analyzed&quot;,
                    &quot;type&quot;: &quot;string&quot;
                  }
                }             
            },
            &quot;field3&quot;: {
                &quot;type&quot;: &quot;string&quot;             
            }
          }
     }
  }
}"><pre class="notranslate"> <span class="pl-ent">"mappings"</span>: {
      <span class="pl-ent">"my_type"</span>: {      
        <span class="pl-ent">"properties"</span>: {
          <span class="pl-ent">"message"</span>: {
               <span class="pl-ent">"index"</span>: <span class="pl-s"><span class="pl-pds">"</span>analyzed<span class="pl-pds">"</span></span>,
                <span class="pl-ent">"omit_norms"</span>: <span class="pl-c1">true</span>,
                <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>string<span class="pl-pds">"</span></span>,
                <span class="pl-ent">"fields"</span>: {
                  <span class="pl-ent">"raw"</span>: {
                    <span class="pl-ent">"ignore_above"</span>: <span class="pl-c1">256</span>,
                    <span class="pl-ent">"index"</span>: <span class="pl-s"><span class="pl-pds">"</span>not_analyzed<span class="pl-pds">"</span></span>,
                    <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>string<span class="pl-pds">"</span></span>
                  }
                }     
            },
           <span class="pl-ent">"field2"</span>: {
                <span class="pl-ent">"index"</span>: <span class="pl-s"><span class="pl-pds">"</span>analyzed<span class="pl-pds">"</span></span>,
                <span class="pl-ent">"omit_norms"</span>: <span class="pl-c1">true</span>,
                <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>string<span class="pl-pds">"</span></span>,
                <span class="pl-ent">"fields"</span>: {
                  <span class="pl-ent">"raw"</span>: {
                    <span class="pl-ent">"ignore_above"</span>: <span class="pl-c1">256</span>,
                    <span class="pl-ent">"index"</span>: <span class="pl-s"><span class="pl-pds">"</span>not_analyzed<span class="pl-pds">"</span></span>,
                    <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>string<span class="pl-pds">"</span></span>
                  }
                }             
            },
            <span class="pl-ent">"field3"</span>: {
                <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>string<span class="pl-pds">"</span></span>             
            }
          }
     }
  }
<span class="pl-ii">}</span></pre></div>
<p dir="auto">It could also be used in dynamic mappings:</p>
<div class="highlight highlight-source-json notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="{
     &quot;settings&quot; : {
        &quot;number_of_shards&quot; : 1,
        &quot;custom_datatypes&quot;: [{
            &quot;string_with_raw&quot;: {
              &quot;mapping&quot;: {
                &quot;index&quot;: &quot;analyzed&quot;,
                &quot;omit_norms&quot;: true,
                &quot;type&quot;: &quot;string&quot;,
                &quot;fields&quot;: {
                  &quot;raw&quot;: {
                    &quot;ignore_above&quot;: 256,
                    &quot;index&quot;: &quot;not_analyzed&quot;,
                    &quot;type&quot;: &quot;string&quot;
                  }
                }
              }
            }
         }],
    },
    &quot;mappings&quot; : {
      &quot;my_type&quot;: {
         &quot;dynamic_templates&quot;: [{
            &quot;message_field&quot;: {
              &quot;mapping&quot;: {
                &quot;omit_norms&quot;: false,
               &quot;type&quot;: &quot;string_with_raw&quot;
              },
              &quot;match_mapping_type&quot;: &quot;string&quot;,
              &quot;match&quot;: &quot;*_with_raw&quot;
            }
        }],
        &quot;properties&quot;: {
          &quot;field1&quot;: {
                &quot;type&quot;: &quot;string_with_raw&quot;             
            },
           &quot;field2&quot;: {
                &quot;type&quot;: &quot;string_with_raw&quot;             
            },
            &quot;field3&quot;: {
                &quot;type&quot;: &quot;string&quot;             
            }            
        }
      }
   }
}"><pre class="notranslate">{
     <span class="pl-ent">"settings"</span> : {
        <span class="pl-ent">"number_of_shards"</span> : <span class="pl-c1">1</span>,
        <span class="pl-ent">"custom_datatypes"</span>: [{
            <span class="pl-ent">"string_with_raw"</span>: {
              <span class="pl-ent">"mapping"</span>: {
                <span class="pl-ent">"index"</span>: <span class="pl-s"><span class="pl-pds">"</span>analyzed<span class="pl-pds">"</span></span>,
                <span class="pl-ent">"omit_norms"</span>: <span class="pl-c1">true</span>,
                <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>string<span class="pl-pds">"</span></span>,
                <span class="pl-ent">"fields"</span>: {
                  <span class="pl-ent">"raw"</span>: {
                    <span class="pl-ent">"ignore_above"</span>: <span class="pl-c1">256</span>,
                    <span class="pl-ent">"index"</span>: <span class="pl-s"><span class="pl-pds">"</span>not_analyzed<span class="pl-pds">"</span></span>,
                    <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>string<span class="pl-pds">"</span></span>
                  }
                }
              }
            }
         }],
    },
    <span class="pl-ent">"mappings"</span> : {
      <span class="pl-ent">"my_type"</span>: {
         <span class="pl-ent">"dynamic_templates"</span>: [{
            <span class="pl-ent">"message_field"</span>: {
              <span class="pl-ent">"mapping"</span>: {
                <span class="pl-ent">"omit_norms"</span>: <span class="pl-c1">false</span>,
               <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>string_with_raw<span class="pl-pds">"</span></span>
              },
              <span class="pl-ent">"match_mapping_type"</span>: <span class="pl-s"><span class="pl-pds">"</span>string<span class="pl-pds">"</span></span>,
              <span class="pl-ent">"match"</span>: <span class="pl-s"><span class="pl-pds">"</span>*_with_raw<span class="pl-pds">"</span></span>
            }
        }],
        <span class="pl-ent">"properties"</span>: {
          <span class="pl-ent">"field1"</span>: {
                <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>string_with_raw<span class="pl-pds">"</span></span>             
            },
           <span class="pl-ent">"field2"</span>: {
                <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>string_with_raw<span class="pl-pds">"</span></span>             
            },
            <span class="pl-ent">"field3"</span>: {
                <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>string<span class="pl-pds">"</span></span>             
            }            
        }
      }
   }
}</pre></div>
<p dir="auto">For the dynamic_templates part, the value set in the mapping should override equivalent field defined in the custom datatype.</p>
<p dir="auto">Is there a way to achieve that at the moment ?</p>
<p dir="auto">Under worst case conditions, a request that is bad for the system, thereby taking a long time to process, often gets repeated as a user--or even an automated system--becomes impatient while they await their response. If an <strong>exactly</strong> duplicated request comes into the same node to run on the same shard (otherwise it's not an exact match of course), then it would be ideal in all but one scenario to simply dupe the eventual response from the original request.</p>
<p dir="auto">Unfortunately, the exact match nature of this eliminates a slew of requests from benefiting from it (at least until 3.0 when we can start to recognize inner parts better to drop irrelevant filters), but it provides a lot of protection from a very common problem: a user hitting refresh to send what was a recoverable situation into a downward spiral leading to <code class="notranslate">OutOfMemoryError</code> if you are lucky or, worse, permanent GCs that never recover.</p>
<p dir="auto">The scenario where this is less ideal is the case where a request expects the most up-to-date information (e.g., details from segments that may have been created since the original request started running). Since search is not real time, and the performance of the system is at stake from a presumably slow, repeated request, this seems like an edge case to me. Perhaps providing a flag to use the existing behavior (just create another shard level request) would be enough to allow those cases to "demand" this behavior.</p>
<p dir="auto">It should be as simple as adding a new response listener to the original request, which entirely side steps adding another thread to do the exact same work, but "simple" ends there as I do not believe we provide any way to easily find/attach to other requests. As a result, this is related to task management in <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="38154842" data-permission-text="Title is private" data-url="https://github.com/elastic/elasticsearch/issues/6914" data-hovercard-type="issue" data-hovercard-url="/elastic/elasticsearch/issues/6914/hovercard" href="https://github.com/elastic/elasticsearch/issues/6914">#6914</a>.</p>