I’m punting this until we have usage data and rationale. Splitting the containers preemptively would add a lot of maintenance burden. Our internal CI, which I’m targeting first, doesn’t need it.

If we still think that these two needs to conflict instead of converging probably we are missing a great opportunity in this refactoring. I hope we could find the right equilibrium to stay easily on the same page with the local environment we distribute and in which the contributor prepare the PR  and the automation that validates it (CI).
I think that this CI vs Dev-env approach it could really go to create some friction quite soon as the CI use case, when “in production”, will be dominant. IMHO it is better to co-design earlier if it is possible.

I have been wondering about how this will turn out. Right now, the cache gets pushed once per day with an inconsistent commit. Using nightly won’t come until the next milestone of work is done. I think what we’ll probably do in the future is to make sure we push the cache with every day’s nightly tag and encourage developers to start from there. Most of the time, I think that should give a good improvement over the current situation.

Not all the commits are the same as you can see in the mentioned forum thread. It seems also from @mihaimaruseac  reported small experiments in the same thread the the llvm “daily syncs” are invalidating many targets (and so the cache).
Working with Github PR over “the last” nightly it could be ok but we really need to understand what is required when we need  resolve conflicts or if and when we are asking to a developer to rebase or to merge master in a under review PR.

I don’t want to accept the hassle of maintaining a user in the image unless it can very easily match the user and group inside the container to the permissions on the volumes, e.g. if my user is 396220:89939 then I shouldn’t end up with files owned by 1000:1000.

I suppose that you know that this it is not possible as it was explored in a quite long thread at upstream level:
  

      github.com/moby/moby
  

  
    
  
	  
  

  
    
      Add ability to mount volume as user other than root
    

    
      
        opened 
        
          
        
        Oct 17, 2013
      
      


      
        
          
          mingfang
        
      
    

    
        
          area/api
        
        
          area/kernel
        
        
          exp/expert
        
        
          kind/enhancement
        
        
          area/volumes
        
    
  


  
    Use case: mount a volume from host to container for use by apache as www user.
T…he problem is currently all mounts are mounted as root inside the container.
For example, this command
 docker run -v /tmp:/var/www ubuntu stat -c "%U %G" /var/www
will print "root root"

I need to mount it as user www inside the container.
  

  

  
    
    
  

  


If we really don’t want a standard 1000 user probably it could be better to not suggest, when we will have again an official devel image, to use an host path with the TF source mounted in the container.
We could suggest to checkout the source in a namedVolume directly so that we don’t mix host path permission with the root permission in the container.
EDIT:
An alternative:
we could suggest to the user to use Docker rootless with uidmap as now it is not experimental anymore.
I still think that having a default user will cover more frequent use cases also for the standard docker root installation.
But if we don’t want to support this we could at least strongly suggest to use Docker rootless as it will not go to create all the permission problem on new files created in the container on an host shared path as we have with the current setup/docs.
Note:
These two alternative solutions are mutually incompatible as with the rootless currently the host user is mapped only with root in the container.
See more:
  

      github.com/moby/moby
  

  
    
  
	  
  

  
    
      Flexible userns uid / subuid mapping
    

    
      
        opened 
        
          
        
        Sep 26, 2020
      
      


      
        
          
          foresto
        
      
    

    
        
          kind/enhancement
        
        
          area/security/userns
        
        
          area/rootless
        
    
  


  
    I use rootless docker, and often need the unprivileged user running dockerd to s…hare resources (e.g. bind mounted files) with an unprivileged container user. Unfortunately, I can't do this by simply mapping the host uid into the container's user namespace, because docker seems to always map my host uid to the container's uid 0.

Running my container application as uid 0 (in the userns) would allow it to share resources with the host user, but that practice is a significant security risk. The kernel has many parts that were not designed with namespaces in mind, and uid 0 in a userns has been exploited to gain privilege escalation over and over again. [(example)](http://www.halfdog.net/Security/2015/UserNamespaceOverlayfsSetuidWriteExec/) [(example)](https://www.cvedetails.com/cve/CVE-2018-18955/) [(examples)](https://www.debian.org/security/2017/dsa-4073)

For shared files, I have been working around the problem as follows:
* Start the container in order to create the user namespace
* Examine the container process to determine which subuid & subgid docker uses for it
* Change ownership of all shared files and directories to that subuid & subgid
* Make all shared files and directories group-writable
* Make all shared directories setgid, so new files/dirs will have the same subgid
* Set up a script to set umask 002 in the container
* Restart the container with bind mounts for the shared directories
* Remember to set umask 002 in the host whenever working with shared files (and sometimes forget, leading to permissions errors some time later)
* Make the host user a member of the subgid group
* (optional) Name the subgid in /etc/group, for easy identification in `ls -l` output

That works, but it's an awful lot of annoying hoops to jump through, and it's error-prone.

All this would be much easier if docker allowed configurable mappings between host and userns uids/gids. LXC has offered this functionality for years, via the `lxc.idmap` config file option.

Even a very limited approach would we a welcome relief. Perhaps making only one uid mapping configurable. Or perhaps allowing only the host user's uid to have a configurable target uid in the userns (instead of always being mapped to uid 0). Podman offers this via the `--userns=keep-id` option.

#27285 seems related, but I think the OP there is asking for a simpler interface to existing functionality (choosing whose subuid ranges are used), which is not the same thing. I'm asking for something new: the ability to map at least one non-subuid into the container as a configurable non-root uid.