Multi-domain authentication options for google app engine
Authentication does not imply authorisation. All that the federated ID system does for your application is give you a username/userid that you can trust. So you can setup your user accounts tied to this infomation and rely on the fact that whenever you see that userid you are talking to the same user. Or in the case of domain-wide applications, whenever you see someone with that domain in their userid.
It is completely up to your application to decide if that userid has any meaning on your application. If I login to your app now with my google account, it should say "Oh I haven't seen you before, would you like to join?" ... it should (depending on your app) not just assume I'm authorised to use your application, just because I told you my name.
I'm not sure where you got the "domain login model" from? The only two choices are Google Account and Open/FederatedID neither of those attempt to restrict user access.
In your final example, users spanning multiple google accounts will see different results depending on if they have enable multiple-signin or not. Most users will be presented with a screen to choose which google account they mean before continuing.
I am looking for some suggestions to implement authentication (and authorization) in our GAE app.  Assuming that our app is called someapp, our requirement is as follows:
someapp is primarily for google apps users of the domain its installed for but can also authenticate users from other google apps domains.  
For example, lets say google apps is configured on domainX.com and domainY.com.  Additionally the admin for domainX.com has added someapp to their domain from the apps marketplace. The admin for domainX.com invites userA@domainX.com and userB@domainY.com log on to the application. Both google app domain users should be able to use their SSO (single sign-on) functionality.
As far as we know, current authentication options in the app engine allow either domain login, which allows only the users of one domain to log in to the app or federated/openid login which would allow the users of any domain to log in to the app.  There is no in-between option which would allow only the users of previously authorized domains to log on to the app.  Does that mean our only option is to leave aside google apps authentication and implement our own custom authentication?
Also in our sample scenario above, what if domainX.com and domainY.com have both added someapp.  If userA@domainX.com navigates to someapp.appspot.com, which installation of the app will be used, the one installed on domainX.com or the one installed on domainY.com.