yes, we can do that also. But main intention is to use weights learn (feature learning) by spliting the network along channels for each layer and merge both for other tasks and use single network for it
RGB => Lab => (Network 1 + Network 2) => fc5 features => classifier