There are a lot of reasons not to split someone’s time between two or more teams. Context switching, meeting overload, multi-tasking, etc.
But the one that never gets talked about is jelling. It’s hard enough for a team to jell, even if everything’s working in its favor. It takes a long time. And unless you hit the jelling jackpot, you have to intentionally work hard at it.
Now try jelling when some of your team members are also trying to jell on other teams. Critical hit. Either the team won’t jell or the part timers will feel like outsiders.
Either dedicate everyone to a single team, or accept that the part timers are consultants rather than team members.