Rust

I have a struct that looks like this:

pub struct Game {
    /// A HashSet with the players waiting to play as account strings.
    lobby: HashSet<String>,
    /// capacity determines  how many people a match contains.
    capacity: u8,
    /// A vector of ongoing matches.
    matches: Vec<Match>,
    /// HashSet indicating for each player which match they are in.
    players: HashMap<String, usize>,
}

I realised that this won't work because if there are 3 matches (0, 1, 2) and I remove 1 because it ends, the players that used to point at 2 will be pointing outside the vector or to an incorrect match.

So I thought the obvious solution was to use a reference to the match: players: HashMap<String, &Match>. But this makes lifetimes very complicated.

What's a good way to deal with a case like these where data are interrelated in the same struct?