@skroo i'm guessing, but it sounds like you're talking about an env_file being referenced from a docker-compose.yml (ie. https://docs.docker.com/compose/compose-file/#env_file)? if so, you wouldn't need to rebuild the images, just restart the application.
what i can't manage to find, though, is whether docker compose restart will pick those changes up, or if you have to do a down and then an up.