Using incorrectly colour coded wire is not ideal, but in some cases unavoidable.
IMHO, if by 'swapping them' you mean undoing and re-connecting them, then take the time to do it properly now.
If it is unavoidable to use the correctly coloured wire (and the wire size / rating is still correct), then use coloured sleeving over the wire to indicate the correct polarity and attach a sticker nearby to explain the situation. This is important as someone else may be needing to perform maintenance in the future.
Do use sleeving though, not just coloured insulating tape, as the latter can fall off over time.