The Fifth amendmentguarantees the right to remain silent . It attaches once there is both custody and interrogation. It also provides for the right to the assistance of counsel during custodial interrogation.
This means that if a person invokes their Fifth Amendment right to remain silent, the police cannot force someone to take a polygraph (they must also cease questioning him at all). Nor can the state use the fact that the defendant refused a polygraph against him at trial.
Once the Sixth amendment right to counsel kicks in, the police would be precluded from giving someone a polygraph without counsel present. The Sixth Amendment initially attaches at the initiation of formal proceedings (at first appearance, at an in person line-up,at arraignment, etc.
Interestingly, however, the Sixth Amendment right to counsel is specific to a particular charge. For instance if the defendant is arrested for a burglary, goes to first appearance and is appointed a lawyer, his Sixth Amendment right to counsel has attached for the burglary.
It would not prevent the police, however, from talking to him about an unrelated murder (or poylgraphing him , with his consent, about that murder) as long as the defendant waives his Fifth Amendment rights and agrees to be questioned about the murder.
As a general rule, polygraph results are NOT admissible in court. So they really would be more of an investigative tool than evidence. However, if the police give a polygraph to someone in violation of their Fifth of Sixth Amendment rights, they risk having any evidence that comes out of the polygraph interview of the polygraph itself.