Those who hold to the Regulative Principle of Worship and recite creeds in public worship typically base this practice on two types of biblical evidence:

• Biblical commands to confess one's faith
• New Testament evidence that creeds were used liturgically in the apostolic age

Biblical commands to confess

Proponents of reciting creeds often point to biblical passages that teach that God's people are to confess their faith publicly and together. Some of the passages cited along these lines include:

the eternal life to which you were called and about which you made the good confession in the presence of many witnesses. [1 Timothy 6:12]¹

that together you may with one voice glorify the God and Father of our Lord Jesus Christ. [Romans 15:6]²

Other cited passages, perhaps less obviously related, include 1 Corinthians 1:10, 1 Peter 3:21, and Hebrews 13:15.^1,2 To some, these passages and others like them are sufficient evidence to justify the corporate use of creeds in worship. Others, however, emphasize more indirect New Testament evidence.

Evidence for creeds in apostolic liturgies

Many Reformed thinkers find evidence for the apostolic practice of reciting creeds in worship. The Orthodox Presbyterian Church, for example, writes:

1 Timothy 3:16 leaves the impression that creeds were uttered in the worship of the early Church.³

That verse is often rendered as poetry in English bibles, reflecting its rhythmic meter:

Great indeed, we confess, is the mystery of godliness:
  He was manifested in the flesh,
     vindicated by the Spirit,
        seen by angels,
  proclaimed among the nations,
     believed on in the world,
        taken up in glory. [ESV]

Robert Rayburn points to several other passages in his defense of creeds:

Whether creedal statements of liturgical origin and usage can likewise be detected in the New Testament is probable, though less certain (cf. 1 Cor. 12:3). The Shema was in use as a liturgical creed in the synagogue by the time of the New Testament. In any case, psalms which functioned as confessions of faith (e.g. 33; 97; 136) provide biblical justification for the liturgical use of creeds.⁴

1 Corinthians 12:3 is also cited as evidence for the use of creeds in the apostolic church by J. N. D. Kelly, though he did not subscribe to the Regulative Principle:

no one can say “Jesus is Lord” except in the Holy Spirit. [ESV]

He furthermore argues that passages like 1 Corinthians 15:3–7, Romans 1:3–5, Romans 8:34, 2 Timothy 2:8, and 1 Peter 3:18–22 are examples of "apostolic kerygma," that is, summaries of apostolic teaching used for teaching, preaching, and confessing.⁵

"The most impressive example," he says, is Philippians 2:6–11, a creedal statement that was "almost certainly an ancient Christian hymn, probably of Palestinian derivation, which was already arranged in rhythmic strophes by the time it fell into St. Paul's hands." Similarly, behind 1 Timothy 6:13–15 he sees "a formal confession of belief" associated with "doctrinal preparation for baptism."⁵

Another scholar, Jerome H. Neyrey, summarizes the scholarship on worship in the early church:

[David] Aune, following Delling, Cullmann, and Martin, identifies a variety of activities that fall under the genus "worship": (1) prayers, creeds and confessions, doxologies, hymns, songs, and psalms; (2) prophecy [...]; (3) sermons and homilies; and (4) public reading of Scripture.⁶ [emphasis added]

Summary

Of course, proponents of the RPW and the recitation of creeds also point to the practical value of creeds. Some, attempting to locate this practice in the Westminster Standards, categorize them as "oaths," as shown in another answer. But for RPW advocates, the primary defenses of this practice are found in the biblical commands to confess one's faith and the New Testament evidence that creeds made up a part of the apostolic liturgy.

References:

1. Edmund Clowney, "Presbyterian Worship," in Worship: Adoration and Action, D. A. Carson, ed., 117. Quoted (approvingly) in D. A. Carson, Worship by the Book, 48. Clowney cites three verses that command confession, and one that is evidence of the use of creeds in the apostolic church.
2. "A Plea for Creeds," British Reformed Journal, No. 21, 1998. (WebCite)
3. "The Doxology, Gloria Patri and the Regulative Principle of Worship." (WebCite)
4. Robert S. Rayburn, "Biblical and Pastoral Basis for Creeds and Confessions," in The Practice of Confessional Subscription (1995). (WebCite)
5. Kelly, Early Christian Creeds, 15–23.
6. Neyrey, Give God the Glory, 170.