Jeremiah Olney was born into an old family from Rhode Island. He formed a company of infantry from that state at the start of the American Revolutionary War. After serving as captain in 1776, he was promoted to lieutenant colonel at the beginning of 1777. As second-in-command of the 2nd Rhode Island Regiment, he fought at Red Bank. After its commander was wounded early in the action, he led Varnum's brigade in bitter fighting at Monmouth in June 1778.

The 2nd Rhode Island Regiment was authorized on 6 May 1775 under Colonel Daniel Hitchcock in the Rhode Island Army of Observation and was organized on 8 May 1775 as eight companies of volunteers from Providence County of the colony of Rhode Island. As part of a brigade organized under Nathanael Greene, the unit participated in the Siege of Boston during the remainder of 1775. Some elements accompanied Benedict Arnold's expedition to Quebec late in the year. The unit was renamed the 11th Continental Regiment on the first day of 1776.