How was law enforcement handled in large US cities before professional police?

General observations on early urban policing

In 1800, only four cities in the US had populations of around 25,000 or more - the largest of these was New York with 60,000 (the others were Philadelphia, Baltimore and Boston) - so the challenges of policing in the US up to the early 1800s were much less than in the major cities of Europe where London was just short of 1 million, with Paris over 500,000. However, population growth over the following decades was rapid and from around the 1830s, city authorities began to make major changes in the organization of law enforcement.

Looking in general at the early history of urban policing,

Police are relative newcomers to the Anglo-American criminal justice system. The Constitution does not mention them. Early city charters do not mention them, either, for the simple reason that, as we know them, police had not been invented. Instead, cities had loosely organized night watches and constables who worked for the courts, supplemented by the private prosecution of offenders through lower-level courts (Steinberg 1989). The night watch and day constable, dating from the Middle Ages... were not replaced until the 1820s, when London police were reorganized by Robert Peel. The police precedent for the United States, as is well known, came from the establishment of the Metropolitan Police of London in 1829.

Source: Eric H. Monkkonen, 'History of Urban Police'. In Crime and Justice, Vol. 15, Modern Policing (1992), pp. 547-580

On these early constables and the night watch,

Constables were responsible to civil and criminal courts. They supported themselves by fees, which came from serving warrants and civil papers and arresting offenders. The victim of an offense had to seek a constable, paying for his actions...

Night watch did just that: they were to raise the hue and cry in case of an offense or to sound an alarm for a fire. The usual criticism of them was that they slept, used their noisy rattles to warn off potential offenders, and ran from real danger....the night watch, who were either citizens doing required volunteer service or, more likely, their paid substitutes, were not in any way a serious crime-fighting organization.

Source: Monkkonen

Also 'imported' from Britain, and operating in American cities from at least the 17th century, were thief catchers and coroners. The former were often, but not always, thieves themselves; despite their name, their job was primarily to recover stolen property and act as informers rather than actually catch thieves. Coroners, on the other hand, played a more active role in actually examining evidence and determining cause of death.

New York

The NCJRS' website has an article on the development of the city's law enforcement, History of New York City Police Department

In the Dutch era from 1625 to 1664, the first professional police department was created in New Amsterdam. Police officers used hand rattles as they patrolled the streets to discourage crime and apprehend criminals. Under British rule from 1664 to 1783, constables were charged with keeping the peace. They focused on such offenses as excessive drinking, gambling, prostitution, and church service disturbances. During the Revolutionary War, the British appointed a military governor and employed citizen patrols to protect New York City residents. After independence, New York adopted the London police model and established a paid professional police force in 1828. The first set of printed rules and regulations was issued to the police force in 1845, and full uniforms were adopted in 1853.

Note that New York had a small population even in 1800, around 60,000 - making it the US' largest city. This was up from just 25,000 in 1776 and just 5,000 in 1700.

Going into more detail and starting from the early days of Dutch settlement (then New Amsterdam, 1625 to 1664), the new arrivals

formed a council to make and interpret the law. The council then appointed a schout-fiscal - Dutch for legal officer - to see that its edicts were obeyed. It was the schout-fiscal's job to punish each lawbreaker in a manner befitting the crime. He served as both sheriff and prosecutor throughout New Netherland, an area covering what is now lower New York State and eastern New Jersey.

Then they got a bit more organized:

In 1651, the first professional police department was created in New Amsterdam – the Rattlewatch. It was a voluntary patrol composed of citizens appointed by the council. In addition to muskets, its members were equipped with the hand rattles that gave the fledgling police force its name. They strolled the streets to discourage crime and search for lawbreakers. In times of emergency, they noisily spun their rattles to summon assistance from fellow Rattlewatch members.

In late 1658, the eight members of the Rattlewatch began drawing pay, making them the first municipally funded police organization.

This seems to have worked reasonably well as, when the British took over in 1664, few changes were made. One was that,

Instead of a schout-fiscal, there was an English constable. His job was to keep the peace, suppress excessive drinking, gambling, prostitution, and prevent disturbances when church services were in progress.

However, this soon changed due to the French and Indian wars:

During the late 1600's, with the first of the French and Indian wars underway, the military assumed responsibility for maintaining law and order in the City. Officials appointed a bellman to do the job. His title came from the bell he rang while making his rounds and calling out the hours. In addition to the bell, he was issued a gun, uniform, badge, shoes and stockings, becoming the City's first uniformed policeman.

In the next stage of development,

The start of the 18th century brought radical changes to the City's system of public protection. The post of bellman was abolished and the English introduced the constable's watch to protect New York's 6,000 residents. What is now lower to mid-Manhattan was divided into six sectors, with a High Constable and 12 sub-constables sworn to "take care, and keep and preserve the peace" throughout the area. In 1731, a watch-house - or Jail - was built at the corner of Wall and Broad Streets. Punishment for "ruffians and evildoers" was carried out in the "cage, stocks, whipping post or ducking stool" in front of City Hall.

Things seem to have worsened after the revolution, though:

By the time the Revolutionary War ended, the City's population had grown to 60,000 - and police protection had become a major problem. But little was done to deal with it. Crime continued to increase through the late 1820's. Although more watchmen were hired, they were widely regarded as incompetent and the protection they provided was considered inadequate. . . For years the situation remained this way. Nothing was done until contempt for the City's weak police force finally gave way to fear - fear that the City's social disintegration was imminent. And with that fear came the realization that something had to be done about providing New York with a strong, effective police department.

Philadelphia

The US' second largest city in 1800 had no professional police force at the time. Like New York, policing evolved as the population grew, from the Town Watch in around 1700 to the first paid (but limited) police agency in 1751:

This agency, comprised of wardens and constables, patrolled the city on a limited basis, usually stationed in "watch boxes."

Then, in 1830,

An ordinance of December 30 of that year first established a "police patrol" which served throughout the day as well as night. Many ordinances extended the scope of the system; in 1833 the City Commissioners' function of lighting the streets was transferred to it and in 1841 a full-scale Police Department was created under the direction of the Mayor.

In common with New York, it steadily developed with guidelines becoming more clearly defined and responsibilities sometimes added and then later transferred:

...in 1859 a Detective Division was formed within the Department and in the following year a River and Harbor Police were added to it. In 1864 the office of Fire Marshal was created within the Department, where it remained until 1937.