Brown was born in Montrose on 21 December 1773. He was the son of James Brown, a minister in the Scottish Episcopal Church with Jacobite convictions so strong that in 1788 he defied his church's decision to give allegiance to George III. His mother was Helen née Taylor, the daughter of a Presbyterian minister. As a child Brown attended the local Grammar School (now called Montrose Academy), then Marischal College at Aberdeen, but withdrew in his fourth year when the family moved to Edinburgh in 1790. His father died late the following year.
Brown enrolled to study medicine at the University of Edinburgh, but developed an interest in botany, and ended up spending more of his time on the latter than the former. He attended the lectures of John Walker; made botanical expeditions into the Scottish Highlands, alone or with nurserymen such as George Don; and wrote out meticulous botanical descriptions of the plants he collected. He also began corresponding with and collecting for William Withering, one of the foremost British botanists of his day. Highlights for Brown during this period include his discovery of a new species of grass, Alopecurus alpinus; and his first botanical paper, "The botanical history of Angus", read to the Edinburgh Natural History Society in January 1792, but not published in print in Brown's lifetime.
Brown dropped out of his medical course in 1793. Late in 1794, he enlisted in the Fifeshire Fencibles, and his regiment was posted to Ireland shortly after. In June 1795 he was appointed Surgeon's Mate. His regiment saw very little action, however, he had a good deal of leisure time, almost all of which he spent on botany. He was frustrated by his itinerant lifestyle, which prevented him from building his personal library and specimen collection as he would have liked, and cut him off from the most important herbaria and libraries.
During this period Brown was especially interested in cryptogams, and these would be the subject of Brown's first, albeit unattributed, publication. Brown began a correspondence with James Dickson, and by 1796 was sending him specimens and descriptions of mosses. Dickson incorporated Brown's descriptions into his Fasciculi plantarum cryptogamicarum britanniae, with Brown's permission but without any attribution.
By 1800, Brown was firmly established amongst Irish botanists, and was corresponding with a number of British and foreign botanists, including Withering, Dickson, James Edward Smith and José Correia da Serra. He had been nominated to the Linnean Society of London; had contributed to Dickson's Fasciculi; was acknowledged in a number of other works; and had had a species of algae, Conferva brownii (now Aegagropila linnaei) named after him by Lewis Weston Dillwyn. He had also begun experimenting with microscopy. However, as an army surgeon stationed in Ireland there seemed little prospect of him attracting the notice of those who could offer him a career in botany.