Virginia had no Constitutional authority to reclaim the territory previously granted to DC.
https://en.wikipedia.org/wiki/District_of_Columbia_retrocession
Constitutionality
The constitutionality of the retrocession has been called into question. The contract clause found in Article One of the United States Constitution prohibits states from breaching contracts to which they are themselves a party. By annexing Alexandria in 1846, Virginia arguably breached its contractual obligation to "forever cede and relinquish" the territory for use as the permanent seat of the United States government.[52] President William Howard Taft also believed the retrocession to be unconstitutional and tried to have the land given back to the District.[42]
The Supreme Court of the United States never issued a firm opinion on whether the retrocession of the Virginia portion of the District of Columbia was constitutional. In the 1875 case of Phillips v. Payne the Supreme Court held that Virginia had de facto jurisdiction over the area returned by Congress in 1846, and dismissed the tax case brought by the plaintiff. The court, however, did not rule on the core constitutional matter of the retrocession. Writing the majority opinion, Justice Noah Haynes Swayne stated only that:
The plaintiff in error is estopped from raising the point which he seeks to have decided. He cannot, under the circumstances, vicariously raise a question, nor force upon the parties to the compact an issue which neither of them desires to make.[55]
https://en.wikipedia.org/wiki/Boundary_markers_of_the_original_District_of_Columbia
The boundary markers of the original District of Columbia are the 40 milestones that marked the four lines forming the boundaries between the states of Maryland and Virginia and the square of 100 square miles (259 km²) of federal territory that became the District of Columbia in 1801
(see: Founding of the District of Columbia). Working under the supervision of three commissioners that President George Washington had appointed in 1790 in accordance with the federal Residence Act, a surveying team that Major Andrew Ellicott led placed these markers in 1791 and 1792.
Today, 36 of the original marker stones survive as the oldest federally placed monuments in the United States. Thirteen of these markers are now within Virginia due to the return of the portion of the District south and west of the Potomac River to Virginia in 1846 (see: District of Columbia retrocession).