Abstract

Star-galaxy separation is a crucial step in creating target catalogues for extragalactic spectroscopic surveys. A classifier biased towards inclusivity risks including high numbers of stars, wasting fibre hours, while a more conservative classifier might overlook galaxies, compromising completeness and hence survey objectives. To avoid bias introduced by a training set in supervised methods, we employ an unsupervised machine learning approach. Using photometry from the Wide Area VISTA Extragalactic Survey (WAVES)-Wide catalogue comprising nine-band u-K-s data, we create a feature space with colours, fluxes, and apparent size information extracted by ProFound. We apply the non-linear dimensionality reduction method UMAP (Uniform Manifold Approximation and Projection) combined with the classifier hdbscan (Hierarchical Density-Based Spatial Clustering of Applications with Noise) to classify stars and galaxies. Our method is verified against a baseline colour and morphological method using a truth catalogue from Gaia, SDSS (Sloan Digital Sky Survey), GAMA (Galaxy And Mass Assembly), and DESI (Dark Energy Spectroscopic Instrument). We correctly identify 99.75 per cent of galaxies within the AB magnitude limit of Z=21.2, with an F1 score of 0.9971 +/- 0.0018 across the entire ground truth sample, compared to 0.9879 +/- 0.0088 from the baseline method. Our method's higher purity (0.9967 +/- 0.0021) compared to the baseline (0.9795 +/- 0.0172) increases efficiency, identifying 11 per cent fewer galaxy or ambiguous sources, saving approximately 70 000 fibre hours on the 4MOST (4-m Multi-Object Spectroscopic Telescope) instrument. We achieve reliable classification statistics for challenging sources including quasars, compact galaxies, and low surface brightness galaxies, retrieving 92.7 per cent, 84.6 per cent, and 99.5 per cent of them, respectively. Angular clustering analysis validates our classifications, showing consistency with expected galaxy clustering, regardless of the baseline classification.