Brain aneurysm detection in Time-Of-Flight Magnetic Resonance Angiography (TOF-MRA) has undergone drastic improvements with the advent of Deep Learning (DL). However, performances of supervised DL models heavily rely on the quantity of labeled samples. To mitigate the recurrent bottleneck of voxel-wise label creation, we investigate the use of weak labels: these are oversized annotations which are considerably faster to create. We present a deep learning algorithm for aneurysm detection that exploits weak labels during training. In addition, our model leverages prior anatomical knowledge by focusing only on plausible locations for aneurysm occurrence. We created a retrospective dataset of 284 TOF-MRA subjects (170 females) out of which 157 are patients (with 198 aneurysms), and 127 are controls. Our open TOF-MRA dataset, the largest in the community, is released on OpenNEURO. To assess model generalizability, we participated in a challenge for aneurysm detection with TOF-MRA data (93 patients, 20 controls, 125 aneurysms). Weak labels were 4 times faster to generate than their voxel-wise counterparts. When using prior anatomical knowledge, our network achieved a sensitivity of 80% on the in-house data, with False Positive (FP) rate of 1.2 per patient. On the public challenge, sensitivity was 68% (FP rate = 2.5), ranking 4th/18 on the open leaderboard. We found no significant difference in sensitivity between aneurysm risk-of-rupture groups (p = 0.75), locations (p = 0.72), or sizes (p = 0.15). Our code is made available for reproducibility.