Disclaimer: This is a question from a seeker and dwells well in the realm of mysticism, hence I may not attempt to reply to it at logical level.
It is an assumption that existence does not want us to remember our past lives. If so, Buddha would not have seen all his past lives to attain enlightenment and Krishna would not have shown past and future to uplift Arjuna from attachment. Therefore, this assumption is grossly misplaced and has been generated by the fears of a seeker about investigating their own self further.
During the birthing process of 9 months, all our memories are in our mind since it is then that there is no subconscious mind or conscious mind. Once the child is born, we begin to create a conscious mind by training the child for set goals, ambitions, rules and prepare them for survival. Often this is not what is stored in the subconscious. In the realm of the Spirit World, there is no such need of surviving because the soul knows that it has to die someday, and that they come to existing to experience certain emotions or resolve certain experiences based on the Karma theory. As we create a structured program for the child, all memories get suppressed further in the subconscious. The child knows that there’s something inside which is drawing him towards them. These memories from the past lives, which contains the purpose of the current birth, begin to come as insights to the child but our upbringing doesn’t have scope to grow the child with them. Although, this also doesn’t mean that if we, as parents, allow our child to listen to these memories and don’t program them for survival simultaneously, all the past lives would open up on their own. For past lives to open up, there is an entirely different process which takes place in a safe environment, in order to be able to handle the information received. Gaining knowledge about the past life brings wisdom to the soul which is a step towards becoming a Buddha. Sadly, no parent wants their child to become Buddha. They want engineers and doctors, who they believe will (a) serve the humanity and (b) raise their family. This is the reason, Islam and Christianity don’t encourage past life practices since they want to create ‘Survivors’, although all Holy books contain great details on work done in the area of past lives.
You don’t have to push yourself to get the information from your past life. It comes when you’re ready in the state of Yog Nidra and curious rather than having a fixed idea about them. However, wisdom about our past life comes to us, irrespective, as insights and intuitions, if we’re connected to ourself.
Traditionally, the task of knowing past lives seem to be the job of a Mystic, who explore the deeper meaning of life and existence. However, we happen to seek guidance from those who themselves know very little about it. In this age of convergence, where the rise of spirituality has taken consciousness to a certain level, common people seem to have gained knowledge on their past lives instead of depending on a Mystic. However, science of past lives, its purpose and usage is often misunderstood in our enthusiasm to become exotic with such information. Such informations should be sucked out when your consciousness is ready for it and when you’re in a fair state to dive deeper inside.
Practitioners of “Science of Health” have discovered safest way of knowing past lives. Their past lives appear in small pieces while they are in Yog Nidra state and leaves them with more clear consciousness, but don’t know about it entirely. This is simplest and holistic way to deepen your consciousness without being ambitious about it and lets you remain focused on current life and its purpose. Regression is another way but that should instead be used as a surgery to handle something mystical which occurs in your life as a pattern and you need to go back to few past lives to discover the source of such pattern.
But we may never know why existence chose to keep it a secret available for few and why it could not create more intelligent human beings by letting them have access to all their past lives? Did existence fear that if everyone became Buddha then who would run families? But then why shall we focus on raising families when we know truth is somewhere else deep down?