It all boils down to the fact that society has to protect the individuals absolute right to the fruits of their own labor, intellectual or otherwise. If you remove patents, you remove protection of intellectual labor - and in this world, most of us do intellectual work at one level or another. If I have no protection of my ideas, then why bother sharing them?
Nobody else has the right to use my ideas in any way except the ways I decide I want them to be used. Patents offer the most rigid protection, but there are other options out there. This is the importance of open source licensing - there are legally protected ways of sharing your work and retaining credit, with control over the sharing, within current patent law.
If you remove patent law from the equation, you end up with the individuals and corporations with the greatest means of production monopolizing ideas, stifling innovation, and profiting off the hard work of others, without giving due recompense for their hard work.
We don’t exist in isolation, so the free trade of ideas needs protection to prevent abuse. This allows situations in which individuals can capitalize on their intellectual labor. Remove the ability to capitalize on intellectual labor, and you remove personal incentive.
The open source mentality is the solution for the problem you are acknowledging. At some point the benefits will be familiar to enough people that the ideas of licensing products and inventions under permissive open sourcing will be common. It will still be possible for corporations like Apple, Microsoft, and Samsung to perpetuate their institutions and protect their intellectual property.
“Should be” doesn’t count for much in my book. Make the best of it. Participate in open source and educate yourself and others to make the best of an imperfect world. Patent laws exist, and aren’t all bad. Open source is the enlightened alternative.