Heuristic Algorithm
Heuristic word used a lot in technical discussions. It's important to understand what it means really and how we can use it in our work.
Heuristic Algorithms are the ones that uses heuristics to solve a problem. Meaning, it uses educated/intelligent guessing to find a solution and the solution may not be optimal or mathematically correct.
Heuristic comes from the same Greek word as Eureka, which means "I have found it."
So the actual meaning of the word isn't exactly "guessing," but "finding a solution that's good enough for the problem at hand."
Examples of Heuristic Algorithms
In most cases, the heuristic based algorithms choose something that's largest, smallest, most recent, etc. based on the problem at hand. This starting point is what makes it a heuristic algorithm, as it's not guaranteed to be the optimal solution, but it's a good enough solution for the problem at hand.
- LRU Cache - It uses the heuristic that the least recently used item is the least likely to be used in the future.
- Search Ranking - Search engines use heuristics to rank search results based on various factors like relevance, popularity, etc.
- Context Selection - pick nearest tokens for the context using embeddings.
- Prompt Truncation - No logic. Just keep the most recent tokens and truncate the rest.
- Many more. When we understand what heuristic means, we can see how frequently we use it in our work without even realizing it.
When we want to find a restaurant for dinner, we don't go through all the restaurants in the city and check their menu, reviews, etc.
Probably we just see a place is crowded and guess it's probably good.