Super-long Prompts
A set of recent experiments with large language models (hashtag#LLM) such as hashtag#chatGPT suggest that using super-long prompts that contain up to hundreds or even thousands of examples outperforms fine-tuning. Given that fine-tuning can be a costly endeavor (although it is often a one-time exercise), it is exciting to see that packing a prompt with examples can be as good or better if the LLM you are using allows for large context windows. This approach, called in-context learning or ICL, trades "finetuning-time cost for increased inference-time compute".
But I see an additional benefit of ICL, which is flexibility, as it alleviates the need for 'fine-retuning' if/when new or different examples become available. Maintaining and growing a library of examples, or as Ethan Mollick would say, a "grimoire" of shots, allows you to assemble any subset needed for a particular use case, the equivalent of having an infinite number of finetuned models at your finger tips.